init
This commit is contained in:
shuhongfan
@@ -0,0 +1,111 @@
|
||||
# Sentinel Envoy RLS - Kubernetes sample
|
||||
|
||||
This sample will illustrate how to use Sentinel RLS token server with Envoy in Kubernetes clusters.
|
||||
|
||||
## Build the Docker image
|
||||
|
||||
We could use the pre-built Docker image: `registry.cn-hangzhou.aliyuncs.com/sentinel-docker-repo/sentinel-envoy-rls-server:latest`
|
||||
|
||||
We can also manually build the Docker image in the `sentinel-cluster-server-envoy-rls` directory:
|
||||
|
||||
```bash
|
||||
docker build -t "sentinel/sentinel-envoy-rls-server:latest" -f ./Dockerfile .
|
||||
```
|
||||
|
||||
## Deploy Sentinel RLS token server
|
||||
|
||||
Next we could deploy the Sentinel RLS token server in the K8S cluster.
|
||||
We've provided a deployment template for Sentinel RLS token server in `sentinel-rls.yml`.
|
||||
It includes:
|
||||
|
||||
- A `ConfigMap` that contains the cluster flow control rule for Envoy global rate limiting.
|
||||
This will be mounted as a file in the target `Deployment`, so that the Sentinel RLS token server
|
||||
could load the rules dynamically as soon as the rule in the `ConfigMap` has been updated.
|
||||
- A `Deployment` for Sentinel RLS token server. It will mount the `ConfigMap` as a volume
|
||||
for dynamic rule configuration.
|
||||
- A `Service` that exports the Sentinel command port (8719) and the RLS gRPC port (by default 10245)
|
||||
on a cluster-internal IP so that the Envoy pods could communicate with the RLS server.
|
||||
|
||||
The sample rate limiting rule in the `sentinel-rule-cm`:
|
||||
|
||||
```yaml
|
||||
domain: foo
|
||||
descriptors:
|
||||
# For requests to the "service_httpbin" cluster, limit the max QPS to 1
|
||||
- resources:
|
||||
- key: "destination_cluster"
|
||||
value: "service_httpbin"
|
||||
count: 1
|
||||
```
|
||||
|
||||
You may enable the access log in the Sentinel RLS token server (output to console)
|
||||
via the `SENTINEL_RLS_ACCESS_LOG` env:
|
||||
|
||||
```yaml
|
||||
env:
|
||||
- name: SENTINEL_RLS_ACCESS_LOG
|
||||
value: "on"
|
||||
```
|
||||
|
||||
You may also append JVM opts via the `JAVA_OPTS` env.
|
||||
|
||||
After preparing the yaml template, you may deploy the Sentinel RLS token server:
|
||||
|
||||
```bash
|
||||
kubectl apply -f sample/k8s/sentinel-rls.yml
|
||||
```
|
||||
|
||||
## Deploy Envoy
|
||||
|
||||
Next we could deploy the Envoy instances in the K8S cluster. If you've already had Envoy instances running,
|
||||
you could configure the address (`sentinel-rls-service`) and the port (`10245`)
|
||||
of the rate limit cluster in your Envoy configuration.
|
||||
|
||||
We've provided a deployment template for Envoy in `envoy.yml`.
|
||||
It includes:
|
||||
|
||||
- A `ConfigMap` that contains the configuration for Envoy.
|
||||
This will be mounted as a file in the target `Deployment`, which will be loaded as the configuration
|
||||
file by Envoy.
|
||||
- A `Deployment` for Envoy. It will mount the `ConfigMap` as a volume
|
||||
for configuration.
|
||||
- A `Service` that exports the Envoy endpoint port (by default 10000) on a cluster-internal IP
|
||||
so that it could be accessible from other pods. If you need external access, you could choose the
|
||||
`LoadBalancer` type or add a frontend ingress.
|
||||
|
||||
In the sample, we have two [Envoy clusters](https://www.envoyproxy.io/docs/envoy/latest/api-v2/clusters/clusters):
|
||||
|
||||
- `service_httpbin`: HTTP proxy to `httpbin.org`
|
||||
- `rate_limit_cluster`: the cluster of the Sentinel RLS token server
|
||||
|
||||
This route configuration tells Envoy to route incoming requests to `httpbin.org`. In the `http_filters` conf item,
|
||||
we added the `envoy.rate_limit` filter to the filter chain so that the global rate limiting is enabled.
|
||||
We set the rate limit domain as `foo`, which matches the item in the Envoy RLS rule.
|
||||
In the `route_config`, we provide the rate limit action: `{destination_cluster: {}}`, which will
|
||||
generate the rate limit descriptor containing the actual target cluster name (e.g. `service_httpbin`).
|
||||
Then we could set rate limit rules for each target clusters.
|
||||
|
||||
After preparing the yaml template, you may deploy the Envoy instance:
|
||||
|
||||
```bash
|
||||
kubectl apply -f sample/k8s/envoy.yml
|
||||
```
|
||||
|
||||
for v3 api：
|
||||
|
||||
```bash
|
||||
kubectl apply -f sample/k8s/envoy-v3-api.yml
|
||||
```
|
||||
|
||||
## Test the rate limiting
|
||||
|
||||
Now it's show time! We could visit the URL `envoy-service:10000/json` in K8S pods.
|
||||
Since we set the max QPS to 1, we could emit concurrent requests to the URL, and we
|
||||
could see the first request passes, while the latter requests are blocked (status 429):
|
||||
|
||||
|
||||
|
||||
## Update the rules dynamically
|
||||
|
||||
You could update the rules in the `sentinel-rule-cm` ConfigMap. Once the content is updated,
|
||||
Sentinel will perceive the changes and load the new rules to `EnvoyRlsRuleManager`.
|
||||
@@ -0,0 +1,141 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: envoy-cm-17
|
||||
data:
|
||||
envoy-yml: |-
|
||||
admin:
|
||||
access_log_path: /tmp/admin_access.log
|
||||
address:
|
||||
socket_address:
|
||||
protocol: TCP
|
||||
address: 127.0.0.1
|
||||
port_value: 9901
|
||||
static_resources:
|
||||
listeners:
|
||||
- name: listener_0
|
||||
address:
|
||||
socket_address:
|
||||
protocol: TCP
|
||||
address: 0.0.0.0
|
||||
port_value: 10000
|
||||
filter_chains:
|
||||
- filters:
|
||||
- name: envoy.filters.network.http_connection_manager
|
||||
typed_config:
|
||||
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
|
||||
stat_prefix: ingress_http
|
||||
route_config:
|
||||
name: local_route
|
||||
virtual_hosts:
|
||||
- name: local_service
|
||||
domains: ["*"]
|
||||
routes:
|
||||
- match:
|
||||
prefix: "/"
|
||||
route:
|
||||
cluster: service_httpbin
|
||||
typed_per_filter_config:
|
||||
envoy.filters.http.dynamic_forward_proxy:
|
||||
"@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.PerRouteConfig
|
||||
host_rewrite_literal: httpbin.org
|
||||
rate_limits:
|
||||
- stage: 0
|
||||
actions:
|
||||
- {destination_cluster: {}}
|
||||
http_filters:
|
||||
- name: envoy.filters.http.ratelimit
|
||||
typed_config:
|
||||
"@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
|
||||
domain: foo
|
||||
request_type: external
|
||||
failure_mode_deny: false
|
||||
stage: 0
|
||||
rate_limit_service:
|
||||
grpc_service:
|
||||
envoy_grpc:
|
||||
cluster_name: rate_limit_cluster
|
||||
timeout: 2s
|
||||
transport_api_version: V3
|
||||
- name: envoy.filters.http.router
|
||||
typed_config:
|
||||
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
|
||||
clusters:
|
||||
- name: service_httpbin
|
||||
connect_timeout: 0.5s
|
||||
type: LOGICAL_DNS
|
||||
# Comment out the following line to test on v6 networks
|
||||
dns_lookup_family: V4_ONLY
|
||||
lb_policy: ROUND_ROBIN
|
||||
load_assignment:
|
||||
cluster_name: service_httpbin
|
||||
endpoints:
|
||||
- lb_endpoints:
|
||||
- endpoint:
|
||||
address:
|
||||
socket_address:
|
||||
address: httpbin.org
|
||||
port_value: 80
|
||||
- name: rate_limit_cluster
|
||||
type: STRICT_DNS
|
||||
connect_timeout: 10s
|
||||
lb_policy: ROUND_ROBIN
|
||||
http2_protocol_options: {}
|
||||
load_assignment:
|
||||
cluster_name: rate_limit_cluster
|
||||
endpoints:
|
||||
- lb_endpoints:
|
||||
- endpoint:
|
||||
address:
|
||||
socket_address:
|
||||
address: sentinel-rls-service
|
||||
port_value: 10245
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: envoy-deployment-basic-17
|
||||
labels:
|
||||
app: envoy-17
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: envoy-17
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: envoy-17
|
||||
spec:
|
||||
containers:
|
||||
- name: envoy
|
||||
image: envoyproxy/envoy:v1.17.3
|
||||
ports:
|
||||
- containerPort: 10000
|
||||
command: ["envoy"]
|
||||
args: ["-c", "/tmp/envoy/envoy.yaml"]
|
||||
volumeMounts:
|
||||
- name: envoy-config
|
||||
mountPath: /tmp/envoy
|
||||
volumes:
|
||||
- name: envoy-config
|
||||
configMap:
|
||||
name: envoy-cm-17
|
||||
items:
|
||||
- key: envoy-yml
|
||||
path: envoy.yaml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: envoy-service-17
|
||||
labels:
|
||||
name: envoy-service-17
|
||||
spec:
|
||||
type: NodePort
|
||||
ports:
|
||||
- port: 10000
|
||||
targetPort: 10000
|
||||
protocol: TCP
|
||||
selector:
|
||||
app: envoy-17
|
||||
@@ -0,0 +1,132 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: envoy-cm
|
||||
data:
|
||||
envoy-yml: |-
|
||||
admin:
|
||||
access_log_path: /tmp/admin_access.log
|
||||
address:
|
||||
socket_address:
|
||||
protocol: TCP
|
||||
address: 127.0.0.1
|
||||
port_value: 9901
|
||||
static_resources:
|
||||
listeners:
|
||||
- name: listener_0
|
||||
address:
|
||||
socket_address:
|
||||
protocol: TCP
|
||||
address: 0.0.0.0
|
||||
port_value: 10000
|
||||
filter_chains:
|
||||
- filters:
|
||||
- name: envoy.http_connection_manager
|
||||
typed_config:
|
||||
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
|
||||
stat_prefix: ingress_http
|
||||
route_config:
|
||||
name: local_route
|
||||
virtual_hosts:
|
||||
- name: local_service
|
||||
domains: ["*"]
|
||||
routes:
|
||||
- match:
|
||||
prefix: "/"
|
||||
route:
|
||||
host_rewrite: httpbin.org
|
||||
cluster: service_httpbin
|
||||
rate_limits:
|
||||
- stage: 0
|
||||
actions:
|
||||
- {destination_cluster: {}}
|
||||
http_filters:
|
||||
- name: envoy.rate_limit
|
||||
config:
|
||||
domain: foo
|
||||
stage: 0
|
||||
rate_limit_service:
|
||||
grpc_service:
|
||||
envoy_grpc:
|
||||
cluster_name: rate_limit_cluster
|
||||
timeout: 0.25s
|
||||
- name: envoy.router
|
||||
clusters:
|
||||
- name: service_httpbin
|
||||
connect_timeout: 0.5s
|
||||
type: LOGICAL_DNS
|
||||
# Comment out the following line to test on v6 networks
|
||||
dns_lookup_family: V4_ONLY
|
||||
lb_policy: ROUND_ROBIN
|
||||
load_assignment:
|
||||
cluster_name: service_httpbin
|
||||
endpoints:
|
||||
- lb_endpoints:
|
||||
- endpoint:
|
||||
address:
|
||||
socket_address:
|
||||
address: httpbin.org
|
||||
port_value: 80
|
||||
- name: rate_limit_cluster
|
||||
type: LOGICAL_DNS
|
||||
connect_timeout: 0.25s
|
||||
lb_policy: ROUND_ROBIN
|
||||
http2_protocol_options: {}
|
||||
load_assignment:
|
||||
cluster_name: rate_limit_cluster
|
||||
endpoints:
|
||||
- lb_endpoints:
|
||||
- endpoint:
|
||||
address:
|
||||
socket_address:
|
||||
address: sentinel-rls-service
|
||||
port_value: 10245
|
||||
---
|
||||
apiVersion: apps/v1beta2
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: envoy-deployment-basic
|
||||
labels:
|
||||
app: envoy
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: envoy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: envoy
|
||||
spec:
|
||||
containers:
|
||||
- name: envoy
|
||||
image: envoyproxy/envoy:v1.12.0
|
||||
ports:
|
||||
- containerPort: 10000
|
||||
command: ["envoy"]
|
||||
args: ["-c", "/tmp/envoy/envoy.yaml"]
|
||||
volumeMounts:
|
||||
- name: envoy-config
|
||||
mountPath: /tmp/envoy
|
||||
volumes:
|
||||
- name: envoy-config
|
||||
configMap:
|
||||
name: envoy-cm
|
||||
items:
|
||||
- key: envoy-yml
|
||||
path: envoy.yaml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: envoy-service
|
||||
labels:
|
||||
name: envoy-service
|
||||
spec:
|
||||
type: ClusterIP
|
||||
ports:
|
||||
- port: 10000
|
||||
targetPort: 10000
|
||||
protocol: TCP
|
||||
selector:
|
||||
app: envoy
|
||||
@@ -0,0 +1,68 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: sentinel-rule-cm
|
||||
data:
|
||||
rule-yaml: |-
|
||||
domain: foo
|
||||
descriptors:
|
||||
- resources:
|
||||
- key: "destination_cluster"
|
||||
value: "service_httpbin"
|
||||
count: 1
|
||||
---
|
||||
apiVersion: apps/v1beta2
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: sentinel-rls-server
|
||||
labels:
|
||||
app: sentinel
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: sentinel
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: sentinel
|
||||
spec:
|
||||
containers:
|
||||
- name: sentinelserver
|
||||
# You could replace the image with your own image here
|
||||
image: "registry.cn-hangzhou.aliyuncs.com/sentinel-docker-repo/sentinel-envoy-rls-server:latest"
|
||||
imagePullPolicy: Always
|
||||
ports:
|
||||
- containerPort: 10245
|
||||
- containerPort: 8719
|
||||
volumeMounts:
|
||||
- name: sentinel-rule-config
|
||||
mountPath: /tmp/sentinel
|
||||
env:
|
||||
- name: SENTINEL_RLS_RULE_FILE_PATH
|
||||
value: "/tmp/sentinel/rule.yaml"
|
||||
volumes:
|
||||
- name: sentinel-rule-config
|
||||
configMap:
|
||||
name: sentinel-rule-cm
|
||||
items:
|
||||
- key: rule-yaml
|
||||
path: rule.yaml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: sentinel-rls-service
|
||||
labels:
|
||||
name: sentinel-rls-service
|
||||
spec:
|
||||
type: ClusterIP
|
||||
ports:
|
||||
- port: 8719
|
||||
targetPort: 8719
|
||||
name: sentinel-command
|
||||
- port: 10245
|
||||
targetPort: 10245
|
||||
name: sentinel-grpc
|
||||
selector:
|
||||
app: sentinel
|
||||
Reference in New Issue
Block a user