feat(consul): filter nodes in upstream with metadata #12448
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| -- | ||
| -- Licensed to the Apache Software Foundation (ASF) under one or more | ||
| -- contributor license agreements. See the NOTICE file distributed with | ||
| -- this work for additional information regarding copyright ownership. | ||
| -- The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| -- (the "License"); you may not use this file except in compliance with | ||
| -- the License. You may obtain a copy of the License at | ||
| -- | ||
| -- http://www.apache.org/licenses/LICENSE-2.0 | ||
| -- | ||
| -- Unless required by applicable law or agreed to in writing, software | ||
| -- distributed under the License is distributed on an "AS IS" BASIS, | ||
| -- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| -- See the License for the specific language governing permissions and | ||
| -- limitations under the License. | ||
| -- | ||
| local core = require("apisix.core") | ||
| local ipairs = ipairs | ||
| local pairs = pairs | ||
|
|
||
| local _M = {} | ||
|
|
||
| local function do_metadata_match(node, metadata_match) | ||
| local metadata = node.metadata | ||
| -- because metadata_match has already been checked in nodes_metadata_match, | ||
| -- there is at least one role, if there is no metadata in node, it's must not matched | ||
| if not metadata then | ||
| return false | ||
| end | ||
| for key, values in pairs(metadata_match) do | ||
| local matched = false | ||
| for _, value in ipairs(values) do | ||
| if metadata[key] == value then | ||
| matched = true | ||
| break | ||
| end | ||
| end | ||
| if not matched then | ||
| return false | ||
| end | ||
| end | ||
| return true | ||
| end | ||
|
|
||
| local function nodes_metadata_match(nodes, metadata_match) | ||
| if not nodes then | ||
| return nil | ||
| end | ||
|
|
||
| -- fast path: there is not metadata_match roles, all nodes are available, | ||
| -- and make a guarantee for do_metadata_match: at least one role | ||
| if not metadata_match then | ||
| return nodes | ||
| end | ||
|
|
||
| local result = {} | ||
| for _, node in ipairs(nodes) do | ||
| if do_metadata_match(node, metadata_match) then | ||
| core.table.insert(result, node) | ||
| end | ||
| end | ||
| return result | ||
| end | ||
| _M.nodes_metadata_match = nodes_metadata_match | ||
|
|
||
| return _M |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -232,6 +232,61 @@ $ curl http://127.0.0.1:9180/apisix/admin/stream_routes/1 -H "X-API-KEY: $admin_ | |
| }' | ||
| ``` | ||
|
|
||
| ### discovery_args | ||
|
|
||
| | Name | Type | Requirement | Default | Valid | Description | | ||
| |----------------| ------ | ----------- | ------- | ----- | ------------------------------------------------------------ | | ||
| | metadata_match | object | optional | {} | | Filter service instances by metadata using containment matching | | ||
|
|
||
| #### Metadata filtering | ||
|
|
||
| APISIX supports filtering service instances based on metadata. When a route is configured with metadata conditions, only service instances whose metadata matched with roles specified in the route's `metadata_match` configuration will be selected. | ||
|
|
||
| Example: If a service instance has metadata `{lane: "a", env: "prod", version: "1.0"}`, it will match routes configured with metadata `{lane: ["a"]}` or `{lane: ["a", "b"], env: "prod"}`, but not routes configured with `{lane: ["c"]}` or `{lane: "a", region: "us"}`. | ||
|
|
||
| Example of routing a request with metadata filtering: | ||
|
|
||
| ```shell | ||
| $ curl http://127.0.0.1:9180/apisix/admin/routes/5 -H "X-API-KEY: $admin_key" -X PUT -i -d ' | ||
| { | ||
| "uri": "/consulWithMetadata/*", | ||
| "upstream": { | ||
| "service_name": "APISIX-CONSUL", | ||
| "type": "roundrobin", | ||
| "discovery_type": "consul", | ||
| "discovery_args": { | ||
| "metadata_match": { | ||
| "version": ["v1", "v2"] | ||
|
There was a problem hiding this comment. I have a question here, is it necessary to set the value to an array? I checked the service meta format of consul https://developer.hashicorp.com/consul/docs/reference/service#meta, and I think setting the value to a string should be sufficient to reduce complexity. There was a problem hiding this comment. Hello @Baoyuantop , this feature has been running in our system since June, and we've benefited repeatedly from the ability to quickly recover from issues using an array rather than a string. To limit the blast radius of new feature releases, we divide service releases into three phases: canary, prod-blue, and prod-green. Canary handles only a minimal amount of traffic to verify system stability, while prod-blue receives 50% of the online traffic to verify business issues. We also implement modular deployments for our online services, ensuring that all services are fully integrated across canary, prod-blue, and prod-green environments. If an issue is discovered in prod-blue, we quickly remove that phase from the gateway entry point and forward all traffic to prod-green. With the current array-based configuration, if canary has some problem, we simply need to modify the apisix routing configuration: From {
"upstream": {
"type": "roundrobin",
"discovery_type": "consul",
"discovery_args": {
"metadata_match": {
"env": ["canary", "prod-blue", "prod-green"]
}
}
}
}To {
"upstream": {
"type": "roundrobin",
"discovery_type": "consul",
"discovery_args": {
"metadata_match": {
"env": ["prod-blue", "prod-green"]
}
}
}
}But if we are using single string value, we must predefine a key likes From {
"upstream": {
"type": "roundrobin",
"discovery_type": "consul",
"discovery_args": {
"metadata_match": {
"env": "prod"
}
}
}
}To {
"upstream": {
"type": "roundrobin",
"discovery_type": "consul",
"discovery_args": {
"metadata_match": {
"env": "prod",
"staging": "prod-blue" // or "prod-green", cannot both "prod-blue" and "prod-green"
}
}
}
}There was a problem hiding this comment. And as the dimensions increase, I have to define multiple routes by cross-product to achieve the functions originally implemented using arrays. |
||
| } | ||
| } | ||
| } | ||
| }' | ||
| ``` | ||
|
|
||
| This route will only route traffic to service instances that have the metadata field `version` set to `v1` or `v2`. | ||
|
|
||
| For multiple metadata criteria: | ||
|
|
||
| ```shell | ||
| $ curl http://127.0.0.1:9180/apisix/admin/routes/6 -H "X-API-KEY: $admin_key" -X PUT -i -d ' | ||
| { | ||
| "uri": "/consulWithMultipleMetadata/*", | ||
| "upstream": { | ||
| "service_name": "APISIX-CONSUL", | ||
| "type": "roundrobin", | ||
| "discovery_type": "consul", | ||
| "discovery_args": { | ||
| "metadata_match": { | ||
| "lane": ["a"], | ||
| "env": ["prod"] | ||
| } | ||
| } | ||
| } | ||
| }' | ||
| ``` | ||
|
|
||
| This route will only route traffic to service instances that have both `lane: "a"` and `env: "prod"` in their metadata. | ||
|
|
||
| You could find more usage in the `apisix/t/discovery/stream/consul.t` file. | ||
|
|
||
| ## Debugging API | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pls take a look, skip current service if the
metadatis invalidThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hello @membphis. I will add cdata check and error log, but I am not sure whether it is appropriate to skip this node.
If it is a container template release, it means that all the containers released in this batch will be skipped, and users only need to pay attention to whether it is valid when they need to use metadata. If it is skipped directly, the service quality will be affected. So I tend to keep these nodes, print logs and leave them empty.