1 # xDS (Load-Balancing) Interop Test Case Descriptions
3 Client and server use [test.proto](../src/proto/grpc/testing/test.proto).
7 The code for the xDS test server can be found at:
8 [Java](https://github.com/grpc/grpc-java/blob/master/interop-testing/src/main/java/io/grpc/testing/integration/XdsTestServer.java) (other language implementations are in progress).
10 Server should accept these arguments:
13 * The port the server will run on.
17 The base behavior of the xDS test client is to send a constant QPS of unary
18 messages and record the remote-peer distribution of the responses. Further, the
19 client must expose an implementation of the `LoadBalancerStatsService` gRPC
20 service to allow the test driver to validate the load balancing behavior for a
21 particular test case (see below for more details).
23 The code for the xDS test client can be at:
24 [Java](https://github.com/grpc/grpc-java/blob/master/interop-testing/src/main/java/io/grpc/testing/integration/XdsTestClient.java) (other language implementations are in progress).
26 Clients should accept these arguments:
28 * --fail_on_failed_rpcs=BOOL
29 * If true, the client should exit with a non-zero return code if any RPCs
30 fail after at least one RPC has succeeded, indicating a valid xDS config
31 was received. This accounts for any startup-related delays in receiving
32 an initial config from the load balancer. Default is false.
33 * --num_channels=CHANNELS
34 * The number of channels to create to the server.
36 * The QPS per channel.
37 * --server=HOSTNAME:PORT
38 * The server host to connect to. For example, "localhost:8080"
40 * The port for to expose the client's `LoadBalancerStatsService`
42 * --rpc_timeout_sec=SEC
43 * The timeout to set on all outbound RPCs. Default is 20.
45 ### XdsUpdateClientConfigureService
47 The xDS test client's behavior can be dynamically changed in the middle of tests.
48 This is achieved by invoking the `XdsUpdateClientConfigureService` gRPC service
49 on the test client. This can be useful for tests requiring special client behaviors
50 that are not desirable at test initialization and client warmup. The service is
54 message ClientConfigureRequest {
55 // Type of RPCs to send.
61 // Metadata to be attached for the given type of RPCs.
68 // The types of RPCs the client sends.
69 repeated RpcType types = 1;
70 // The collection of custom metadata to be attached to RPCs sent by the client.
71 repeated Metadata metadata = 2;
74 message ClientConfigureResponse {}
76 service XdsUpdateClientConfigureService {
77 // Update the tes client's configuration.
78 rpc Configure(ClientConfigureRequest) returns (ClientConfigureResponse);
82 The test client changes its behavior right after receiving the
83 `ClientConfigureRequest`. Currently it only supports configuring the type(s)
84 of RPCs sent by the test client and metadata attached to each type of RPCs.
88 Note that, unlike our other interop tests, neither the client nor the server has
89 any notion of which of the following test scenarios is under test. Instead, a
90 separate test driver is responsible for configuring the load balancer and the
91 server backends, running the client, and then querying the client's
92 `LoadBalancerStatsService` to validate load balancer behavior for each of the
93 tests described below.
95 ## LoadBalancerStatsService
97 The service is defined as:
100 message LoadBalancerStatsRequest {
101 // Request stats for the next num_rpcs sent by client.
103 // If num_rpcs have not completed within timeout_sec, return partial results.
104 int32 timeout_sec = 2;
107 message LoadBalancerStatsResponse {
108 // The number of completed RPCs for each peer.
109 map<string, int32> rpcs_by_peer = 1;
110 // The number of RPCs that failed to record a remote peer.
111 int32 num_failures = 2;
114 message LoadBalancerAccumulatedStatsRequest {}
116 message LoadBalancerAccumulatedStatsResponse {
117 // The total number of RPCs have ever issued for each type.
118 map<string, int32> num_rpcs_started_by_method = 1;
119 // The total number of RPCs have ever completed successfully for each type.
120 map<string, int32> num_rpcs_succeeded_by_method = 2;
121 // The total number of RPCs have ever failed for each type.
122 map<string, int32> num_rpcs_failed_by_method = 3;
125 service LoadBalancerStatsService {
126 // Gets the backend distribution for RPCs sent by a test client.
127 rpc GetClientStats(LoadBalancerStatsRequest)
128 returns (LoadBalancerStatsResponse) {}
129 // Gets the accumulated stats for RPCs sent by a test client.
130 rpc GetClientAccumulatedStats(LoadBalancerAccumulatedStatsRequest)
131 returns (LoadBalancerAccumulatedStatsResponse) {}
135 Note that the `LoadBalancerStatsResponse` contains the remote peer distribution
136 of the next `num_rpcs` *sent* by the client after receiving the
137 `LoadBalancerStatsRequest`. It is important that the remote peer distribution be
138 recorded for a block of consecutive outgoing RPCs, to validate the intended
139 distribution from the load balancer, rather than just looking at the next
140 `num_rpcs` responses received from backends, as different backends may respond
147 This test verifies that every backend receives traffic.
153 1. --fail_on_failed_rpc=true
155 Load balancer configuration:
157 1. 4 backends are created in a single managed instance group (MIG).
161 1. All backends receive at least one RPC
165 This test verifies that RPCs are evenly routed according to an unweighted round
172 1. --fail_on_failed_rpc=true
174 Load balancer configuration:
176 1. 4 backends are created in a single MIG.
178 Test driver asserts that:
180 1. Once all backends receive at least one RPC, the following 100 RPCs are
181 evenly distributed across the 4 backends.
185 This test verifies that the load balancer will resume sending traffic to a set
186 of backends that is stopped and then resumed.
193 Load balancer configuration:
195 1. 4 backends are created in a single MIG.
199 1. All backends receive at least one RPC.
201 The test driver records the peer distribution for a subsequent block of 100 RPCs
202 then stops the backends.
206 1. No RPCs from the client are successful.
208 The test driver resumes the backends.
212 1. Once all backends receive at least one RPC, the distribution for a block of
213 100 RPCs is the same as the distribution recorded prior to restart.
215 ### secondary_locality_gets_requests_on_primary_failure
217 This test verifies that backends in a secondary locality receive traffic when
218 all backends in the primary locality fail.
225 Load balancer configuration:
227 1. The primary MIG with 2 backends in the same zone as the client
228 1. The secondary MIG with 2 backends in a different zone
232 1. All backends in the primary locality receive at least 1 RPC.
233 1. No backends in the secondary locality receive RPCs.
235 The test driver stops the backends in the primary locality.
239 1. All backends in the secondary locality receive at least 1 RPC.
241 The test driver resumes the backends in the primary locality.
245 1. All backends in the primary locality receive at least 1 RPC.
246 1. No backends in the secondary locality receive RPCs.
248 ### secondary_locality_gets_no_requests_on_partial_primary_failure
250 This test verifies that backends in a failover locality do not receive traffic
251 when at least one of the backends in the primary locality remain healthy.
253 **Note:** Future TD features may change the expected behavior and require
254 changes to this test case.
261 Load balancer configuration:
263 1. The primary MIG with 2 backends in the same zone as the client
264 1. The secondary MIG with 2 backends in a different zone
268 1. All backends in the primary locality receive at least 1 RPC.
269 1. No backends in the secondary locality receive RPCs.
271 The test driver stops one of the backends in the primary locality.
275 1. All backends in the primary locality receive at least 1 RPC.
276 1. No backends in the secondary locality receive RPCs.
278 ### remove_instance_group
280 This test verifies that a remaining instance group can successfully serve RPCs
281 after removal of another instance group in the same zone.
288 Load balancer configuration:
290 1. Two MIGs with two backends each, using rate balancing mode.
294 1. All backends receive at least one RPC.
296 The test driver removes one MIG.
300 1. All RPCs are directed to the two remaining backends (no RPC failures).
302 ### change_backend_service
304 This test verifies that the backend service can be replaced and traffic routed
311 1. --fail_on_failed_rpc=true
313 Load balancer configuration:
315 1. One MIG with two backends
319 1. All backends receive at least one RPC.
321 The test driver creates a new backend service containing a MIG with two backends
322 and changes the TD URL map to point to this new backend service.
326 1. All RPCs are directed to the new backend service.
328 ### traffic_splitting
330 This test verifies that the traffic will be distributed between backend
331 services with the correct weights when route action is set to weighted
339 Load balancer configuration:
341 1. One MIG with one backend
345 1. Once all backends receive at least one RPC, the following 1000 RPCs are
348 The test driver adds a new MIG with 1 backend, and changes the route action
349 to weighted backend services with {a: 20, b: 80}.
353 1. Once all backends receive at least one RPC, the following 1000 RPCs are
354 distributed across the 2 backends as a: 20, b: 80.
357 This test verifies that the traffic for a certain RPC can be routed to a
358 specific cluster based on the RPC path.
364 1. –fail_on_failed_rpc=true
365 1. –rpc=“EmptyCall,UnaryCall”
367 Load balancer configuration:
369 1. 2 MIGs, each with 1 backend
375 1. UnaryCall RPCs are sent to MIG_default
376 1. EmptyCall RPCs are sent to MIG_default
378 The test driver changes route and asserts RPCs are sent to expected backends. **Note** that the default route `"/"` is always pointing to MIG_default, so all RPCs not matching the new route will be sent to MIG_default.
380 - {path: `/grpc.testing.TestService/EmptyCall`}: MIG_2
381 - UnaryCall -> MIG_default
384 - {prefix: `/grpc.testing.TestService/Unary`}: MIG_2
386 - EmptyCall -> MIG_default
388 - {prefix: `/grpc.testing.TestService/Unary`}: MIG_default & {path: `/grpc.testing.TestService/EmptyCall`}: MIG_2
389 - UnaryCall -> MIG_default
392 - {regex: `^\/.*\/UnaryCall$`}: MIG_2
394 - EmptyCall -> MIG_default
396 - {path: `/gRpC.tEsTinG.tEstseRvice/empTycaLl`, ignoreCase: `True`}: MIG_2
397 - UnaryCall -> MIG_default
402 This test verifies that the traffic for a certain RPC can be routed to a
403 specific cluster based on the RPC header (metadata).
409 1. –fail_on_failed_rpc=true
410 1. –rpc=“EmptyCall,UnaryCall”
411 1. –rpc=“EmptyCall:xds_md:exact_match”
413 Load balancer configuration:
415 1. 2 MIGs, each with 1 backend
421 1. UnaryCall RPCs are sent to MIG_default
422 1. EmptyCall RPCs are sent to MIG_default
424 The test driver changes route and asserts RPCs are sent to expected backends. **Note** that the default route `"/"` is always pointing to MIG_default, so all RPCs not matching the new route will be sent to MIG_default.
426 - {header `xds_md`, exact: `empty_ytpme`}: MIG_2
427 - Unary -> MIG_default
430 - {header `xds_md`, prefix: `un`}: MIG_2
431 - `un` is the prefix of metadata sent with UnaryCall
433 - Empty -> MIG_default
435 - {header `xds_md`, suffix: `me`}: MIG_2
436 - `me` is the suffix of metadata sent with EmptyCall
437 - Unary -> MIG_default
440 - {header `xds_md_numeric`, present: `True`}: MIG_2
441 - Unary is sent with the metadata, so will be sent to alternative
443 - Empty -> MIG_default
445 - {header `xds_md`, exact: `unary_yranu`, invert: `True`}: MIG_2
446 - Unary is sent with the metadata, so this will not match Unary, but will match Empty
447 - Unary -> MIG_default
450 - {header `xds_md_numeric`, range `[100,200]`}: MIG_2
451 - Unary is sent with the metadata in range
453 - Empty -> MIG_default
455 - {header `xds_md`, regex: `^em.*me$`}: MIG_2
456 - EmptyCall is sent with the metadata
457 - Unary -> MIG_default
462 This test verifies that traffic is partially diverted to a secondary locality
463 when > 50% of the instances in the primary locality are unhealthy.
470 Load balancer configuration:
472 1. The primary MIG with 3 backends in the same zone as the client
473 1. The secondary MIG with 2 backends in a different zone
477 1. All backends in the primary locality receive at least 1 RPC.
478 1. No backends in the secondary locality receive RPCs.
480 The test driver stops 2 of 3 backends in the primary locality.
484 1. All backends in the secondary locality receive at least 1 RPC.
485 1. The remaining backend in the primary locality receives at least 1 RPC.
487 The test driver resumes the backends in the primary locality.
491 1. All backends in the primary locality receive at least 1 RPC.
492 1. No backends in the secondary locality receive RPCs.
495 ### load_based_failover
497 This test verifies that traffic is partially diverted to a secondary locality
498 when the QPS is greater than the configured RPS in the priority locality.
505 Load balancer configuration:
507 1. The primary MIG with 2 backends in the same zone as the client
508 1. The secondary MIG with 2 backends in a different zone
512 1. All backends in the primary locality receive at least 1 RPC.
513 1. No backends in the secondary locality receive RPCs.
515 The test driver sets `balancingMode` is `RATE`, and `maxRate` to 20 in the primary locality.
519 1. All backends in the primary locality receive at least 1 RPC.
520 1. All backends in the secondary locality receive at least 1 RPC.
522 The test driver set `maxRate` to 120 in the primary locality.
526 1. All backends in the primary locality receive at least 1 RPC.
527 1. No backends in the secondary locality receive RPCs.
532 This test verifies that the maximum number of outstanding requests is limited
533 by circuit breakers of the backend service.
540 Load balancer configuration:
542 1. Two MIGs with each having two backends.
544 The test driver configures the backend services with:
546 1. path{“/grpc.testing.TestService/UnaryCall"}: MIG_1
547 1. path{“/grpc.testing.TestService/EmptyCall"}: MIG_2
548 1. MIG_1 circuit_breakers with max_requests = 500
549 1. MIG_2 circuit breakers with max_requests = 1000
551 The test driver configures the test client to send both UnaryCall and EmptyCall,
552 with all RPCs keep-open.
556 1. After reaching steady state, there are 500 UnaryCall RPCs in-flight
557 and 1000 EmptyCall RPCs in-flight.
559 The test driver updates MIG_1's circuit breakers with max_request = 800.
563 1. After reaching steady state, there are 800 UnaryCall RPCs in-flight.
567 This test verifies that traffic along a route with a `max_stream_duration` set
568 will cause timeouts on streams open longer than that duration.
572 1. `--num_channels=1`
579 1. Path match for `/grpc.testing.TestService/UnaryCall`, with a `route_action`
580 containing `max_stream_duration` of 3 seconds.
581 1. Default route containing no `max_stream_duration` setting.
583 There are four sub-tests:
585 1. `app_timeout_exceeded`
586 1. Test client configured to send UnaryCall RPCs with a 1s application
587 timeout, and metadata of `rpc-behavior: sleep-2`.
588 1. Test driver asserts client recieves ~100% status `DEADLINE_EXCEEDED`.
589 1. `timeout_not_exceeded`
590 1. Test client configured to send UnaryCall RPCs with the default
591 application timeout (20 seconds), and no metadata.
592 1. Test driver asserts client recieves ~100% status `OK`.
593 1. `timeout_exceeded` (executed with the below test case)
594 1. `timeout_different_route`
595 1. Test client configured to send UnaryCall RPCs and EmptyCall RPCs with
596 the default application timeout (20 seconds), and metadata of
597 `rpc-behavior: sleep-4`.
598 1. Test driver asserts client recieves ~100% status `OK` for EmptyCall
599 and ~100% status `DEADLINE_EXCEEDED` for UnaryCall.