Skip to content

Commit 41987ef

Browse files
rnorthrnorth
authored
Add Toxiproxy module (#1330)
Based on https://gist.github.com/rnorth/4c3666d62fa93bf0daa813b282e4ebff, in turn based on [hotels.com’s blog post](https://medium.com/hotels-com-technology/i-dont-know-about-resilience-testing-and-so-can-you-b3c59d80012d) where they show use of Testcontainers with Toxiproxy. Having seen Toxiproxy and experimented with it, I'm inclined to go with this and suggest that we not move forward with #283 (which is a great PR, but we've unfortunately failed to land after quite a long time). While Pumba may be nice and transparent, there are a lot of moving parts and complexity under the covers, vs Toxiproxy, which is a bit less magical but easier to understand.
1 parent 31b3610 commit 41987ef

File tree

6 files changed

+318
-4
lines changed

6 files changed

+318
-4
lines changed

‎.travis.yml‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,6 @@ language: java
22
jdk:
33
- oraclejdk8
44

5-
branches:
6-
only:
7-
- master
8-
95
sudo: required
106
services:
117
- docker

‎docs/modules/toxiproxy.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Toxiproxy Module
2+
3+
Testcontainers module for Shopify's [Toxiproxy](https://github.com/Shopify/toxiproxy).
4+
This TCP proxy can be used to simulate network failure conditions in between tests and containers.
5+
6+
[toxiproxy-java](https://github.com/trekawek/toxiproxy-java) is used as a client.
7+
8+
## Usage example
9+
10+
A Toxiproxy container can be placed in between test code and a container, or in between containers.
11+
In either scenario, it is necessary to create a `ToxiproxyContainer` instance on the same Docker network, as follows:
12+
13+
<!--codeinclude-->
14+
[Creating a Toxiproxy container](../../modules/toxiproxy/src/test/java/org/testcontainers/containers/ToxiproxyTest.java) inside_block:creatingProxy
15+
<!--/codeinclude-->
16+
17+
Next, it is necessary to instruct Toxiproxy to start proxying connections.
18+
Each `ToxiproxyContainer` can proxy to many target containers if necessary.
19+
20+
We do this as follows:
21+
22+
<!--codeinclude-->
23+
[Starting proxying connections to a target container](../../modules/toxiproxy/src/test/java/org/testcontainers/containers/ToxiproxyTest.java) inside_block:obtainProxyObject
24+
<!--/codeinclude-->
25+
26+
Then, to establish a connection via Toxiproxy, we obtain **Toxiproxy's** proxy host IP and port:
27+
28+
<!--codeinclude-->
29+
[Obtaining proxied host and port](../../modules/toxiproxy/src/test/java/org/testcontainers/containers/ToxiproxyTest.java) inside_block:obtainProxiedHostAndPort
30+
<!--/codeinclude-->
31+
32+
Code under test, or other containers, should connect to this proxied host IP and port.
33+
34+
Having done this, it is possible to trigger failure conditions ('Toxics') through the `proxy.toxics()` object:
35+
36+
* `bandwidth` - Limit a connection to a maximum number of kilobytes per second.
37+
* `latency` - Add a delay to all data going through the proxy. The delay is equal to `latency +/- jitter`.
38+
* `slicer` - Slices TCP data up into small bits, optionally adding a delay between each sliced "packet".
39+
* `slowClose` - Delay the TCP socket from closing until `delay` milliseconds has elapsed.
40+
* `timeout` - Stops all data from getting through, and closes the connection after `timeout`. If `timeout` is `0`, the connection won't close, and data will be delayed until the toxic is removed.
41+
* `limitData` - Closes connection when transmitted data exceeded limit.
42+
43+
Please see the [Toxiproxy documentation](https://github.com/Shopify/toxiproxy#toxics) for full details on the available Toxics.
44+
45+
As one example, we can introduce latency and random jitter to proxied connections as follows:
46+
47+
<!--codeinclude-->
48+
[Adding latency to a connection](../../modules/toxiproxy/src/test/java/org/testcontainers/containers/ToxiproxyTest.java) inside_block:addingLatency
49+
<!--/codeinclude-->
50+
51+
Additionally we can disable the proxy to simulate a complete interruption to the network connection:
52+
53+
<!--codeinclude-->
54+
[Cutting a connection](../../modules/toxiproxy/src/test/java/org/testcontainers/containers/ToxiproxyTest.java) inside_block:disableProxy
55+
<!--/codeinclude-->
56+
57+
## Adding this module to your project dependencies
58+
59+
Add the following dependency to your `pom.xml`/`build.gradle` file:
60+
61+
```groovy tab='Gradle'
62+
testCompile "org.testcontainers:toxiproxy:{{latest_version}}"
63+
```
64+
65+
```xml tab='Maven'
66+
<dependency>
67+
<groupId>org.testcontainers</groupId>
68+
<artifactId>toxiproxy</artifactId>
69+
<version>{{latest_version}}</version>
70+
<scope>test</scope>
71+
</dependency>
72+
```
73+
74+
## Acknowledgements
75+
76+
This module was inspired by a [hotels.com blog post](https://medium.com/hotels-com-technology/i-dont-know-about-resilience-testing-and-so-can-you-b3c59d80012d).

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@ nav:
5656
- modules/pulsar.md
5757
- modules/localstack.md
5858
- modules/mockserver.md
59+
- modules/toxiproxy.md
5960
- modules/nginx.md
6061
- modules/vault.md
6162
- Test framework integration:

‎modules/toxiproxy/build.gradle‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
description = "Testcontainers :: Toxiproxy"
2+
3+
dependencies {
4+
compile project(':testcontainers')
5+
compile 'eu.rekawek.toxiproxy:toxiproxy-java:2.1.3'
6+
7+
testCompile 'redis.clients:jedis:3.0.1'
8+
testCompile 'org.rnorth.visible-assertions:visible-assertions:2.1.2'
9+
}

‎modules/toxiproxy/src/main/java/org/testcontainers/containers/ToxiproxyContainer.java‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
package org.testcontainers.containers;
2+
3+
import com.github.dockerjava.api.command.InspectContainerResponse;
4+
import eu.rekawek.toxiproxy.Proxy;
5+
import eu.rekawek.toxiproxy.ToxiproxyClient;
6+
import eu.rekawek.toxiproxy.model.ToxicDirection;
7+
import eu.rekawek.toxiproxy.model.ToxicList;
8+
import lombok.AccessLevel;
9+
import lombok.Getter;
10+
import lombok.RequiredArgsConstructor;
11+
import org.testcontainers.containers.wait.strategy.HttpWaitStrategy;
12+
13+
import java.io.IOException;
14+
import java.util.HashMap;
15+
import java.util.Map;
16+
import java.util.concurrent.atomic.AtomicInteger;
17+
18+
/**
19+
* Container for resiliency testing using <a href="https://github.com/Shopify/toxiproxy">Toxiproxy</a>.
20+
*/
21+
public class ToxiproxyContainer extends GenericContainer<ToxiproxyContainer> {
22+
23+
private static final String IMAGE_NAME = "shopify/toxiproxy:2.1.0";
24+
private static final int TOXIPROXY_CONTROL_PORT = 8474;
25+
private static final int FIRST_PROXIED_PORT = 8666;
26+
private static final int LAST_PROXIED_PORT = 8666 + 31;
27+
28+
private ToxiproxyClient client;
29+
private final Map<String, ContainerProxy> proxies = new HashMap<>();
30+
private final AtomicInteger nextPort = new AtomicInteger(FIRST_PROXIED_PORT);
31+
32+
public ToxiproxyContainer() {
33+
this(IMAGE_NAME);
34+
}
35+
36+
public ToxiproxyContainer(String imageName) {
37+
super(imageName);
38+
addExposedPorts(TOXIPROXY_CONTROL_PORT);
39+
setWaitStrategy(new HttpWaitStrategy().forPath("/version").forPort(TOXIPROXY_CONTROL_PORT));
40+
41+
// allow up to 32 ports to be proxied (arbitrary value). Here we make the ports exposed; whether or not
42+
// Toxiproxy will listen is controlled at runtime using getProxy(...)
43+
for (int i = FIRST_PROXIED_PORT; i <= LAST_PROXIED_PORT; i++) {
44+
addExposedPort(i);
45+
}
46+
}
47+
48+
@Override
49+
protected void containerIsStarted(InspectContainerResponse containerInfo) {
50+
client = new ToxiproxyClient(getContainerIpAddress(), getMappedPort(TOXIPROXY_CONTROL_PORT));
51+
}
52+
53+
/**
54+
* Obtain a {@link ContainerProxy} instance for target container that is managed by Testcontainers. The target
55+
* container should be routable from this <b>from this {@link ToxiproxyContainer} instance</b> (e.g. on the same
56+
* Docker {@link Network}).
57+
*
58+
* @param container target container
59+
* @param port port number on the target service that should be proxied
60+
* @return a {@link ContainerProxy} instance
61+
*/
62+
public ContainerProxy getProxy(GenericContainer<?> container, int port) {
63+
return this.getProxy(container.getNetworkAliases().get(0), port);
64+
}
65+
66+
/**
67+
* Obtain a {@link ContainerProxy} instance for a specific hostname and port, which can be for any host
68+
* that is routable <b>from this {@link ToxiproxyContainer} instance</b> (e.g. on the same
69+
* Docker {@link Network} or on routable from the Docker host).
70+
*
71+
* <p><em>It is expected that {@link ToxiproxyContainer#getProxy(GenericContainer, int)} will be more
72+
* useful in most scenarios, but this method is present to allow use of Toxiproxy in front of containers
73+
* or external servers that are not managed by Testcontainers.</em></p>
74+
*
75+
* @param hostname hostname of target server to be proxied
76+
* @param port port number on the target server that should be proxied
77+
* @return a {@link ContainerProxy} instance
78+
*/
79+
public ContainerProxy getProxy(String hostname, int port) {
80+
String upstream = hostname + ":" + port;
81+
82+
return proxies.computeIfAbsent(upstream, __ -> {
83+
try {
84+
final int toxiPort = nextPort.getAndIncrement();
85+
if (toxiPort > LAST_PROXIED_PORT) {
86+
throw new IllegalStateException("Maximum number of proxies exceeded");
87+
}
88+
89+
final Proxy proxy = client.createProxy("name", "0.0.0.0:" + toxiPort, upstream);
90+
return new ContainerProxy(proxy, getContainerIpAddress(), getMappedPort(toxiPort));
91+
} catch (IOException e) {
92+
throw new RuntimeException("Proxy could not be created", e);
93+
}
94+
});
95+
}
96+
97+
@RequiredArgsConstructor(access = AccessLevel.PROTECTED)
98+
public static class ContainerProxy {
99+
private static final String CUT_CONNECTION_DOWNSTREAM = "CUT_CONNECTION_DOWNSTREAM";
100+
private static final String CUT_CONNECTION_UPSTREAM = "CUT_CONNECTION_UPSTREAM";
101+
private final Proxy toxi;
102+
@Getter private final String containerIpAddress;
103+
@Getter private final int proxyPort;
104+
private boolean isCurrentlyCut;
105+
106+
public ToxicList toxics() {
107+
return toxi.toxics();
108+
}
109+
110+
/**
111+
* Cuts the connection by setting bandwidth in both directions to zero.
112+
* @param shouldCutConnection true if the connection should be cut, or false if it should be re-enabled
113+
*/
114+
public void setConnectionCut(boolean shouldCutConnection) {
115+
try {
116+
if (shouldCutConnection) {
117+
toxics().bandwidth(CUT_CONNECTION_DOWNSTREAM, ToxicDirection.DOWNSTREAM, 0);
118+
toxics().bandwidth(CUT_CONNECTION_UPSTREAM, ToxicDirection.UPSTREAM, 0);
119+
isCurrentlyCut = true;
120+
} else if (isCurrentlyCut) {
121+
toxics().get(CUT_CONNECTION_DOWNSTREAM).remove();
122+
toxics().get(CUT_CONNECTION_UPSTREAM).remove();
123+
isCurrentlyCut = false;
124+
}
125+
} catch (IOException e) {
126+
throw new RuntimeException("Could not control proxy", e);
127+
}
128+
}
129+
}
130+
}

‎modules/toxiproxy/src/test/java/org/testcontainers/containers/ToxiproxyTest.java‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
package org.testcontainers.containers;
2+
3+
import eu.rekawek.toxiproxy.model.ToxicDirection;
4+
import org.junit.Rule;
5+
import org.junit.Test;
6+
import redis.clients.jedis.Jedis;
7+
import redis.clients.jedis.exceptions.JedisConnectionException;
8+
9+
import java.io.IOException;
10+
11+
import static java.lang.String.format;
12+
import static org.rnorth.visibleassertions.VisibleAssertions.*;
13+
14+
public class ToxiproxyTest {
15+
16+
// creatingProxy {
17+
// Create a common docker network so that containers can communicate
18+
@Rule
19+
public Network network = Network.newNetwork();
20+
21+
// the target container - this could be anything
22+
@Rule
23+
public GenericContainer redis = new GenericContainer("redis:5.0.4")
24+
.withExposedPorts(6379)
25+
.withNetwork(network);
26+
27+
// Toxiproxy container, which will be used as a TCP proxy
28+
@Rule
29+
public ToxiproxyContainer toxiproxy = new ToxiproxyContainer()
30+
.withNetwork(network);
31+
// }
32+
33+
@Test
34+
public void testDirect() {
35+
final Jedis jedis = new Jedis(redis.getContainerIpAddress(), redis.getFirstMappedPort());
36+
jedis.set("somekey", "somevalue");
37+
38+
final String s = jedis.get("somekey");
39+
assertEquals("direct access to the container works OK", "somevalue", s);
40+
}
41+
42+
@Test
43+
public void testLatencyViaProxy() throws IOException {
44+
// obtainProxyObject {
45+
final ToxiproxyContainer.ContainerProxy proxy = toxiproxy.getProxy(redis, 6379);
46+
// }
47+
48+
// obtainProxiedHostAndPort {
49+
final String ipAddressViaToxiproxy = proxy.getContainerIpAddress();
50+
final int portViaToxiproxy = proxy.getProxyPort();
51+
// }
52+
53+
final Jedis jedis = new Jedis(ipAddressViaToxiproxy, portViaToxiproxy);
54+
jedis.set("somekey", "somevalue");
55+
56+
checkCallWithLatency(jedis, "without interference", 0, 250);
57+
58+
// addingLatency {
59+
proxy.toxics()
60+
.latency("latency", ToxicDirection.DOWNSTREAM, 1_100)
61+
.setJitter(100);
62+
// from now on the connection latency should be from 1000-1200 ms.
63+
// }
64+
65+
checkCallWithLatency(jedis, "with interference", 1_000, 1_500);
66+
}
67+
68+
@Test
69+
public void testConnectionCut() {
70+
final ToxiproxyContainer.ContainerProxy proxy = toxiproxy.getProxy(redis, 6379);
71+
final Jedis jedis = new Jedis(proxy.getContainerIpAddress(), proxy.getProxyPort());
72+
jedis.set("somekey", "somevalue");
73+
74+
assertEquals("access to the container works OK before cutting the connection", "somevalue", jedis.get("somekey"));
75+
76+
// disableProxy {
77+
proxy.setConnectionCut(true);
78+
79+
// for example, expect failure when the connection is cut
80+
assertThrows("calls fail when the connection is cut",
81+
JedisConnectionException.class, () -> {
82+
jedis.get("somekey");
83+
});
84+
85+
proxy.setConnectionCut(false);
86+
87+
// and with the connection re-established, expect success
88+
assertEquals("access to the container works OK after re-establishing the connection", "somevalue", jedis.get("somekey"));
89+
// }
90+
}
91+
92+
private void checkCallWithLatency(Jedis jedis, final String description, int expectedMinLatency, long expectedMaxLatency) {
93+
final long start = System.currentTimeMillis();
94+
String s = jedis.get("somekey");
95+
final long end = System.currentTimeMillis();
96+
final long duration = end - start;
97+
98+
assertEquals(format("access to the container %s works OK", description), "somevalue", s);
99+
assertTrue(format("%s there is at least %dms latency", description, expectedMinLatency), duration >= expectedMinLatency);
100+
assertTrue(format("%s there is no more than %dms latency", description, expectedMaxLatency), duration < expectedMaxLatency);
101+
}
102+
}

0 commit comments

Comments
 (0)