Simple, configurable, local (VM) chaotic operation orchestrator for measuring the resiliency of cloud services by injecting bedlam (machine resource and networking faults) into underlying virtual machines. This version is meant to run inside Linux VMs (and containers).
Step 0.
Just change XML settings to meet your specific chaotic needs. The default config will run CPU, Memory and Networking chaos. You can remove the CPU and Memory nodes and just do Network emulation or remove Network and just do CPU/Mem. It's configurable, so do what you want!
Currently supported network emulation operations:
Packet Corruption
Packet Loss
Packet Reordering
Bandwidth Rate Limiting
Latency
For example, the below configuration XML sequentially runs (according to specified run order) a CPU pressure fault of 90% CPU utilization across all CPUs for 15 seconds, Memory pressure fault eating 90% of available memory for 15 seconds, and Network Latency emulation fault of 3000ms delay for 30 seconds for specified target endpoints. The experiment runs 2 times successively (Repeat=”1”). See Chaos.config for more info on available configuration settings, including samples. CloudBedlam will execute (and log) the orchestration of these bedlam operations. You just need to modify some XML and then experiment away. Enjoy!
<ChaosConfiguration Orchestration="Sequential" Duration="60" RunDelay="0" Repeat="1">
<CpuPressure RunOrder="0">
<PressureLevel>90</PressureLevel>
<Duration>15</Duration>
</CpuPressure>
<MemoryPressure RunOrder="1">
<PressureLevel>90</PressureLevel>
<Duration>15</Duration>
</MemoryPressure>
<NetworkEmulation RunOrder="2">
<EmulationType>Latency</EmulationType>
<LatencyDelay>3000</LatencyDelay>
<TargetEndpoints>
<Endpoint Port="443" Uri="https://www.bing.com" />
<Endpoint Port="80" Uri="http://www.msn.com" />
<Endpoint Port="443" Uri="https://www.google.com" />
</TargetEndpoints>
<Duration>30</Duration>
</NetworkEmulation>
</ChaosConfiguration>
Step 1. (Mono)
Install MonoDevelop: http://www.monodevelop.com/download/linux/
Step 2:
Clone project:
git clone https://github.com/GitTorre/CloudBedlamLinux.gitOpen sln in MonoDevelop, build.
If you don't want to install the dependencies and build CloudBedlam, then just grab the latest release and install CloudBedlam and dependencies from a package (file name will be CloudBedlam-linux-amd64):
https://github.com/GitTorre/CloudBedlamLinux/releases
CloudBedlam must run as sudo:
sudo mono CloudBedlam.exe
When running CloudBedlam, a bedlamlogs folder will be created in the folder where the CloudBedlam binary is running. Output file will contain INFO and ERROR lines (ERROR lines will include error messages and stack traces).
Of course, please help make this better 😊 – and add whatever you need around and inside the core bedlam engine (which is what this is, really). The focus for us is on making a very easy to use, simple to configure, lightweight solution for chaos engineering and experimentation inside virtual machines.
Have fun and hopefully this proves useful to you in your service resiliency experimentation. It should be clear that this is a development tool at this stage and not a DevOps workflow orchestrator. You should run this in individual VMs to vet the quality of your code in terms of resiliency and fault tolerance.
Any and all feedback very welcome. Let us know if you use this and if it helps uncover resiliency/fault tolerance issues in your service implementation. You can send mail to [email protected] and/or create Issues here. Thank you! This will continue to evolve and your contributions, in whatever form (words or code), will be greatly appreciated!
Make bedlam, not war!
--CloudBedlam Team