1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
|
# How to Use Antithesis
## Context
Antithesis is a third party vendor with an environment that can perform network fuzzing. We can
upload images containing `docker-compose.yml` files, which represent various MongoDB topologies, to
the Antithesis Docker registry. Antithesis runs `docker-compose up` from these images to spin up
the corresponding multi-container application in their environment and run a test suite. Network
fuzzing is performed on the topology while the test suite runs & a report is generated by
Antithesis identifying bugs. Check out
https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis to see an example of how we
use Antithesis today.
## Base Images
The `base_images` directory consists of the building blocks for creating a MongoDB test topology.
These images are uploaded to the Antithesis Docker registry weekly during the
`antithesis_image_build` task. For more visibility into how these images are built and uploaded to
the Antithesis Docker registry, please see `evergreen/antithesis_image_build.sh`.
### mongo_binaries
This image contains the latest `mongo`, `mongos` and `mongod` binaries. It can be used to
start a `mongod` instance, `mongos` instance or execute `mongo` commands. This is the main building
block for creating the System Under Test topology.
### workload
This image contains the latest `mongo` binary as well as the `resmoke` test runner. The `workload`
container is not part of the actual toplogy. The purpose of a `workload` container is to execute
`mongo` commands to complete the topology setup, and to run a test suite on an existing topology
like so:
```shell
buildscript/resmoke.py run --suite antithesis_concurrency_sharded_with_stepdowns_and_balancer --shellConnString "mongodb://mongos:27017"
```
**Every topology must have 1 workload container.**
Note: During `workload` image build, `buildscripts/antithesis_suite.py` runs, which generates
"antithesis compatible" test suites and prepends them with `antithesis_`. These are the test suites
that can run in antithesis and are available from witihin the `workload` container.
## Topologies
The `topologies` directory consists of subdirectories representing various mongo test topologies.
Each topology has a `Dockerfile`, a `docker-compose.yml` file and a `scripts` directory.
### Dockerfile
This assembles an image with the necessary files for spinning up the corresponding topology. It
consists of a `docker-compose.yml`, a `logs` directory, a `scripts` directory and a `data`
directory. If this is structured properly, you should be able to copy the files & directories
from this image and run `docker-compose up` to set up the desired topology.
Example from `buildscripts/antithesis/topologies/replica_set/Dockerfile`:
```Dockerfile
FROM scratch
COPY docker-compose.yml /
ADD scripts /scripts
ADD logs /logs
ADD data /data
```
All topology images are built and uploaded to the Antithesis Docker registry during the
`antithesis_image_build` task in the `evergreen/antithesis_image_build.sh` script. Some of these
directories are created in `evergreen/antithesis_image_build.sh` such as `/data` and `/logs`.
Note: These images serve solely as a filesystem containing all necessary files for a topology,
therefore use `FROM scratch`.
### docker-compose.yml
This describes how to construct the corresponding topology using the
`mongo-binaries` and `workload` images.
Example from `buildscripts/antithesis/topologies/replica_set/docker-compose.yml`:
```yml
version: '3.0'
services:
database1:
container_name: database1
hostname: database1
image: mongo-binaries:evergreen-latest-master
command: /bin/bash /scripts/database_init.sh
volumes:
- ./logs/database1:/var/log/mongodb/
- ./scripts:/scripts/
- ./data/database1:/data/db/
networks:
antithesis-net:
ipv4_address: 10.20.20.3
# Set the an IPv4 with an address of 10.20.20.130 or higher
# to be ignored by the fault injector
#
database2: ...
database3: ...
workload:
container_name: workload
hostname: workload
image: workload:evergreen-latest-master
command: /bin/bash /scripts/workload_init.sh
volumes:
- ./logs/workload:/var/log/resmoke/
- ./scripts:/scripts/
depends_on:
- "database1"
- "database2"
- "database3"
networks:
antithesis-net:
ipv4_address: 10.20.20.130
# The subnet provided here is an example
# An alternative subnet can be used
networks:
antithesis-net:
driver: bridge
ipam:
config:
- subnet: 10.20.20.0/24
```
Each container must have a `command`in `docker-compose.yml` that runs an init script. The init
script belongs in the `scripts` directory, which is included as a volume. The `command` should be
set like so: `/bin/bash /scripts/[script_name].sh`. This is a requirement for the topology to start
up properly in Antithesis.
When creating `mongod` or `mongos` instances, route the logs like so:
`--logpath /var/log/mongodb/mongodb.log` and utilize `volumes` -- as in `database1`.
This enables us to easily retrieve logs if a bug is detected by Antithesis.
The `ipv4_address` should be set to `10.20.20.130` or higher if you do not want that container to
be affected by network fuzzing. For instance, you would likely not want the `workload` container
to be affected by network fuzzing -- as shown in the example above.
Use the `evergreen-latest-master` tag for all images. This is updated automatically in
`evergreen/antithesis_image_build.sh` during the `antithesis_image_build` task -- if needed.
### scripts
Example from `buildscripts/antithesis/topologies/replica_set/scripts/workload_init.sh`:
```shell
sleep 5s
mongo --host database1 --port 27017 --eval "config={\"_id\" : \"RollbackFuzzerTest\",\"protocolVersion\" : 1,\"members\" : [{\"_id\" : 0,\"host\" : \"database1:27017\"}, {\"_id\" : 1,\"host\" : \"database2:27017\"}, {\"_id\" : 2,\"host\" : \"database3:27017\"} ],\"settings\" : {\"chainingAllowed\" : false,\"electionTimeoutMillis\" : 500, \"heartbeatTimeoutSecs\" : 1, \"catchUpTimeoutMillis\": 700}}; rs.initiate(config)"
# this cryptic statement keeps the workload container running.
tail -f /dev/null
```
The `sleep` command can be useful to ensure that other containers have had a chance to start. In
this example, the `workload` container waits 5 seconds while the database containers start.
After that, it initiates the replica set. The `tail -f /dev/null` is required for `workload`
containers otherwise the container shuts down.
## How do I create a new topology for Antithesis testing?
To create a new topology for Antithesis testing is easy & requires a few simple steps.
1. Add a new directory in `buildscripts/antithesis/topologies` to represent your desired topology.
You can use existing topologies as an example.
2. Update the `evergreen/antithesis_image_build.sh` file so that your new topology image is
uploaded to the Antithesis Docker registry.
3. Reach out to #server-testing on Slack & provide the new topology image name as well as the
desired test suite to run.
4. Include a member of the STM team on the code review.
These are the required updates to `evergreen/antithesis_image_build.sh`:
- Add the following command for each of your `mongos` and `mongod` containers in your topology to
create your log directories.
```shell
mkdir -p antithesis/topologies/[topology_name]/{logs,data}/[container_name]
```
- Build an image for your new topology ending in `-config`
```shell
cd [your_topology_dir]
sed -i s/evergreen-latest-master/$tag/ docker-compose.yml
sudo docker build . -t [your-topology-name]-config:$tag
```
- Push your new image to the Antithesis Docker registry
```shell
sudo docker tag "[your-topology-name]-config:$tag" "us-central1-docker.pkg.dev/molten-verve-216720/mongodb-repository/[your-topology-name]-config:$tag"
sudo docker push "us-central1-docker.pkg.dev/molten-verve-216720/mongodb-repository/[your-topology-name]-config:$tag"
```
## Additional Resources
If you are interested in leveraging Antithesis feel free to reach out to #server-testing on Slack.
|
