# How to Use Antithesis

## Context
Antithesis is a third party vendor with an environment that can perform network fuzzing. We can 
upload images containing `docker-compose.yml` files, which represent various MongoDB topologies, to 
the Antithesis Docker registry. Antithesis runs `docker-compose up` from these images to spin up 
the corresponding multi-container application in their environment and run a test suite. Network 
fuzzing is performed between the containers while the test suite runs & a report is generated by 
Antithesis identifying bugs.

## Base Images
The `base_images` directory consists of the building blocks for creating a MongoDB test topology. 
These images are uploaded to the Antithesis Docker registry weekly during the 
`antithesis_image_build` task. For more visibility into how these images are built and uploaded to 
the Antithesis Docker registry, please see `evergreen/antithesis_image_build.sh`.

### mongo_binaries
This image contains the latest `mongo`, `mongos` and `mongod` binaries. It can be used to 
start a `mongod` instance, `mongos` instance or execute `mongo` commands. This is the main building 
block for creating the System Under Test topology.

### workload
This image contains the latest `mongo` binary as well as the `resmoke` test runner. The `workload` 
container is not part of the actual toplogy. The purpose of a `workload` container is to execute 
`mongo` commands to complete the topology setup, and to run a test suite on an existing topology 
like so:
```shell
buildscript/resmoke.py run --suite antithesis_concurrency_sharded_with_stepdowns_and_balancer --shellConnString "mongodb://mongos:27017"
```

**Every topology must have 1 workload container.**

Note: During `workload` image build, `buildscripts/antithesis_suite.py` runs, which generates 
"antithesis compatible" test suites and prepends them with `antithesis_`. These are the test suites 
that can run in antithesis and are available from witihin the `workload` container.

## Topologies
The `topologies` directory consists of subdirectories representing various mongo test topologies. 
Each topology has a `Dockerfile`, a `docker-compose.yml` file and an optional `scripts` directory.

### Dockerfile
This assembles an image with the necessary files for spinning up the corresponding topology. It 
consists of a `docker-compose.yml`, a `logs` directory and an optional `scripts` directory. If this 
is structured properly, you should be able to copy the files from this image and run 
`docker-compose up` to set up the desired multi-container topology.

Example from `buildscripts/antithesis/topologies/replica_set/Dockerfile`:
```Dockerfile
FROM scratch
COPY docker-compose.yml /
ADD scripts /scripts
ADD logs /logs
```

All topology images are built and uploaded to the Antithesis Docker registry during the 
`antithesis_image_build` task in the `evergreen/antithesis_image_build.sh` script.

Note: These image serve solely as a filesystem containing all necessary files for a topology, 
therefore use `FROM scratch`.

### docker-compose.yml
 This describes how to construct the corresponding topology using the 
 `mongo-binaries` and `workload` images.

Example from `buildscripts/antithesis/topologies/replica_set/docker-compose.yml`:
```yml
version: '3.0'

services:
        database1:
                container_name: database1
                hostname: database1
                image: mongo-binaries:evergreen-latest-master
                command: /usr/bin/mongod --bind_ip 0.0.0.0 --replSet RollbackFuzzerTest --logpath /var/log/mongodb/mongodb.log --setParameter enableTestCommands=1
                volumes:
                  - ./logs/database1:/var/log/mongodb/
                networks:
                        antithesis-net:
                                ipv4_address: 10.20.20.3
                                # Set the an IPv4 with an address of 10.20.20.130 or higher
                                # to be ignored by the fault injector
                                #
        database2: ...
        database3: ...
        workload:
                container_name: workload
                hostname: workload
                image: workload:evergreen-latest-master
                command: /bin/bash /scripts/workload_init.sh
                volumes:
                  - ./scripts:/scripts/
                depends_on:
                        - "database1"
                        - "database2"
                        - "database3"
                networks:
                        antithesis-net:
                                ipv4_address: 10.20.20.130
                                # The subnet provided here is an example
                                # An alternative subnet can be used

networks:
        antithesis-net:
                driver: bridge
                ipam:
                        config:
                        - subnet: 10.20.20.0/24

```

Each container must have a `command`. If it is a single line it can be included directly in the 
`docker-compose.yml` -- as in `database1`. If it is more involved, it can be put in a shell script 
in the `scripts` directory, included as a volume and `command` can be set to 
`/bin/bash /scripts/[script_name].sh` -- as in `workload`. 

When creating `mongod` or `mongos` instances, route the logs like so: 
`--logpath /var/log/mongodb/mongodb.log` and utilize `volumes` -- as in `database1`. This enables 
us to easily retrieve logs if a bug is detected by Antithesis.  

The `ipv4_address` should be set to `10.20.20.130` or higher if you do not want that container to 
be affected by network fuzzing. For instance, you would likely not want the `workload` container  
to be affected by network fuzzing -- as shown in the example above.

Use the `evergreen-latest-master` tag for all images. This is updated automatically in 
`evergreen/antithesis_image_build.sh` during the `antithesis_image_build` task -- if needed. 

### scripts
This is an optional directory, which can be used for the reason described above.

Example from `buildscripts/antithesis/topologies/replica_set/scripts/workload_init.sh`:
```shell
sleep 5s
mongo --host database1 --port 27017 --eval "config={\"_id\" : \"RollbackFuzzerTest\",\"protocolVersion\" : 1,\"members\" : [{\"_id\" : 0,\"host\" : \"database1:27017\"}, {\"_id\" : 1,\"host\" : \"database2:27017\"}, {\"_id\" : 2,\"host\" : \"database3:27017\"} ],\"settings\" : {\"chainingAllowed\" : false,\"electionTimeoutMillis\" : 500, \"heartbeatTimeoutSecs\" : 1, \"catchUpTimeoutMillis\": 700}}; rs.initiate(config)"

# this cryptic statement keeps the workload container running.
tail -f /dev/null
```
The `sleep` command can be useful to ensure that other containers have had a chance to start. In 
this example, the `workload` container waits 5 seconds while the database containers start. 
After that, it initiates the replica set. It is now ready to run a test suite using `resmoke`.
The `tail -f /dev/null` is required for `workload` containers otherwise the container shuts down.

## How do I create a new topology for Antithesis testing?
To create a new topology for Antithesis testing is easy & requires a few simple steps. 
1. Add a new directory in `buildscripts/antithesis/topologies` to represent your desired topology. 
You can use existing topologies as an example.
2. Update the `evergreen/antithesis_image_build.sh` file so that your new topology image is 
uploaded to the Antithesis Docker registry.
3. Reach out to #server-testing on Slack & provide the new topology image name as well as the 
   desired test suite to run.
4. Include a member of the STM team on the code review.
   
These are the required updates to `evergreen/antithesis_image_build.sh`:
- Add the following command for each of your `mongos` and `mongod` containers in your topology to 
create your log directories.
```shell
mkdir -p antithesis/topologies/[topology_name]/logs/[container_name]
```
     
- Build an image for your new topology ending in `-config`
```shell
cd [your_topology_dir]
sed -i s/evergreen-latest-master/$tag/ docker-compose.yml
docker build . -t [your-topology-name]-config:$tag
```
- Push your new image to the Antithesis Docker registry
```shell
docker tag "[your-topology-name]-config:$tag" "us-central1-docker.pkg.dev/molten-verve-216720/mongodb-repository/[your-topology-name]-config:$tag"
docker push "us-central1-docker.pkg.dev/molten-verve-216720/mongodb-repository/[your-topology-name]-config:$tag"
```