diff options
| -rw-r--r-- | doc/container_registry/README.md | 4 | ||||
| -rw-r--r-- | doc/container_registry/img/mitmproxy-docker.png | bin | 0 -> 407004 bytes | |||
| -rw-r--r-- | doc/container_registry/troubleshooting.md | 139 |
3 files changed, 143 insertions, 0 deletions
diff --git a/doc/container_registry/README.md b/doc/container_registry/README.md index 55077197ff9..3db351811a8 100644 --- a/doc/container_registry/README.md +++ b/doc/container_registry/README.md @@ -90,6 +90,10 @@ your `.gitlab-ci.yml`, you have to follow the [Using a private Docker Registry][private-docker] documentation. This workflow will be simplified in the future. +## Troubleshooting + +See [the GitLab Docker registry troubleshooting guide](troubleshooting.md). + [ce-4040]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4040 [docker-docs]: https://docs.docker.com/engine/userguide/intro/ [private-docker]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#using-a-private-docker-registry diff --git a/doc/container_registry/img/mitmproxy-docker.png b/doc/container_registry/img/mitmproxy-docker.png Binary files differnew file mode 100644 index 00000000000..4e3e37b413d --- /dev/null +++ b/doc/container_registry/img/mitmproxy-docker.png diff --git a/doc/container_registry/troubleshooting.md b/doc/container_registry/troubleshooting.md new file mode 100644 index 00000000000..e21fc9a2c61 --- /dev/null +++ b/doc/container_registry/troubleshooting.md @@ -0,0 +1,139 @@ +# Troubleshooting the GitLab Container Registry + +## Basic Troubleshooting + +1. Check to make sure that the system clock on your Docker client and GitLab server have + been synchronized (e.g. via NTP). + +2. If you are using an S3-backed registry, double check that the IAM + permissions and the S3 credentials (including region) are correct. See [the + sample IAM policy](https://docs.docker.com/registry/storage-drivers/s3/) + for more details. + +3. Check the registry logs (e.g. `/var/log/gitlab/registry/current`) and the GitLab production logs + for errors (e.g. `/var/log/gitlab/gitlab-rails/production.log`). You may be able to find clues + there. + +# Advanced Troubleshooting + +NOTE: The following section is only recommended for experts. + +Sometimes it's not obvious what is wrong, and you may need to dive deeper into +the communication between the Docker client and the registry to find out +what's wrong. We will use a concrete example in the past to illustrate how to +diagnose a problem with the S3 setup. + +## Example: Unexpected 403 error during push + +A user attempted to enable an S3-backed registry. The `docker login` step went +fine. However, when pushing an image, the output showed: + +``` +The push refers to a repository [s3-testing.myregistry.com:4567/root/docker-test] +dc5e59c14160: Pushing [==================================================>] 14.85 kB +03c20c1a019a: Pushing [==================================================>] 2.048 kB +a08f14ef632e: Pushing [==================================================>] 2.048 kB +228950524c88: Pushing 2.048 kB +6a8ecde4cc03: Pushing [==> ] 9.901 MB/205.7 MB +5f70bf18a086: Pushing 1.024 kB +737f40e80b7f: Waiting +82b57dbc5385: Waiting +19429b698a22: Waiting +9436069b92a3: Waiting +error parsing HTTP 403 response body: unexpected end of JSON input: "" +``` + +This error is ambiguous, as it's not clear whether the 403 is coming from the GitLab Rails +application, the Docker registry, or something else. In this case, we know that since +the login succeeded, we probably need to look at the communication between the client +and the registry. + +The REST API between the Docker client and registry is [described +here](https://docs.docker.com/registry/spec/api/). Normally, one would just +use Wireshark or tcpdump to capture the traffic and see where things went +wrong. However, since all communication between Docker clients and servers +are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even +if you know the private key. What can we do instead? + +## mitmproxy + +Enter [mitmproxy](https://mitmproxy.org/). This tool allows you to place a +proxy between your client and server to inspect all traffic. One wrinkle is +that your system needs to trust the mitmproxy SSL certificates for this +to work. + +The following installation instructions assume you are running Ubuntu: + +1. Install mitmproxy (see http://docs.mitmproxy.org/en/stable/install.html) + +2. Run `mitmproxy --port 9000` to generate its certificates. Enter CTRL-C to quit. + +3. Install the certificate from ~/.mitmproxy to your system: + + ```sh + sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt + sudo update-ca-certificates + ``` + +If successful, the output should indicate that a certificate was added: + +```sh +Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done. +Running hooks in /etc/ca-certificates/update.d....done. +``` + +## Verifying mitmproxy certifiactes + +To verify that the certificates are properly install, run: + +```sh +mitmproxy --port 9000 +``` + +This will run mitmproxy on port 9000. In another window, run: + +```sh +curl --proxy http://localhost:9000 https://httpbin.org/status/200 +``` + +If everything is setup correctly, then you will see information on the mitmproxy window and +no errors from the curl commands. + +## Running the Docker daemon with a proxy + +For Docker to connect through a proxy, you must start the Docker daemon with the +proper environment variables. The easiest way is to shutdown Docker (e.g. `sudo initctl stop docker`) +and then run Docker by hand. As root, run: + +```sh +export HTTP_PROXY="http://localhost:9000" +export HTTPS_PROXY="https://localhost:9000" +docker daemon --debug +``` + +This will launch the Docker daemon and proxy all connections through mitmproxy. + +## Running the Docker client + +Now that we have mitmproxy and Docker running, we can now attempt to login and push a container +image. You may need to run as root to do this. For example: + +```sh +docker login s3-testing.myregistry.com:4567 +docker push s3-testing.myregistry.com:4567/root/docker-test +``` + +In the example above, we see the following trace on the mitmproxy window: + + + +The above image shows: + +* The initial PUT requests went through fine with a 201 status code. +* The 201 redirected the client to the S3 bucket. +* The HEAD request to the AWS bucket reported a 403 Unauthorized. + +What does this mean? This strongly suggests that the S3 user does not have the right +[permissions to perform a HEAD request](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html). +The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/). +Once the right permissions were set, the error went away. |
