From d707c91f706a26ff30b9d4862bb2c50b45a35fff Mon Sep 17 00:00:00 2001 From: Stan Hu Date: Mon, 1 Aug 2016 22:57:15 -0700 Subject: Add guide for debugging issues with the Docker container registry [ci skip] --- doc/container_registry/README.md | 4 + doc/container_registry/img/mitmproxy-docker.png | Bin 0 -> 407004 bytes doc/container_registry/troubleshooting.md | 139 ++++++++++++++++++++++++ 3 files changed, 143 insertions(+) create mode 100644 doc/container_registry/img/mitmproxy-docker.png create mode 100644 doc/container_registry/troubleshooting.md diff --git a/doc/container_registry/README.md b/doc/container_registry/README.md index 55077197ff9..3db351811a8 100644 --- a/doc/container_registry/README.md +++ b/doc/container_registry/README.md @@ -90,6 +90,10 @@ your `.gitlab-ci.yml`, you have to follow the [Using a private Docker Registry][private-docker] documentation. This workflow will be simplified in the future. +## Troubleshooting + +See [the GitLab Docker registry troubleshooting guide](troubleshooting.md). + [ce-4040]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4040 [docker-docs]: https://docs.docker.com/engine/userguide/intro/ [private-docker]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#using-a-private-docker-registry diff --git a/doc/container_registry/img/mitmproxy-docker.png b/doc/container_registry/img/mitmproxy-docker.png new file mode 100644 index 00000000000..4e3e37b413d Binary files /dev/null and b/doc/container_registry/img/mitmproxy-docker.png differ diff --git a/doc/container_registry/troubleshooting.md b/doc/container_registry/troubleshooting.md new file mode 100644 index 00000000000..e21fc9a2c61 --- /dev/null +++ b/doc/container_registry/troubleshooting.md @@ -0,0 +1,139 @@ +# Troubleshooting the GitLab Container Registry + +## Basic Troubleshooting + +1. Check to make sure that the system clock on your Docker client and GitLab server have + been synchronized (e.g. via NTP). + +2. If you are using an S3-backed registry, double check that the IAM + permissions and the S3 credentials (including region) are correct. See [the + sample IAM policy](https://docs.docker.com/registry/storage-drivers/s3/) + for more details. + +3. Check the registry logs (e.g. `/var/log/gitlab/registry/current`) and the GitLab production logs + for errors (e.g. `/var/log/gitlab/gitlab-rails/production.log`). You may be able to find clues + there. + +# Advanced Troubleshooting + +NOTE: The following section is only recommended for experts. + +Sometimes it's not obvious what is wrong, and you may need to dive deeper into +the communication between the Docker client and the registry to find out +what's wrong. We will use a concrete example in the past to illustrate how to +diagnose a problem with the S3 setup. + +## Example: Unexpected 403 error during push + +A user attempted to enable an S3-backed registry. The `docker login` step went +fine. However, when pushing an image, the output showed: + +``` +The push refers to a repository [s3-testing.myregistry.com:4567/root/docker-test] +dc5e59c14160: Pushing [==================================================>] 14.85 kB +03c20c1a019a: Pushing [==================================================>] 2.048 kB +a08f14ef632e: Pushing [==================================================>] 2.048 kB +228950524c88: Pushing 2.048 kB +6a8ecde4cc03: Pushing [==> ] 9.901 MB/205.7 MB +5f70bf18a086: Pushing 1.024 kB +737f40e80b7f: Waiting +82b57dbc5385: Waiting +19429b698a22: Waiting +9436069b92a3: Waiting +error parsing HTTP 403 response body: unexpected end of JSON input: "" +``` + +This error is ambiguous, as it's not clear whether the 403 is coming from the GitLab Rails +application, the Docker registry, or something else. In this case, we know that since +the login succeeded, we probably need to look at the communication between the client +and the registry. + +The REST API between the Docker client and registry is [described +here](https://docs.docker.com/registry/spec/api/). Normally, one would just +use Wireshark or tcpdump to capture the traffic and see where things went +wrong. However, since all communication between Docker clients and servers +are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even +if you know the private key. What can we do instead? + +## mitmproxy + +Enter [mitmproxy](https://mitmproxy.org/). This tool allows you to place a +proxy between your client and server to inspect all traffic. One wrinkle is +that your system needs to trust the mitmproxy SSL certificates for this +to work. + +The following installation instructions assume you are running Ubuntu: + +1. Install mitmproxy (see http://docs.mitmproxy.org/en/stable/install.html) + +2. Run `mitmproxy --port 9000` to generate its certificates. Enter CTRL-C to quit. + +3. Install the certificate from ~/.mitmproxy to your system: + + ```sh + sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt + sudo update-ca-certificates + ``` + +If successful, the output should indicate that a certificate was added: + +```sh +Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done. +Running hooks in /etc/ca-certificates/update.d....done. +``` + +## Verifying mitmproxy certifiactes + +To verify that the certificates are properly install, run: + +```sh +mitmproxy --port 9000 +``` + +This will run mitmproxy on port 9000. In another window, run: + +```sh +curl --proxy http://localhost:9000 https://httpbin.org/status/200 +``` + +If everything is setup correctly, then you will see information on the mitmproxy window and +no errors from the curl commands. + +## Running the Docker daemon with a proxy + +For Docker to connect through a proxy, you must start the Docker daemon with the +proper environment variables. The easiest way is to shutdown Docker (e.g. `sudo initctl stop docker`) +and then run Docker by hand. As root, run: + +```sh +export HTTP_PROXY="http://localhost:9000" +export HTTPS_PROXY="https://localhost:9000" +docker daemon --debug +``` + +This will launch the Docker daemon and proxy all connections through mitmproxy. + +## Running the Docker client + +Now that we have mitmproxy and Docker running, we can now attempt to login and push a container +image. You may need to run as root to do this. For example: + +```sh +docker login s3-testing.myregistry.com:4567 +docker push s3-testing.myregistry.com:4567/root/docker-test +``` + +In the example above, we see the following trace on the mitmproxy window: + +![mitmproxy output from Docker](img/mitmproxy-docker.png) + +The above image shows: + +* The initial PUT requests went through fine with a 201 status code. +* The 201 redirected the client to the S3 bucket. +* The HEAD request to the AWS bucket reported a 403 Unauthorized. + +What does this mean? This strongly suggests that the S3 user does not have the right +[permissions to perform a HEAD request](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html). +The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/). +Once the right permissions were set, the error went away. -- cgit v1.2.1 From 46385e4e5a88a4ac614f680094b9226778cee64a Mon Sep 17 00:00:00 2001 From: Stan Hu Date: Tue, 2 Aug 2016 15:20:36 -0700 Subject: Add a note about setting up an insecure registry [ci skip] --- doc/container_registry/troubleshooting.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/doc/container_registry/troubleshooting.md b/doc/container_registry/troubleshooting.md index e21fc9a2c61..c24c80518dd 100644 --- a/doc/container_registry/troubleshooting.md +++ b/doc/container_registry/troubleshooting.md @@ -55,12 +55,17 @@ wrong. However, since all communication between Docker clients and servers are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even if you know the private key. What can we do instead? +One way would be to disable HTTPS by setting up an [insecure +registry](https://docs.docker.com/registry/insecure/). This could introduce a +security hole and is only recommended for local testing. If you have a +production system and can't or don't want to do this, there is another way: +use mitmproxy, which stands for Man-in-the-Middle Proxy. + ## mitmproxy -Enter [mitmproxy](https://mitmproxy.org/). This tool allows you to place a -proxy between your client and server to inspect all traffic. One wrinkle is -that your system needs to trust the mitmproxy SSL certificates for this -to work. +[mitmproxy](https://mitmproxy.org/) allows you to place a proxy between your +client and server to inspect all traffic. One wrinkle is that your system +needs to trust the mitmproxy SSL certificates for this to work. The following installation instructions assume you are running Ubuntu: -- cgit v1.2.1 From 2c9cce0feb8bd4e10f3406493eff30e783782d15 Mon Sep 17 00:00:00 2001 From: Stan Hu Date: Tue, 2 Aug 2016 15:24:15 -0700 Subject: Grammar improvements [ci skip] --- doc/container_registry/troubleshooting.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/container_registry/troubleshooting.md b/doc/container_registry/troubleshooting.md index c24c80518dd..8008bf29935 100644 --- a/doc/container_registry/troubleshooting.md +++ b/doc/container_registry/troubleshooting.md @@ -43,10 +43,10 @@ a08f14ef632e: Pushing [==================================================>] 2.04 error parsing HTTP 403 response body: unexpected end of JSON input: "" ``` -This error is ambiguous, as it's not clear whether the 403 is coming from the GitLab Rails -application, the Docker registry, or something else. In this case, we know that since -the login succeeded, we probably need to look at the communication between the client -and the registry. +This error is ambiguous, as it's not clear whether the 403 is coming from the +GitLab Rails application, the Docker registry, or something else. In this +case, since we know that since the login succeeded, we probably need to look +at the communication between the client and the registry. The REST API between the Docker client and registry is [described here](https://docs.docker.com/registry/spec/api/). Normally, one would just -- cgit v1.2.1 From c91168c04a71301d8423b881c1219cfd510d5784 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 3 Aug 2016 17:33:37 +0300 Subject: Small refactor on Registry troubleshooting [ci skip] --- doc/container_registry/troubleshooting.md | 49 +++++++++++++++---------------- 1 file changed, 23 insertions(+), 26 deletions(-) diff --git a/doc/container_registry/troubleshooting.md b/doc/container_registry/troubleshooting.md index 8008bf29935..14c4a7d9a63 100644 --- a/doc/container_registry/troubleshooting.md +++ b/doc/container_registry/troubleshooting.md @@ -5,27 +5,27 @@ 1. Check to make sure that the system clock on your Docker client and GitLab server have been synchronized (e.g. via NTP). -2. If you are using an S3-backed registry, double check that the IAM +2. If you are using an S3-backed Registry, double check that the IAM permissions and the S3 credentials (including region) are correct. See [the sample IAM policy](https://docs.docker.com/registry/storage-drivers/s3/) for more details. -3. Check the registry logs (e.g. `/var/log/gitlab/registry/current`) and the GitLab production logs +3. Check the Registry logs (e.g. `/var/log/gitlab/registry/current`) and the GitLab production logs for errors (e.g. `/var/log/gitlab/gitlab-rails/production.log`). You may be able to find clues there. -# Advanced Troubleshooting +## Advanced Troubleshooting -NOTE: The following section is only recommended for experts. +>**NOTE:** The following section is only recommended for experts. Sometimes it's not obvious what is wrong, and you may need to dive deeper into -the communication between the Docker client and the registry to find out +the communication between the Docker client and the Registry to find out what's wrong. We will use a concrete example in the past to illustrate how to diagnose a problem with the S3 setup. -## Example: Unexpected 403 error during push +### Unexpected 403 error during push -A user attempted to enable an S3-backed registry. The `docker login` step went +A user attempted to enable an S3-backed Registry. The `docker login` step went fine. However, when pushing an image, the output showed: ``` @@ -44,11 +44,11 @@ error parsing HTTP 403 response body: unexpected end of JSON input: "" ``` This error is ambiguous, as it's not clear whether the 403 is coming from the -GitLab Rails application, the Docker registry, or something else. In this +GitLab Rails application, the Docker Registry, or something else. In this case, since we know that since the login succeeded, we probably need to look -at the communication between the client and the registry. +at the communication between the client and the Registry. -The REST API between the Docker client and registry is [described +The REST API between the Docker client and Registry is [described here](https://docs.docker.com/registry/spec/api/). Normally, one would just use Wireshark or tcpdump to capture the traffic and see where things went wrong. However, since all communication between Docker clients and servers @@ -56,12 +56,12 @@ are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even if you know the private key. What can we do instead? One way would be to disable HTTPS by setting up an [insecure -registry](https://docs.docker.com/registry/insecure/). This could introduce a +Registry](https://docs.docker.com/registry/insecure/). This could introduce a security hole and is only recommended for local testing. If you have a production system and can't or don't want to do this, there is another way: use mitmproxy, which stands for Man-in-the-Middle Proxy. -## mitmproxy +### mitmproxy [mitmproxy](https://mitmproxy.org/) allows you to place a proxy between your client and server to inspect all traffic. One wrinkle is that your system @@ -70,10 +70,9 @@ needs to trust the mitmproxy SSL certificates for this to work. The following installation instructions assume you are running Ubuntu: 1. Install mitmproxy (see http://docs.mitmproxy.org/en/stable/install.html) - -2. Run `mitmproxy --port 9000` to generate its certificates. Enter CTRL-C to quit. - -3. Install the certificate from ~/.mitmproxy to your system: +1. Run `mitmproxy --port 9000` to generate its certificates. + Enter CTRL-C to quit. +1. Install the certificate from `~/.mitmproxy` to your system: ```sh sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt @@ -87,24 +86,22 @@ Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d....done. ``` -## Verifying mitmproxy certifiactes - -To verify that the certificates are properly install, run: +To verify that the certificates are properly installed, run: ```sh mitmproxy --port 9000 ``` -This will run mitmproxy on port 9000. In another window, run: +This will run mitmproxy on port `9000`. In another window, run: ```sh curl --proxy http://localhost:9000 https://httpbin.org/status/200 ``` -If everything is setup correctly, then you will see information on the mitmproxy window and +If everything is setup correctly, you will see information on the mitmproxy window and no errors from the curl commands. -## Running the Docker daemon with a proxy +### Running the Docker daemon with a proxy For Docker to connect through a proxy, you must start the Docker daemon with the proper environment variables. The easiest way is to shutdown Docker (e.g. `sudo initctl stop docker`) @@ -118,10 +115,10 @@ docker daemon --debug This will launch the Docker daemon and proxy all connections through mitmproxy. -## Running the Docker client +### Running the Docker client -Now that we have mitmproxy and Docker running, we can now attempt to login and push a container -image. You may need to run as root to do this. For example: +Now that we have mitmproxy and Docker running, we can attempt to login and push +a container image. You may need to run as root to do this. For example: ```sh docker login s3-testing.myregistry.com:4567 @@ -141,4 +138,4 @@ The above image shows: What does this mean? This strongly suggests that the S3 user does not have the right [permissions to perform a HEAD request](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html). The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/). -Once the right permissions were set, the error went away. +Once the right permissions were set, the error will go away. -- cgit v1.2.1