1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
|
# Troubleshooting the GitLab Container Registry
## Basic Troubleshooting
1. Check to make sure that the system clock on your Docker client and GitLab server have
been synchronized (e.g. via NTP).
2. If you are using an S3-backed registry, double check that the IAM
permissions and the S3 credentials (including region) are correct. See [the
sample IAM policy](https://docs.docker.com/registry/storage-drivers/s3/)
for more details.
3. Check the registry logs (e.g. `/var/log/gitlab/registry/current`) and the GitLab production logs
for errors (e.g. `/var/log/gitlab/gitlab-rails/production.log`). You may be able to find clues
there.
# Advanced Troubleshooting
NOTE: The following section is only recommended for experts.
Sometimes it's not obvious what is wrong, and you may need to dive deeper into
the communication between the Docker client and the registry to find out
what's wrong. We will use a concrete example in the past to illustrate how to
diagnose a problem with the S3 setup.
## Example: Unexpected 403 error during push
A user attempted to enable an S3-backed registry. The `docker login` step went
fine. However, when pushing an image, the output showed:
```
The push refers to a repository [s3-testing.myregistry.com:4567/root/docker-test]
dc5e59c14160: Pushing [==================================================>] 14.85 kB
03c20c1a019a: Pushing [==================================================>] 2.048 kB
a08f14ef632e: Pushing [==================================================>] 2.048 kB
228950524c88: Pushing 2.048 kB
6a8ecde4cc03: Pushing [==> ] 9.901 MB/205.7 MB
5f70bf18a086: Pushing 1.024 kB
737f40e80b7f: Waiting
82b57dbc5385: Waiting
19429b698a22: Waiting
9436069b92a3: Waiting
error parsing HTTP 403 response body: unexpected end of JSON input: ""
```
This error is ambiguous, as it's not clear whether the 403 is coming from the GitLab Rails
application, the Docker registry, or something else. In this case, we know that since
the login succeeded, we probably need to look at the communication between the client
and the registry.
The REST API between the Docker client and registry is [described
here](https://docs.docker.com/registry/spec/api/). Normally, one would just
use Wireshark or tcpdump to capture the traffic and see where things went
wrong. However, since all communication between Docker clients and servers
are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even
if you know the private key. What can we do instead?
## mitmproxy
Enter [mitmproxy](https://mitmproxy.org/). This tool allows you to place a
proxy between your client and server to inspect all traffic. One wrinkle is
that your system needs to trust the mitmproxy SSL certificates for this
to work.
The following installation instructions assume you are running Ubuntu:
1. Install mitmproxy (see http://docs.mitmproxy.org/en/stable/install.html)
2. Run `mitmproxy --port 9000` to generate its certificates. Enter CTRL-C to quit.
3. Install the certificate from ~/.mitmproxy to your system:
```sh
sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt
sudo update-ca-certificates
```
If successful, the output should indicate that a certificate was added:
```sh
Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.
```
## Verifying mitmproxy certifiactes
To verify that the certificates are properly install, run:
```sh
mitmproxy --port 9000
```
This will run mitmproxy on port 9000. In another window, run:
```sh
curl --proxy http://localhost:9000 https://httpbin.org/status/200
```
If everything is setup correctly, then you will see information on the mitmproxy window and
no errors from the curl commands.
## Running the Docker daemon with a proxy
For Docker to connect through a proxy, you must start the Docker daemon with the
proper environment variables. The easiest way is to shutdown Docker (e.g. `sudo initctl stop docker`)
and then run Docker by hand. As root, run:
```sh
export HTTP_PROXY="http://localhost:9000"
export HTTPS_PROXY="https://localhost:9000"
docker daemon --debug
```
This will launch the Docker daemon and proxy all connections through mitmproxy.
## Running the Docker client
Now that we have mitmproxy and Docker running, we can now attempt to login and push a container
image. You may need to run as root to do this. For example:
```sh
docker login s3-testing.myregistry.com:4567
docker push s3-testing.myregistry.com:4567/root/docker-test
```
In the example above, we see the following trace on the mitmproxy window:
The above image shows:
* The initial PUT requests went through fine with a 201 status code.
* The 201 redirected the client to the S3 bucket.
* The HEAD request to the AWS bucket reported a 403 Unauthorized.
What does this mean? This strongly suggests that the S3 user does not have the right
[permissions to perform a HEAD request](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html).
The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/).
Once the right permissions were set, the error went away.
|
