1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
|
---
stage: Configure
group: Configure
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Troubleshooting the GitLab agent for Kubernetes
When you are using the GitLab agent for Kubernetes, you might experience issues you need to troubleshoot.
You can start by viewing the service logs:
```shell
kubectl logs -f -l=app=gitlab-agent -n gitlab-kubernetes-agent
```
If you are a GitLab administrator, you can also view the [GitLab agent server logs](../../../administration/clusters/kas.md#troubleshooting).
## Transport: Error while dialing failed to WebSocket dial
```json
{
"level": "warn",
"time": "2020-11-04T10:14:39.368Z",
"msg": "GetConfiguration failed",
"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://gitlab-kas:443/-/kubernetes-agent\\\": dial tcp: lookup gitlab-kas on 10.60.0.10:53: no such host\""
}
```
This error occurs when there are connectivity issues between the `kas-address`
and your agent pod. To fix this issue, make sure the `kas-address` is accurate.
```json
{
"level": "error",
"time": "2021-06-25T21:15:45.335Z",
"msg": "Reverse tunnel",
"mod_name": "reverse_tunnel",
"error": "Connect(): rpc error: code = Unavailable desc = connection error: desc= \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 301\""
}
```
This error occurs when the `kas-address` doesn't include a trailing slash. To fix this issue, make sure that the
`wss` or `ws` URL ends with a trailing slash, like `wss://GitLab.host.tld:443/-/kubernetes-agent/`
or `ws://GitLab.host.tld:80/-/kubernetes-agent/`.
## ValidationError(Deployment.metadata)
```json
{
"level": "info",
"time": "2020-10-30T08:56:54.329Z",
"msg": "Synced",
"project_id": "root/kas-manifest001",
"resource_key": "apps/Deployment/kas-test001/nginx-deployment",
"sync_result": "error validating data: [ValidationError(Deployment.metadata): unknown field \"replicas\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"selector\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"template\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta]"
}
```
This error occurs when a manifest file is malformed and Kubernetes can't
create the specified objects. Make sure that your manifest files are valid.
For additional troubleshooting, try to use the manifest files to create objects in Kubernetes directly.
## Error while dialing failed to WebSocket dial: failed to send handshake request
```json
{
"level": "warn",
"time": "2020-10-30T09:50:51.173Z",
"msg": "GetConfiguration failed",
"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent\\\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \\\"\\\\x00\\\\x00\\\\x06\\\\x04\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x05\\\\x00\\\\x00@\\\\x00\\\"\""
}
```
This error occurs when you configured `wss` as `kas-address` on the agent side,
but the agent server is not available at `wss`. To fix this issue, make sure the
same schemes are configured on both sides.
## Decompressor is not installed for grpc-encoding
```json
{
"level": "warn",
"time": "2020-11-05T05:25:46.916Z",
"msg": "GetConfiguration.Recv failed",
"error": "rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding \"gzip\""
}
```
This error occurs when the version of the agent is newer that the version of the agent server (KAS).
To fix it, make sure that both `agentk` and the agent server are the same version.
## Certificate signed by unknown authority
```json
{
"level": "error",
"time": "2021-02-25T07:22:37.158Z",
"msg": "Reverse tunnel",
"mod_name": "reverse_tunnel",
"error": "Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent/\\\": x509: certificate signed by unknown authority\""
}
```
This error occurs when your GitLab instance is using a certificate signed by an internal
certificate authority that is unknown to the agent.
To fix this issue, you can present the CA certificate file to the agent
by using a Kubernetes `configmap` and mount the file in the agent `/etc/ssl/certs` directory from where it
will be picked up automatically.
For example, if your internal CA certificate is `myCA.pem`:
```plaintext
kubectl -n gitlab-kubernetes-agent create configmap ca-pemstore --from-file=myCA.pem
```
Then in `resources.yml`:
```yaml
spec:
serviceAccountName: gitlab-kubernetes-agent
containers:
- name: agent
image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:<version>"
args:
- --token-file=/config/token
- --kas-address
- wss://kas.host.tld:443 # replace this line with the line below if using Omnibus GitLab or GitLab.com.
# - wss://gitlab.host.tld:443/-/kubernetes-agent/
# - wss://kas.gitlab.com # for GitLab.com users, use this KAS.
# - grpc://host.docker.internal:8150 # use this attribute when connecting from Docker.
volumeMounts:
- name: token-volume
mountPath: /config
- name: ca-pemstore-volume
mountPath: /etc/ssl/certs/myCA.pem
subPath: myCA.pem
volumes:
- name: token-volume
secret:
secretName: gitlab-kubernetes-agent-token
- name: ca-pemstore-volume
configMap:
name: ca-pemstore
items:
- key: myCA.pem
path: myCA.pem
```
Alternatively, you can mount the certificate file at a different location and specify it for the
`--ca-cert-file` agent parameter:
```yaml
containers:
- name: agent
image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:<version>"
args:
- --ca-cert-file=/tmp/myCA.pem
- --token-file=/config/token
- --kas-address
- wss://kas.host.tld:443 # replace this line with the line below if using Omnibus GitLab or GitLab.com.
# - wss://gitlab.host.tld:443/-/kubernetes-agent/
# - wss://kas.gitlab.com # for GitLab.com users, use this KAS.
# - grpc://host.docker.internal:8150 # use this attribute when connecting from Docker.
volumeMounts:
- name: token-volume
mountPath: /config
- name: ca-pemstore-volume
mountPath: /tmp/myCA.pem
subPath: myCA.pem
```
## Project not found
```json
{
"level ":"error ",
"time ":"2022-01-05T15:18:11.331Z",
"msg ":"GetObjectsToSynchronize.Recv failed ",
"mod_name ":"gitops ",
"error ":"rpc error: code = NotFound desc = project not found ",
}
```
This error occurs when the project where you keep your manifests is not public. To fix it, make sure your project is public or your manifest files
are stored in the repository where the agent is configured.
|
