summaryrefslogtreecommitdiff
path: root/internal
Commit message (Collapse)AuthorAgeFilesLines
* Configure a default ttl for personal access tokensJoe Woodward2023-05-112-6/+3
| | | | | | | | | | | | | Prior to this change personal access tokens without a ttl would never expire. In Gitlab 15.4 we deprecated non-expiring tokens and are scheduled for removal in 16.0. https://gitlab.com/gitlab-org/gitlab/-/issues/369122 This change alters the gitlab-shell command for creating tokens to ensure add a default limit of 30 days. Closes https://gitlab.com/gitlab-org/gitlab-shell/-/issues/640
* Update golang-crypto fork versionIgor Drozdov2023-05-041-2/+2
| | | | | It updates golang-crypto version and fixes the test that verifies the order of the supported algorithms
* Acceptance test for Geo pushIgor Drozdov2023-03-171-0/+29
| | | | | It imitates a push to the secondary and verifies that the push is redirected to the primary
* Configure Gitaly storage acceptance testsPatrick Cyiza2023-03-151-1/+1
|
* Perform HTTP request to primary on Geo pushIgor Drozdov2023-03-039-7/+431
| | | | | | | | | Currently, we perform a request to Gitlab Rails that proxies the request to primary However, it causes timeouts on big pushes and consumes large amount of memory. We can perform an HTTP request directly from Gitlab Shell instead and stream the response to the user
* sshd: exclude gssapi when building without cgoLorenz Brun2023-02-233-0/+38
| | | | | | | | | | | MR #682 broke building without cgo enabled as it introduced a dependency on a Kerberos library. This can only be disabled at runtime and thus static builds of gitlab-sshd are no longer possible. This change introduces an alternative implementation of the GSSAPI structure which just rejects attempts to use it. That alternative implementation gets automatically activated in case the user is building without cgo.
* Add DNS discovery support for Gitaly/PraefectQuang-Minh Nguyen2023-02-144-152/+180
| | | | | | | | | | All the implementations of DNS discovery were done in this epic: https://gitlab.com/groups/gitlab-org/-/epics/8971. Gitaly allows clients to configure DNS discovery via dial option. This MR adds the exposed dial options to client connection creation in Gitlab-shell. Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/4722 Changelog: added
* feat: make retryable http default clientSteve Azzopardi2023-01-308-81/+59
| | | | | | | | | | | | | | | | | | | | | | What --- Make the retryableHTTP client introduced in https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703 the default HTTP client. Why --- In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1254964426 we've seen a 99% error reduction on `git` commands from `gitlab-shell` when the retryableHTTP client is used. This has been running in production for over 2 weeks in `us-east1-b` and 5 days fleet-wide so we should be confident that this client works as expected. Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>
* Merge branch '196-add-kerberos-support' into 'main'Igor Drozdov2023-01-2313-39/+332
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add support for the gssapi-with-mic auth method Closes #196 See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/682 Merged-by: Igor Drozdov <idrozdov@gitlab.com> Approved-by: Alejandro Rodríguez <alejandro@gitlab.com> Approved-by: Patrick Bajao <ebajao@gitlab.com> Approved-by: Costel Maxim <cmaxim@gitlab.com> Approved-by: Igor Drozdov <idrozdov@gitlab.com> Reviewed-by: Alejandro Rodríguez <alejandro@gitlab.com> Reviewed-by: Igor Drozdov <idrozdov@gitlab.com> Reviewed-by: Patrick Bajao <ebajao@gitlab.com> Reviewed-by: Rohit Shambhuni <rshambhuni@gitlab.com> Co-authored-by: Lee Tickett <ltickett@gitlab.com> Co-authored-by: Marin Hannache <git@mareo.fr>
| * Add support for the gssapi-with-mic auth methodMarin Hannache2023-01-2313-39/+332
| |
* | feat: put retryablehttp.Client behind feature flagSteve Azzopardi2023-01-128-57/+81
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | What --- - Update the `client.HttpClient` fields to have `http.Client` and `retryablehttp.Client`, one of them will be `nil` depending on the feature flag toggle. - Create new method `newRetryableRequest` which will create a `retryablehttp.Request` and use that if the `FF_GITLAB_SHELL_RETRYABLE_HTTP` feature flag is turned on. - Add checks for `FF_GITLAB_SHELL_RETRYABLE_HTTP` everywhere we use the http client to use the `retryablehttp.Client` or the default `http.Client` - New job `tests-integration-retryableHttp` to run the integraiton tests with the new retryablehttp client. We didn't update go tests because some assertions are different and will break table driven tests. Why --- As discussed in https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703#note_1229645097 we want to put the client behind a feature flag, not just the retry logic. This does bring extra risk for accessing a `nil` field but there should be checks everytime we access `RetryableHTTP` and `HTTPClient`. Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>
* | feat: retry on errorSteve Azzopardi2023-01-127-39/+33
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | What --- Change the default `HTTP.Client` to `github.com/hashicorp/go-retryablehttp.Client` to get automatic retries and exponential backoff. We retry the request 2 times resulting in 3 attempts of sending the request, the min retry wait is 1 second, and the maximum is 15 seconds. Hide the retry logic behind a temporary feature flag `FF_GITLAB_SHELL_RETRYABLE_HTTP` to easily roll this out in GitLab.com. When we verify that this works as expected we will remove `FF_GITLAB_SHELL_RETRYABLE_HTTP` and have the retry logic as the default logic. Why --- In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 users end up seeing the following errors when trying to `git-clone(1)` a repository locally on in CI. ```shell remote: =============================== remote: remote: ERROR: Internal API unreachable remote: remote: ================================ ``` When we look at the application logs we see the following error: ```json { "err": "http://gitlab-webservice-git.gitlab.svc:8181/api/v4/internal/allowed": dial tcp 10.69.184.120:8181: connect: connection refused", "msg": "Internal API unreachable"} ``` In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1222670120 we've correlated these `connection refused` errors with infrastructure events that remove the git pods that are hosting `gitlab-webservice-git` service. We could try to make the underlying infrastructure more reactive to these changes as suggested in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1225164944 but we can still end up serving bad requests. Implementing retry logic for 5xx or other errors would allow users to still be able to `git-clone(1)` reposirories, although it being slower. This is espically important during CI runs so users don't have to retry jobs themselves. Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 Closes: https://gitlab.com/gitlab-org/gitlab-shell/-/issues/604 Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>
* Use blocking reader to fix race in testPatrick Bajao2022-11-281-1/+1
| | | | | | | | | | | | | | The `TestCanceledContext` test in `twofactorverify_test.go` tests the scenario wherein the request context gets canceled in the middle of the request. A race can occur though when the input gets read before the context is canceled which can result to a different error: instead of having the context canceled, the OTP will be blank. To fix it, we use a blocking reader as input to simulate "waiting for input" scenario. This way, reading the input will never be finished and the context cancelation can be done appropriately.
* Merge branch 'id-add-sshd-dev-docs' into 'main'Ash McKenzie2022-11-242-0/+48
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | Add developer documentation to sshd package See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/683 Merged-by: Ash McKenzie <amckenzie@gitlab.com> Approved-by: Alejandro Rodríguez <alejandro@gitlab.com> Approved-by: Oscar Tovar <otovar@gitlab.com> Approved-by: James Fargher <proglottis@gmail.com> Approved-by: John Cai <jcai@gitlab.com> Approved-by: Amy Qualls <aqualls@gitlab.com> Approved-by: Ash McKenzie <amckenzie@gitlab.com> Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>
| * Add developer documentation to sshd packageIgor Drozdov2022-11-232-0/+48
| |
* | sshd: Return error when proxy policy is misconfiguredJames Fargher2022-11-231-7/+12
| | | | | | | | | | MustStrictWhiteListPolicy panics when configured incorrectly. So here we use the error returning version instead.
* | sshd: Add ProxyAllowed setting to limit PROXY protocol IP addressesJames Fargher2022-11-233-6/+66
| | | | | | | | Changelog: added
* | sshd: Extract static proxy policy handlerJames Fargher2022-11-231-7/+13
| | | | | | | | | | | | | | Instead of interpreting the configuration for every new connection, we can rely on a closure to simplify the proxy handler path. This is more similar to how the provided MustStrictWhiteListPolicy works which will be added in a later commit.
* | Fix broken Gitaly integration testsPatrick Bajao2022-11-231-2/+2
|/ | | | | | | | | | | Gitaly started to require `Repository` information for certain commands like upload-pack, upload-archive, and receive-pack. However, gitlab-shell tests are sending `git_object_directory` and `git_alternate_object_directories` that do not exist. To fix the tests, we set those to an existing directory in the created repository.
* Improve error message for Gitaly `LimitError`sAlejandro Rodríguez2022-10-242-1/+50
|
* Merge branch 'id-add-documentation-to-command' into 'main'Patrick Bajao2022-09-211-0/+30
|\ | | | | | | | | | | | | | | | | Add developer documentation to command package See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/594 Merged-by: Patrick Bajao <ebajao@gitlab.com> Approved-by: Patrick Bajao <ebajao@gitlab.com> Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>
| * Specify all packages that use commandsPatrick Bajao2022-08-031-1/+5
| |
| * Add developer documentation to command packageIgor Drozdov2022-08-031-0/+26
| |
* | Update Gitaly to v15Igor Drozdov2022-08-058-13/+13
|/ | | | | | | | | | | This commit also excludes gitlab-shell from dependencies: Gitaly specifies Gitlab Shell as a dependency as well in order to use gitlabnet client to perform API endpoints to Gitlab Rails. As a result, Gitlab Shell requires Gitaly -> Gitaly requires an older version of Gitlab Shell -> that version requires an older version of Gitlab Shell, etc. Let's use exclude to break the chain earlier
* Simplify 2FA Push auth processingIgor Drozdov2022-07-208-707/+448
| | | | Use a single channel to handle both Push Auth and OTP results
* Implement Push Auth support for 2FA verificationkmcknight2022-07-186-59/+484
| | | | | | | | | | | | | | | | | When `2fa_verify` command is executed: - A user is asked to enter OTP - A blocking call for push auth is performed Then: - If the push auth request fails, the user is still able to enter OTP - If OTP is invalid, the `2fa_verify` command ends the execution - If OTP is valid or push auth request succeeded, then the user is successfully authenticated - If 30 seconds passed while no OTP or Push have been provided, then the `2fa_verify` command ends the execution
* Fix flaky race testStan Hu2022-07-141-1/+2
| | | | | | | | `ignoredError.err` was being used in a Goroutine handler, but the value of `ignoredError` changes with each test case. To avoid a race, make a local copy of the error before each Goroutine runs. Closes https://gitlab.com/gitlab-org/gitlab-shell/-/issues/590
* go: Bump major version to v14Patrick Steinhardt2022-07-0573-271/+271
| | | | | | | | | | | | | While gitlab-shell currently has a major version of v14, the module path it exposes is not using that major version like it is required by the Go standard. This makes it impossible for dependents to import gitlab-shell as a dependency without using a commit as version. Fix this by changing the module path of gitlab-shell to instead be `gitlab.com/gitlab-org/gitlab-shell/v14` and adjust all imports accordingly. Changelog: fixed
* Pass original IP from PROXY requests to internal API callsAlejandro Rodríguez2022-06-304-20/+46
|
* gitlab-sshd: Add support for configuring host certificatesStan Hu2022-06-2611-8/+187
| | | | | | | | | | | | | | | | | This adds support for specifying host certificates via the `host_cert_files` option and advertises the signed key to the client. This acts similarly to OpenSSH's `HostCertificate` parameter: gitlab-sshd attempts to match a host key to its certificate, and then substitutes the matching host key with a certificate signed by a trusted certificate authority's key. This is the first requirement to supporting SSH certificates. This will enable the client to trust the server if both trust a common certificate authority. The `TrustedUserCAKeys` option will need to be supported later for the server to trust all user keys signed by this certificate authority. Relates to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/495
* Ignore "not our ref" errors from gitlab-sshd error metricsStan Hu2022-06-062-1/+7
| | | | | | | | | | | | If a client requests a ref that cannot be found in the repository, previously gitlab-sshd would record it as part of its service level indicator metric. This is really an application error between the client and the Git repository, so we exclude it from our metrics. Relates to https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15848 Changelog: fixed
* Exclude disallowed command from error rateIgor Drozdov2022-06-012-26/+26
|
* Calculate session start after the connection is establishedIgor Drozdov2022-05-251-2/+1
|
* Merge branch 'id-session-duration' into 'main'Stan Hu2022-05-253-19/+20
|\ | | | | | | | | Improve establish session duration metrics See merge request gitlab-org/gitlab-shell!651
| * Improve establish session duration metricsIgor Drozdov2022-05-253-19/+20
| | | | | | | | | | | | | | | | | | Before we took into account the time a user takes to authenticate Now it only measures the time between a connection established and a command started to being executed It's still can be controlled by a user, but it's something we can measure and restrict if necessary
* | Log gitlab-sshd session level indicator errorsStan Hu2022-05-241-2/+4
|/ | | | | | In production, we saw gitlab-sshd error metrics rise, but it was not clear why. We now log a message every time we encounter a session error that affects the service level indicator counter.
* Add missing SshdSessionEstablishedDuration metricsIgor Drozdov2022-05-231-0/+1
|
* Abort long-running unauthenticated SSH connectionsIgor Drozdov2022-05-234-11/+50
| | | | | | | | | The config option is basically a copy of LoginGraceTime OpenSSH option. If an SSH connection is hanging unauthenticated, after some period of time, the connection gets canceled. The value is configurable, the server waits for 60 seconds by default.
* Close the connection when context is canceledIgor Drozdov2022-05-232-6/+43
| | | | | | | | | | | When graceful shutdown timeout expires, the global context is canceled. All the operations dependent on it are canceled as well. Unfortunately, some of the operations doesn't respect the context. For example, SSH connection initialization. In this case, we need to manually close the connection. One of the options is to wait for ctx.Done() and close the connection
* Move connection init into connection.goIgor Drozdov2022-05-233-69/+78
|
* Return support for diffie-hellman-group14-sha1Igor Drozdov2022-05-231-0/+1
| | | | | It seems that a lot of users rely on this, let's return it and deprecated later to make the migration less disruptive
* Downgrade auth EOF messages from warning to debugIgor Drozdov2022-05-211-1/+1
| | | | | The errors happen when a client closes a connection on handshake They can be ignored to avoid noise
* Exclude Gitaly unavailable error from error rateIgor Drozdov2022-05-214-24/+26
| | | | | | | When a user hits repository rate limit, Gitaly returns an error that the request can't be handled (Gitaly unavailable) We should avoid this error to avoid exceeding the error rate
* Display constistently in gitlab-sshd and gitlab-shellIgor Drozdov2022-05-214-18/+31
| | | | | - Use console package to format the errors in gitlab-sshd - Suppress internal Gitaly errors in client output
* Merge branch 'sh-downgrade-host-key-errors' into 'main'Igor Drozdov2022-05-211-2/+15
|\ | | | | | | | | Downgrade host key mismatch messages from warning to debug See merge request gitlab-org/gitlab-shell!639
| * Downgrade handleConn start message to debugStan Hu2022-05-201-1/+1
| | | | | | | | This message doesn't provide that much value, so let's just drop it.
| * Downgrade host key mismatch messages from warning to debugStan Hu2022-05-201-1/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In production, we often see SSH key scans requesting host key algorithms that we don't support, such as `sk-ssh-ed25519@openssh.com` or `sk-ecdsa-sha2-nistp256@openssh.com`. These messages might be useful if someone forgets to configure a host key that should be supported, but most of the time they are noise. This commit downgrades these messages to DEBUG. Relates to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/581 Changelog: changed
* | Introduce a GitLab-SSHD server version during handshakeHendrik Meyer2022-05-211-0/+1
|/
* Narrow supported kex algorithmsIgor Drozdov2022-05-202-19/+22
| | | | | | We don't support diffie-hellman-group14-sha1 via OpenSSH currently Let's avoid introducing it in gitlab-sshd because it's using weak hashing algorithm
* Make ProxyHeaderTimeout configurableIgor Drozdov2022-05-194-8/+11
| | | | | | | | Issue: https://gitlab.com/gitlab-org/gitlab-shell/-/issues/576 ProxyHeaderTimeout must be small to avoid DoS risk Let's make the value configurable and 500ms by default
View Lorry Controller Status