1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
|
---
status: proposed
creation-date: "2022-11-25"
authors: [ "@theoretick" ]
coach: "@DylanGriffith"
approvers: [ "@connorgilbert", "@amarpatel" ]
owning-stage: "~devops::secure"
participating-stages: []
---
# Secret Detection as a platform-wide experience
## Summary
Today's secret detection feature is built around containerized scans of repositories
within a pipeline context. This feature is quite limited compared to where leaks
or compromised tokens may appear and should be expanded to include a much wider scope.
Secret detection as a platform-wide experience encompasses detection across
platform features with high risk of secret leakage, including repository contents,
job logs, and project management features such as issues, epics, and MRs.
## Motivation
### Goals
- Support asynchronous secret detection for:
- push events
- issuable creation
- issuable updates
- issuable comments
### Non-Goals
The current proposal is limited to asynchronous detection and alerting only.
**Blocking** secrets on push events is high-risk to a critical path and
would require extensive performance profiling before implementing. See
[a recent example](https://gitlab.com/gitlab-org/gitlab/-/issues/246819#note_1164411983)
of a customer incident where this was attempted.
Secret revocation and rotation is also beyond the scope of this new capability.
Scanned object types beyond the scope of this MVC include:
- Media types (JPEGs, PDFs,...)
- Snippets
- Wikis
## Proposal
To achieve scalable secret detection for a variety of domain objects a dedicated
scanning service must be created and deployed alongside the GitLab distribution.
This is referred to as the `SecretScanningService`.
This service must be:
- highly performant
- horizontally scalable
- generic in domain object scanning capability
Platform-wide secret detection should be enabled by-default on GitLab SaaS as well
as self-managed instances.
## Challenges
- Secure authentication to GitLab.com infrastructure
- Performance of scanning against large blobs
- Performance of scanning against volume of domain objects (such as push frequency)
## Design and implementation details
The critical paths as outlined under [goals above](#goals) cover two major object
types: Git blobs (corresponding to push events) and arbitrary text blobs.
The detection flow for push events relies on subscribing to the PostReceive hook
and enqueueing Sidekiq requests to the `SecretScanningService`. The `SecretScanningService`
service fetches enqueued refs, queries Gitaly for the ref blob contents, scans
the commit contents, and notifies the Rails application when a secret is detected.
See [Push event detection flow](#push-event-detection-flow) for sequence.
The detection flow for arbitrary text blobs, such as issue comments, relies on
subscribing to `Notes::PostProcessService` (or equivalent service) and enqueueing
Sidekiq requests to the `SecretScanningService` to process the text blob by object type
and primary key of domain object. The `SecretScanningService` service fetches the
relevant text blob, scans the contents, and notifies the Rails application when a secret
is detected.
The detection flow for job logs requires processing the log during archive to object
storage. See discussion [in this issue](https://gitlab.com/groups/gitlab-org/-/epics/8847#note_1116647883)
around scanning during streaming and the added complexity in buffering lookbacks
for arbitrary trace chunks.
In any case of detection, the Rails application manually creates a vulnerability
using the `Vulnerabilities::ManuallyCreateService` to surface the finding within the
existing Vulnerability Management UI.
See [technical discovery](https://gitlab.com/gitlab-org/gitlab/-/issues/376716)
for further background exploration.
### Token types
The existing Secret Detection configuration covers ~100 rules across a variety
of platforms. To reduce total cost of execution and likelihood of false positives
the dedicated service targets only well-defined tokens. A well-defined token is
defined as a token with a precise definition, most often a fixed substring prefix or
suffix and fixed length.
Token types to identify in order of importance:
1. Well-defined GitLab tokens (including Personal Access Tokens and Pipeline Trigger Tokens)
1. Verified Partner tokens (including AWS)
1. Remainder tokens currently included in Secret Detection CI configuration
### Detection engine
Our current secret detection offering utilizes [Gitleaks](https://github.com/zricethezav/gitleaks/)
for all secret scanning within pipeline contexts. By using its `--no-git` configuration
we can scan arbitrary text blobs outside of a repository context and continue to
utilize it for non-pipeline scanning.
Given our existing familiarity with the tool and its extensibility, it should
remain our engine of choice. Changes to the detection engine are out of scope
unless benchmarking unveils performance concerns.
### Push event detection flow
```mermaid
sequenceDiagram
autonumber
actor User
User->>+Workhorse: git push
Workhorse->>+Gitaly: tcp
Gitaly->>+Rails: grpc
Sidekiq->>+Rails: poll job
Rails->>-Sidekiq: PostReceive worker
Sidekiq-->>+Sidekiq: enqueue PostReceiveSecretScanWorker
Sidekiq->>+Rails: poll job
loop PostReceiveSecretScanWorker
Rails->>-Sidekiq: PostReceiveSecretScanWorker
Sidekiq->>+SecretScanningSvc: ScanBlob(ref)
SecretScanningSvc->>+Sidekiq: accepted
Note right of SecretScanningSvc: Scanning job enqueued
Sidekiq-->>+Rails: done
SecretScanningSvc->>+Gitaly: retrieve blob
SecretScanningSvc->>+SecretScanningSvc: scan blob
SecretScanningSvc->>+Rails: secret found
end
```
## Iterations
1. Requirements definition for detection coverage and actions
1. PoC of secret scanning service
1. gRPC commit retrieval from Gitaly
1. blob scanning
1. benchmarking of issuables, comments, job logs and blobs to gain confidence that the total costs will be viable
1. Implementation of secret scanning service MVC (targeting individual commits)
1. Security and readiness review
1. Deployment and monitoring
1. Implementation of secret scanning service MVC (targeting arbitrary text blobs)
1. Deployment and monitoring
1. High priority domain object rollout (priority `TBD`)
1. Issuable comments
1. Issuable bodies
1. Job logs
|
