Add latest changes from gitlab-org/gitlab@13-9-stable-eev13.9.0-rc42

7 files changed, 196 insertions, 91 deletions

diff --git a/doc/development/agent/gitops.md b/doc/development/agent/gitops.md
index 8c8586326fa..f183ba86aa1 100644
--- a/doc/development/agent/gitops.md
+++ b/doc/development/agent/gitops.md
@@ -4,7 +4,7 @@ group: Configure
 info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
 ---
 
-# GitOps with the Kubernetes Agent **(PREMIUM ONLY)**
+# GitOps with the Kubernetes Agent **(PREMIUM SELF)**
 
 The [GitLab Kubernetes Agent](../../user/clusters/agent/index.md) supports the
 [pull-based version](https://www.gitops.tech/#pull-based-deployments) of
diff --git a/doc/development/agent/identity.md b/doc/development/agent/identity.md
index 884ce015a02..49d20d2fd87 100644
--- a/doc/development/agent/identity.md
+++ b/doc/development/agent/identity.md
@@ -4,7 +4,7 @@ group: Configure
 info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
 ---
 
-# Kubernetes Agent identity and authentication **(PREMIUM ONLY)**
+# Kubernetes Agent identity and authentication **(PREMIUM SELF)**
 
 This page uses the word `agent` to describe the concept of the
 GitLab Kubernetes Agent. The program that implements the concept is called `agentk`.
diff --git a/doc/development/agent/index.md b/doc/development/agent/index.md
index 95661c8ddbd..112162f8f90 100644
--- a/doc/development/agent/index.md
+++ b/doc/development/agent/index.md
@@ -4,7 +4,7 @@ group: Configure
 info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
 ---
 
-# Kubernetes Agent development **(PREMIUM ONLY)**
+# Kubernetes Agent development **(PREMIUM SELF)**
 
 This page contains developer-specific information about the GitLab Kubernetes Agent.
 [End-user documentation about the GitLab Kubernetes Agent](../../user/clusters/agent/index.md)
diff --git a/doc/development/agent/local.md b/doc/development/agent/local.md
index 47246a6a6d3..603364567bf 100644
--- a/doc/development/agent/local.md
+++ b/doc/development/agent/local.md
@@ -4,7 +4,7 @@ group: Configure
 info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
 ---
 
-# Run the Kubernetes Agent locally **(PREMIUM ONLY)**
+# Run the Kubernetes Agent locally **(PREMIUM SELF)**
 
 You can run `kas` and `agentk` locally to test the [Kubernetes Agent](index.md) yourself.
 
diff --git a/doc/development/agent/repository_overview.md b/doc/development/agent/repository_overview.md
new file mode 100644
index 00000000000..b9eea286a3e
--- /dev/null
+++ b/doc/development/agent/repository_overview.md
@@ -0,0 +1,98 @@
+---
+stage: Configure
+group: Configure
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
+# Kubernetes Agent repository overview **(PREMIUM SELF)**
+
+This page describes the subfolders of the Kubernetes Agent repository.
+[Development information](index.md) and
+[end-user documentation](../../user/clusters/agent/index.md) are both available.
+
+<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>
+For a video overview, see
+[GitLab Kubernetes Agent repository overview](https://www.youtube.com/watch?v=j8CyaCWroUY).
+
+## `build`
+
+Various files for the build process.
+
+### `build/deployment`
+
+A [`kpt`](https://googlecontainertools.github.io/kpt/) package that bundles some
+[Kustomize](https://kustomize.io/) layers and components. Can be used as-is, or
+to create a custom package to install `agentk`.
+
+## `cmd`
+
+Commands are binaries that this repository produces. They are:
+
+- `kas` is the GitLab Kubernetes Agent Server binary.
+- `agentk` is the GitLab Kubernetes Agent binary.
+
+Each of these directories contain application bootstrap code for:
+
+- Reading configuration.
+- Applying defaults to it.
+- Constructing the dependency graph of objects that constitute the program.
+- Running it.
+
+### `cmd/agentk`
+
+- `agentk` initialization logic.
+- Implementation of the agent modules API.
+
+### `cmd/kas`
+
+- `kas` initialization logic.
+- Implementation of the server modules API.
+
+## `examples`
+
+Git submodules for the example projects.
+
+## `internal`
+
+The main code of both `gitlab-kas` and `agentk`, and various supporting building blocks.
+
+### `internal/api`
+
+Structs that represent some important pieces of data.
+
+### `internal/gitaly`
+
+Items to work with [Gitaly](../../administration/gitaly/index.md).
+
+### `internal/gitlab`
+
+GitLab REST client.
+
+### `internal/module`
+
+Modules that implement server and agent-side functionality.
+
+### `internal/tool`
+
+Various building blocks. `internal/tool/testing` contains mocks and helpers
+for testing. Mocks are generated with [`gomock`](https://pkg.go.dev/github.com/golang/mock).
+
+## `it`
+
+Contains scaffolding for integration tests. Unused at the moment.
+
+## `pkg`
+
+Contains exported packages.
+
+### `pkg/agentcfg`
+
+Contains protobuf definitions of the `agentk` configuration file. Used to configure
+the agent through a configuration repository.
+
+### `pkg/kascfg`
+
+Contains protobuf definitions of the `gitlab-kas` configuration file. Contains an
+example of that configuration file along with the test for it. The test ensures
+the configuration file example is in sync with the protobuf definitions of the
+file and defaults, which are applied when the file is loaded.
diff --git a/doc/development/agent/routing.md b/doc/development/agent/routing.md
index 43cc78ccdfb..9a7d6422d47 100644
--- a/doc/development/agent/routing.md
+++ b/doc/development/agent/routing.md
@@ -4,7 +4,7 @@ group: Configure
 info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
 ---
 
-# Routing `kas` requests in the Kubernetes Agent **(PREMIUM ONLY)**
+# Routing `kas` requests in the Kubernetes Agent **(PREMIUM SELF)**
 
 This document describes how `kas` routes requests to concrete `agentk` instances.
 GitLab must talk to GitLab Kubernetes Agent Server (`kas`) to:
@@ -134,90 +134,97 @@ of the `kas` receiving the request from the _external_ endpoint to retry and re-
 requests. This method ensures a single central component for each request can determine
 how a request is routed, rather than distributing the decision across several `kas` instances.
 
-### API definitions
+### Reverse gRPC tunnel
+
+This section explains how the `agentk` -> `kas` reverse gRPC tunnel is implemented.
+
+<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>
+For a video overview of how some of the blocks map to code, see
+[GitLab Kubernetes Agent reverse gRPC tunnel architecture and code overview
+](https://www.youtube.com/watch?v=9pnQF76hyZc).
+
+#### High level schema
+
+In this example, `Server side of module A` exposes its API to get the `Pod` list
+on the `Public API gRPC server`. When it receives a request, it must determine
+the agent ID from it, then call the proxying code which forwards the request to
+a suitable `agentk` that can handle it.
+
+The `Agent side of module A` exposes the same API on the `Internal gRPC server`.
+When it receives the request, it needs to handle it (such as retrieving and returning
+the `Pod` list).
+
+This schema describes how reverse tunneling is handled fully transparently
+for modules, so you can add new features:
+
+```mermaid
+graph TB
+    subgraph kas
+        server-internal-grpc-server[Internal gRPC server]
+        server-api-grpc-server[Public API gRPC server]
+        server-module-a[Server side of module A]
+        server-module-b[Server side of module B]
+    end
+    subgraph agentk
+        agent-internal-grpc-server[Internal gRPC server]
+        agent-module-a[Agent side of module A]
+        agent-module-b[Agent side of module B]
+    end
+
+    agent-internal-grpc-server -- request --> agent-module-a
+    agent-internal-grpc-server -- request --> agent-module-b
+
+    server-module-a-. expose API on .-> server-internal-grpc-server
+    server-module-b-. expose API on .-> server-api-grpc-server
+
+    server-internal-grpc-server -- proxy request --> agent-internal-grpc-server
+    server-api-grpc-server -- proxy request --> agent-internal-grpc-server
+```
+
+#### Implementation schema
+
+`HandleTunnelConnection()` is called with the server-side interface of the reverse
+tunnel. It registers the connection and blocks, waiting for a request to proxy
+through the connection.
+
+`HandleIncomingConnection()` is called with the server-side interface of the incoming
+connection. It registers the connection and blocks, waiting for a matching tunnel
+to proxy the connection through.
+
+After it has two connections that match, `Connection registry` starts bi-directional
+data streaming:
 
-```proto
-syntax = "proto3";
-
-import "google/protobuf/timestamp.proto";
-
-message KasAddress {
-    string ip = 1;
-    uint32 port = 2;
-}
-
-message ConnectedAgentInfo {
-    // Agent id.
-    int64 id = 1;
-    // Identifies a particular agentk->kas connection. Randomly generated when agent connects.
-    int64 connection_id = 2;
-    string version = 3;
-    string commit = 4;
-    // Pod namespace.
-    string pod_namespace = 5;
-    // Pod name.
-    string pod_name = 6;
-    // When the connection was established.
-    google.protobuf.Timestamp connected_at = 7;
-    KasAddress kas_address = 8;
-    // What else do we need?
-}
-
-message KasInstanceInfo {
-    string version = 1;
-    string commit = 2;
-    KasAddress address = 3;
-    // What else do we need?
-}
-
-message ConnectedAgentsForProjectRequest {
-    int64 project_id = 1;
-}
-
-message ConnectedAgentsForProjectResponse {
-    // There may 0 or more agents with the same id, depending on the number of running Pods.
-    repeated ConnectedAgentInfo agents = 1;
-}
-
-message ConnectedAgentsByIdRequest {
-    int64 agent_id = 1;
-}
-
-message ConnectedAgentsByIdResponse {
-    repeated ConnectedAgentInfo agents = 1;
-}
-
-// API for use by GitLab.
-service KasApi {
-    // Connected agents for a particular configuration project.
-    rpc ConnectedAgentsForProject (ConnectedAgentsForProjectRequest) returns (ConnectedAgentsForProjectResponse) {
-    }
-    // Connected agents for a particular agent id.
-    rpc ConnectedAgentsById (ConnectedAgentsByIdRequest) returns (ConnectedAgentsByIdResponse) {
-    }
-    // Depends on the need, but here is the call from the example above.
-    rpc GetPods (GetPodsRequest) returns (GetPodsResponse) {
-    }
-}
-
-message Pod {
-    string namespace = 1;
-    string name = 2;
-}
-
-message GetPodsRequest {
-    int64 agent_id = 1;
-    int64 connection_id = 2;
-}
-
-message GetPodsResponse {
-    repeated Pod pods = 1;
-}
-
-// Internal API for use by kas for kas -> kas calls.
-service KasInternal {
-    // Depends on the need, but here is the call from the example above.
-    rpc GetPods (GetPodsRequest) returns (GetPodsResponse) {
-    }
-}
+```mermaid
+graph TB
+    subgraph kas
+        server-tunnel-module[Server tunnel module]
+        connection-registry[Connection registry]
+        server-internal-grpc-server[Internal gRPC server]
+        server-api-grpc-server[Public API gRPC server]
+        server-module-a[Server side of module A]
+        server-module-b[Server side of module B]
+    end
+    subgraph agentk
+        agent-internal-grpc-server[Internal gRPC server]
+        agent-tunnel-module[Agent tunnel module]
+        agent-module-a[Agent side of module A]
+        agent-module-b[Agent side of module B]
+    end
+
+    server-tunnel-module -- "HandleTunnelConnection()" --> connection-registry
+    server-internal-grpc-server -- "HandleIncomingConnection()" --> connection-registry
+    server-api-grpc-server -- "HandleIncomingConnection()" --> connection-registry
+    server-module-a-. expose API on .-> server-internal-grpc-server
+    server-module-b-. expose API on .-> server-api-grpc-server
+
+    agent-tunnel-module -- "establish tunnel, receive request" --> server-tunnel-module
+    agent-tunnel-module -- make request --> agent-internal-grpc-server
+    agent-internal-grpc-server -- request --> agent-module-a
+    agent-internal-grpc-server -- request --> agent-module-b
 ```
+
+### API definitions
+
+- [`agent_tracker/agent_tracker.proto`](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/internal/module/agent_tracker/agent_tracker.proto)
+- [`agent_tracker/rpc/rpc.proto`](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/internal/module/agent_tracker/rpc/rpc.proto)
+- [`reverse_tunnel/rpc/rpc.proto`](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/internal/module/reverse_tunnel/rpc/rpc.proto)
diff --git a/doc/development/agent/user_stories.md b/doc/development/agent/user_stories.md
index 2929573ffd3..609be47a3cb 100644
--- a/doc/development/agent/user_stories.md
+++ b/doc/development/agent/user_stories.md
@@ -4,7 +4,7 @@ group: Configure
 info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
 ---
 
-# Kubernetes Agent user stories **(PREMIUM ONLY)**
+# Kubernetes Agent user stories **(PREMIUM SELF)**
 
 The [personas in action](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#user-personas)
 for the Kubernetes Agent are: