diff options
Diffstat (limited to 'doc/development/experiment_guide/index.md')
| -rw-r--r-- | doc/development/experiment_guide/index.md | 83 |
1 files changed, 35 insertions, 48 deletions
diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index f7af1113b6e..b140cce34fc 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -6,47 +6,46 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Experiment Guide -Experiments can be conducted by any GitLab team, most often the teams from the [Growth Sub-department](https://about.gitlab.com/handbook/engineering/development/growth/). Experiments are not tied to releases because they primarily target GitLab.com. - -Experiments are run as an A/B/n test, and are behind an [experiment feature flag](../feature_flags/#experiment-type) to turn the test on or off. Based on the data the experiment generates, the team decides if the experiment had a positive impact and should be made the new default, or rolled back. - -## Experiment rollout issue - -Each experiment should have an [experiment rollout](https://gitlab.com/groups/gitlab-org/-/boards/1352542) issue to track the experiment from rollout through to cleanup and removal. -The rollout issue is similar to a feature flag rollout issue, and is also used to track the status of an experiment. -When an experiment is deployed, the due date of the issue should be set (this depends on the experiment but can be up to a few weeks in the future). -After the deadline, the issue needs to be resolved and either: - -- It was successful and the experiment becomes the new default. -- It was not successful and all code related to the experiment is removed. - -In either case, an outcome of the experiment should be posted to the issue with the reasoning for the decision. - -## Code reviews - -Experiments' code quality can fail our standards for several reasons. These -reasons can include not being added to the codebase for a long time, or because -of fast iteration to retrieve data. However, having the experiment run (or not -run) shouldn't impact GitLab availability. To avoid or identify issues, -experiments are initially deployed to a small number of users. Regardless, -experiments still need tests. - -Experiments must have corresponding [frontend or feature tests](../testing_guide/index.md) to ensure they -exist in the application. These tests should help prevent the experiment code from -being removed before the [experiment cleanup process](https://about.gitlab.com/handbook/engineering/development/growth/experimentation/#experiment-cleanup-issue) starts. - -If, as a reviewer or maintainer, you find code that would usually fail review -but is acceptable for now, mention your concerns with a note that there's no -need to change the code. The author can then add a comment to this piece of code -and link to the issue that resolves the experiment. The author or reviewer can add a link to this concern in the -experiment rollout issue under the `Experiment Successful Cleanup Concerns` section of the description. -If the experiment is successful and becomes part of the product, any items that appear under this section will be addressed. +Experiments can be conducted by any GitLab team, most often the teams from the +[Growth Sub-department](https://about.gitlab.com/handbook/engineering/development/growth/). +Experiments are not tied to releases because they primarily target GitLab.com. + +Experiments are run as an A/B/n test, and are behind an [experiment feature flag](../feature_flags/#experiment-type) +to turn the test on or off. Based on the data the experiment generates, the team decides +if the experiment had a positive impact and should be made the new default, or rolled back. + +Experiments in GitLab are tightly coupled with the concepts provided by +[Feature flags in development of GitLab](../feature_flags/index.md). You're strongly encouraged +to read and understand the [Feature flags in development of GitLab](../feature_flags/index.md) +portion of the documentation before considering running experiments. Experiments add additional +concepts which may seem confusing or advanced without understanding the underpinnings of how GitLab +uses feature flags in development. One concept: experiments can be run with multiple variants, +which are sometimes referred to as A/B/n tests. + +We use the [`gitlab-experiment` gem](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment), +sometimes referred to as GLEX, to run our experiments. The gem exists in a separate repository +so it can be shared across any GitLab property that uses Ruby. You should feel comfortable reading +the documentation on that project if you want to dig into more advanced topics or open issues. Be +aware that the documentation there reflects what's in the main branch and may not be the same as +the version being used within GitLab. + +## Glossary of terms + +To ensure a shared language, you should understand these fundamental terms we use +when communicating about experiments: + +- `experiment`: Any deviation of code paths we want to run at some times, but not others. +- `context`: A consistent experience we provide in an experiment. +- `control`: The default, or "original" code path. +- `candidate`: Defines an experiment with only one code path. +- `variant(s)`: Defines an experiment with multiple code paths. +- `behaviors`: Used to reference all possible code paths of an experiment, including the control. ## Implementing an experiment [`GLEX`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment) - or `Gitlab::Experiment`, the `gitlab-experiment` gem - is the preferred option for implementing an experiment in GitLab. -For more information, see [Implementing an A/B/n experiment using GLEX](gitlab_experiment.md). +For more information, see [Implementing an A/B/n experiment using GLEX](implementing_experiments.md). This uses [experiment](../feature_flags/index.md#experiment-type) feature flags. @@ -64,15 +63,3 @@ We recommend the following workflow: 1. **If the experiment is a success**, designers add the new icon or illustration to the Pajamas UI kit as part of the cleanup process. Engineers can then add it to the [SVG library](https://gitlab-org.gitlab.io/gitlab-svgs/) and modify the implementation based on the [Frontend Development Guidelines](../fe_guide/icons.md#usage-in-hamlrails-2). - -## Turn off all experiments - -When there is a case on GitLab.com (SaaS) that necessitates turning off all experiments, we have this control. - -You can toggle experiments on SaaS on and off using the `gitlab_experiment` [feature flag](../feature_flags). - -This can be done via chatops: - -- [disable](../feature_flags/controls.md#disabling-feature-flags): `/chatops run feature set gitlab_experiment false` -- [enable](../feature_flags/controls.md#process): `/chatops run feature delete gitlab_experiment` - - This allows the `default_enabled` [value of true in the yml](https://gitlab.com/gitlab-org/gitlab/-/blob/016430f6751b0c34abb24f74608c80a1a8268f20/config/feature_flags/ops/gitlab_experiment.yml#L8) to be honored. |