summaryrefslogtreecommitdiff
path: root/doc/development/snowplow/implementation.md
blob: 0d81b4428504534e54f9fe4299ec73471cabdae6 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
---
stage: Growth
group: Product Intelligence
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---

# Implement Snowplow tracking

This page describes how to:

- Implement Snowplow frontend and backend tracking
- Test Snowplow events

## Snowplow JavaScript frontend tracking

GitLab provides a `Tracking` interface that wraps the [Snowplow JavaScript tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/) 
to track custom events. 

For the recommended frontend tracking implementation, see [Usage recommendations](#usage-recommendations).

Tracking implementations must have an `action` and a `category`. You can provide additional 
categories from the [structured event taxonomy](index.md#structured-event-taxonomy) with an `extra` object
that accepts key-value pairs.

| Field      | Type   | Default value              | Description                                                                                                                                                                                                    |
|:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `category` | string | `document.body.dataset.page` | Page or subsection of a page in which events are captured.                                                                                                                                            |
| `action`   | string | generic                  | Action the user is taking. Clicks must be `click` and activations must be `activate`. For example, focusing a form field is `activate_form_input`, and clicking a button is `click_button`. |
| `data`     | object | `{}`                         | Additional data such as `label`, `property`, `value`, `context` as described in [Structured event taxonomy](index.md#structured-event-taxonomy), and `extra` (key-value pairs object). |

### Usage recommendations

- Use [data attributes](#implement-data-attribute-tracking) on HTML elements that emit `click`, `show.bs.dropdown`, or `hide.bs.dropdown` events.
- Use the [Vue mixin](#implement-vue-component-tracking) for tracking custom events, or if the supported events for data attributes are not propagating.
- Use the [tracking class](#implement-raw-javascript-tracking) when tracking raw JavaScript files.

### Implement data attribute tracking

To implement tracking for HAML or Vue templates, add a [`data-track` attribute](#data-track-attributes) to the element.

The following example shows `data-track-*` attributes assigned to a button:

```haml
%button.btn{ data: { track: { action: "click_button", label: "template_preview", property: "my-template" } } }
```

```html
<button class="btn"
  data-track-action="click_button"
  data-track-label="template_preview"
  data-track-property="my-template"
  data-track-extra='{ "template_variant": "primary" }'
/>
```

#### `data-track` attributes

| Attribute             | Required | Description |
|:----------------------|:---------|:------------|
| `data-track-action`    | true     | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field is `activate_form_input` and clicking a button is `click_button`. Replaces `data-track-event`, which was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290962) in GitLab 13.11. |
| `data-track-label`    | false    | The specific element or object to act on. This can be: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. |
| `data-track-property` | false    | Any additional property of the element, or object being acted on. |
| `data-track-value`    | false    | Describes a numeric value or something directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. If omitted, this is the element's `value` property or `undefined`. For checkboxes, the default value is the element's checked attribute or `0` when unchecked. |
| `data-track-extra` | false    | A key-value pair object passed as a valid JSON string. This attribute is added to the `extra` property in our [`gitlab_standard`](schemas.md#gitlab_standard) schema. |
| `data-track-context`  | false    | The `context` as described in our [Structured event taxonomy](index.md#structured-event-taxonomy). |

#### Event listeners

Event listeners bind at the document level to handle click events in elements with data attributes.
This allows them to be handled when the DOM re-renders or changes. Document-level binding reduces 
the likelihood that click events stop propagating up the DOM tree. 

If click events stop propagating, you must implement listeners and [Vue component tracking](#implement-vue-component-tracking) or [raw JavaScript tracking](#implement-raw-javascript-tracking).

#### Helper methods

Use the following Ruby helper:

```ruby
tracking_attrs(label, action, property) # { data: { track_label... } }

%button{ **tracking_attrs('main_navigation', 'click_button', 'navigation') }
```

If you use the GitLab helper method [`nav_link`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/helpers/tab_helper.rb#L76), you must wrap `html_options` under the `html_options` keyword argument. If you
use the `ActionView` helper method [`link_to`](https://api.rubyonrails.org/v5.2.3/classes/ActionView/Helpers/UrlHelper.html#method-i-link_to), you don't need to wrap `html_options`.

```ruby
# Bad
= nav_link(controller: ['dashboard/groups', 'explore/groups'], data: { track_label: "explore_groups",
track_action: "click_button" })

# Good
= nav_link(controller: ['dashboard/groups', 'explore/groups'], html_options: { data: { track_label: 
"explore_groups", track_action: "click_button" } })

# Good (other helpers)
= link_to explore_groups_path, title: _("Explore"), data: { track_label: "explore_groups", track_action:
"click_button" }
```

### Implement Vue component tracking

For custom event tracking, use a Vue `mixin` in components. Vue `mixin` exposes the `Tracking.event`
static method and the `track` method called from components or templates. You can specify tracking 
options in `data` or `computed`. These options override any defaults and allow the values to be dynamic 
from props or based on state. 

Default options are passed when an event is tracked from the component. If you don't specify an option,
the default `document.body.dataset.page` is used. The default options are: 

- `category`
- `label`
- `property`
- `value`

To implement Vue component tracking:

1. Import the `Tracking` library and request a `mixin`:

    ```javascript
    import Tracking from '~/tracking';
    const trackingMixin = Tracking.mixin;
    ```

1. Provide categories to track the event from the component. For example, to track all events in a 
component with a label, use the `label` category:

    ```javascript
    import Tracking from '~/tracking';
    const trackingMixin = Tracking.mixin({ label: 'right_sidebar' });
    ```

1. In the component, declare the Vue `mixin`:

    ```javascript
    export default {
      mixins: [trackingMixin],
      // ...[component implementation]...
      data() {
        return {
          expanded: false,
          tracking: {
            label: 'left_sidebar',
          },
        };
      },
    };
    ```

1. To receive event data as a tracking object or computed property:
   - Declare it in the `data` function. Use a `tracking` object when default event properties are dynamic or provided at runtime:

      ```javascript
      export default {
        name: 'RightSidebar',
        mixins: [Tracking.mixin()],
        data() {
          return {
            tracking: {
              label: 'right_sidebar',
              // category: '',
              // property: '',
              // value: '',
              // experiment: '',
              // extra: {},
            },
          };
        },
      };
      ```

   - Declare it in the event data in the `track` function. This object merges with any previously provided options:

      ```javascript
      this.track('click_button', {
        label: 'right_sidebar',
      });
      ```

1. Optional. Use the `track` method in a template:

    ```html
    <template>
      <div>
        <button data-testid="toggle" @click="toggle">Toggle</button>

        <div v-if="expanded">
          <p>Hello world!</p>
          <button @click="track('click_action')">Track another event</button>
        </div>
      </div>
    </template>
    ```

The following example shows an implementation of Vue component tracking:

```javascript
export default {
  name: 'RightSidebar',
  mixins: [Tracking.mixin({ label: 'right_sidebar' })],
  data() {
    return {
      expanded: false,
    };
  },
  methods: {
    toggle() {
      this.expanded = !this.expanded;
      // Additional data will be merged, like `value` below
      this.track('click_toggle', { value: Number(this.expanded) });
    }
  }
};
```

#### Testing example

```javascript
import { mockTracking } from 'helpers/tracking_helper';
// mockTracking(category, documentOverride, spyMethod)

describe('RightSidebar.vue', () => {
  let trackingSpy;
  let wrapper;

  beforeEach(() => {
    trackingSpy = mockTracking(undefined, wrapper.element, jest.spyOn);
  });

  const findToggle = () => wrapper.find('[data-testid="toggle"]');

  it('tracks turning off toggle', () => {
    findToggle().trigger('click');

    expect(trackingSpy).toHaveBeenCalledWith(undefined, 'click_toggle', {
      label: 'right_sidebar',
      value: 0,
    });
  });
});
```

### Implement raw JavaScript tracking

To call custom event tracking and instrumentation directly from the JavaScript file, call the `Tracking.event` static function.

The following example demonstrates tracking a click on a button by manually calling `Tracking.event`.

```javascript
import Tracking from '~/tracking';

const button = document.getElementById('create_from_template_button');

button.addEventListener('click', () => {
  Tracking.event('dashboard:projects:index', 'click_button', {
    label: 'create_from_template',
    property: 'template_preview',
    extra: {
      templateVariant: 'primary',
      valid: 1,
    },
  });
});
```

#### Testing example

```javascript
import Tracking from '~/tracking';

describe('MyTracking', () => {
  let wrapper;

  beforeEach(() => {
    jest.spyOn(Tracking, 'event');
  });

  const findButton = () => wrapper.find('[data-testid="create_from_template"]');

  it('tracks event', () => {
    findButton().trigger('click');

    expect(Tracking.event).toHaveBeenCalledWith(undefined, 'click_button', {
      label: 'create_from_template',
      property: 'template_preview',
      extra: {
        templateVariant: 'primary',
        valid: true,
      },
    });
  });
});
```

### Form tracking

To enable Snowplow automatic [form tracking](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracking-specific-events/#form-tracking): 

1. Call `Tracking.enableFormTracking` when the DOM is ready. 
1. Provide a `config` object that includes at least one of the following elements:
    - `forms` determines the forms to track. Identified by the CSS class name.
    - `fields` determines the fields inside the tracked forms to track. Identified by the field `name`.
1. Optional. Provide a list of contexts as the second argument. The [`gitlab_standard`](schemas.md#gitlab_standard) schema is excluded from these events.
    
```javascript
Tracking.enableFormTracking({
  forms: { allow: ['sign-in-form', 'password-recovery-form'] },
  fields: { allow: ['terms_and_conditions', 'newsletter_agreement'] },
});
```

#### Testing example

```javascript
import Tracking from '~/tracking';

describe('MyFormTracking', () => {
  let formTrackingSpy;

  beforeEach(() => {
    formTrackingSpy = jest
      .spyOn(Tracking, 'enableFormTracking')
      .mockImplementation(() => null);
  });

  it('initialized with the correct configuration', () => {
    expect(formTrackingSpy).toHaveBeenCalledWith({
      forms: { allow: ['sign-in-form', 'password-recovery-form'] },
      fields: { allow: ['terms_and_conditions', 'newsletter_agreement'] },
    });
  });
});
```

## Implement Ruby backend tracking

`Gitlab::Tracking` is an interface that wraps the [Snowplow Ruby Tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/ruby-tracker/) for tracking custom events.
Backend tracking provides:

- User behavior tracking
- Instrumentation to monitor and visualize performance over time in a section or aspect of code.

To add custom event tracking and instrumentation, call the `GitLab::Tracking.event` class method. 
For example:

```ruby
class Projects::CreateService < BaseService
  def execute
    project = Project.create(params)

    Gitlab::Tracking.event('Projects::CreateService', 'create_project', label: project.errors.full_messages.to_sentence,
                           property: project.valid?.to_s, project: project, user: current_user, namespace: namespace)
  end
end
```

Use the following arguments:

| Argument   | Type                      | Default value | Description                                                                                                                       |
|------------|---------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------|
| `category` | String                    |               | Area or aspect of the application. For example,  `HealthCheckController` or `Lfs::FileTransformer`.                  |
| `action`   | String                    |               | The action being taken. For example, a controller action such as `create`, or an Active Record callback. |
| `label`    | String                    | nil           | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created.                                                          |
| `property` | String                    | nil           | Any additional property of the element, or object being acted on.                                                          |
| `value`    | Numeric                   | nil           | Describes a numeric value or something directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility.                                                          |
| `context`  | Array\[SelfDescribingJSON\] | nil           | An array of custom contexts to send with this event. Most events should not have any custom contexts.                             |
| `project`  | Project                   | nil           | The project associated with the event. |
| `user`     | User                      | nil           | The user associated with the event. |
| `namespace` | Namespace                | nil           | The namespace associated with the event. |
| `extra`   | Hash                | `{}`         | Additional keyword arguments are collected into a hash and sent with the event. |

### Unit testing

To test backend Snowplow events, use the `expect_snowplow_event` helper. For more information, see 
[testing best practices](../testing_guide/best_practices.md#test-snowplow-events).

### Performance

We use the [AsyncEmitter](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/ruby-tracker/emitters/#the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development.

## Develop and test Snowplow

To develop and test a Snowplow event, there are several tools to test frontend and backend events:

| Testing Tool                                 | Frontend Tracking  | Backend Tracking    | Local Development Environment | Production Environment | Production Environment |
|----------------------------------------------|--------------------|---------------------|-------------------------------|------------------------|------------------------|
| Snowplow Analytics Debugger Chrome Extension | Yes | No | Yes            | Yes     | Yes     |
| Snowplow Inspector Chrome Extension          | Yes | No | Yes            | Yes     | Yes     |
| Snowplow Micro                               | Yes | Yes  | Yes            | No    | No    |

### Test frontend events

Before you test frontend events in development, you must:

1. [Enable Snowplow tracking in the Admin Area](index.md#enable-snowplow-tracking).
1. Turn off ad blockers that could prevent Snowplow JavaScript from loading in your environment.
1. Turn off "Do Not Track" (DNT) in your browser.

All URLs are pseudonymized. The entity identifier [replaces](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracker-setup/other-parameters-2/#Setting_a_custom_page_URL_and_referrer_URL) personally identifiable
information (PII). PII includes usernames, group, and project names.

#### Snowplow Analytics Debugger Chrome Extension

[Snowplow Analytics Debugger](https://www.iglooanalytics.com/blog/snowplow-analytics-debugger-chrome-extension.html) is a browser extension for testing frontend events. It works in production, staging, and local development environments.

1. Install the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension.
1. Open Chrome DevTools to the Snowplow Analytics Debugger tab.

#### Snowplow Inspector Chrome Extension

Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works in production, staging, and local development environments.

1. Install [Snowplow Inspector](https://chrome.google.com/webstore/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm?hl=en).
1. To open the extension, select the Snowplow Inspector icon beside the address bar.
1. Click around on a webpage with Snowplow to see JavaScript events firing in the inspector window.

### Test backend events

#### Snowplow Micro

[Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) is a
Docker-based solution for testing backend and frontend in a local development environment. Snowplow Micro
records the same events as the full Snowplow pipeline. To query events, use the Snowplow Micro API. 

To install and run Snowplow Micro, complete these steps to modify the 
[GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit):

1. Ensure Docker is installed and running.

1. To install Snowplow Micro, clone the settings in 
[this project](https://gitlab.com/gitlab-org/snowplow-micro-configuration).

1. Navigate to the directory with the cloned project, 
   and start the appropriate Docker container:

   ```shell
   ./snowplow-micro.sh
   ```

1. Use GDK to start the PostgreSQL terminal and connect 
   to the `gitlabhq_development` database:

   ```shell
   gdk psql -d gitlabhq_development
   ```

1. Update your instance's settings to enable Snowplow events and 
   point to the Snowplow Micro collector:

   ```shell
   update application_settings set snowplow_collector_hostname='localhost:9090', snowplow_enabled=true, snowplow_cookie_domain='.gitlab.com';
   ```

1. Update `DEFAULT_SNOWPLOW_OPTIONS` in `app/assets/javascripts/tracking/constants.js` to remove `forceSecureTracker: true`:

   ```diff
   diff --git a/app/assets/javascripts/tracking/constants.js b/app/assets/javascripts/tracking/constants.js
   index 598111e4086..eff38074d4c 100644
   --- a/app/assets/javascripts/tracking/constants.js
   +++ b/app/assets/javascripts/tracking/constants.js
   @@ -7,7 +7,6 @@ export const DEFAULT_SNOWPLOW_OPTIONS = {
      appId: '',
      userFingerprint: false,
      respectDoNotTrack: true,
   -  forceSecureTracker: true,
      eventMethod: 'post',
      contexts: { webPage: true, performanceTiming: true },
      formTracking: false,
   ```

1. Update `options` in `lib/gitlab/tracking.rb` to add `protocol` and `port`:

   ```diff
   diff --git a/lib/gitlab/tracking.rb b/lib/gitlab/tracking.rb
   index 618e359211b..e9084623c43 100644
   --- a/lib/gitlab/tracking.rb
   +++ b/lib/gitlab/tracking.rb
   @@ -41,7 +41,9 @@ def options(group)
              cookie_domain: Gitlab::CurrentSettings.snowplow_cookie_domain,
              app_id: Gitlab::CurrentSettings.snowplow_app_id,
              form_tracking: additional_features,
   -          link_click_tracking: additional_features
   +          link_click_tracking: additional_features,
   +          protocol: 'http',
   +          port: 9090
            }.transform_keys! { |key| key.to_s.camelize(:lower).to_sym }
          end
   ```

1. Update `emitter` in `lib/gitlab/tracking/destinations/snowplow.rb` to change `protocol`:

   ```diff
   diff --git a/lib/gitlab/tracking/destinations/snowplow.rb b/lib/gitlab/tracking/destinations/snowplow.rb
   index 4fa844de325..5dd9d0eacfb 100644
   --- a/lib/gitlab/tracking/destinations/snowplow.rb
   +++ b/lib/gitlab/tracking/destinations/snowplow.rb
   @@ -40,7 +40,7 @@ def tracker
            def emitter
              SnowplowTracker::AsyncEmitter.new(
                Gitlab::CurrentSettings.snowplow_collector_hostname,
   -            protocol: 'https'
   +            protocol: 'http'
              )
            end
          end

   ```

1. Restart GDK:

   ```shell
   gdk restart
   ```

1. Send a test Snowplow event from the Rails console:

   ```ruby
   Gitlab::Tracking.event('category', 'action')
   ```

1. Navigate to `localhost:9090/micro/good` to see the event.

#### Useful links

- [Snowplow Micro repository](https://github.com/snowplow-incubator/snowplow-micro)
- [Installation guide recording](https://www.youtube.com/watch?v=OX46fo_A0Ag)

### Troubleshoot

To control content security policy warnings when using an external host, modify `config/gitlab.yml`
to allow or disallow them. To allow them, add the relevant host for `connect_src`. For example, for
`https://snowplow.trx.gitlab.net`:

```yaml
development:
  <<: *base
  gitlab:
    content_security_policy:
      enabled: true
      directives:
        connect_src: "'self' http://localhost:* http://127.0.0.1:* ws://localhost:* wss://localhost:* ws://127.0.0.1:* https://snowplow.trx.gitlab.net/"
```