summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
...
| | * | | | | | Use the `IssuableBaseService` lifecycle hooks to cache ↵Timothy Andrew2016-09-219-19/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | `MergeRequestsClosingIssues` - Instead of overriding `create` and `update` in `MergeRequests::BaseService` - Get all merge request service specs passing
| | * | | | | | Implement a second round of review comments from @DouweM.Timothy Andrew2016-09-2123-205/+252
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Don't use `TableReferences` - using `.arel_table` is shorter! - Move some database-related code to `Gitlab::Database` - Remove the `MergeRequest#issues_closed` and `Issue#closed_by_merge_requests` associations. They were either shadowing or were too similar to existing methods. They are not being used anywhere, so it's better to remove them to reduce confusion. - Use Rails 3-style validations - Index for `MergeRequest::Metrics#first_deployed_to_production_at` - Only include `CycleAnalyticsHelpers::TestGeneration` for specs that need it. - Other minor refactorings.
| | * | | | | | Move JSON generation (cycle analytics) into a controller method.Timothy Andrew2016-09-202-37/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Helper methods are meant for views
| | * | | | | | Add CHANGELOG entry.Timothy Andrew2016-09-201-0/+1
| | | | | | | |
| | * | | | | | Fix rubocop spec.Timothy Andrew2016-09-205-12/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | And `scss_lint`
| | * | | | | | Implement (some) comments from @DouweM's review.Timothy Andrew2016-09-2010-123/+127
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Move things common to `Issue` and `MergeRequest` into `Issuable` - Move more database-specific functions into `Gitlab::Database` - Indentation changes and other minor refactorings.
| | * | | | | | Update schema.rbTimothy Andrew2016-09-201-12/+12
| | | | | | | |
| | * | | | | | Fix spec failures.Timothy Andrew2016-09-206-12/+16
| | | | | | | |
| | * | | | | | Fix `ON DELETE CASCADE` migrations.Timothy Andrew2016-09-203-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Incorrect syntax.
| | * | | | | | Cycle analytics should be accessible to all for public projects.Timothy Andrew2016-09-201-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Public projects - anyone can access - Private projects - any member (guest level and above) can access
| | * | | | | | Implement review comments from @yorickpeterseTimothy Andrew2016-09-2017-225/+147
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Change multiple updates to a single `update_all` 2. Use cascading deletes 3. Extract an average function for the database median. 4. Move database median to `lib/gitlab/database` 5. Use `delete_all` instead of `destroy_all` 6. Minor refactoring
| | * | | | | | Merge remote-tracking branch 'origin/master' into 21170-cycle-analyticsTimothy Andrew2016-09-20528-5141/+11166
| | |\ \ \ \ \ \
| | * | | | | | | Cycle Analytics: A value of 0 is changed to `nil`.Timothy Andrew2016-09-201-1/+1
| | | | | | | | |
| | * | | | | | | Implement a database median strategy for MySQL.Timothy Andrew2016-09-202-23/+76
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Dispatch between the two strategies automatically based on the current database type. 2. The MySQL version needs to run multiple statements, so the `cycle_analytics` model is modified to support this.
| | * | | | | | | Miscellaneous cycle-analytics-related changes.Timothy Andrew2016-09-202-7/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Add indexes to `CreateMergeRequestsClosingIssues` columns. 2. Remove an extraneous `check_if_open` check that is redundant now. It would've been better to rebase this in, but that's not possible because more people are working on this branch.
| | * | | | | | | Fix all cycle analytics specs.Timothy Andrew2016-09-207-16/+82
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A number of failures were introduced due to performance improvements (like pre-calculating metrics).
| | * | | | | | | CycleAnalytics operates on merge requests that have been deployed.Timothy Andrew2016-09-201-23/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Look for merge requests (and issues that they close) that have been deployed to production in the last X days (where X is given by the `from` parameter). 2. Cycle analytics queries only operate on this fitered set of merge requests and issues.
| | * | | | | | | Add a uniqueness constraint to `MergeRequestsClosingIssues`Timothy Andrew2016-09-201-0/+5
| | | | | | | | |
| | * | | | | | | Test the `CycleAnalytics::Summary` model.Timothy Andrew2016-09-204-6/+83
| | | | | | | | |
| | * | | | | | | Simplify vue template for cycle analytics.Fatih Acet2016-09-202-11/+11
| | | | | | | | |
| | * | | | | | | Finalise cycle analytics frontend.Fatih Acet2016-09-195-139/+164
| | | | | | | | |
| | * | | | | | | Do _not_ set `stats` to `null` when data is not available.Timothy Andrew2016-09-191-2/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - For cycle analytics. - Instead, make each individual `value` `null`, since the titles and descriptions are used by the frontend even when there is no data.
| | * | | | | | | Make changes to the cycle analytics JSON endpoint.Timothy Andrew2016-09-193-5/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Add `summary` section. 2. `stats` is `null` if no stats are present. 3. `stats` and `summary` are both arrays.
| | * | | | | | | Remove unused merge request metrics.Timothy Andrew2016-09-196-103/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - These are not being used anymore. - Consolidate all issue metrics into a single migration. - Consolidate all merge request metrics into a single migration.
| | * | | | | | | Test all cycle analytics pre-calculation code.Timothy Andrew2016-09-1912-10/+352
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | All the code that pre-calculates metrics for use in the cycle analytics page. - Ci::Pipeline -> build start/finish - Ci::Pipeline#merge_requests - Issue -> record default metrics after save - MergeRequest -> record default metrics after save - Deployment -> Update "first_deployed_to_production_at" for MR metrics - Git Push -> Update "first commit mention" for issue metrics - Merge request create/update/refresh -> Update "merge requests closing issues"
| | * | | | | | | Remove extraneous code that was previously added for performance.Timothy Andrew2016-09-192-4/+2
| | | | | | | | |
| | * | | | | | | Move cycle analytics JSON generation to a helper.Timothy Andrew2016-09-173-8/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Use a new format, with each stage having a `title`, `description`, and `value.
| | * | | | | | | Move cycle analytics calculations to SQL.Timothy Andrew2016-09-173-36/+144
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Use Arel for composable queries. 2. For a project with ~10k issues, the page loads in around 600ms. Previously, a project with ~5k issues would have a ~20s page load time.
| | * | | | | | | Add a "populate metrics directly" option to the cycle analytics seed.Timothy Andrew2016-09-171-3/+61
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - The normal seed creates all the data for cycle analytics the "right" way. It creates issues, merge requests, commits, branches, deployments, etc. This is good, but too slow for perf testing. Generating a 1000 sets of records this way takes more than an hour. - When the `CYCLE_ANALYTICS_POPULATE_METRICS_DIRECTLY` environment variable is passed in, the seed only creates issues and merge requests. It then adds the `metrics` for each issue and merge request directly, to save time. - The seed now takes about 4 minutes to run for 1000 sets of records.
| | * | | | | | | Fix layout for cycle analytics.Fatih Acet2016-09-162-39/+41
| | | | | | | | |
| | * | | | | | | Implement Cycle Analytics frontend.Fatih Acet2016-09-151-3/+2
| | | | | | | | |
| | * | | | | | | Improve performance of the cycle analytics page.Timothy Andrew2016-09-1518-120/+335
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
| | * | | | | | | Implement Cycle Analytics frontend.Fatih Acet2016-09-158-52/+255
| | | | | | | | |
| | * | | | | | | wip: perfTimothy Andrew2016-09-143-17/+20
| | | | | | | | |
| | * | | | | | | Add a `seed_fu` fixture to generate data for cycle analytics.Timothy Andrew2016-09-141-0/+173
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - The fixture generates data for every stage in the cycle analytics dashboard. Once this fixture has run, you shouldn't be seeing any "<not enough data>" messages for cycle analytics. - This is probably not necessary for every fixture run, so it might be moved behind an env var in the future.
| | * | | | | | | Add a JSON version of the `CycleAnalytics` page.Timothy Andrew2016-09-083-0/+13
| | | | | | | | |
| | * | | | | | | Merge remote-tracking branch 'origin/master' into 21170-cycle-analyticsTimothy Andrew2016-09-07478-3473/+7528
| | |\ \ \ \ \ \ \
| | * | | | | | | | Fix rubocop spec.Timothy Andrew2016-09-071-2/+2
| | | | | | | | | |
| | * | | | | | | | Consolidate all cycle analytics helper methods.Timothy Andrew2016-09-078-264/+184
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Move the test generation to `CycleAnalyticsHelpers::TestGeneration` 2. Move all helper methods (previously placed in each individual spec file) to `CycleAnalyticsHelpers`
| | * | | | | | | | Add a test case for "end condition happens before start condition".Timothy Andrew2016-09-071-0/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - In the cycle analytics specs.
| | * | | | | | | | Test the `test` cycle analytics phase.Timothy Andrew2016-09-072-2/+92
| | | | | | | | | |
| | * | | | | | | | Test the `production` cycle analytics phase.Timothy Andrew2016-09-072-1/+86
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a `before_end_fn` option to the code that generates cycle analytics specs. `before_end_fn` is called before the end conditions are. Used for data setup that needs to be called after the start conditions and before the end conditions.
| | * | | | | | | | Test the `staging` cycle analytics phase.Timothy Andrew2016-09-073-4/+85
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove overlap from the "start + end" durations in the happy test case. For the `staging` phase, the end time is the _first_ deployment that happens after the MR merge. If we have 5 MRs where the `start_time`s (merge time) are the same, and all the `end_time`s (deploy to production) a few days from now, only the earliest deploy will get picked up, because that consitutes a deploy for _all_ the MRs. We fix this by removing overlap. Every `start_time` is now generated to be _after_ the preceding `end_time`.
| | * | | | | | | | Generate branch/file names more randomly to avoid failures.Timothy Andrew2016-09-074-6/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Tests would randomly fail because of naming conflicts. - Use a `random_git_name` method instead of using `FFaker` directly.
| | * | | | | | | | Test the 'review' cycle analytics phase.Timothy Andrew2016-09-071-0/+45
| | | | | | | | | |
| | * | | | | | | | Test the `code` cycle analytics phase.Timothy Andrew2016-09-074-25/+77
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Move the "data belongs to other project" test case into the generated tests, and remove the explicit tests from the `code` and `plan` phases.
| | * | | | | | | | Refactor cycle analytics specs.Timothy Andrew2016-09-023-180/+133
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Generalise the specs that will be common across all cycle analytics phases. 2. Rewrite specs `issue` and `plan` to use this abstracted testing strategy. 3. Specs that are specific to a given phase, or unwieldy to test in an abstracted manner, are added to each phase's spec.
| | * | | | | | | | Add tests for the `plan` cycle analytics phase.Timothy Andrew2016-09-022-63/+188
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Clean up the `issue` tests as well
| | * | | | | | | | Tweak cycle analytics query to match the current requirements.Timothy Andrew2016-09-022-8/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - The `review` phase ends when a MR is merged, not "merged OR closed". - The `code` phase starts when a MR is first mentioned in a commit, and ends when a merge request closing the issue is created. - The `plan` phase ends when the issue first mentioned in a commit. --- - Fix the `median` function so it sorts the incoming data points. - A data point where `end_time` is prior to `start_time` is invalid.
| | * | | | | | | | Authorize access to cycle analytics.Timothy Andrew2016-08-262-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | All users with "Reporter" access (and above) to a project can see the Cycle Analytics page.