| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
| |
By passing commit data to this worker we remove the need for querying
the Git repository for every job. This in turn reduces the time spent
processing each job.
The migration included migrates jobs from the old format to the new
format. For this to work properly it requires downtime as otherwise
workers may start producing errors until they're using a newer version
of the worker code.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When I proposed using serializable transactions I was hoping we would be
able to refresh data of individual users concurrently. Unfortunately
upon closer inspection it was revealed this was not the case. This could
result in a lot of queries failing due to serialization errors,
overloading the database in the process (given enough workers trying to
update the target table).
To work around this we're now using a Redis lease that is cancelled upon
completion. This ensures we can update the data of different users
concurrently without overloading the database.
The code will try to obtain the lease until it succeeds, waiting at
least 1 second between retries. This is necessary as we may otherwise
end up _not_ updating the data which is not an option.
|
| |\
| |
| |
| |
| |
| |
| | |
Preserve optional second parameter in NewNoteWorker jobs
Closes #24678
See merge request !7602
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
If there are any old or retries in the Sidekiq queue, NewNoteWorker
will fail with the error:
wrong number of arguments (given 2, expected 1)
This change allows the optional second argument to be used
to preserve backwards compatibility. It can be removed later.
Closes #24678
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This refactors repository caching so it's possible to selectively
refresh certain caches, instead of just expiring and refreshing
everything.
To allow this the various methods that were cached (e.g. "tag_count" and
"readme") use a similar pattern that makes expiring and refreshing
their data much easier.
In this new setup caches are refreshed as follows:
1. After a commit (but before running ProjectCacheWorker) we expire some
basic caches such as the commit count and repository size.
2. ProjectCacheWorker will recalculate the commit count, repository
size, then refresh a specific set of caches based on the list of
files changed in a push payload.
This requires a bunch of changes to the various methods that may be
cached. For one, data should not be cached if a branch used or the
entire repository does not exist. To prevent all these methods from
handling this manually this is taken care of in
Repository#cache_method_output. Some methods still manually check for
the existence of a repository but this result is also cached.
With selective flushing implemented ProjectCacheWorker no longer uses an
exclusive lease for all of its work. Instead this worker only uses a
lease to limit the number of times the repository size is updated as
this is a fairly expensive operation.
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Precalculate authorized projects in database
## What does this MR do?
It caches user's authorized projects in database instead of using multiple unions, which should simplify and speed-up things since this operation (getting authorized projects) is used a lot.
## Are there points in the code the reviewer needs to double check?
Did we miss a scenario where we need to refresh the list of projects?
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] ~~[Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)~~
- [ ] ~~API support added~~
- Tests
- [x] Added for this feature/bug
- [x] All builds are passing
- [x] Conform by the [merge request performance guides](http://docs.gitlab.com/ce/development/merge_request_performance_guidelines.html)
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if it does - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
## What are the relevant issue numbers?
#23150
See merge request !6839
|
| | |
| |
| |
| | |
Closes #23150
|
| |\ \
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Add environment info to builds page
Closes #23205
See merge request !7251
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | |/
|/|
| |
| | |
worker to populate this field.
|
| |\ \
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Add button to delete all merged branches
## What does this MR do?
It adds a button to the branches page that the user can use to delete all the branches that are already merged. This can be used to clean up all the branches that were forgotten to delete while merging MRs.
**Note**
~~This MR is WIP until MR !6408 is merged.~~
## Are there points in the code the reviewer needs to double check?
The UX of the actual "Delete merged branches" button.
## Why was this MR needed?
Fixes #21076
## Screenshots
Before:
After:
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [x] API support added
- Tests
- [x] Added for this feature/bug
- [x] All builds are passing
- [x] Conform by the [merge request performance guides](http://docs.gitlab.com/ce/development/merge_request_performance_guidelines.html)
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
## What are the relevant issue numbers?
Closes #21076
See merge request !6449
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
It adds a button to the branches page that the user can use to delete
all the branches that are already merged. This can be used to clean up
all the branches that were forgotten to delete while merging MRs.
Fixes #21076.
|
| | | |
| | |
| | |
| | | |
- Also remove unnecessary param
|
| |\ \ \
| | |/
| |/|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
* upstream/master: (70 commits)
Fix routing spec for group controller
Add small improvements to constrainers and specs
Faster search
Fix broken commits search
Changed helper method to check for none on params Moved if statements around in view
API: Return 400 when creating a systemhook fails
Update non-exist group spinach test to match routing
Bump omniauth-gitlab to 1.0.2 to fix incompatibility with omniauth-oauth2
Replace trigger with the new ID of the docs project
Refactor method name
17492 Update link color for more accessible contrast
Fixed todos empty state when filtering
Refactor namespace regex
implements reset incoming email token on issues modal and account page, reactivates all tests and writes more tests for it
Use separate email-friendly token for incoming email and let incoming email token be reset
Use the Gitlab Workhorse HTTP header in the admin dashboard
Refactor project routing
Fix 404 when visit /projects page
Rewritten spinach git_blame tests to rspec feature tests
Add tests for project#index routing
...
|
| | |\ \
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Processing Commits Using Sidekiq
This moves the code of `GitPushService#process_commit_messages` into a separate Sidekiq worker. This allows processing of commits to happen in parallel, speeding up the process. See the individual commit (messages) for more information.
Part of https://gitlab.com/gitlab-org/gitlab-ce/issues/15463
See merge request !6802
|
| | | |/
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This moves the code used for processing commits from GitPushService to
its own Sidekiq worker: ProcessCommitWorker.
Using a Sidekiq worker allows us to process multiple commits in
parallel. This in turn will lead to issues being closed faster and cross
references being created faster. Furthermore by isolating this code into
a separate class it's easier to test and maintain the code.
The new worker also ensures it can efficiently check which issues can be
closed, without having to run numerous SQL queries for every issue.
|
| | |/ |
|
| |\ \
| |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* upstream/master: (26 commits)
Add a `--force` option to bin/changelog
Update examples in changelog docs to use single quotes around title
Use the server's base URL without relative URL part when creating links in JIRA
Make ESLint ignore instrumented files for coverage analysis (!7236)
Check that JavaScript file names match convention (!7238)
Removed z-index for filters on issue boards
GitLab 8.13 not 13
Replace MR Description Format links
Fix gdb backtrace command
Update gitlab.yml.example
remove extra spaces from app/workers/post_receive.rb
Add Rake task to create/repair GitLab Shell hooks symlinks
Added guide for upgrading Postgres using Slony
Ensure hook tokens are write-only in the API
Add support for token attr in project hooks API
Add a CHANGELOG entry
Fix edit button wiki
Updated Sortable JS plugin
Allow owners to fetch source code in CI builds
fixes milestone dropdown not select issue
...
|
| | | |
|
| |\ \
| |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* upstream/master: (216 commits)
Remove invalid changelog entries
require rails/generators for generators
Adds variable. Fixes changelog
Fix haml_lint
Fixes after review
Update changelog
Remove tooltips from project header
Makes table pagination responsive
23545 Fix distorted project and group avatars
Fix spinach tests
Update links in side nav and header
Allow to search for user by secondary email address in the admin interface
Rename :name search parameter to :search_query at /admin/users
Initialize Sidekiq with the list of queues used by GitLab
Fix project features default values
Add a link to the Issue Boards API in main README
Add validation errors to Merge Request form
Fix Markdown styling inside reference links
Fix relative links in Markdown wiki when displayed in "Project" tab
Adds label description to issue board title
...
|
| | |\
| | |
| | |
| | | |
# Conflicts:
# app/workers/project_web_hook_worker.rb
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | | |
This changes ProjectCacheWorker.perform_async so it only schedules a job
when no lease for the given project is present. This ensures we don't
end up scheduling hundreds of jobs when they won't be executed anyway.
|
| | | | |
|
| | | |
| | |
| | |
| | | |
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6342#note_17353703
|
| |\ \ \
| | |/
| |/|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
* upstream/master: (33 commits)
removes extra line for empty milestone description
code formatting corrected
Fix status code expectation
Stop clearing the database cache on rake cache:clear
Fix error in generating labels
Fix bug where e-mails were not being sent out via Sidekiq
Document link syntax introduced by !5586
Fix documents and comments on Build API `scope`. #23146 #19131
adds entry in CHANGELOG
removes extra line for empty issue description
Re-organize queues to use for Sidekiq
Fix wrong endpoint in api/users documentation, fix same typo in spec describe blocks
Update CHANGELOG
Fix object data to be sent to fetch analytics data
Fixed compare ellipsis messing with layout
Change "Group#web_url" to return "/groups/twitter" rather than "/twitter".
Fix broken label uniqueness label migration
Fix project member access levels
fix font weight of project feature settings
Add hover to trash icon in notes
...
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Dumping too many jobs in the same queue (e.g. the "default" queue) is a
dangerous setup. Jobs that take a long time to process can effectively
block any other work from being performed given there are enough of
these jobs.
Furthermore it becomes harder to monitor the jobs as a single queue
could contain jobs for different workers. In such a setup the only
reliable way of getting counts per job is to iterate over all jobs in a
queue, which is a rather time consuming process.
By using separate queues for various workers we have better control over
throughput, we can add weight to queues, and we can monitor queues
better. Some workers still use the same queue whenever their work is
related. For example, the various CI pipeline workers use the same
"pipeline" queue.
This commit includes a Rails migration that moves Sidekiq jobs from the
old queues to the new ones. This migration also takes care of doing the
inverse if ever needed. This does require downtime as otherwise new jobs
could be scheduled in the old queues after this migration completes.
This commit also includes an RSpec test that blacklists the use of the
"default" queue and ensures cron workers use the "cronjob" queue.
Fixes gitlab-org/gitlab-ce#23370
|
| |\ \
| |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* upstream/master: (43 commits)
Disable warming of the asset cache in Spinach tests under CI
Trim project_path whitespace on form submit
added skipped definition
updated some links in definitions
Don't use Hash#slice since it's not supported in Ruby 2.1
Create protected branches bundle
[ci skip] Add a comment explaining validate_board_limit callback
Fix: Backup restore doesn't clear cache
Fix GitLab project import when a user has access only to their default namespace.
Test GitLab project import for a user with only their default namespace.
We want to release this in 8.13.0
Add CHANGELOG.md entry
Return truncation for notification descriptions, fix minor bugs with rendering
Use guard clause instead of if-else statement
Tests for markdown HipChat notifications
Clean up Banzai HTML for HipChat
Ensure absolute URLs for single lines from Banzai for HipChat
Absolute URLs for Banzai HTML for HipChat
Also render commit titles in HipChat notifications
Full Banzai rendering for HipChat notifications
...
|
| | |
| |
| |
| |
| |
| |
| | |
This ensures ProjectCacheWorker jobs for a given project are performed
at most once per 15 minutes. This should reduce disk load a bit in cases
where there are multiple pushes happening (which should schedule
multiple ProjectCacheWorker jobs).
|
| |\ \
| |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* upstream/master: (221 commits)
Differentiate the expire from leave event
Use LabelsFinder on Fogbuz importer
Use LabelsFinder on Google Code importer
Change the order of tested methods in project_members_controller_spec
Remove show_menu_above attribute from issuable dropdowns.
Spaces before `}`!
Avoid touch label links that does not belongs to project when moving it
Remove order by label type on LabelsFinder
Rename Labels::CreateService to Labels::FindOrCreateService
Add self.project_foreign_key on both Issue and MergeRequest
Only show label type for projects that belong to a group
Disable subscribing to group-level labels
Remove unused method Project#all_labels
Update specs to cope with new label types and priorities
Fix max number of permitted priorities per project label
Fix GitHub importer spec
Update CHANGELOG
Add support to group labels to SlashCommands::InterpretService
Use join instead of subquery on Label.unprioritized scope
Warn user deleting a group label affect all projects within the group
...
|
| | |
| |
| |
| |
| |
| |
| |
| | |
It may happen that job meant to remove expired artifacts will be
executed asynchronously when, in the meantime, project associated with
given build gets removed by another asynchronous job. In that case we
should not remove artifacts because such build will be removed anyway,
when project removal is complete.
|
| |\ \
| |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* pipeline-emails: (58 commits)
Update endpoint to username validator
change border color to variable
Add todo for deprecated user routes and more information about deprecation to changelog
Provide better error message to the user
Apply better hierarchy to markdown headers and issue/mr titles
Swapped button text manipulation outcomes for the toggle query
Fixed find file keyboard navigation
Update CHANGELOG for 8.12.7
Added download-button class and applied button margin
Enable activerecord_sane_schema_dumper for test
Updated logo from @luke
Fix broken specs on MySQL after https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6896
Fix Test Env (proper error handling when gitlab-shell is not clonned)
Fix randomly crashing spinach test for merge request
[Great spinach fix] Replace gsub with delete
Remove carriage returns from commit description as summary is on a newline and will always include carriage returns
Convert due_date_select.js filetype to es6.
Stop directly parsing due_date with Date.parse, prefer parsing implicitly.
Improve spec for pipeline metrics worker
Add Pipeline metrics worker
...
|
| | | |
|
| | | |
|
| | |
| |
| |
| | |
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6342/diffs#note_17038807
|
| | |
| |
| |
| | |
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6342/diffs#note_17039533
|
| |/
|
|
| |
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6342#note_16118195
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Execute pipeline hooks asynchronously
## What does this MR do?
This MR makes it possible to execute pipeline hooks asynchronously, what should help to improve performance of CI pipeline processing.
## What are the relevant issue numbers?
Closes #23056
See merge request !6824
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
