| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| ... | |
| | |/ |
|
| |/
|
|
|
|
|
|
|
|
|
|
|
| |
The AddVisibilityLevelToGroups migration introduced a visibility_level for
namespaces and specified that projects should always have a visibility level
less than or equal to their namespace. However, some invalid rows could have
been created.
This commit introduces a migration that updates the invalid rows, setting the
invalid project to have the same visibility_level as their namespaces. This
will make some projects internal or private when they would previously have
been public or internal, but this is better than silently making an internal
or private group public.
|
| |\
| |
| |
| |
| | |
Use separate email-friendly token for incoming email
See merge request !5914
|
| | |
| |
| |
| | |
reactivates all tests and writes more tests for it
|
| | |
| |
| |
| | |
email token be reset
|
| |\ \
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Use more than one kind of Git garbage collection
Replaces https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6588 by @jacobvosmaer to get the builds to pass :)
Closes #22729
See merge request !7321
|
| | | | |
|
| |\ \ \
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Fixups to "Round-robin repository storage"
## What does this MR do?
* Simplifies a method in application_settings.rb
* Correctly marks a migration as needing downtime
* Documents the requirement for renamed columns to be
## Are there points in the code the reviewer needs to double check?
Should any of these changes be split out? Ideally we'd get this into the same point release as !7273
## Why was this MR needed?
Post-facto review of !7273
## Screenshots (if relevant)
## Does this MR meet the acceptance criteria?
- [ ] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG.md) entry added
- [X] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [X] API support added
- Tests
- [X] Added for this feature/bug
- [x] All builds are passing
- [X] Conform by the [merge request performance guides](http://docs.gitlab.com/ce/development/merge_request_performance_guidelines.html)
- [X] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [X] Branch has no merge conflicts with `master` (if it does - rebase it please)
- [X] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
## What are the relevant issue numbers?
Related to #24059
/cc @yorickpeterse @rspeicher
See merge request !7287
|
| | |/ / |
|
| | |/
|/|
| |
| |
| |
| |
| |
| |
| |
| | |
performance
We see that many slow queries on GitLab.com are dominated by finding
the project import data for a specific project. Adding an index is
the most straightforward way of fixing this.
Closes #23748
|
| |/
|
|
|
|
| |
are resolved
Signed-off-by: Rémy Coutable <remy@rymai.me>
|
| |
|
|
| |
round-robin between them
|
| |
|
|
|
|
|
|
| |
These are regular Rails migrations that are executed by default. A user
can opt-out of these migrations by setting an environment variable
during the deployment process.
Fixes gitlab-org/gitlab-ce#22133
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Use optimistic locking
## What does this MR do?
Removes the usage of pessimistic locking in favor of optimistic which is way cheaper and doesn't block database operation.
Since this is very simple change it should be safe. If we receive `StaleObjectError` message we will reload object a retry operations in lock.
However, I still believe that we need this one: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7005 as this will reduce a load on Database and FS.
This changes a behavior from:
### Pesimistic locking (previous behavior)
#### For updating
1. SELECT * FOR UPDATE (other updates wait on this)
2. we update ci_pipeline
3. latest_build_status
4. enqueue: (use: transition :created -> :pending)
5. [state_machine] we are in state created, we can go to pending
6. [state_machine] ci_pipeline.status = created
7. [state_machine] ci_pipeline.save
8. [state_machine] after_transition: (if for success): PipelineSuccessWorker on Sidekiq
9. release DB lock
#### If no update is required
1. SELECT * FOR UPDATE (other updates wait on this)
2. we update ci_pipeline
3. latest_build_status
4. we are in pending, we can't transition to pending, because it's forbidden
5. release DB lock
### Optimistic locking (implemented by this MR)
#### For updating
1. latest_build_status
2. enqueue: (use `transition :created -> :pending`)
3. [state_machine] we are in state created, we can go to pending
4. [state_machine] ci_pipeline.status = created
5. [state_machine] ci_pipeline.save
6. [state_machine] [save] where(lock_version: ci_pipeline.lock_version).update_all(status: :created, updated_at: Time.now)
7. [state_machine] [save] unless we_updated_row then raise ObjectInconsistentError
#### If no update is required
1. we update ci_pipeline
2. latest_build_status
3. we are in pending, we can't transition to pending, because it's forbidden
## Why was this MR needed?
We have been seeing a number of problems when we migrated Pipeline/Build processing to Sidekiq. Especially we started seeing a lot of blocking queries.
We used a pessimistic locking which doesn't seem to be required. This effectively allows us to fix our issues with blocked queries by using more efficient method of operation.
## What are the relevant issue numbers?
Issues: https://gitlab.com/gitlab-com/infrastructure/issues/623 and https://gitlab.com/gitlab-com/infrastructure/issues/584, but also there's a bunch of Merge Requests that try to improve behavior of scheduled jobs.
cc @pcarranza @yorickpeterse @stanhu
See merge request !7040
|
| | | |
|
| | |
| |
| |
| | |
Code improvements, bug fixes, finish documentation and specs
|
| |/ |
|
| |
|
|
|
|
|
|
|
|
| |
mail_room was configured to deliver mail to the `incoming_email`
queue while `EmailReceiveWorker` was reading the `email_receiver`
queue. Adds a migration that repeats the work of a previous
migration to ensure all mails that wound up in the old
queue get processed.
Closes #23689
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Dumping too many jobs in the same queue (e.g. the "default" queue) is a
dangerous setup. Jobs that take a long time to process can effectively
block any other work from being performed given there are enough of
these jobs.
Furthermore it becomes harder to monitor the jobs as a single queue
could contain jobs for different workers. In such a setup the only
reliable way of getting counts per job is to iterate over all jobs in a
queue, which is a rather time consuming process.
By using separate queues for various workers we have better control over
throughput, we can add weight to queues, and we can monitor queues
better. Some workers still use the same queue whenever their work is
related. For example, the various CI pipeline workers use the same
"pipeline" queue.
This commit includes a Rails migration that moves Sidekiq jobs from the
old queues to the new ones. This migration also takes care of doing the
inverse if ever needed. This does require downtime as otherwise new jobs
could be scheduled in the old queues after this migration completes.
This commit also includes an RSpec test that blacklists the use of the
"default" queue and ensures cron workers use the "cronjob" queue.
Fixes gitlab-org/gitlab-ce#23370
|
| |\
| |
| |
| |
| |
| |
| |
| |
| | |
Fix project member access levels
Migrate invalid project members (owner -> master)
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/18616
See merge request !6957
|
| | | |
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The previous implementation of the migration failed on staging because
the migration was attempted to remove labels from projects that did not
actually have duplicates. This occurred because the SQL query did not
account for the project ID when selecting the labels.
To replicate the problem:
1. Disable the uniqueness validation in app/models/label.rb.
2. Create a duplicate label "bug" in project A.
3. Create the same label in project B with label "bug".
The migration will attempt to remove the label in B even if there are no duplicates.
Closes #23609
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\ |
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| | |
use separate renders
|
| |\ \
| |/ |
|
| | |
| |
| |
| | |
Signed-off-by: Rémy Coutable <remy@rymai.me>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This commit introduces a Sidekiq worker that precalculates the list of
trending projects on a daily basis. The resulting set is stored in a
database table that is then queried by Project.trending.
This setup means that Unicorn workers no longer _may_ have to calculate
the list of trending projects. Furthermore it supports filtering without
any complex caching mechanisms.
The data in the "trending_projects" table is inserted in the same order
as the project ranking. This means that getting the projects in the
correct order is simply a matter of:
SELECT projects.*
FROM projects
INNER JOIN trending_projects ON trending_projects.project_id = projects.id
ORDER BY trending_projects.id ASC;
Such a query will only take a few milliseconds at most (as measured on
GitLab.com), opposed to a few seconds for the query used for calculating
the project ranks.
The migration in this commit does not require downtime and takes care of
populating an initial list of trending projects.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This commit adds a number of _html columns and, with the exception of Note,
starts updating them whenever the content of their partner fields changes.
Note has a collision with the note_html attr_accessor; that will be fixed later
A background worker for clearing these cache columns is also introduced - use
`rake cache:clear` to set it off. You can clear the database or Redis caches
separately by running `rake cache:clear:db` or `rake cache:clear:redis`,
respectively.
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Changes include:
- Ensure Member.add_user is not called directly when not necessary
- New GroupMember.add_users_to_group to have the same abstraction level as for Project
- Refactor Member.add_user to take a source instead of an array of members
- Fix Rubocop offenses
- Always use Project#add_user instead of project.team.add_user
- Factorize users addition as members in Member.add_users_to_source
- Make access_level a keyword argument in GroupMember.add_users_to_group and ProjectMember.add_users_to_projects
- Destroy any requester before adding them as a member
- Improve the way we handle access requesters in Member.add_user
Instead of removing the requester and creating a new member,
we now simply accepts their access request. This way, they will
receive a "access request granted" email.
- Fix error that was previously silently ignored
- Stop raising when access level is invalid in Member, let Rails validation do their work
Signed-off-by: Rémy Coutable <remy@rymai.me>
|
| |
|
|
|
|
|
| |
* Add newline to user organization spec according to test guide
* Remove unnecessary comments from user organization database migration
Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>
|
| |
|
|
| |
Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Fix database seeds for development environment
## What does this MR do?
This MR fixes database seeds for development environment and adds CI test for it.
## Why was this MR needed?
Database seeds for development environment are often broken, and we are not able to catch that when someone modified `db/fixtures` and forgets to reseed database.
Closes #22422
See merge request !6475
|
| | | |
|
| |/ |
|
| |
|
|
|
| |
It seems this wasn't updated properly when migrations were added in a
previous commit.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Don't use `TableReferences` - using `.arel_table` is shorter!
- Move some database-related code to `Gitlab::Database`
- Remove the `MergeRequest#issues_closed` and
`Issue#closed_by_merge_requests` associations. They were either
shadowing or were too similar to existing methods. They are not being
used anywhere, so it's better to remove them to reduce confusion.
- Use Rails 3-style validations
- Index for `MergeRequest::Metrics#first_deployed_to_production_at`
- Only include `CycleAnalyticsHelpers::TestGeneration` for specs that
need it.
- Other minor refactorings.
|
