| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
| |
The flag is on by default, but allows us to revert back
to the old behaviour if we encounter any problems.
|
| | |
|
| |\
| |
| |
| |
| |
| |
| | |
Resolve "Hashed storage migration should not be attempted if project does not validate"
Closes #56618
See merge request gitlab-org/gitlab-ce!25753
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This is a fix for the Hashed Storage migration and Rollback procedure
to ignore any project-level validation error that can happen in a
long-running instance.
There are many situations where defaults and acceptable values changed
but, because we didn't provide a migration to "valid" attributes, it
can happen that project will not be `valid? => true`.
Because the changes we are making are limited to setting a project as
read_only or changing the storage_level, it's safe to bypass validation.
|
| |\ \
| |/
|/| |
|
| | | |
|
| |\ \
| | |
| | |
| | |
| | |
| | |
| | | |
into 'master'
Hashed Storage rollback mechanism
See merge request gitlab-org/gitlab-ce!23955
|
| | | |
| | |
| | |
| | | |
Exceptions were also simplified from 2 to 1.
|
| | | |
| | |
| | |
| | |
| | | |
`try_to_set_repository_read_only!` is now on the Base class.
Simplified Exception classes from 2 to 1 with a more descriptive name.
|
| | | |
| | |
| | |
| | |
| | |
| | | |
This change takes the case where repository doesn't exist on disk
and make it ignore but succeed the step, increasing or decreasing the
version depending on the operation.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Rollback is done similar to Migration for the Hashed Storage.
It also shares the same ExclusiveLease key to prevent both happening
at the same time.
All Hashed Storage related workers now share the same queue namespace
which allows for assigning dedicated workers easily.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | | |
We are adding sidekiq workers and service classes to allow to rollback
a hashed storage migration. There are some refactoring involved as well
as part of the code can be reused by both the migration and the rollback
logic.
|
| |/ /
| |
| |
| |
| |
| | |
- Move some specs into contexts
- Let get_slugs method take a parameter and return a specific slug.
- Add rescues when using Addressable::URI.
|
| |/
|
|
|
|
| |
The API get projects/:id/traffic/fetches allows user with write
access to the repository to get the number of clones for the
last 30 days.
|
| |
|
|
|
|
|
| |
When hashed storage is in use, it's helpful to have the project
name associated with the request.
Closes https://gitlab.com/gitlab-org/gitaly/issues/1394
|
| |\
| |
| |
| |
| | |
Update pages config only when changed
See merge request gitlab-org/gitlab-ce!24424
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
RubyZip allows us to perform strong validation of
expanded paths where we do extract file.
We introduce the following additional checks
to extract routines:
1. None of path components can be symlinked,
2. We drop privileges support for directories,
3. Symlink source needs to point within the target directory,
like `public/`,
4. The symlink source needs to exist ahead of time.
|
| | | |
|
| | | |
|
| | |
| |
| |
| | |
We need this new state for the Geo event logic in EE
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
Specs were reviewed and improved to better cover the current behavior.
There was some standardization done as well to facilitate the
implementation of the rollback functionality.
StorageMigratorWorker was extracted to HashedStorage namespace were
RollbackerWorker will live one as well.
|
| | |
| |
| |
| |
| | |
This is part of the refactor to include a RollbackService into
HashedStorage module
|
| |\ \
| | |
| | |
| | |
| | |
| | |
| | | |
Container repository cleanup API
Closes #55978
See merge request gitlab-org/gitlab-ce!24303
|
| | |/
| |
| |
| |
| |
| | |
This includes a set of APIs to manipulate container registry.
This includes also an ability to delete tags based on requested
criteria, like keep-last-n, matching-name, older-than.
|
| | |
| |
| |
| | |
Signed-off-by: Rémy Coutable <remy@rymai.me>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
We still rely on the Dirty API for project rename (before/after) values,
but we don't access the dirty api from the service class anymore.
The previous value is now part of the initialization, which makes it
easier to test and the behavior is clearer.
The same was done with the `rename_repo` on the Storage classes, we now
provide before and after values as part of the method signature.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
During a previous refactor on project model, code related to the
hashed storage was extracted into AfterRenameService, see
4b9c17f196bab6075563f62d01f9db65c1a0515c.
The "path_before" was changed from using `previous_changes['path']` to
`path_was`. They are not equivalent. `path_was` exists reliably only
*before* persisting to the database. After database persistence is
confirmed, the value is moved to `previous_changes[:attribute_name]`.
Because the repository/attachments rename or storage upgrade happens
after it was persisted to the database, we were in fact not informing
the right parameters (and therefore not doing what it was supposed to).
|
| |\ \
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
'master'
Cleanup stale +deleted repo paths on project removal (adjusts project removal bug)
Closes #46146
See merge request gitlab-org/gitlab-ce!24269
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
1. When removing projects, we can end-up leaving the +deleted
repo path dirty and not successfully removing the non-deleted
namespace (mv process is not atomic and can be killed without
fully moving the path).
2. In order to solve that, we're adding a clean-up phase on
ensure which will schedule possible staled +deleted path deletion.
Note that we don't check the current state (if there is or not a
repo) in order to schedule the deletion. That's intentional
in order to leverage Gitlab::GitalyClient::NamespaceService#remove
idempotency and ensure consistency.
|
| | |/
|/|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This refactors some of the logic used for protecting default branches,
in particular Project#after_create_default_branch. The logic for this
method is moved into a separate service class. Ideally we'd get rid of
Project#after_create_default_branch entirely, but unfortunately
Project#after_import depends on it. This means it has to stick around
until we also refactor Project#after_import.
For branch protection levels we introduce
Gitlab::Access::BranchProtection, which provides a small wrapper around
Integer based branch protection levels. Using this class removes the
need for having to constantly refer to Gitlab::Access::PROTECTION_*
constants.
|
| |/ |
|
| |\
| |
| |
| |
| | |
Hashed Storage: Only set as `read_only` when starting the per-project migration
See merge request gitlab-org/gitlab-ce!24128
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
In the previous code, we locked the project during the migration
scheduling step, which works fine for small setups, but can be
problematic in really big installations.
We now moved the logic to inside the worker, so we minimize the time a
project will be read-only. We also make sure we only do that if
reference counter is `0` (no current operation is in progress).
|
| | |
| |
| |
| | |
Re-use operations controller which already handles tracing settings.
|
| |/
|
|
|
| |
This commit prepares the structure for the upcoming feature error
tracking.
|
| |\
| |
| |
| |
| | |
[master] Security todos not redacted for guests
See merge request gitlab/gitlabhq!2697
|
| | |
| |
| |
| |
| | |
Fix leaking information of confidential issues on TODOs
when user is downgraded to guest access.
|
| |\ \
| | |
| | |
| | |
| | | |
[master] SSRF in project imports with LFS
See merge request gitlab/gitlabhq!2720
|
| | | | |
|
| | | | |
|
| |/ / |
|
| |/ |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When a project is forked, the new repository used to be a deep copy of everything
stored on disk by leveraging `git clone`. This works well, and makes isolation
between repository easy. However, the clone is at the start 100% the same as the
origin repository. And in the case of the objects in the object directory, this
is almost always going to be a lot of duplication.
Object Pools are a way to create a third repository that essentially only exists
for its 'objects' subdirectory. This third repository's object directory will be
set as alternate location for objects. This means that in the case an object is
missing in the local repository, git will look in another location. This other
location is the object pool repository.
When Git performs garbage collection, it's smart enough to check the
alternate location. When objects are duplicated, it will allow git to
throw one copy away. This copy is on the local repository, where to pool
remains as is.
These pools have an origin location, which for now will always be a
repository that itself is not a fork. When the root of a fork network is
forked by a user, the fork still clones the full repository. Async, the
pool repository will be created.
Either one of these processes can be done earlier than the other. To
handle this race condition, the Join ObjectPool operation is
idempotent. Given its idempotent, we can schedule it twice, with the
same effect.
To accommodate the holding of state two migrations have been added.
1. Added a state column to the pool_repositories column. This column is
managed by the state machine, allowing for hooks on transitions.
2. pool_repositories now has a source_project_id. This column in
convenient to have for multiple reasons: it has a unique index allowing
the database to handle race conditions when creating a new record. Also,
it's nice to know who the host is. As that's a short link to the fork
networks root.
Object pools are only available for public project, which use hashed
storage and when forking from the root of the fork network. (That is,
the project being forked from itself isn't a fork)
In this commit message I use both ObjectPool and Pool repositories,
which are alike, but different from each other. ObjectPool refers to
whatever is on the disk stored and managed by Gitaly. PoolRepository is
the record in the database.
|
| | |
|
| |
|
|
| |
This reflects how we now create or update
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
AFAIK the only relevant place is Projects::CreateService, this gets
called when user creates a new project, forks a new project and does
those things via the api.
Also create k8s namespace for new group hierarchy
when transferring project between groups
Uses new Refresh service to create k8s namespaces
- Ensure we use Cluster#cluster_project
If a project has multiple clusters (EE), using Project#cluster_project
is not guaranteed to return the cluster_project for this cluster. So
switch to using Cluster#cluster_project - at this stage a cluster can
only have 1 cluster_project.
Also, remove rescue so that sidekiq can retry
|
| |
|
|
|
| |
Clears the import related columns and code from the Project
model over to the ProjectImportState model
|
| | |
|
