| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |\ |
|
| | |
| |
| |
| | |
Change-Id: I6c0bc1570f6a48439de5a029a86f1b582f30f8a6
|
| |\ \ |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This patch provides the necessary error handling while unpickling
a builder file. Earlier if a builder file is empty/invalid/corrupted,
the stacktrace was shown to user with an exit code of 1. This fixes it
to show a user-friendly message and also returns the exit code of 2,
indicating there was a failure.
Change-Id: I51eb24702c422299629f8053d4591dd10f5863f8
Closes-Bug: #1370680
|
| |\ \ \ |
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
This is "obviously" the right thing to do, except of course it's
pure sugar: the return value is not used anywhere. Except if someone
has a script that imports the whole thing somehow and then does
isinstance(dict). Because that is so much easier than submitting
a patch, I can imagine someone, somewhere doing that.
The fix came in a patch by dfg, see review 121851.
Change-Id: I39ddf70dc2d027b7db6e31097400248dc66eb137
|
| |\ \ \ \
| |_|_|/
|/| | | |
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
The "logging" available in eventlet.wsgi.server/BaseHTTPServer doesn't
generally suite our needs, so it should be bypassed using a NullLogger in
production. But in development it can be useful if tracebacks generated from
inside eventlet.wsgi (say a NameError in DiskFile.__iter__) end up in logs.
Since we already have eventlet_debug parsed inside of run_server we can skip
the NullLogger bypass and let stuff blast out to STDERR when configured for
development/debug logging.
Change-Id: I20a9e82c7fed8948bf649f1f8571b4145fca201d
|
| |\ \ \ \ |
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
When adding a new tier (region, zone, node, device) to an existing,
already balanced ring all existing partitions in the existing tiers of
the same level are gathered for reassigning, even when there is
not enough space in the new tier. This will create a lot of unnecessary
replication traffic in the backend network.
For example, when only one region exists in the ring and a new region is
added, all existing parts are selected to reassign, even when the new
region has a total weight of 0. Same for zones, nodes and devices.
This patch limits the number of partitions that are choosen to reassign
by checking for devices on other tiers that are asking for more
partitions.
Failed devices are not considered when applying the limit.
Co-Authored By: Florent Flament <florent.flament-ext@cloudwatt.com>
Change-Id: I6178452e47492da4677a8ffe4fb24917b5968cd9
Closes-Bug: 1367826
|
| |\ \ \ \ \ |
|
| | | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
Currently metadata limits are checked on a per request basis. If
multiple requests are sent within the per request limits, it is
possible to exceed the overall limits. This patch adds an overall
metadata check to ensure that multiple requests to add metadata to
an account/container will check overall limits before adding
the additional metadata.
Change-Id: Ib9401a4ee05a9cb737939541bd9b84e8dc239c70
Closes-Bug: 1365350
|
| |\ \ \ \ \ \ |
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
1) Added comment for H231, which we were already enforcing. H231
is for Python 3.x compatible except statements.
2) Added check for H201, which we were enforcing in reviews
but waiting on hacking checks to be updated. H201 is for bare
except statements, and the update in upstream hacking is to
support the " # noqa" flag on it.
The H201 check catches some existing bare excepts that are fixed.
Change-Id: I68638aa9ea925ef62f9035a426548c2c804911a8
|
| |\ \ \ \ \ \ \ |
|
| | | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
Currently, the fork_child() function in auditor.py does not
handle the case where run_audit() encounters an exception
properly.
A simple case is where the /srv directory is set
with permissions such that the 'swift' user cannot access it.
Such a situation causes a os.listdir() to return an OSError
exception. When this happens the fork_child() process does not
run to completion and sys.exit() is not executed. The process
that was forked off continues to run as a result. Execution goes
back up to the audit_loop function which restarts the whole process. The
end result is an increasing number of processes on the system
until the parent is terminated. This can quickly exhaust the
process descriptors on a system.
This change wraps run_audit() in a try block and adds an
exception handler that prints what exception was encountered.
The sys.exit() was moved to a finally: block so that it will
always be run, avoiding the creation of zombies.
Change-Id: I89d7cd27112445893852e62df857c3d5262c27b3
Closes-bug: 1375348
|
| |/ / / / / / /
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
If you adjust the replica count after adding devices but before the
first rebalance, the ring builder would crash.
To reproduce:
$ swift-ring-builder crashy.builder create 8 3 1
$ swift-ring-builder crashy.builder add r1z1-10.1.1.1:6000/sda1 1
$ swift-ring-builder crashy.builder set_replicas 5
$ swift-ring-builder crashy.builder rebalance
Change-Id: Id91cf8680961ccf6e3db153577e99cad545bee9d
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
Change-Id: I6310390f756bc0d326e56fceef32b630f0fe1ca1
|
| |\ \ \ \ \ \ \
| |_|/ / / / /
|/| | | | | |
| | | | | | | |
TestObjectReplicator"
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
TestObjectReplicator
Remove the duplicated statements.
Closes-Bug: #1374783
Change-Id: If2b55e864fea497d7a7b218adf11eb7749c27765
|
| |\ \ \ \ \ \ \
| |_|_|_|_|/ /
|/| | | | | | |
|
| | |/ / / / /
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
You should use the suffix variable instead of the hash_ variable when
updates the hashes dictionary in the get_hashes().
Fixes Bug1374777
Change-Id: If9a7ab5b6d271385e4fdc79ff9c216047a2f4d7e
|
| |\ \ \ \ \ \ |
|
| | | |/ / / /
| |/| | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
Everyone else cooperates to store the container_info cache in the env the
first time it's requested. This should save a duplicate memcache hit on
requests that ratelimiting looks at.
Change-Id: Ic6411d4619db6b53fafe9fdbf1d0a370d1258c38
|
| |\ \ \ \ \ \ |
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
disk_chunk_size is unused by the ObjectReplicator class. Note that it
*is* used by the object replicator process; it's just that
DiskFileManager is the one that pulls it out of the conf and pays
attention to it. Keeping it as an attribute on ObjectReplicator is
unnecessary.
Change-Id: I1eeef7b33873b4c8bb269ca02dcb067098b6fded
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
Change-Id: I59b68ecb6f07e2b71e4655699d0165007d764718
|
| |\ \ \ \ \ \ \ |
|
| | | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
The call to _get_direct_account_container in direct_get_account
has several of its args =None instead of set to the value passed
to direct_get_account.
The same applies to _get_direct_account_container in
direct_get_container.
The direct_get_container is only called by the account-reaper
and this bug will have limited impact on it. The marker,
maintained in reap_container, is ignored by direct_get_container.
This is not as bad as it sounds, if the account-reaper successfully
deletes the first 10K objects, assuming the container has > 10K
objects, the next call to direct_get_container will in fact return
the next 10K objects even though it sets marker=None (assuming the
first 10K objects were successfully deleted).
This patch also updates test_direct_get_account and
test_direct_get_container to ensure the appropriate
args are included in the connection query_string.
Closes-Bug: #1369558
Change-Id: If1c8aa1240d38354ebc9b1ebca92dc1c8c36cb5f
|
| | |_|_|/ / / /
|/| | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
This patch fixes the unit tests to remove the temporary directories
created during run of unit tests. Some of unit tests did not tear down
correctly, whatever it had set it up for running. This would over period
of time bloat up the tmp directory. As on date, there were around 49 tmp
directories left uncleared per round of unit tests. This patch fixes it.
Change-Id: If591375ca9cc87d52c7c9c6dc16c9fb4b49e99fc
|
| |\ \ \ \ \ \ \
| |_|_|_|_|_|/
|/| | | | | | |
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
This commit lets the object server use splice() and tee() to move data
from disk to the network without ever copying it into user space.
Requires Linux. Sorry, FreeBSD folks. You still have the old
mechanism, as does anyone who doesn't want to use splice. This
requires a relatively recent kernel (2.6.38+) to work, which includes
the two most recent Ubuntu LTS releases (Precise and Trusty) as well
as RHEL 7. However, it excludes Lucid and RHEL 6. On those systems,
setting "splice = on" will result in warnings in the logs but no
actual use of splice.
Note that this only applies to GET responses without Range headers. It
can easily be extended to single-range GET requests, but this commit
leaves that for future work. Same goes for PUT requests, or at least
non-chunked ones.
On some real hardware I had laying around (not a VM), this produced a
37% reduction in CPU usage for GETs made directly to the object
server. Measurements were done by looking at /proc/<pid>/stat,
specifically the utime and stime fields (user and kernel CPU jiffies,
respectively).
Note: There is a Python module called "splicetee" available on PyPi,
but it's licensed under the GPL, so it cannot easily be added to
OpenStack's requirements. That's why this patch uses ctypes instead.
Also fixed a long-standing annoyance in FakeLogger:
>>> fake_logger.warn('stuff')
>>> fake_logger.get_lines_for_level('warn')
[]
>>>
This, of course, is because the correct log level is 'warning'. Now
you get a KeyError if you call get_lines_for_level with a bogus log
level.
Change-Id: Ic6d6b833a5b04ca2019be94b1b90d941929d21c8
|
| |\ \ \ \ \ \ \ |
|
| | | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
Change-Id: I363651046cf414e14f15affd834043aabd5427c0
|
| |\ \ \ \ \ \ \ \
| | |_|_|_|_|/ /
| |/| | | | | | |
|
| | | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
When a ring is rebalanced the number of changed partitions is counted.
Before this patch partitions might be rebalanced, but actually no data
is moved - for example, when a partition is assigned to the same device
as before. This results in a wrong number of reassigned partitions that
is shown to the user.
This patch remembers the partition allocation before the rebalance, and
compares it to the new allocation after a rebalance. Only partitions
that are stored on a different device than before are counted.
Partial-Bug: 1367826
Also-By: Florent Flament <florent.flament-ext@cloudwatt.com>
Change-Id: Iacfd514df3af351791f9191cef78cff1b3e2645f
|
| |\ \ \ \ \ \ \ \
| |_|_|_|_|_|/ /
|/| | | | | | |
| | | | | | | | |
mount using it."
|
| | | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
using it.
Change-Id: Idcaf16a278d6c34770af9b1f17d69bdd94fb86b7
|
| |\ \ \ \ \ \ \ \ |
|
| | | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
Start tracking the container count per policy including reporting
it in account HEAD and supporting installations where the DB
existed before the updated schema.
Migration is triggered by the account audtior; if the database is
un-migrated it will continue to report policy_stats without the per
policy container_count keys.
Closes-Bug: #1367514
Change-Id: I07331cea177e19b3df303609a4ac510765a19162
|
| |\ \ \ \ \ \ \ \ \ |
|
| | | |_|_|_|/ / / /
| |/| | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
RFC 7233 says that servers MAY reject egregious range-GET requests
such as requests with hundreds of ranges, requests with non-ascending
ranges, and so on.
Such requests are fairly hard for Swift to process. Consider a Range
header that asks for the first byte of every 10th MiB in a 4 GiB
object, but in some random order. That'll cause a lot of seeks on the
object server, but the corresponding response body is quite small in
comparison to the workload.
This commit makes Swift reject, with a 416 response, any ranged GET
request with more than fifty ranges, more than three overlapping
ranges, or more than eight non-increasing ranges.
This is a necessary prerequisite for supporting multi-range GETs on
large objects. Otherwise, a malicious user could construct a Range
header with hundreds of byte ranges where each individual byterange
requires the proxy to contact a different object server. If seeking
all over a disk is bad, connecting all over the cluster is way worse.
DocImpact
Change-Id: I4dcedcaae6c3deada06a0223479e611094d57234
|
| |\ \ \ \ \ \ \ \ \
| |_|_|_|_|_|_|/ /
|/| | | | | | | | |
|
| | | |_|_|_|/ / /
| |/| | | | | |
| | | | | | | |
| | | | | | | | |
Change-Id: Ibd8882766a87c6d77e786f7635b1290391e43f10
|
| |\ \ \ \ \ \ \ \ |
|
| | | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
Cleanup and add clarification to the documentation
for using Keystone auth.
Update to refer to auth_token middleware being
distributed as part of the keystomemiddelware project
rather than keystone.
Include capabilities (/info) in the list of reasons
why delay_auth_decision might need to be set in
auth_token middleware config.
Add description of the project_id:user_id format
for container ACLs and emphasize that ids rather than
names should be used since this patch has now merged:
https://review.openstack.org/#/c/86430
DocImpact
blueprint keystone-v3-support
Change-Id: Idda4a3dcf8240474f1d2d163016ca2d40ec2d589
|
| |\ \ \ \ \ \ \ \ \
| |_|/ / / / / / /
|/| | | | | | | | |
|
| | | |_|_|/ / / /
| |/| | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
This change allows the user to use a "--no-overlap" parameter when
running the tool multiple times. It will increase the coverage by
whatever is specified in the dispersion_coverage field of the conf
file in a manner where existing container/objects are left in place
and no partition is populated more than once.
Related-Bug: #1233045
Change-Id: I139fed2f4c967ba18d073b7ecd1e946ed4da1271
|
| |\ \ \ \ \ \ \ \ |
|
| | | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
ContainerBroker.merge_items() had a bug in it where non-ASCII Unicode
names would possibly result in duplicate entries in container
databases. AccountBroker.merge_items() doesn't do the same
bulk-operations tricks that ContainerBroker does, so it doesn't
currently have the bug. This commit just adds a test to ensure the bug
doesn't creep in should someone decide to make AccountBroker look more
like ContainerBroker someday.
Change-Id: Id2ac129828dbdf55b609d839ce4d9d42437ee0a3
|
