summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
...
| * | | ssync: fix decoding of ts_meta when ts_data has offsetAlistair Coles2023-02-274-5/+116
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The SsyncSender encodes object file timestamps in a compact form and the SsyncReceiver decodes the timestamps and compares them to its object file set. The encoding represents the meta file timestamp as a delta from the data file timestamp, NOT INCLUDING the data file timestamp offset. Previously, the decoding was erroneously calculating the meta file timestamp as the sum of the delta plus the data file timestamp INCLUDING the offset. For example, if the SssyncSender has object file timestamps: ts_data = t0_1.data ts_meta = t1.data then the receiver would erroneously perceive that the sender has: ts_data = t0_1.data ts_meta = t1_1.data As described in the referenced bug report, this erroneous decoding could cause the SsyncReceiver to request that the SsyncSender sync an object that is already in sync, which results in a 409 Conflict at the receiver. The 409 causes the ssync session to terminate, and the same process repeats on the next attempt. Closes-Bug: #2007643 Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Change-Id: I74a0aac0ac29577026743f87f4b654d85e8fcc80
* | | | Merge "InternalClient: error if allow_modify_pipeline is True"Zuul2023-04-142-0/+23
|\ \ \ \
| * | | | InternalClient: error if allow_modify_pipeline is TrueAlistair Coles2023-04-142-0/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | allow_modify_pipeline is no longer supported, but if a caller is still setting it to True then raise ValueError, because the InternalClient instance will no longer behave in the way the caller previously expected. Change-Id: I24015b8becc7289a7d72f9a5863d201e27bcc955
* | | | | Merge "internal_client: Remove allow_modify_pipeline option"Zuul2023-04-147-10/+96
|\ \ \ \ \ | |/ / / /
| * | | | internal_client: Remove allow_modify_pipeline optionMatthew Oliver2023-04-147-10/+96
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The internal client is suppose to be internal to the cluster, and as such we rely on it to not remove any headers we decide to send. However if the allow_modify_pipeline option is set the gatekeeper middleware is added to the internal client's proxy pipeline. So firstly, this patch removes the allow_modify_pipeline option from the internal client constructor. And when calling loadapp allow_modify_pipeline is always passed with a False. Further, an op could directly put the gatekeeper middleware into the internal client config. The internal client constructor will now check the pipeline and raise a ValueError if one has been placed in the pipeline. To do this, there is now a check_gatekeeper_loaded staticmethod that will walk the pipeline which called from the InternalClient.__init__ method. Enabling this walking through the pipeline, we are now stashing the wsgi pipeline in each filter so that we don't have to rely on 'app' naming conventions to iterate the pipeline. Co-Authored-By: Alistair Coles <alistairncoles@gmail.com> Change-Id: Idcca7ac0796935c8883de9084d612d64159d9f92
* | | | | Pull libc-related functions out to a separate moduleTim Burke2023-04-125-1024/+1099
| |_|/ / |/| | | | | | | | | | | | | | | Partial-Bug: #2015274 Change-Id: I3e26f8d4e5de0835212ebc2314cac713950c85d7
* | | | Merge "Pull timestamp-related functions out to a separate module"Zuul2023-04-125-1226/+1295
|\ \ \ \
| * | | | Pull timestamp-related functions out to a separate moduleTim Burke2023-04-055-1226/+1295
| | | | | | | | | | | | | | | | | | | | | | | | | Partial-Bug: #2015274 Change-Id: I5b7ab3b2c150ec1513b3e6ebc4b27808d5df042c
* | | | | Merge "Test nested (Metrics)PrefixLoggerAdapters"Zuul2023-04-122-0/+53
|\ \ \ \ \
| * | | | | Test nested (Metrics)PrefixLoggerAdaptersTim Burke2023-04-052-0/+53
| |/ / / / | | | | | | | | | | | | | | | Change-Id: I71ad4de0ab3af8e7e865cb924f96e5c415935654
* | | | | ring-builder: Abort cleanly on ^C, ^DTim Burke2023-04-101-1/+5
|/ / / / | | | | | | | | | | | | Change-Id: If8fbcaff8e5676accb10e6c3c49387bc0de0cdb9
* | | | Merge "Start common.utils refactor"Zuul2023-04-051-0/+0
|\ \ \ \
| * | | | Start common.utils refactorClay Gerrard2023-04-041-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | Related-Bug: #2015274 Change-Id: I6e7c1a19a39f51e4520dabfcfad65817534b42a2
* | | | | Merge "Patch policies for more per-policy quota tests"Zuul2023-04-041-3/+6
|\ \ \ \ \ | |/ / / / |/| | | |
| * | | | Patch policies for more per-policy quota testsTim Burke2023-04-031-3/+6
| | |_|/ | |/| | | | | | | | | | | | | | Related-Change: Ib25c2f667e5b81301f8c67375644981a13487cfe Change-Id: I1cc32bfe607454c6756f08ed9d95ac027369e638
* | | | Merge "tests: Skip s3api functional tests when no s3api user configured"Zuul2023-04-031-0/+4
|\ \ \ \
| * | | | tests: Skip s3api functional tests when no s3api user configuredTim Burke2023-03-101-0/+4
| | | | | | | | | | | | | | | | | | | | Change-Id: I61f141a71eddcac600058d66ddf802306df455c1
* | | | | Merge "tests: Let func tests run with test users 1 and 2 but not 3"Zuul2023-04-031-39/+42
|\ \ \ \ \ | |/ / / / | | / / / | |/ / / |/| | |
| * | | tests: Let func tests run with test users 1 and 2 but not 3Tim Burke2023-03-101-39/+42
| | | | | | | | | | | | | | | | Change-Id: Ia564f2ee70f5d04acab1c38e17d1936642a01447
* | | | Merge "sharder: fix host logged when send_shard_ranges fails"Zuul2023-03-272-38/+35
|\ \ \ \
| * | | | sharder: fix host logged when send_shard_ranges failsAlistair Coles2023-03-272-38/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The ContainerSharder._send_shard_ranges method sets an 'x-backend-use-replication-network' header with value 'true', so if the PUT to the root container fails the log message should show the replication ip and port of the container server. Change-Id: I8c84f6ee15e6999f71b092bbeed414065a22ee8b
* | | | | Merge "sharder: fix 'Cleaving not completed' warning log"Zuul2023-03-272-10/+10
|\ \ \ \ \ | |/ / / /
| * | | | sharder: fix 'Cleaving not completed' warning logAlistair Coles2023-03-272-10/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix missing space between the epoch time and "DB state:" in the log message. Change-Id: Ib654ba58cdcbf245458816460a15c964dfdb073c
* | | | | Update urlWei LingFei2023-03-244-4/+4
|/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | The OpenStack project is currently maintained on opendev.org, with github.com serving as a mirror repository. Replace the source code repository address for the python-swiftclient project from github.com to opendev.org. Change-Id: I650a80cb45febc457c42360061faf3a9799e6131
* | | | Merge "quotas: Add account-level per-policy quotas"Zuul2023-03-244-42/+192
|\ \ \ \
| * | | | quotas: Add account-level per-policy quotasTim Burke2023-03-214-42/+192
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Reseller admins can set new headers on accounts like X-Account-Quota-Bytes-Policy-<policy-name>: <quota> This may be done to limit consumption of a faster, all-flash policy, for example. This is independent of the existing X-Account-Meta-Quota-Bytes header, which continues to limit the total storage for an account across all policies. Change-Id: Ib25c2f667e5b81301f8c67375644981a13487cfe
* | | | | Forward-port stable-release CHANGELOG entriesTim Burke2023-03-231-0/+173
| | | | | | | | | | | | | | | | | | | | Change-Id: I155bd0d64b9662fe79323c9a5d3dbfe8e9109d79
* | | | | Merge "Allow internal container POSTs to not update put_timestamp"Zuul2023-03-235-3/+239
|\ \ \ \ \ | |/ / / / |/| | | |
| * | | | Allow internal container POSTs to not update put_timestampAlistair Coles2023-03-205-3/+239
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There may be circumstances when an internal client wishes to modify container sysmeta that is hidden from the user. It is desirable that this happens without modifying the put-timestamp and therefore the last-modified time that is reported in responses to client HEADs and GETs. This patch modifies the container server so that a POST will not update the container put_timestamp if an X-Backend-No-Timestamp-Update header is included with the request and has a truthy value. Note: there are already circumstances in which container sysmeta is modified without changing the put_timestamp: - PUT requests with shard range content do not update put_timestamp. - the sharder updates sysmeta directly via the ContainerBroker without modifying put_timestamp. Change-Id: I835b2dd58bc1d4fb911629e4da2ea4b9697dd21b
* | | | | proxy: add tests for listing with shard gapsAlistair Coles2023-03-171-5/+142
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The existing test_GET_sharded_container_gap_in_shards does not set memcache in the request environ and therefore does not exercise the proxy container controller code path for caching fetched shard ranges. This patch add two new tests: - test_GET_sharded_container_gap_in_shards_memcache which is similar to the existing but sets memcache in the request environ so that the fetched shard ranges (with gaps) are cached by the controller. - test_GET_sharding_container_gap_in_shards_memcache which sets memcache in the request environ but has the root pretend to be in sharding state so that the fetched shard ranges (with gaps) are not cached by the controller. In all cases the pretend root response headers are fixed to include 'X-Backend-Override-Shard-Name-Filter': 'true', which the real container server would return. This is another precondition for shard ranges being cached by the controller. Change-Id: I1f13690174cc2bddeca152a87cc978dee2a136d6
* | | | | Merge "Rename ShardRange*Bound to Namespace*Bound"Zuul2023-03-162-541/+545
|\ \ \ \ \ | |_|_|/ / |/| | | |
| * | | | Rename ShardRange*Bound to Namespace*BoundAlistair Coles2023-03-152-541/+545
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Also: - move some tests to test_utils.TestNamespace. - move ShardName class in file (no change to class) - move end_marker method from ShardRange to Namespace Related-Change: If98af569f99aa1ac79b9485ce9028fdd8d22576b Change-Id: Ibd5614d378ec5e9ba47055ba8b67a42ab7f7453c
* | | | | Merge "Proxy: restructure cached updating shard ranges"Zuul2023-03-156-272/+518
|\ \ \ \ \ | |/ / / /
| * | | | Proxy: restructure cached updating shard rangesJianjian Huo2023-03-066-272/+518
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Restructure the shard ranges that are stored in memcache for object updating to only persist the essential attributes of shard ranges in memcache (lower bounds and names), so the aggregate of memcache values is much smaller and retrieval will be much faster too. Co-Authored-By: Alistair Coles <alistairncoles@gmail.com> Co-Authored-By: Tim Burke <tim.burke@gmail.com> UpgradeImpact ============= The cache key for updating shard ranges in memcached is renamed from 'shard-updating/<account>/<container>' to 'shard-updating-v2/<account>/<container>', and cache data is changed to be a list of [lower bound, name]. As a result, this will invalid all existing updating shard ranges stored in the memcache cluster. Change-Id: If98af569f99aa1ac79b9485ce9028fdd8d22576b
* | | | | Merge "Update master for stable/2023.1"Zuul2023-03-132-13/+7
|\ \ \ \ \
| * | | | | Update master for stable/2023.1OpenStack Release Bot2023-03-132-13/+7
| | |_|/ / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add file to the reno documentation build to show release notes for stable/2023.1. Use pbr instruction to increment the minor version number automatically so that master versions are higher than the versions on stable/2023.1. Sem-Ver: feature Change-Id: I291be8e9911c40850002d629072ed3ab830ea626
* | | | | Drop bandit check B309Tim Burke2023-03-101-2/+1
|/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It was retired upstream: https://github.com/PyCQA/bandit/pull/858 Continuing to list it causes CI failures like ERROR Unknown test found in profile: B309 Change-Id: I52c535bd2078f25bb57c42a2c2c966247cd3094b
* | | | Authors/ChangeLog for 2.31.12.31.1Tim Burke2023-03-093-1/+71
| | | | | | | | | | | | | | | | Change-Id: I7ee960dee1329aee6ea535c6c64634cc05ca679a
* | | | Merge "quotas: Move account-level handling to a separate function"Zuul2023-03-081-20/+25
|\ \ \ \ | |/ / / |/| | |
| * | | quotas: Move account-level handling to a separate functionTim Burke2023-02-171-20/+25
| | | | | | | | | | | | | | | | Change-Id: I1ab7376b5e68a3deaad5aca113ad55bde00b2238
* | | | Merge "proxy: Reduce round-trips to memcache and backend on info misses"Zuul2023-03-023-2/+40
|\ \ \ \
| * | | | proxy: Reduce round-trips to memcache and backend on info missesTim Burke2023-03-013-2/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Following a memcache restart in a SAIO, I've seen the following happen during an object HEAD: - etag_quoter wants to get account/container info to decide whether to quote-wrap or not - account info is a cache miss, so we make a no-auth'ed HEAD to the next filter in the pipeline - eventually this gets down to ratelimit, which *also* wants to get account info - still a cache miss, so we make a *separate* HEAD that eventually talks to the backend and populates cache - ratelimit realizes it can't ratelimit the request and lets the original HEAD through to the backend There's a related bug about how something similar can happen when the backend gets overloaded, but *everything is working* -- we just ought to be talking straight to the proxy app. Note that there's likely something similar going on with container info, but the hardcoded 10% sampling rate makes it harder to see if you're monitoring raw metric streams. I thought I fixed this in the related change, but no :-/ Change-Id: I49447c62abf9375541f396f984c91e128b8a05d5 Related-Change: If9249a42b30e2a2e7c4b0b91f947f24bf891b86f Related-Bug: #1883214
* | | | | Merge "sharder: show path and db file in logs"Zuul2023-03-022-246/+438
|\ \ \ \ \
| * | | | | sharder: show path and db file in logsAlistair Coles2023-03-012-246/+438
| | |_|/ / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Make all logs that are associated with a ContainerBroker include the path to the DB file and the namespace path to the container. UpgradeImpact: There is a change in the format of sharder log messages, including some warning and error level logs. Change-Id: I7d2fe064175f002055054a72f348b87dc396772b
* | | | | Merge "ring-builder: require part_power>=0, give better error messages"Zuul2023-03-023-1/+14
|\ \ \ \ \ | |_|/ / / |/| | | |
| * | | | ring-builder: require part_power>=0, give better error messagesTim Burke2023-02-173-1/+14
| | |_|/ | |/| | | | | | | | | | | | | | Closes-Bug: #1697860 Change-Id: I500a86de390b24b9d08a478d695a7d62c447e779
* | | | Merge "docs: Add memcache.conf config doc"Zuul2023-02-283-1/+78
|\ \ \ \
| * | | | docs: Add memcache.conf config docMatthew Oliver2023-02-223-1/+78
| |/ / / | | | | | | | | | | | | Change-Id: I29d00e939a3842bd064382575955fa3e255242eb
* | | | Merge "Use FakeSwift in account quota tests"Zuul2023-02-281-158/+120
|\ \ \ \ | | |_|/ | |/| |
| * | | Use FakeSwift in account quota testsTim Burke2023-02-171-158/+120
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This lets us be much more controlled about which headers come back in which responses. Change-Id: I9823d00e78479e3335f7a25d0d2c99b63c1f9c12
View Lorry Controller Status