summaryrefslogtreecommitdiff
path: root/swift/common/middleware/s3api
Commit message (Collapse)AuthorAgeFilesLines
* Merge "s3token: Only replace access_key_id in account"Zuul2023-02-101-2/+2
|\
| * s3token: Only replace access_key_id in accountTim Burke2022-05-311-2/+2
| | | | | | | | | | Change-Id: Ifccedbe7662925db55d0d8cd9e2e66a03126f661 Closes-Bug: #1816181
* | Switch from pkg_resources to importlibTim Burke2023-01-311-2/+18
| | | | | | | | | | | | setuptools seems to be in the process of deprecating pkg_resources. Change-Id: I64f1434a5acab99057beb4f397adca85bdcc4ab6
* | s3api: Prevent XXE injectionsAymeric Ducroquetz2023-01-171-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, clients could use XML external entities (XXEs) to read arbitrary files from proxy-servers and inject the content into the request. Since many S3 APIs reflect request content back to the user, this could be used to extract any secrets that the swift user could read, such as tempauth credentials, keymaster secrets, etc. Now, disable entity resolution -- any unknown entities will be replaced with an empty string. Without resolving the entities, the request is still processed. [CVE-2022-47950] Closes-Bug: #1998625 Co-Authored-By: Romain de Joux <romain.de-joux@ovhcloud.com> Change-Id: I84494123cfc85e234098c554ecd3e77981f8a096
* | Merge "s3api errors for unsupported headers x-delete-at, x-delete-after"Zuul2022-12-053-12/+15
|\ \
| * | s3api errors for unsupported headers x-delete-at, x-delete-afterindianwhocodes2022-12-013-12/+15
| | | | | | | | | | | | | | | | | | | | | We need to support the aforementioned headers in our s3 apis and raise an InvalidArgumentError if a s3 client makes a request Change-Id: I2c5b18e52da7f33b31ba386cdbd042f90b69ef97
* | | Malformed CompleteMultipartUpload request should 400indianwhocodes2022-12-011-2/+6
|/ / | | | | | | | | Closes-Bug: #1883172 Change-Id: Ie44288976ac5a507c27bd175c5f56c9b0bd04fe0
* | proxy: Add a chance to skip memcache for get_*_info callsTim Burke2022-08-301-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If you've got thousands of requests per second for objects in a single container, you basically NEVER want that container's info to ever fall out of memcache. If it *does*, all those clients are almost certainly going to overload the container. Avoid this by allowing some small fraction of requests to bypass and refresh the cache, pushing out the TTL as long as there continue to be requests to the container. The likelihood of skipping the cache is configurable, similar to what we did for shard range sets. Change-Id: If9249a42b30e2a2e7c4b0b91f947f24bf891b86f Closes-Bug: #1883324
* | Add MPU to s3api testsClay Gerrard2022-08-241-3/+4
| | | | | | | | | | | | | | ... and reword some mpu listing logic Related-Change-Id: I923033e863b2faf3826a0f5ba84307addc34f986 Change-Id: If1909bb7210622908f2ecc5e06d53cd48250572a
* | s3api: Be more consistent about CompleteMultipartUpload loggingTim Burke2022-08-051-0/+6
| | | | | | | | | | Co-Authored-By: Jianjian Huo <jhuo@nvidia.com> Change-Id: I7837a2ec7dee9a657e36147c208c524b5a01671d
* | s3api: Better handle 503s in get_container_info, tooTim Burke2022-06-241-1/+3
| | | | | | | | | | Change-Id: I269a59559a943fbf2781224d6962b25f6e07d30c Related-Change: Iadb0a40092b8347eb5c04785cc14d1324cc9396f
* | s3api: Make grentee types case insensitiveAymeric Ducroquetz2022-06-201-5/+6
| | | | | | | | | | | | This allows to be always more compatible with AWS S3. Change-Id: Icf6da9e9abba4abb825a5b109ff978e586319fbb
* | Fix s3api cross policy copyClay Gerrard2022-06-171-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | In s3api's request object we copy backend headers into the request environ for logging after we call get_response. The problem with s3api copy is that we make a pre-flight HEAD request to the source object using the same request object, so the first response backend headers pollute the request and the proxy won't over-ride the backend header with the correct storage policy. As a possible fix we simply remove the problematic header from the request object after the pre-flight HEAD request finishes. Change-Id: I40b252446b3a1294a5ca8b531f224ce9c16f9aba
* | s3api: Add best-effort support for more canned ACLsTim Burke2022-05-311-1/+6
|/ | | | | | | | | | Using pure Swift ACLs, there is no real difference between private, bucket-owner-full-control, and bucket-owner-read. Drive-By: Return NotImplemented on log-delivery-write, just like we do for authenticated-read. Change-Id: I79761b0f1f5f90f2602005e3e0428d201b5c813e
* Round s3api listing LastModified to integer resolutionAlistair Coles2022-05-103-5/+16
| | | | | | | | | | | | | | | | | | | | | s3api bucket listing elements currently have LastModified values with millisecond precision. This is inconsistent with the value of the Last-Modified header returned with an object GET or HEAD response which has second precision. This patch reduces the precision to seconds in bucket listings and upload part listings. This is also consistent with observation of an aws listing response. The last modified values in the swift native listing *up* to the nearest second to be consistent with the seconds-precision Last-Modified time header that is returned with an object GET or HEAD. However, we continue to include millisecond digits set to 0 in the last-modified string, e.g.: '2014-06-10T22:47:32.000Z'. Also, fix the last modified time returned in an object copy response to be consistent with the last modified time of the object that was created. Previously it was rounded down, but it should be rounded up. Change-Id: I8c98791a920eeedfc79e8a9d83e5032c07ae86d3
* s3api: Improve error message when bucket is not emptyAymeric Ducroquetz2022-04-152-2/+12
| | | | | | | When the versioning is enabled (or suspended), AWS specifies in the error message that all versions should be deleted. Change-Id: I3da9469a5cfed031a2cee85e1dfcd78bbe54695a
* s3api: Use constant-time string comparisons in check_signatureTim Burke2022-04-131-3/+4
| | | | Change-Id: Ibe514a7ab22d475517b1efc50de676f47d741a4c
* Merge "s3api: Copy more headers from MPU marker to final object"Zuul2022-03-271-0/+3
|\
| * s3api: Copy more headers from MPU marker to final objectTim Burke2022-03-251-0/+3
| | | | | | | | | | Closes-Bug: 1966396 Change-Id: I253d8e3e8678fad3fde43259ed3225df4048a458
* | s3api: Make the 'Quiet' key value case insensitiveAymeric Ducroquetz2022-03-242-5/+2
|/ | | | | | | | | | | | | | | | When deleting multiple objects, S3 allows to enable a quiet mode with the 'Quiet' key. At AWS S3, the value of this key is case-insensitive. - Quiet mode is enabled if the value is 'true' (regardless of case). - Otherwise, in all other cases (even a non-boolean value), this mode will be disabled. Also, some tools (like Minio's python API) send the value 'True' (and not 'true'). Change-Id: Id9d1da2017b8d13242ae1f410347febb013e9ce1
* Comply with AWS signature calculation (s3v4)Thibault Person2022-03-151-1/+2
| | | | | | | | | | | | | | | The current implementation of s3 signature calculation rely on WSGI Url encoding which is discouraged by AWS: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html. This leads to reject requests with valid signature. This update encode only characters specified by AWS except 'A'-'Z', 'a'-'z', '0'-'9', '-', '.', '_', and '~' to comply AWS signature calculation. Fixes LP Bug #1961841 Change-Id: Ifa8f94544224c3379e7f2805f6f86d0b0a47279a
* Merge "s3api: Fix multi_delete with object names using non-ASCII characters"Zuul2022-03-101-1/+2
|\
| * s3api: Fix multi_delete with object names using non-ASCII charactersAymeric Ducroquetz2022-03-011-1/+2
| | | | | | | | | | Co-Authored-By: Florent Vennetier <florent.vennetier@ovhcloud.com> Change-Id: I635bc91faa7709f9df9cdf3aec157a21c08923ca
* | Merge "s3api: Delete all parts when aborting MPU with non-ASCII characters"Zuul2022-03-091-1/+2
|\ \
| * | s3api: Delete all parts when aborting MPU with non-ASCII charactersAymeric Ducroquetz2022-03-091-1/+2
| |/ | | | | | | Change-Id: Idcda76f7a880a18c3bac699e0fb2435e4a54abbd
* | s3api: Properly decode MPU request parameters before using themAymeric Ducroquetz2022-03-011-17/+19
|/ | | | | | | Specifically, parameters that may contain non-ASCII characters, such as the prefix and marker to list current uploads. Change-Id: Icfae68825f94ddf2412c0274c3d500e265117e8e
* Trim sensitive information in the logs (CVE-2017-8761)Matthew Oliver2022-02-091-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several headers and query params were previously revealed in logs but are now redacted: * X-Auth-Token header (previously redacted in the {auth_token} field, but not the {headers} field) * temp_url_sig query param (used by tempurl middleware) * Authorization header and X-Amz-Signature and Signature query parameters (used by s3api middleware) This patch adds some new middleware helper methods to track headers and query parameters that should be redacted by proxy-logging. While instantiating the middleware, authors can call either: register_sensitive_header('case-insensitive-header-name') register_sensitive_param('case-sensitive-query-param-name') to add items that should be redacted. The redaction uses proxy-logging's existing reveal_sensitive_prefix config option to determine how much to reveal. Note that query params will still be logged in their entirety if eventlet_debug is enabled. UpgradeImpact ============= The reveal_sensitive_prefix config option now applies to more items; operators should review their currently-configured value to ensure it is appropriate for these new contexts. In particular, operators should consider reducing the value if it is more than 20 or so, even if that previously offered sufficient protection for auth tokens. Co-Authored-By: Tim Burke <tim.burke@gmail.com> Closes-Bug: #1685798 Change-Id: I88b8cfd30292325e0870029058da6fb38026ae1a
* Merge "Fix multipart upload listings"Zuul2022-02-041-21/+29
|\
| * Fix multipart upload listingsTimur Alperovich2022-02-021-21/+29
| | | | | | | | | | | | | | | | | | When listing current multipart uploads, we have to keep listing the container until we either list all the entries or there are enough MPUs to return to the caller. Otherwise, it is impossible to list all of the multipart uploads when some of them have > 1000 parts. Change-Id: I923033e863b2faf3826a0f5ba84307addc34f986
* | Merge "Move *_swift_info functions into a new registry module"Zuul2022-02-046-10/+13
|\ \ | |/ |/|
| * Move *_swift_info functions into a new registry moduleMatthew Oliver2022-02-036-10/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The *_swift_info functions use in module global dicts to provide a registry mechanism for registering and getting swift info. This is an abnormal pattern and doesn't quite fit into utils. Further we looking at following this pattern for sensitive info to trim in the future. So this patch does some house cleaning and moves this registry to a new module swift.common.registry. And updates all the references to it. For backwards compat we still import the *_swift_info methods into utils for any 3rd party tools or middleware. Change-Id: I71fd7f50d1aafc001d6905438f42de4e58af8421
* | Merge "s3api: Fix non-ascii MPUs"Zuul2022-01-312-8/+15
|\ \
| * | s3api: Fix non-ascii MPUsTim Burke2022-01-282-8/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previous problems included: - returning wsgi strings quoted assuming UTF-8 on py3 when initiating or completing multipart uploads - trying to str() some unicode on py2 when listing parts, leading to UnicodeEncodeErrors Change-Id: Ibc1d42c8deffe41c557350a574ae80751e9bd565
* | | Merge "s3api: Allow multiple storage domains"Zuul2022-01-283-10/+11
|\ \ \ | |_|/ |/| |
| * | s3api: Allow multiple storage domainsTim Burke2022-01-243-10/+11
| |/ | | | | | | | | | | | | | | Sometimes a cluster might be accessible via more than one set of domain names. Allow operators to configure them such that virtual-host style requests work with all names. Change-Id: I83b2fded44000bf04f558e2deb6553565d54fd4a
* | s3api: Fix types in /infoTim Burke2022-01-271-7/+9
|/ | | | | | | Having the defaults be ints/booleans but configured values be strings is weird, at best. Change-Id: I4a46bd650a53f88c642d402e697869df28bd2fd3
* Do not fetch content of container/object to retrieve S3 ACLsAymeric Ducroquetz2022-01-132-1/+4
| | | | | | | | | | | Bucket ACLs: The contents of the container are unnecessarily listed. Object ACLs: The content of the object is unnecessarily fetched. Additionally, because the data is skipped, a 499 error is returned on a subrequest. Change-Id: I1e6ccc8ec4a54375b5817498c4ac7f995656a794
* Fix some imports for py310Tim Burke2021-11-251-1/+4
| | | | | | | Between this and the (unreleased) pyeclib fix, I see unit and func tests passing on py310. Haven't tried probe tests, yet. Change-Id: Iacf66eda75fed6bf96900107250f393227c57ae5
* Merge "Bug: fix s3api multipart parts listings"Zuul2021-10-031-5/+13
|\
| * Bug: fix s3api multipart parts listingsTimur Alperovich2021-09-281-5/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | s3api returns multipart parts listings out of order and possibly missing. For example, if there are 2000 parts, the first 12 parts returned by s3api currently will be: 1, 10-19, 100. Then after part 199, the following part is 1000, and so on. The change fixes this behavior by internally listing all of the parts (with default settings, this should be 1 listing request, as the 10000 parts limit matches the Swift listing limit). After that, the parts are sorted and delimited/marker settings are applied to craft the response for the client. Change-Id: I150cf53b07e7d2d8de1d6e8c1fb08c07b9afe842
* | s3api: fix the copy of non-ASCII objectsFlorent Vennetier2021-09-291-1/+2
|/ | | | | | | | | | | | | Trying to copy an object with non-ASCII characters in its name results in, depending on the pipeline: - an error code 412 because of a badly urlencoded path - an error code 500 "TypeError: Expected a WSGI string" This commit fixes the problem by calling str_to_wsgi on the object name after it has been urldecoded. We do not need to call this on the container name because it is supposed to contain only ASCII characters. Change-Id: If837d4e55735b10a783c85d91f37fbea5e3baf1d
* Merge "s3api: Allow CORS preflights for pre-signed URLs"Zuul2021-09-091-9/+27
|\
| * s3api: Allow CORS preflights for pre-signed URLsTim Burke2021-07-141-9/+27
| | | | | | | | | | | | | | Looks like browsers *do* send the query string in the OPTIONS request. Change-Id: Id10e6e32890f1c9a09c91990e5a6ee729bf4d973 Related-Change: I985143bf03125a05792e79bc5e5f83722d6431b3
* | Merge "s3api: Fix (async) multi-delete of MPUs"Zuul2021-08-021-2/+6
|\ \
| * | s3api: Fix (async) multi-delete of MPUsTim Burke2021-07-301-2/+6
| | | | | | | | | | | | Change-Id: I2347a73ff23c5c7d415f23d864fc29147e4a1754
* | | Merge "s3api: Return KeyTooLongError when upload name exceeds constraints"Zuul2021-07-273-3/+12
|\ \ \ | |/ / |/| |
| * | s3api: Return KeyTooLongError when upload name exceeds constraintsTim Burke2021-06-093-3/+12
| | | | | | | | | | | | Change-Id: I8ce73e2e21e9216484130ba3bd1e77b45eb1d77c
* | | s3api: Pass through 409s from SLOTim Burke2021-07-232-1/+10
| | | | | | | | | | | | Change-Id: Ie91a90fbb3488af63a51dcd18fa2c60ad00e234d
* | | Handle ClientDisconnect on s3api object PUTClay Gerrard2021-07-051-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Mostly when I disconnect during upload with s3api clients I see the proxy log a traceback for EPIPE, but if I set my my client_timeout low and the proxy initiates the disconnect s3api will get surprised by the 499 response and return 500. Now s3api will handle it the same as a RequestTimeout, which looks like a 400 on the wire if anyone is still there. Change-Id: I08be94fc5cf16679f41a2fd08ce1d52ce6300871
* | | Include status code when we fail to parse an SLO-delete responseTim Burke2021-06-281-2/+2
|/ / | | | | | | | | | | | | We've occasionally seen errors here where the body is empty. Hopefully knowing more about the response will shed some light on what happened. Change-Id: I69c748ebf721579a5fae85333ce3d4e999b9eb2a
View Lorry Controller Status