summaryrefslogtreecommitdiff
path: root/swiftclient/client.py
Commit message (Collapse)AuthorAgeFilesLines
* Clean up warnings from newer flake8Tim Burke2019-06-271-1/+1
| | | | Change-Id: I18a6327b3acdd4db5ae80097080c043f7c20c353
* Make proper functions instead of assigning lambdasTim Burke2019-02-211-1/+3
| | | | Change-Id: I89255f6923c649c7b9d3d36e96c09f8bc4f51a3c
* Update hacking versionZhijunWei2019-01-031-7/+7
| | | | | | | 1. update hacking version to latest 2. fix pep8 failed Change-Id: Ifc3bfeff4038c93d8c8cf2c9d7814c3003e73504
* Add delimiter to get_account().Timur Alperovich2018-11-301-7/+12
| | | | | | | Exposes the delimiter parameter, which the Swift API supports for container listings. Change-Id: Id8dfce01a9b64de9d1222aab9a4a682ce9e0f2b7
* Stop leaking quite so many connectionsTim Burke2018-11-091-12/+16
| | | | | | | | | | | | | | While investigating the failures when you move func tests to py3, I noticed a whole bunch of ResourceWarning: unclosed <socket.socket ...> noise. This should fix it. While we're at it, make get_capabilities less stupid. Change-Id: I3913e9334090b04a78143e0b70f621aad30fc642 Related-Change: I86d24104033b490a35178fc504d88c1e4a566628
* Stop lazy importing keystoneclientTim Burke2018-09-071-23/+30
| | | | | | | | | | | | | | | | There were two basic problems: - We'd try to import on every attempt at getting auth, even when we already know keystoneclient isn't available. - Sometimes devs would hit some crazy import race involving (some combination of?) greenthreads and OS threads. So let's just try the imports *once*, at import time, and have None sentinels if it fails. Try both versions separately to decouple failures; this should let us support a wider range of keystoneclient versions. Change-Id: I2367310aac74f1b7c5ea0cb1a822a491e4ba8e68
* Remove unnecessary calls to parse_header_string().Timur Alperovich2018-07-241-8/+2
| | | | | | | | Since we define the getheader() method on the response from HTTPConnection, we don't have to call parse_header_string, as the values will already be converted properly. Change-Id: Ia81e8674b828b3ff1f014454126b469e41adfc23
* Merge "Properly handle unicode headers."Zuul2018-07-241-5/+24
|\
| * Properly handle unicode headers.Timur Alperovich2018-07-231-5/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix unicode handling in Python 3 and Python 2. There are currently two failure modes. In python 2, swiftclient fails to log in debug mode if the account name has a non-ASCII character. This is because the account name will appear in the storage URL, which we attempt to pass to the logger as a byte string (whereas it should be a unicode string). This patch changes the behavior to convert the path strings into unicode by calling the parse_header_string() function. The second failure mode is with Python 3, where http_lib returns headers that are latin-1 encoded, but swiftclient expects UTF-8. The patch automatically converts headers from latin-1 (iso-8859-1) to UTF-8, so that we can properly handle non-ASCII headers in responses. Change-Id: Ifa7f3d5af71bde8127129f1f8603772d80d063c1
* | Add close() to _RetryBody.Timur Alperovich2018-07-201-0/+3
|/ | | | | | | Allows clients to give up on reading the rest of the server response, if they so choose. Change-Id: Iccc95b1b5e7d066470966ee0c62a3beb260846e5
* Merge "Stop mutating header dicts"Zuul2018-07-171-31/+34
|\
| * Stop mutating header dictsTim Burke2017-08-251-31/+34
| | | | | | | | Change-Id: Ia1638c216eff9db6fbe416bc0570c27cfdcfe730
* | Make OS_AUTH_URL work in DevStack by defaultClay Gerrard2018-06-201-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | An earlier change added support for versionless authurls, but the huristic to detect them didn't work for some configurations I've encountered. Now we use a little bit tighter pattern matching and support auth_url values with more than one path component. Change-Id: I5a99c7b4e957ee7c8a5b5470477db49ab2ddba4b Related-Change-Id: If7ecb67776cb77828f93ad8278cc5040015216b7
* | Merge "Use a valid default for auth_version"Zuul2018-06-061-1/+1
|\ \
| * | Use a valid default for auth_versionPete Zaitcev2018-05-041-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The valid set of values for auth_version does not include values starting with the 'v'. In this particular function, the auth_version variable is only used for comparisons with v3. So, the code worked correctly. However, let's clean this up in order to reduce review confusion and defuse possible future landmine in case of code changes. Change-Id: I671016d7992a1922b786b4eb8876b3fbb2532e15
* | | Make swiftclient respect region_name when using sessionsErik Olof Gunnar Andersson2018-05-161-1/+3
|/ / | | | | | | Change-Id: I94aca6f1120c34616562be7345f0e5aa51a69499
* | Add force auth retry mode in swiftclientKota Tsuyuzaki2018-03-131-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch attemps to add an option to force get_auth call while retrying an operation even if it gets errors other than 401 Unauthorized. Why we need this: The main reason why we need this is current python-swiftclient requests could never get succeeded under certion situation using third party proxies/load balancers between the client and swift-proxy server. I think, it would be general situation of the use case. Specifically describing nginx case, the nginx can close the socket from the client when the response code from swift is not 2xx series. In default, nginx can wait the buffers from the client for a while (default 30s)[1] but after the time past, nginx will close the socket immediately. Unfortunately, if python-swiftclient has still been sending the data into the socket, python-swiftclient will get socket error (EPIPE, BrokenPipe). From the swiftclient perspective, this is absolutely not an auth error, so current python-swiftclient will continue to retry without re-auth. However, if the root cause is sort of 401 (i.e. nginx got 401 unauthorized from the swift-proxy because of token expiration), swiftclient will loop 401 -> EPIPE -> 401... until it consume the max retry times. In particlar, less time to live of the token and multipart object upload with large segments could not get succeeded as below: Connection Model: python-swiftclient -> nginx -> swift-proxy -> swift-backend Case: Try to create slo with large segments and the auth token expired with 1 hour 1. client create a connection to nginx with successful response from swift-proxy and its auth 2. client continue to put large segment objects (e.g. 1~5GB for each and the total would 20~30GB, i.e. 20~30 segments) 3. after some of segments uploaded, 1 hour past but client is still trying to send remaining segment objects. 4. nginx got 401 from swift-proxy for a request and wait that the connection is closed from the client but timeout past because the python-swiftclient is still sending much data into the socket before reading the 401 response. 5. client got socket error because nginx closed the connection during sending the buffer. 6. client retries a new connection to nginx without re-auth... <loop 4-6> 7. finally python-swiftclient failed with socket error (Broken Pipe) In operational perspective, setting longer timeout for lingering close would be an option but it's not complete solution because any other proxy/LB may not support the options. If we actually do THE RIGHT THING in python-swiftclient, we should send expects: 100-continue header and handle the first response to re-auth correctly. HOWEVER, the current python's httplib and requests module used by python-swiftclient doesn't support expects: 100-continue header [2] and the thread proposed a fix [3] is not super active. And we know the reason we depends on the library is to fix a security issue that existed in older python-swiftclient [4] so that we should touch around it super carefully. In the reality, as the hot fix, this patch try to mitigate the unfortunate situation described above WITHOUT 100-continue fix, just users can force to re-auth when any errors occurred during the retries that can be accepted in the upstream. 1: http://nginx.org/en/docs/http/ngx_http_core_module.html#lingering_close 2: https://github.com/requests/requests/issues/713 3: https://bugs.python.org/issue1346874 4: https://review.openstack.org/#/c/69187/ Change-Id: I3470b56e3f9cf9cdb8c2fc2a94b2c551927a3440
* | Add a query_string option to head_object().Timur Alperovich2018-03-051-3/+6
| | | | | | | | | | | | | | | | Submitting a path parameter with a HEAD request on an object can be useful if one is trying to find out information about an SLO/DLO without retrieving the manifest. Change-Id: I39efd098e72bd31de271ac51d4d75381929c9638
* | Merge "Add support for versionless endpoints"Jenkins2017-08-291-2/+15
|\ \ | |/ |/|
| * Add support for versionless endpointsChristian Schwede2017-06-131-2/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Newer deployments are using versionless Keystone endpoints, and most OpenStack clients already support this. This patch enables this for Swift: if an auth_url without any path component is found, it assumes a versionless endpoint will be used. In this case the v3 suffix will be appended to the path if none auth_version is set, and v2.0 is appended if auth_version requires v2. Closes-Bug: 1554885 Related-Bug: 1691106 Change-Id: If7ecb67776cb77828f93ad8278cc5040015216b7
* | Do not set Content-Type to '' with new requests.Timur Alperovich2017-06-131-2/+4
|/ | | | | | | | | | | | | | | | | | Previously, python-swiftclient worked around a requests issue where Content-Type could be set to application/x-www-form-urlencoded when using python3. This issue has been resolved and a fix released in requests 2.4 (fixed in subsequent releases as well). The patch makes the workaround conditional on the requests version, so that with sufficiently new requests libraries, the Content-Type is not set. For reference, requests 2.4 was released August 29th, 2014. The specific issue filed in the requests tracker is: https://github.com/requests/requests/issues/2071. Related-Change: I035f8b4b9c9ccdc79820b907770a48f86d0343b4 Closes-Bug: #1433767 Change-Id: Ieb2243d2ff5326920a27ce8c3c6f0f5c396701ed
* Fix logging of the gzipped bodyVitaly Gridnev2017-03-081-2/+5
| | | | | Change-Id: I6d7ccbf4ef9b46e890ecec58842c5cdd2804c7a9 Closes-bug: 1670620
* Merge "Add additional headers for HEAD/GET/DELETE requests."Jenkins2016-11-081-23/+42
|\
| * Add additional headers for HEAD/GET/DELETE requests.Charles Hsu2016-11-071-23/+42
| | | | | | | | | | Change-Id: I69276ba711057c122f97deac412e492e313c34dd Closes-Bug: 1615830
* | Low-level API: Don't log just before raising an exceptionTim Burke2016-10-281-7/+2
| | | | | | | | | | | | | | | | | | | | | | | | The only logging we should do is when we've encountered a problem *and we've dealt with it ourselves*. When we're raising an exception, it should be up to the caller to decide whether to log anything about it. Anything else is just rude. Change-Id: I1c96b76d90a78b7a10ffe63e4a7440c8f579147c Closes-Bug: 1213179 Related-Bug: 1202229
* | Merge "Adding keystoneauth sessions support"Jenkins2016-10-261-10/+25
|\ \
| * | Adding keystoneauth sessions supportPaulo Ewerton2016-05-191-10/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch allows authentication in swiftclient with a keystonauth session. Co-Authored-By: Tim Burke <tim@swiftstack.com> Change-Id: Ia3fd947ff619c11ff0ce474897533dcf7b49d9b3 Closes-Bug: 1518938
* | | Remove redundant space in docstringAnh Tran2016-09-281-4/+4
| | | | | | | | | | | | | | | | | | TrivialFix Change-Id: I7012fb7bbbedfba30b8e0450a449787e9076f4ef
* | | TrivialFix: Removed redundant 'the'Anh Tran2016-09-271-2/+2
| |/ |/| | | | | Change-Id: I3b3c0e7e4d5c8a1934378e2083a64bb686754f18
* | Merge "Accept gzip-encoded API responses"Jenkins2016-09-011-3/+5
|\ \
| * | Accept gzip-encoded API responsesTim Burke2016-08-301-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, we would accept gzip-encoded responses, but only because we were letting requests decode *all* responses (even object data). This restores the previous capability, but with tighter controls about which requests will accept gzipped responses and where the decoding happens. Change-Id: I4fd8b97207b9ab01b1bcf825cc16efd8ad46344a Related-Bug: 1282861 Related-Bug: 1338464
* | | Merge "Convert numeric and boolean header values to strings"Jenkins2016-08-311-0/+7
|\ \ \ | |/ / |/| |
| * | Convert numeric and boolean header values to stringsTim Burke2016-08-251-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Recently, requests got a bit more picky about what types of data it will accept as header values [1]. The reasons for this are generally sound; str()ing arbitrary objects just before pushing them out a socket may not do what the developer wanted/expected. However, there are a few standard types that developers may be sending that we should convert for them as a convenience. Now, we'll convert all int, float, and bool values to strings before sending them on to requests. Change-Id: I6c2f451009cb03cb78812f54e4ed8566076de821 Closes-Bug: 1614932
* | | Merge "client: renew token on 401 even if retries is 0"Jenkins2016-08-261-4/+4
|\ \ \ | |/ / |/| |
| * | client: renew token on 401 even if retries is 0Julien Danjou2016-06-071-4/+4
| |/ | | | | | | | | | | | | | | | | | | | | | | Gnocchi uses a client with retries=0 to maximize throughtput and not retry N times on e.g. 404 when checking existence of an object. However, this as the side effect of never renewing the token since there' no retry on 401 either. This patches change the behavior so that 401 errors are always retried, whatever the retries value is. Closes-Bug: #1589926 Change-Id: Ie06adf4cf17ea4592b5bbd7bbde9828e5e134e3e
* | Merge "Add copy object method"Jenkins2016-08-241-0/+71
|\ \
| * | Add copy object methodMarek Kaleta2016-08-231-0/+71
| |/ | | | | | | | | | | | | | | | | | | | | Implement copy object method in swiftclient Connection, Service and CLI. Although COPY functionality can be accomplished with 'X-Copy-From' header in PUT request, using copy is more convenient especially when using copy for updating object metadata non-destructively. Closes-Bug: 1474939 Change-Id: I1338ac411f418f4adb3d06753d044a484a7f32a4
* | Merge "Query string functionality for containers"Jenkins2016-06-141-9/+25
|\ \ | |/ |/|
| * Query string functionality for containersAndrew Welleck2016-06-091-9/+25
| | | | | | | | | | | | | | | | | | Added functionality for arbitrary query strings to be passed into container functions. Additionally a minor typo correction in the README. Added unit tests for query string functionality. Closes-Bug: #1542459 Change-Id: Ica2cb3ea439632588388e748d8d2e944e9ed4fa4
* | Merge "Support client certificate/key"Jenkins2016-05-191-3/+36
|\ \
| * | Support client certificate/keyCedric Brandily2016-04-101-3/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | This change enables to specify a client certificate/key with: * usual CLI options (--os-cert/--os-key) * usual environment variables ($OS_CERT/$OS_KEY) Closes-Bug: #1565112 Change-Id: I12e151adcb6084d801c6dfed21d82232a3259aea
* | | Correct doc string for http_conn argumentKota Tsuyuzaki2016-05-121-27/+27
| |/ |/| | | | | | | | | | | | | Previous doc string describes "http_conn" for a bunch of functions as "HTTPConnection object" but it is actually "a tuple of (parsed_url, HTTPConnection object)" so now this patch correct the mis-consistency. Change-Id: I6ff90a941e6df6a1b3b5dc74fd5f4262654c9f0a
* | Merge "Check responses when retrying bodies"Jenkins2016-05-111-0/+17
|\ \ | |/ |/|
| * Check responses when retrying bodiesTim Burke2016-05-041-0/+17
| | | | | | | | | | | | | | | | | | | | | | | | Previously, if a Range request came back 200 OK (rather than 206 Partial Content), we would mangle the response body. This could happen if there was a middleware that would silently drop Range headers, for example. Now, if the response does not include a Content-Range header, we will log a warning and seek to our previous position in the stream. If the Content-Range header has an unexpected value, we will raise an exception. Change-Id: I94d4536cc1489968d45a2b6ba7edd70c85800275
* | Include response headers in ClientExceptionsTim Burke2016-03-031-71/+19
| | | | | | | | | | | | | | | | | | | | | | Now, client applications can get to things like transaction IDs for failures without needing to turn on all of logging. While we're at it, add a from_response factory method for ClientException. Co-Authored-By: Alexander Corwin <ancorwin@gmail.com> Change-Id: Ib46d5f8fc7f36f651f5908bb9d900316fdaebce3
* | Merge "Fix wrong args for get_container with full listing"Jenkins2016-02-291-2/+2
|\ \
| * | Fix wrong args for get_container with full listingAlistair Coles2016-02-221-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In client get_container(), when full_listing is true, the calls back to get_container() pass service_token as a positional arg which maps its value to the full_listing arg. It should use a keyword. Change-Id: Iac2af45df124ff33fcb7fbaf1ba959ef06c96378 Closes-Bug: #1496093
* | | Merge "Force header keys/values to bytes/unicode before coercing to unicode"Jenkins2016-02-271-0/+2
|\ \ \
| * | | Force header keys/values to bytes/unicode before coercing to unicodeTim Burke2016-02-261-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, parse_header_string was only called with data coming out of requests, which would be either bytes or unicode. Now that we're sending it request headers as well (see related change), we need to be more defensive. If the value given is neither bytes nor unicode, convert it to a native string. This will allow developers using the client API to continue sending header dicts like {'X-Delete-After': 2} ...as in Swift's test/probe/test_object_expirer.py Change-Id: Ie57a93274507b184af5cad4260f244359a585f09 Related-Change: I43dd7254f7281d4db59b286aa2145643c64e1705
* | | | Fix test for redacting sensitive data in client.http_log()Joel Wright2016-02-261-1/+1
|/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | The test should have included utf8 encoded unicode data to test that encoded unicode data stored in headers was parsed correctly. Also fixes the docstring for swiftclient.safe_value() Change-Id: Id0def0b3af7a364f1257cc22f67b71c0cc5d8479
View Lorry Controller Status