summaryrefslogtreecommitdiff
path: root/tests/unit/test_swiftclient.py
Commit message (Collapse)AuthorAgeFilesLines
...
* | | Merge "Retry file uploads via SwiftService"Jenkins2016-01-121-14/+48
|\ \ \ | |_|/ |/| |
| * | Retry file uploads via SwiftServiceTim Burke2016-01-111-14/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When we introduced LengthWrapper, we neglected to make it resettable. As a result, upload failures result in errors like: put_object(...) failure and no ability to reset contents for reupload. Now, LengthWrappers will be resettable if their _readable has seek/tell. Related-Change: I6c8bc1366dfb591a26d934a30cd21c9e6b9a04ce Change-Id: I21f43f06e8c78b24d1fc081efedf2687942e042f
* | | Retry download of object bodyStuart McLaren2016-01-081-0/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently the swift client retries establishing a connection to the server (by default up to 5 times). However, when downloading an object, once the connection has been established and the inital headers have been returned, no attempt is made to retry. So, for example, if 99MB of a 100MB object have been downloaded and the connection is then lost, the download will fail. This patch changes the behaviour to re-establish the connection and fetch the remaining bytes using the 'Range' header to offset. Data retry is not yet supported if the original request is for a subset of the object data (ie uses the 'Range' header), or if resp_chunk_size has not been set. The object's etag is checked using If-Match to make sure the object data hasn't changed since the start of the download. Change-Id: Iab47f10081ff39f6d344dbc2479cbc3bfd1c5b29
* | | Merge "Replace assertEqual(arg, None) with assertIsNone(arg)"Jenkins2016-01-061-8/+8
|\ \ \ | |/ / |/| |
| * | Replace assertEqual(arg, None) with assertIsNone(arg)SaiKiran2015-12-301-8/+8
| | | | | | | | | | | | | | | | | | | | | | | | In python-swiftclient some test cases using asserEqual(arg, None) instead of assertIsNone(arg).assertIsNone method provides clear error message. Change-Id: I4d673ede0965408344325c9c234c5c4b1ae4146a Closes-Bug: #1527556
* | | Merge "Wrap raw iterators to ensure we send entire contents to server"Jenkins2016-01-061-0/+20
|\ \ \
| * | | Wrap raw iterators to ensure we send entire contents to serverTim Burke2015-12-301-0/+20
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, if you attempt to stream an upload from an iterator, as in def data(): yield 'foo' yield '' yield 'bar' conn.put_object('c', 'o', data()) ... requests will faithfully emit a zero-length chunk, ending the transfer. Swift will then close the connection, possibly (if Connection: keep-alive was set) after attempting to parse the next chunk as a new request. Now, Swift will receive all of the bytes from the iterable, and any zero-byte chunks will be ignored. This will be fixed in requests [1], but not until an eventual 3.0.0 release. [1] https://github.com/kennethreitz/requests/pull/2631 Change-Id: I19579ed7a0181ac3f488433e7c1839f7f7a040b8
* | | Fix some typosTim Burke2015-12-301-2/+2
|/ / | | | | | | Change-Id: Iaf7f30a7ae0c2ac76fc5cdcee31ea74c08ce601e
* | Test 'string' behaviour of get_objectStuart McLaren2015-12-151-0/+6
| | | | | | | | | | | | | | | | Add a unit test to test the 'string' like behaviour of get_object when it is called without resp_chunk_size set. Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Change-Id: I496032a76036141d027c30b076c810b34bc6bef0
* | Merge "Centralize header parsing"Jenkins2015-11-121-0/+37
|\ \
| * | Centralize header parsingTim Burke2015-09-031-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | All response headers are now exposed as unicode objects. Any url-encoding is interpretted as UTF-8; if that causes decoding to fail, the url-encoded form is returned. As a result, deleting DLOs with unicode characters will no longer raise UnicodeEncodeErrors under Python 2. Related-Bug: #1431866 Change-Id: Idb111c5bf3ac1f5ccfa724b3f4ede8f37d5bfac4
* | | Merge "Add tests and param definitions for headers parameter"Jenkins2015-11-041-0/+53
|\ \ \
| * | | Add tests and param definitions for headers parameterAlistair Coles2015-09-231-0/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Cleanups for change I35c3b266b3c733f6b1629de4c683ea7d40128032 Add missing param definitions to client get_container and head_object docstrings. For consistency, add headers parameter to the Connection class head_object and head_container wrapper methods. Add tests to verify that the headers parameter of Connection get_container, head_container and head_object methods is passed to the module functions. Change-Id: Ib40d5b626b2793840727c58cffbf725bea55651f
* | | | swiftclient content-type headerLisak, Peter2015-10-221-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | According to help `swift upload -h` you can add a customized request header 'Content-Type'. But actually it is ignored (cleared and default is used) if subcommand is upload. Subcommand post works as expected in help. Bug fix: Use 'Content-Type' from the customized request headers also if uploading. Change-Id: If0d1354b6214b909527341078fe1769aa6587457
* | | | Miscellaneous (mostly test) cleanupTim Burke2015-10-071-76/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Always use testtools.TestCase, since we're relying on testtools * Always use mock (as opposed to unittest.mock) since we're relying on mock * Add note about when a missing logging handler was added * Stop %-formatting the giant usage string that doesn't actually need any formatting * Prefer assertIs, assertIn, assertIsInstance over assertTrue * Use else-self.fail instead of sentinel values * Check resp.get('error') is None before checking resp['success'] is True, so test failures actually tell you something useful * Tighten some isinstance assertions * Import MockHttpTest from correct location * Only populate clean_os_environ once * Use setUp for setup, not __init__ * Replace assertIn(key, dict) and assertEqual(foo, dict[key]) with assertEqual(foo, dict.get(key)) when key is a literal and foo is not None * Use mock.patch.object instead of manually patching for tests * Use six.binary_type instead of type(''.encode('utf-8')) * Stop shadowing builtin bytes * Reclaim some margin * Stop checking the return-type of encode_utf8; we already know it's bytes Change-Id: I2138ea553378ce88810b7353147c8645a8f8c90e
* | | | Make more assertions in client unit testsAlistair Coles2015-09-231-6/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some tests rely on the fake connection checking expected request parameters, but that assumes that the fake ocnnection is even called, which is not being checked. Add more explicit assertions that requests are in fact made. Change-Id: Id1c48235d7d97fd1b0feec6c19ed59a87bebdf89
* | | | Actually make assertions when testing get_account and get_containerTim Burke2015-09-221-23/+80
|/ / / | | | | | | | | | Change-Id: Ibb1301b00d1bc99ec089ead02f944aa94972120a
* | | Add headers parameterMahati2015-09-161-0/+27
|/ / | | | | | | | | | | | | Headers parameter is required when passing client key for encryption. It is missing for get_container and head_object. Change-Id: I35c3b266b3c733f6b1629de4c683ea7d40128032
* | Stop Connection class modifying os_options parameterAlistair Coles2015-08-251-0/+27
|/ | | | | | | | | | | | | | | | | When a caller passes an os_options dict to the Connection class constructor, the constructor may modify the os_options dict, which can surprise the caller if they re-use the os_options dict. Specifically the os_options tenant_name and object_storage_url may be modified, and the changed values would then leak through to a subsequent Connection constructed using the same os_options dict. This fix simply constructs a new dict from the supplied os_options. The patch also adds a test that covers this and also verifies that a preauth_url passed as a keyword arg to Connection() will take precedence over any object_storage_url in an os_options parameter. Closes-Bug: 1488070 Change-Id: Ic6b5cf3ac68c505de155619f2610be9529e15432
* Merge "Add minimal working service token support."Jenkins2015-08-251-1/+306
|\
| * Add minimal working service token support.Stuart McLaren2015-08-241-1/+306
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add client changes to allow accessing alternative reseller_prefixes via a service token. ie client changes for this server side spec: https://review.openstack.org/#/c/105228 We assume that the service storage url has been passed in as a preauthurl. We rely on get_auth preserving this url. Change-Id: I1cfda178f0b6c8add46cfebd6bf38440caae2036
* | flake8 ignores same hacks as swiftHiroshi Miura2015-08-181-2/+2
|/ | | | | | | | | | | - blacklisted flake8 hacking - fix against E122 continuation line missing indentation or outdented Closes-bug: #1475516 Change-Id: I708d0a3466a1f85c84e478873e142821ce0774cb Signed-off-by: Hiroshi Miura <miurahr@nttdata.co.jp>
* Merge "change deprecated assertEquals to assertEqual"Jenkins2015-08-111-14/+14
|\
| * change deprecated assertEquals to assertEqualHiroshi Miura2015-08-061-14/+14
| | | | | | | | | | | | | | | | | | | | fix against H234: assertEquals() logs a DeprecationWarning in Python3.x. use assertEqual() instead. Closes-bug: #1480776 Change-Id: Iffda6bb5f2616d4af4567eeea37bb26531e34371 Signed-off-by: Hiroshi Miura <miurahr@nttdata.co.jp>
* | make Connection.get_auth set url and token attributes on selfZack M. Davis2015-07-241-0/+11
|/ | | | | | | | | | | | | | | | When a Connection is first __init__ialized (without providing a preauthurl or preauthtoken), the url and token attributes are None; they get set (to be reused on future requests) after one of the wrapper methods internally using _retry (head_account, get_container, put_object, and similar friends) is called. However, this had not been the case for get_auth, much to the momentary confusion and disappointment of programmers using swiftclient who expected to be able to get the token or storage URL off the Connection object after calling get_auth (perhaps in order to make an unusual kind of request that swiftclient doesn't already have a function for). This commit makes get_auth set the url and token attributes as one might expect. Change-Id: I0d9593eb5b072c8e3fa84f7d5a4c948c0bc6037a
* Add connection release testChristian Schwede2015-06-051-1/+13
| | | | | | | | | | | This patch adds a small test to ensure a connection is released after all chunks have been consumed. It's a follow up to commit 8756591b and added to ensure there will be no regression in the future (this test fails also with that patch not applied). Change-Id: I6a6fcd26879eb2070f418c8770a395ff6c30aa51
* Merge "Allow reading from object body on download"Jenkins2015-06-041-0/+35
|\
| * Allow reading from object body on downloadStuart McLaren2015-03-241-0/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, get_object returns a generator. This allows access to the object's data in multiples of 'resp_chunk_size'. This patch adds a read function to also allow accessing the object data using read(size). This allows, for example, the consumer of an object (where no byte range has been specified) to read up to certain boundaries while streaming to a new Large Object with segments of a specified size. Reading and chunking can be safely mixed. Related-Bug: 1367925 Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Change-Id: I1cfb67f45afc7015fd896f1a89bebae048871769
* | Make default get_auth timeout be NoneAlistair Coles2015-06-031-2/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Setting timeout to a default of False in get_auth() results in a requests timeout of 0.0 in keystoneclient, which causes a connection failure. This bug will cause func tests to not run when using keystoneauth. Added unit tests to verify correct default timeout is set in get_auth(). Drive-by: remove what seems like a stale TODO comment Change-Id: I17c781ce160a682b1768d315422ade0cdd2df198
* | Remove simplejson dependencyTim Burke2015-05-201-39/+7
| | | | | | | | | | | | | | In addition to removing an unnecessary dependency, this closes another hole that was allowing raw bytes to appear in user-facing messages. Change-Id: Ia0b76426a38e5a5c368c4c7e7ba2aef286758aca
* | Add test for timeout being passed to keystone clientAlistair Coles2015-04-281-1/+23
| | | | | | | | | | | | | | | | | | Extends existing unit test for timeout being passed to get_auth to cover v2.0 auth when keystone client should get the timeout kwarg. Related-Bug: 1447847 Change-Id: Ie9cfc86fa2156b94b45d290ac12e3f71b20d6c4f
* | Use a socket timeout in get_authPete Zaitcev2015-04-261-4/+53
| | | | | | | | | | | | | | | | | | | | A previous change added socket timeouts, but they weren't plumbed through to the get_auth code path. Make sure we're passing them along everywhere. Original-Author: Monty Taylor <mordred@inaugust.com> Change-Id: I398241898248e66d1f8c436c8ed2ec7a0e9387f6 Closes-bug: 1447847
* | Merge "Add socket-level read timeout parameter"Jenkins2015-04-241-1/+2
|\ \
| * | Add socket-level read timeout parameterMonty Taylor2015-03-301-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The underlying requests library supports a timeout input parameter. The other python-*client libraries support passing it in, so while working on shade, it seemed like a great idea to be able to pass in timeout to swiftclient too. For reference, there's a change in shade: I095c1240693abf024bda2315dd77f4400b24a45b that shows interaction with the other libs, and a tentative patch that would consume this one. Change-Id: I699ebb1e092aa010af678de7ba15712da6ed5315
* | | Merge "Include unsupported url scheme with ClientException"Jenkins2015-03-311-0/+23
|\ \ \ | |/ / |/| |
| * | Include unsupported url scheme with ClientExceptionAlistair Coles2015-02-231-0/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This was an attempt to reveal some more info about failing jenkins jobs such as here [1] where glance_store is receiving a swiftclient exception because of 'unsupported scheme in url'. It seems worth merging the more helpful exception message. [1] http://logs.openstack.org/91/155291/7/check/gate-tempest-dsvm-neutron-src-python-swiftclient/4adac1e/ Change-Id: I0f3411efd42d045b07d6d1000f59a06865d8034b
* | | Add improvements to MD5 validation.Daniel Wakefield2015-03-041-2/+3
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | With MD5Sum checking being added, a concern was brought up that It was a change with no possibility of reverting to the old behaviour. This change adds the flag '--ignore-checksum' to the upload subcommand allowing the checks to be turned off. Changed occurrences of the magic string for a null md5 to use a descriptive constant instead. Updated Error messages generated when validation fails. They should now be more descriptive and not output a literal newline sequence. Change-Id: Id1756cbb6700bb7e38f0ee0e75bc535e37f777ed
* | Verify MD5 of uploaded objects.Daniel Wakefield2015-02-191-37/+97
|/ | | | | | | | | | | Changed existing code to calculate the MD5 of the object during the upload stream. Checks this MD5 against the etag returned in the response. An exception is raised if they do not match. Closes-Bug: 1379263 Change-Id: I6c8bc1366dfb591a26d934a30cd21c9e6b9a04ce
* Make preauth params workClay Gerrard2014-12-171-123/+390
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If you specify a token and storage url when creating a Connection, regardless of the auth api version the first request will be made directly to swift. You can either provide a preauthurl and preauthtoken or fall back to os_options' object_storage_url and auth_token keys (exposed as --os-storage-url and --os-auth-token on the command line or OS_STORAGE_URL and OS_AUTH_TOKEN in the environment). If a _retry wrapped request on a Connection fails because of invalid authentication (401) the Connection's cached token and url will be invalidated. If the Connection's retries attribute is > 0 the subsequent attempt will call get_auth to refresh the token, but the pre-configured storage_url will always be re-used. This is consistent with current auth v2 behavior and less surprising for auth v1. The pre-existing, but previously undocumented behavior/interface of get_auth would override the storage_url returned by the auth service if the 'os_storage_url' option was provided in the os_options dict. To ensure that this behavior is consistent across auth v1 and v2 from the command line and when using the Connection class as a library - the preauthurl is stashed in the os_options dict when provided. Improved Connection.get_capabilities storage_url handling to better support the consistent behavior of a preauthurl/object_storage_url on the connection regardless of auth version. Fixed up some test infrastructure to enable setting up and testing multiple requests/responses. Change-Id: I6950fb73f3e28fdddb62760cae9320e2f4336776
* Add unit tests for _encode_meta_headersDaniel Wakefield2014-11-131-0/+20
| | | | | Also move it to a module level function. Change-Id: I2da64876815619a6db7ff208061df2191767806c
* Fix KeyError raised from client ConnectionAlistair Coles2014-10-201-0/+98
| | | | | | | | | | | | | | | | | Some client.Connection methods will raise a KeyError if a response_dict argument is passed and an error occurs during authentication or making the request. The fix is straightforward: add a test for existence of a response_dict before attempting to get it from kwargs. The bulk of this patch is adding unit tests for the response_dict feature. Closes-Bug: 1381304 Change-Id: Ic7e1b3dfae33909533931c52ac97355867a08a07
* Fix bug with some OS options not being passed to clientAlistair Coles2014-09-241-46/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes a regression that is preventing swiftclient from authenticating using keystone v3 options. swiftclient/shell.py processes the dict of command line options to set up a child dict ('os_options') of keystone auth specific options (line 960). The processing includes stripping --os- prefixes from the command line options before adding keys to os_options. A recent patch https://review.openstack.org/#/c/85453/ introduced a duplication of this option processing in service.py (line 79) which replaces the os_options created in shell.py, but omits keystone v3 related options. Consequently the keystone v3 options are not being passed to the client get_auth() method. This patch adds the keystone v3 related options to the option processing in service.py. For pragmatic reasons (i.e. fixing the bug quickly) the option processing code has not been removed from parse_args in shell.py. It is likely that the code in parse_args is now redundant, but all code paths between parse_args and process_options should be inspected and test coverage added before removing that code. Unit tests have been added in test_shell.py to verify that command line options are correctly passed to the client get_auth method. The MockHttpTest class is re-used in test_shell.py, so it is moved from test_swiftclient.py to tests/unit/utils.py Closes-bug: #1372465 Change-Id: I4fed013cdb8936509609d06093337cc147ade0d6
* Add keystone v3 auth supportanc2014-07-231-14/+63
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Enables swiftclient to authenticate using the keystone v3 API, allowing user id's, user domains and tenant/project domains to be specified. Since swiftclient imports keystoneclient, the main changes in swiftclient/client.py are to selectively import the correct keystoneclient library version and pass a number of new options to it via the get_auth() function. In addition the get_keystoneclient_2_0 method has been renamed get_auth_keystone to better reflect its purpose since it now deals with both v2 and v3 use cases. In swiftclient/shell.py the new options are added to the parser. To make the default help message shorter, help for all the --os-* options (including the existing v2 options) is only displayed when explicitly requested usng a new --os-help option. A new set of unit tests is added to test_shell.py to verify the parser. A comment in tests/sample.conf explains how to configure the existing functional tests to run using keystone v3 API. Note that to use keystone v3 with swift you will need to set auth_version = v3.0 in the auth_token middleware config section of proxy-server.conf. Change-Id: Ifda0b3263eb919a8c6a1b204ba0a1215ed6f642f
* Relax requirement for tenant_name in get_auth()anc2014-06-051-2/+20
| | | | | | | | | | | get_auth() in client.py raises an exception if tenant_name is not included in the os_options dict. This is overly constrained since tenant_id is equally sufficient. This patch modifies get_auth to require either tenant_name or tenant_id. Change-Id: Ibbcda1704637eb887efa5895579d260a1e072327
* Merge "change assertEquals to assertEqual"Jenkins2014-05-281-4/+4
|\
| * change assertEquals to assertEqualChristian Berendt2014-05-141-4/+4
| | | | | | | | | | | | | | According to http://docs.python.org/2/library/unittest.html assertEquals is a deprecated alias of assertEqual. Change-Id: Ibf4d548b86c53f30830b7e34b019fa8f67997cd8
* | Merge "Remove testtools.main() call from tests"Jenkins2014-05-241-4/+0
|\ \
| * | Remove testtools.main() call from testsChristian Schwede2014-05-201-4/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is no testtools.main() at all (this was a unittest.main() before). Let's remove this unused and non-working code. The following code can be used if someone needs to start tests manually: python -m unittest <test_file.py> Change-Id: Id5162ac73825584df6c23dbe68786ff3355fc6ae
* | | Fix wrong assertions in unit testsanc2014-05-221-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | A couple of assertTrue in the test_swiftclient.py unit tests should be assertEqual. Also, the expected values now need to be bytes literals. Change-Id: I7cc1bd60d9ba82d1a28fbae2e1243d3c799451bd
* | | Fix Python3 bugs2.1.0Christian Schwede2014-05-211-1/+19
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch fixes three issues that were found running functional tests with Python 3: 1. python-requests sets 'application/x-www-form-urlencoded' as content-type if the input is not a string object and no content-type is given. 2. Encoding of the headers is now only done if required. This prevents comparisons between unencoded headers and encoded prefixes and avoids unnecessary forloop-iterations. One unittest was extended to ensure it works for unencoded and encoded headers with or without the prefix. 3. Functional tests recently switched to using byte data for testing, thus the comparison needs to be a byte object as well. Change-Id: I035f8b4b9c9ccdc79820b907770a48f86d0343b4
View Lorry Controller Status