summaryrefslogtreecommitdiff
path: root/swiftclient
Commit message (Collapse)AuthorAgeFilesLines
* Merge "Always decode command-line arguments as UTF-8"2.5.0Jenkins2015-07-161-6/+9
|\
| * Always decode command-line arguments as UTF-8Tim Burke2015-07-071-6/+9
| | | | | | | | | | | | | | | | | | There was always an implicit assumption that they were UTF-8 before, and by converting them to unicode we close another hole allowing raw bytes to appear in user-facing messages. Closes-Bug: #1431866 Change-Id: If2e41d9a592c3ad02818e9c6f0959fd4b73cd0e0
* | Merge "Correct the help message of swift tempurl"Jenkins2015-07-111-4/+4
|\ \
| * | Correct the help message of swift tempurlYangLei2015-06-301-4/+4
| | | | | | | | | | | | | | | | | | | | | correct the help message of swift tempurl use <> instead of [] in Positional arguments. Change-Id: Ib60ce97cef03e0423082c497604525eba2300fa9
* | | Merge "Add some bash helpers for auth stuff"Jenkins2015-07-111-4/+50
|\ \ \
| * | | Add some bash helpers for auth stuffClay Gerrard2015-06-151-4/+50
| | | | | | | | | | | | | | | | Change-Id: If61ac9a050e7a115f37dbf4e74b904ac5dfd2052
* | | | Merge "Add ability to download objects to particular folder."Jenkins2015-07-112-3/+36
|\ \ \ \ | |_|_|/ |/| | |
| * | | Add ability to download objects to particular folder.Charles Hsu2015-07-082-3/+36
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds "--output-dir" and "--remove-prefix" options to the "download" command and unit tests for it. Example: $ swift list example --prefix swift2.2 swift2.2/bin/swift-object-auditor swift2.2/bin/swift-object-expirer swift2.2/bin/swift-object-info swift2.2/bin/swift-object-replicator swift2.2/bin/swift-object-server swift2.2/bin/swift-object-updater When given "--output-dir <directory>", client downloads objects to <directory>. $ swift download example --prefix swift2.2 \ --output-dir new/swift/dir The folder structure: . └── new └── swift └── dir └── swift2.2 └── bin ├── swift-object-auditor ├── swift-object-expirer ├── swift-object-info ├── swift-object-replicator ├── swift-object-server └── swift-object-updater When given "--remove-prefix", client downloads objects without <prefix>. $ swift download example --prefix swift2.2 \ --remove-prefix \ --output-dir swift The folder structure: . └── swift └── bin ├── swift-object-auditor ├── swift-object-expirer ├── swift-object-info ├── swift-object-replicator ├── swift-object-server └── swift-object-updater Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Change-Id: I7463fe2941cc94f9a50a4756a97c2ccdf946294d Implements: blueprint swiftclient-download-pseudo-folder-to-specific-target
* | | Merge "SwiftClient object upload beginning with / or "./""Jenkins2015-07-071-4/+4
|\ \ \
| * | | SwiftClient object upload beginning with / or "./"Pradeep Kumar Singh2015-06-171-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently SwiftClient populate response dictionary before removing "./" or "/" at begining of object name. This patch fixes that by changing that order. Closes-bug: #1412425 Change-Id: I80222754caba5d42a468f4677ac539e46682dd31
* | | | Merge "Added check for negative segment-size"Jenkins2015-07-071-0/+3
|\ \ \ \
| * | | | Added check for negative segment-sizePradeep Kumar Singh2015-06-231-0/+3
| | |_|/ | |/| | | | | | | | | | | | | | | | | | Closes-Bug: #1453135 Change-Id: Ia9c2b27d998e6ac1889cc74c12e456c06ecd84d9
* | | | Merge "Remove all DLO segments on upload of replacement"Jenkins2015-07-021-12/+20
|\ \ \ \ | |/ / / |/| | |
| * | | Remove all DLO segments on upload of replacementTim Burke2015-03-231-12/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, only the first container-listing's worth of segments was deleted, which would leave behind orphaned segments when the object was very large with small segments or the server's container_listing_limit was small. In addition, process DLO and SLO deletions on the segment thread pool, rather than the object thread pool. Change-Id: I1587375261a6237fa55a9cb96bda8dae918cc795 Related-Bug: #1418007
* | | | Fix docstring typo for SwiftService.uploadClint Byrum2015-06-181-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The option from argparse in shell is 'header' and that is what is used in the code. Change-Id: I5c889192ef7c46c299dc0ec7cbc7c4d027dae6d5
* | | | Fix inconsistent usage of "Positional argument"Andreas Jaeger2015-06-081-1/+1
| |_|/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | All help texts uses "Positional argument" with the exception of tempurl. Update tempurl to use this as well so that the formatting tools work fine and can show this nicely on http://docs.openstack.org/cli-reference/content/swiftclient_commands.html#swiftclient_subcommand_tempurl like it's done for other options. Change-Id: Ib5502c23b236986bea5a4d4a63a46fca411a8494 Closes-Bug: #1463081
* | | Merge "Allow reading from object body on download"Jenkins2015-06-041-7/+32
|\ \ \
| * | | Allow reading from object body on downloadStuart McLaren2015-03-241-7/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, get_object returns a generator. This allows access to the object's data in multiples of 'resp_chunk_size'. This patch adds a read function to also allow accessing the object data using read(size). This allows, for example, the consumer of an object (where no byte range has been specified) to read up to certain boundaries while streaming to a new Large Object with segments of a specified size. Reading and chunking can be safely mixed. Related-Bug: 1367925 Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Change-Id: I1cfb67f45afc7015fd896f1a89bebae048871769
* | | | Make default get_auth timeout be NoneAlistair Coles2015-06-031-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Setting timeout to a default of False in get_auth() results in a requests timeout of 0.0 in keystoneclient, which causes a connection failure. This bug will cause func tests to not run when using keystoneauth. Added unit tests to verify correct default timeout is set in get_auth(). Drive-by: remove what seems like a stale TODO comment Change-Id: I17c781ce160a682b1768d315422ade0cdd2df198
* | | | Merge "Compare each chunk of large objects when downloading"Jenkins2015-06-011-12/+60
|\ \ \ \
| * | | | Compare each chunk of large objects when downloadingTim Burke2015-05-271-12/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This allows the use of --skip-identical when downloading SLOs and DLOs, while still only requiring one request for normal objects. Change-Id: I237ec5a8cc4d1119b6d3f166db81a977a50a4e4e Related-Bug: #1379252
* | | | | Merge "Stop decoding object content"Jenkins2015-05-291-1/+2
|\ \ \ \ \ | |/ / / / |/| | | |
| * | | | Stop decoding object contentTim Burke2015-05-221-1/+2
| | |_|/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, we had urllib3 (via requests) automatically decode all responses with a Content-Encoding of deflate or gzip. This included object downloads, which would in turn cause etag or content-length mismatch errors. (See bug 1338464) This was apparently added in response to a third-party proxy sitting between the client and server which, having observed that the client would accept gzip-encoded content while the server sent an unencoded response, would perform the compression. (See bug 1282861) Now, we'll no longer let requests send any default headers, nor do any decoding. Change-Id: I6cc30a5c12e37de06d7322533a3c36ad15397cc8 Closes-Bug: 1338464 Related-Bug: 1282861
* | | | Merge "be more explicit in the --version CLI option"Jenkins2015-05-261-1/+1
|\ \ \ \
| * | | | be more explicit in the --version CLI optionJohn Dickinson2015-05-131-1/+1
| |/ / / | | | | | | | | | | | | Change-Id: I14f33c7ea90ab1fe58aa67f1ff70e527c88cd141
* | | | Remove simplejson dependencyTim Burke2015-05-203-16/+26
|/ / / | | | | | | | | | | | | | | | | | | In addition to removing an unnecessary dependency, this closes another hole that was allowing raw bytes to appear in user-facing messages. Change-Id: Ia0b76426a38e5a5c368c4c7e7ba2aef286758aca
* | | Merge "Use a socket timeout in get_auth"Jenkins2015-05-041-6/+13
|\ \ \
| * | | Use a socket timeout in get_authPete Zaitcev2015-04-261-6/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A previous change added socket timeouts, but they weren't plumbed through to the get_auth code path. Make sure we're passing them along everywhere. Original-Author: Monty Taylor <mordred@inaugust.com> Change-Id: I398241898248e66d1f8c436c8ed2ec7a0e9387f6 Closes-bug: 1447847
* | | | Merge "Compare each chunk of large objects when uploading"Jenkins2015-04-302-36/+71
|\ \ \ \ | |/ / / |/| | |
| * | | Compare each chunk of large objects when uploadingTim Burke2015-03-232-36/+71
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, we compared the ETag from Swift against the MD5 of the entire large object. However, the ETag for large objects is generally the MD5 of the concatenation of the ETags for each segment, unless the object is a DLO whose segments span more than one page of a container listing. Rather than worry about ETags, just compare each chunk of the segmented file. This allows the use of --skip-identical when uploading SLOs and DLOs. Additionally, there are several test-related improvements: * The default arguments for OutputManager are now evaluated on construction, rather than on definition, so that TestOutputManager.test_instantiation will succeed when using nosetest as a test runner. (See also: bug 1251507) * An account_username option is now available in the functional tests config file for auth systems that do not follow the account:username format. * CaptureOutput no longer writes to the captured stream, and MockHttpTest now captures output. These were polluting test output unnecessarily. (See also: bug 1201376) Change-Id: Ic484e9a0c186c9283c4012c6a2fa77b96b8edf8a Closes-Bug: #1201376 Closes-Bug: #1379252 Related-Bug: #1251507
* | | Merge "Add socket-level read timeout parameter"Jenkins2015-04-241-3/+10
|\ \ \
| * | | Add socket-level read timeout parameterMonty Taylor2015-03-301-3/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The underlying requests library supports a timeout input parameter. The other python-*client libraries support passing it in, so while working on shade, it seemed like a great idea to be able to pass in timeout to swiftclient too. For reference, there's a change in shade: I095c1240693abf024bda2315dd77f4400b24a45b that shows interaction with the other libs, and a tentative patch that would consume this one. Change-Id: I699ebb1e092aa010af678de7ba15712da6ed5315
* | | | Merge "Include unsupported url scheme with ClientException"Jenkins2015-03-311-1/+2
|\ \ \ \ | |/ / / |/| | |
| * | | Include unsupported url scheme with ClientExceptionAlistair Coles2015-02-231-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This was an attempt to reveal some more info about failing jenkins jobs such as here [1] where glance_store is receiving a swiftclient exception because of 'unsupported scheme in url'. It seems worth merging the more helpful exception message. [1] http://logs.openstack.org/91/155291/7/check/gate-tempest-dsvm-neutron-src-python-swiftclient/4adac1e/ Change-Id: I0f3411efd42d045b07d6d1000f59a06865d8034b
* | | | Mention --segment-size option after 413 response2.4.0Tim Burke2015-03-051-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, it would only output something like ClientException: Object PUT failed <url> 413 Request Entity Too Large Your request is too large. ... which is misleading as it implies that Swift simply can't handle large objects. Change-Id: Icec9e76b2671490c3a2249ec08992950107c6828
* | | | Merge "Unindent a chunk of st_list"Jenkins2015-03-051-57/+52
|\ \ \ \
| * | | | Unindent a chunk of st_listPete Zaitcev2015-02-261-57/+52
| | |/ / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | The chunk looks like nicely containable, but it pushes against the right edge of the screen annoyingly. This patch _only_ prettifies, without any observable changes. Change-Id: I96fcb5848e3e5e0e8a8ebbe398b1d4a0bea5fa24
* | | | Add improvements to MD5 validation.Daniel Wakefield2015-03-043-23/+34
|/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With MD5Sum checking being added, a concern was brought up that It was a change with no possibility of reverting to the old behaviour. This change adds the flag '--ignore-checksum' to the upload subcommand allowing the checks to be turned off. Changed occurrences of the magic string for a null md5 to use a descriptive constant instead. Updated Error messages generated when validation fails. They should now be more descriptive and not output a literal newline sequence. Change-Id: Id1756cbb6700bb7e38f0ee0e75bc535e37f777ed
* | | Merge "Release connection after consuming the content"Jenkins2015-02-251-3/+14
|\ \ \
| * | | Release connection after consuming the contentIan Cordasco2015-02-251-3/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When using stream=True, we need to make sure we release a connection back to the connection pool to avoid warnings clogging up the logs. We can do this by releasing the connection once we've read all the content from the response. In situations where stream=False, requests already does this for us. Related-bug: 1424732 Closes-bug: 1341777 Change-Id: Id1c72ad86135a49d4b985992d788736b65f7dbda
* | | | Merge "Fix crash when stat'ing objects with non-ascii names"Jenkins2015-02-251-3/+1
|\ \ \ \ | |/ / / |/| | |
| * | | Fix crash when stat'ing objects with non-ascii namesJoel Wright2015-02-171-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch fixes a crash and stack trace in stat when an object name contains non-ascii characters. Change-Id: Ib6dc686771e593243de56cafc100b17e51d9d9d5 Closes-Bug: 1411665
* | | | Merge "Fix crash with -l, -d /, and pseudo folders"Jenkins2015-02-251-3/+6
|\ \ \ \ | |_|_|/ |/| | |
| * | | Fix crash with -l, -d /, and pseudo foldersPete Zaitcev2015-02-181-3/+6
| | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Using -l, -d /, or a pseduo folder by itself works fine, but if you combine all of them, this happens: [zaitcev@guren s]$ swift list -d / testcont amazon_lili.jpg pseudodir/ rin_shibuya_pixiv.jpg swift-c134855-5a.diff [zaitcev@guren s]$ swift list -d / -l testcont 151954 2014-06-20 17:13:52 amazon_lili.jpg Traceback (most recent call last): File "/q/zaitcev/hail/python-swiftclient-tip/bin/swift", line 24, in <module> sys.exit(main()) File "/q/zaitcev/hail/python-swiftclient-tip/swiftclient/shell.py", line 1301, in main globals()['st_%s' % args[0]](parser, argv[1:], output) File "/q/zaitcev/hail/python-swiftclient-tip/swiftclient/shell.py", line 414, in st_list total_bytes += item_bytes TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType' The patch is a straightforward fix that assigns expected variables. Change-Id: If24dcc2491e8b8982455107d2aca30c0491a398a Closes-Bug: 1399678
* | | Verify MD5 of uploaded objects.Daniel Wakefield2015-02-193-42/+140
|/ / | | | | | | | | | | | | | | | | | | | | Changed existing code to calculate the MD5 of the object during the upload stream. Checks this MD5 against the etag returned in the response. An exception is raised if they do not match. Closes-Bug: 1379263 Change-Id: I6c8bc1366dfb591a26d934a30cd21c9e6b9a04ce
* | Fix missing ca-certificate parameter to get_auth.Charles Hsu2015-02-071-2/+4
|/ | | | | | | | | Missing cacert parameter in get_auth_1_0() and it makes requests_args['verify'] is True in default(likes "swift --os-cacert ssl.crti list"). The requests_args['verify'] should be a ca-certificate file name("ssl.crt"). Change-Id: Id3110f3818f55922ad19eeb1ead80fface824009 Closes-Bug: 1418473
* Fix deleting SLO segments on overwriteJoel Wright2015-02-041-10/+9
| | | | | | | | This patch fixes a bug that results in SLO segments being left behind regardless of the setting of --leave-segments. Change-Id: I2b3c8ab4817df6facb4b5961dbe570645441a552 Closes-Bug: 1418007
* Merge "This patch fixes downloading files to stdout."Jenkins2015-02-033-60/+73
|\
| * This patch fixes downloading files to stdout.Joel Wright2015-01-233-60/+73
| | | | | | | | | | | | | | | | | | | | | | | | This patch fixes downloading files to stdout and modifies _SwiftReader to operate as an iterator that performs file checks at the end of iteration as well as a context manager. File verification checks have been removed from __exit__ and added to __iter__. Change-Id: I3250bdeeef8484a9122c4b5b854756a7c8f8731e Closes-Bug: 1395922 Closes-Bug: 1387376
* | Merge "Fix cross account upload using --os-storage-url"Jenkins2015-02-023-50/+45
|\ \
View Lorry Controller Status