summaryrefslogtreecommitdiff
path: root/swiftclient/multithreading.py
Commit message (Collapse)AuthorAgeFilesLines
* docs: Fix warning treated as errorTim Burke2019-09-041-0/+6
| | | | Change-Id: I669533334419e94ca925e859f2b0d5d2afe9f7f1
* Isolate docs requirementsTim Burke2019-06-271-0/+8
| | | | | | | | | | | ...since modern sphinx won't install on py27. While we're at it, clean up some warnings and treat warnings as errors. Also, fix up how we parse test configs so we can run func tests. Related-Change: Id3c2ed87230c5918c18e2c01d086df8157f036b1 Change-Id: I3718f69610545b0dbcb0a2ab45b400da3a45682c
* Add new doc structure and contents for swiftclientJoel Wright2016-04-291-1/+0
| | | | | | | | | | | | | | | | | | | | As a result of the Hackathon we have produced a new documentation structure for the python-swiftclient. This patch introduces the new structure and adds the required content. The intention is to document the CLI, the SwiftService and Connection API. Importantly, we also provide guidance on important considerations when using a swift object store, such as which aspect of the python-swiftclient to use for various use cases, common authentication patterns and some useful examples. Co-Authored-By: Alexandra Settle <alexandra.settle@rackspace.com> Co-Authored-By: Mohit Motiani <mohit.motiani@intel.com> Co-Authored-By: Hisashi Osanai <osanai.hisashi@jp.fujitsu.com> Change-Id: I9eb41f8e9137efa66cead67dc264a76a3c03fbda
* Python 3: Replacing unicode with six.text_type for py3 compatibilityjanonymous2015-06-281-1/+1
| | | | | | | | | | | The "unicode" type was renamed to "str" in Python 3. Use six.text_type to make swiftclient compatible with Python 3. For more information about changes needed for py3 compatibility, see: https://wiki.openstack.org/wiki/Python3 Change-Id: Ic65607a69935652a1874340928f626fbcc35c014
* Compare each chunk of large objects when uploadingTim Burke2015-03-231-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, we compared the ETag from Swift against the MD5 of the entire large object. However, the ETag for large objects is generally the MD5 of the concatenation of the ETags for each segment, unless the object is a DLO whose segments span more than one page of a container listing. Rather than worry about ETags, just compare each chunk of the segmented file. This allows the use of --skip-identical when uploading SLOs and DLOs. Additionally, there are several test-related improvements: * The default arguments for OutputManager are now evaluated on construction, rather than on definition, so that TestOutputManager.test_instantiation will succeed when using nosetest as a test runner. (See also: bug 1251507) * An account_username option is now available in the functional tests config file for auth systems that do not follow the account:username format. * CaptureOutput no longer writes to the captured stream, and MockHttpTest now captures output. These were polluting test output unnecessarily. (See also: bug 1201376) Change-Id: Ic484e9a0c186c9283c4012c6a2fa77b96b8edf8a Closes-Bug: #1201376 Closes-Bug: #1379252 Related-Bug: #1251507
* Fix crash when stat'ing objects with non-ascii namesJoel Wright2015-02-171-3/+1
| | | | | | | | This patch fixes a crash and stack trace in stat when an object name contains non-ascii characters. Change-Id: Ib6dc686771e593243de56cafc100b17e51d9d9d5 Closes-Bug: 1411665
* Merge "This patch fixes downloading files to stdout."Jenkins2015-02-031-0/+11
|\
| * This patch fixes downloading files to stdout.Joel Wright2015-01-231-0/+11
| | | | | | | | | | | | | | | | | | | | | | | | This patch fixes downloading files to stdout and modifies _SwiftReader to operate as an iterator that performs file checks at the end of iteration as well as a context manager. File verification checks have been removed from __exit__ and added to __iter__. Change-Id: I3250bdeeef8484a9122c4b5b854756a7c8f8731e Closes-Bug: 1395922 Closes-Bug: 1387376
* | Fix cross account upload using --os-storage-urlAlistair Coles2015-01-061-2/+8
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Removes an account stat from the object upload path. This stat fails when user is not account admin even though the user may have container ACL permission to write objects. Reduces the severity of the CLI output message when upload fails to create the given container (this is not an error since the container may exist - the user just does not have permission to PUT or POST the container). Changes the 'swift upload' exit return code from 1 to 0 if container PUT fails but object PUT succeeds. For segment uploads, makes the attempt to create the segment container conditional on it not being the same as the manifest container. This avoids an unnecessary container PUT. Fixes another bug that became apparent: with segmented upload a container HEAD may be attempted to determine the policy to be used for the segment container. When this failed the result dict has headers=None which was causing an exception in the shell result handler. Add unit tests for object upload/download and container list with --os-storage-url option. Closes-Bug: #1371650 Change-Id: If1f8a02ee7459ea2158ffa6e958f67d299ec529e
* Change tests to use new CaptureOutput class.Daniel Wakefield2014-12-051-0/+3
| | | | | | | | | | Added the ability to clear the buffers in the CaptureOutput class so it can be easily used multiple times in the same context manager. Also added a option to suppress the SystemExit associated with printing an error. Change-Id: Ib59bbbe88256f215eed0a8ebc8282e02181d4377
* Add importable SwiftService incorporating shell.py logicJoel Wright2014-08-261-198/+88
| | | | | | | | | | | | | | | | This patch adds a SwiftService class that incorporates the high level logic from swiftclient/shell.py. It also ports shell.py to use the new class, and updates the code in swiftclient/multithreading.py to allow the SwiftService to be used for multiple operations whilst using only one thread pool. Currently, code that imports swiftclient has to have its own logic for things like creating large objects, parallel uploads, and parallel downloads. This patch adds a SwiftService class that makes that functionality available in Python code as well as through the shell. Change-Id: I08c5796b4c01001d79fd571651c3017c16462ffd Implements: blueprint bin-swift-logic-as-importable-library
* Clean up raw policy stats in account statYuan Zhou2014-08-061-2/+15
| | | | | | | | | | | | | | | | | | | | | Storage policy stats was not well parsed in account stat. This patch parses the stats and print out the stats in a format like below: $swift -A http://swift_cluster/auth/v1.0 -U test:tester -K testing stat Account: AUTH_test Containers: 5 Objects: 1 Bytes: 2097152 Objects in policy "golden": 1 Bytess in policy "golden": 2097152 Objects in policy "silver": 0 Bytes in policy "silver": 0 X-Timestamp: 1404697760.88809 X-Trans-Id: txec519e24b44a413abb705-0053da2dcb Content-Type: text/plain; charset=utf-8 Accept-Ranges: bytes Change-Id: I7ad0ee6d88f8393e3a93e90cd52b9b592da7072d
* Fix test_multithreading on Python 3Victor Stinner2014-04-151-4/+7
| | | | | | | | | | | * On Python 3, the printer doesn't encode Unicode to utf8 anymore, since print() expects a Unicode string. * Update unit tests for Python 3 since repr() doesn't escape non-ASCII characters in Unicode strings anymore: http://legacy.python.org/dev/peps/pep-3138/ Change-Id: I89471019d691a46651312d6a49964b719192148a
* Replaced print statements with print function.Alex Gaynor2014-03-301-1/+4
| | | | | | This is needed for Python3 compatibility. Change-Id: Iadd21e4b3a936b601a69f1db2aba8e1597f13fc3
* Python 3: Fix module names in importVictor Stinner2014-03-281-1/+1
| | | | | | Use six.moves to fix imports on Python 3. Change-Id: I35b9405690e9f0607b24d79aa7c00830df954c41
* Replace xrange in for loop with rangeWu Wenxiang2014-01-141-1/+1
| | | | | | | | | xrange is not supported in python 3, so replace it with range since no large size lists involved. Closes-Bug: #1237717 Change-Id: I4e5e0782153d32c8beee03f7d871722ed4352859
* Add verbose output to all stat commandsClay Gerrard2013-10-091-0/+24
| | | | | | | | | | | | | When you stat a container or object with the verbose flag the full path of the reousrce will be displayed with the token similarlly to how an account stat displays the auth url and token. * move some logic out of bin/swift.st_stat to test it * new module swiftclient.commnad_helpers for code you want to test * moved prt_bytes into swiftclient.utils to test it * fixed IndexError with prt_bytes on sizes >= 1024Y Change-Id: Iaaa96e0308b08c554205b0055b8a04de581fefa4
* Move multi-threading code to a library.Darrell Bishop2013-07-281-0/+241
This patch extracts the multi-threading code from bin/swift into swiftclient/multithreading and adds tests. In particular, this new way of doing it (with context managers) will prevent non-daemonic threads from wedging the process when unexpected exceptions happen. I enabled reporting of which lines, specifically, are not covered by unit tests (added -m option to "coverage report" in .unittests). This patch includes a drive-by fix for uploading a segmented file with --use-slo when that object already exists. A key of "name" was used instead of "path", raising KeyError. There's also another drive-by fix for uploading segmented objects with --use-slo. Commit 874e0e4427b80e1b15b74a1557b73ba9d61443ca regressed this by removing the capturing of thread-worker results in QueueFunctionThread.run(). This patch restores that functionality and the feature (uploading SLO objects). Change-Id: I0b4f677e4a734e83d1a25088d9a74f7d46384e53
View Lorry Controller Status