| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
| |
In addition to being less confusing for devs, this lets us actually run
tempauth tests in swiftclient dsvm jobs.
The job definition (over in the swift repo) specifies test/sample.conf,
which does not exist in this repo. As a result, those tests would skip
with
SKIPPING FUNCTIONAL TESTS DUE TO NO CONFIG
Change-Id: I558dbf9a657d442e6e19468e543bbec855129eeb
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, when deleting a symlink that points to an xLO, we'd clean
up the xLO's segments then delete the symlink, leaving the xLO itself
busted.
Similar trouble would come from overwriting a symlink pointing to an
xLO. Check for a Content-Location in the HEAD response and leave such
segments.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I45b210cf380a68bd88187c91fa2d63a8b2bb709b
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When uploading from standard input, swiftclient should turn the upload
into an SLO in the case of large objects. This patch picks the
threshold as 10MB (and uses that as the default segment size). The
consumers can also supply the --segment-size option to alter that
threshold and the SLO segment size. The patch does buffer one segment
in memory (which is why 10MB default was chosen).
(test is updated)
Change-Id: Ib13e0b687bc85930c29fe9f151cf96bc53b2e594
|
|\ |
|
| |
| |
| |
| |
| |
| |
| | |
Previously, the --meta option was only allowed on post or copy subcommands.
Change-Id: I87bf0338c34b5e89aa946505bee68dbeb37d784c
Closes-Bug: #1616238
|
| |
| |
| |
| |
| |
| |
| | |
Otherwise, Python defaults to 8k reads which seems kinda terrible.
Change-Id: I3160626e947083af487fd1c3cb0aa6a62646527b
Closes-Bug: #1671621
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently, the swiftclient upload command passes a custom metadata
header for each object (called object-meta-mtime), whose value is
the current UNIX timestamp. When downloading such an object with the
swiftclient, the mtime header is parsed and passed as the atime and
mtime for the newly created file.
There are use-cases where this is not desired, for example when using
tmp or scratch directories in which files older than a specific date
are deleted. This commit provides a boolean option for ignoring the
mtime header.
Change-Id: If60b389aa910c6f1969b999b5d3b6d0940375686
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| | |
If we get back some partial content, we can't validate the MD5.
That's OK.
Change-Id: Ic1d65272190af0d3d982f3cd06833cac5c791a1e
Closes-Bug: 1642021
|
|\ \
| |/ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Since time immemorial, Swift has returned unquoted ETags for plain-old
Swift objects -- I hear tell that we once tried to change this, but
quickly backed it out when some clients broke.
However, some proxies (such as nginx) apparently may force the ETag to
adhere to the RFC, which states [1]:
An entity-tag consists of an opaque *quoted* string
(emphasis mine). See the related bug for an instance of this happening.
Since we can still get the original ETag easily, we should tolerate the
more-compliant format.
[1] https://tools.ietf.org/html/rfc2616.html#section-3.11 or, if you
prefer the new ones, https://tools.ietf.org/html/rfc7232#section-2.3
Change-Id: I7cfacab3f250a9443af4b67111ef8088d37d9171
Closes-Bug: 1681529
Related-Bug: 1678976
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, using SwiftService to delete "many" objects would use
bulk delete if available, but it would not respect the bulk delete
page size. If the number of objects to delete exceeded the bulk delete
page size, SwiftService would ignore the error and nothing would be
deleted.
This patch changes _should_bulk_delete() to be _bulk_delete_page_size();
instead of returning a simple True/False, it returns the page size for
the bulk deleter, or 1 if objects should be deleted one at a time.
Delete SDK calls are then spread across multiple bulk DELETEs if the
requested number of objects to delete exceeds the returned page size.
Fixed the logic in _should_bulk_delete() so that if the object list
is exactly 2x the thread count, it will not bulk delete. This is the
natural conclusion following the logic that existed previously: if
the delete request can be satisfied by every worker thread doing one
or two tasks, don't bulk delete. But if it requires a worker thread
to do three or more tasks, do a bulk delete instead. Previously, the
logic would mean that if every worker thread did exactly two tasks, it
would bulk delete. This patch changes a "<" to a "<=".
Closes-Bug: 1679851
Change-Id: I3c18f89bac1170dc62187114ef06dbe721afcc2e
|
|
|
|
|
|
|
|
| |
The opened file for upload is not closed.
This fix prevents possible file handle leak.
Closes-Bug: #1559079
Change-Id: Ibc58667789e8f54c74ae2bbd32717a45f7b30550
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, we only accepted iterables of strings like 'Header: Value'.
Now, we'll also accept lists of tuples like ('Header', 'Value') as well
as dictionaries like {'Header': 'Value'}.
This should be more intuitive for application developers, who are
already used to being able to pass dicts or lists of tuples to libraries
like requests.
Change-Id: I93ed2f1e8305f0168b7a4bd90c205b04730da836
|
|
|
|
|
| |
Change-Id: I69276ba711057c122f97deac412e492e313c34dd
Closes-Bug: 1615830
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| | |
New versions of requests will raise an InvalidHeader error otherwise.
Change-Id: Idf3bcd8ac359bdda9a847bf99a78988943374134
Closes-Bug: #1614280
Closes-Bug: #1613814
|
|\ \
| |/
|/| |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Implement copy object method in swiftclient Connection, Service and CLI.
Although COPY functionality can be accomplished with 'X-Copy-From'
header in PUT request, using copy is more convenient especially when
using copy for updating object metadata non-destructively.
Closes-Bug: 1474939
Change-Id: I1338ac411f418f4adb3d06753d044a484a7f32a4
|
| |
| |
| |
| |
| |
| | |
For example: self.assertEqual(a,None) is equal to self.assertIsNone(a)
Change-Id: I063145d034979cf3d36645a00a30cc27defac7c4
|
|/
|
|
|
|
|
|
| |
This patch is to add an option of disable etag
check on downloads.
Change-Id: I9ad389dd691942dea6db470ca3f0543eb6e9703e
Closes-bug: #1581147
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Previously, we were using a content-type of text/directory, but that is
already defined in RFC 2425 and doesn't reflect our usage:
The text/directory Content-Type is defined for holding a variety
of directory information, for example, name, or email address,
or logo.
(From there it goes on to describe a superset of the vCard format
defined in RFC 2426.)
application/directory, on the other hand, is used by Static Web [1] and
is used by cloudfuse [2]. Seems like as sane a choice as any to
standardize on.
[1] https://github.com/openstack/swift/blob/2.5.0/swift/common/middleware/staticweb.py#L71-L75
[2] https://github.com/redbo/cloudfuse/blob/1.0/README#L105-L106
Change-Id: I19e30484270886292d83f50e7ee997b6e1623ec7
|
|\ \ |
|
| |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
...using a new "application/swiftclient-segment" content-type.
Segments uploaded by swiftclient are expected to have a many-to-one
relationship to large objects, rather than the more-general many-to-many
relationship that SLO and DLO generally allow. Later, we may use this
information to make more intelligent decisions, such as when to
automatically clean up segments.
Change-Id: Ie56a3aa10065db754ac572cc37d93f2c901aac60
|
|/
|
|
|
|
|
|
|
| |
The documentation of "swift download" hints that "marker" option
is supported, but in reality we forgot to patch it through, so
all downloads were always done with the default, empty marker.
Closes-Bug: #1565393
Change-Id: I38bd29d2baa9188b61397dec75ce1d864041653c
|
|
|
|
| |
Change-Id: Iac93ced6344d4a6fee7e6390e891fde765814c03
|
|
|
|
|
|
|
|
|
|
|
| |
SwiftService().post(cont, [SwiftPostObject(obj, options]) currently
ignores options['header'], raises exception when options['headers']
is set and make malformed metadata when options['meta'] is set.
Fix tipos in code, add unittest for SwiftService().post
Closes-Bug: #1560052
Change-Id: Ie460f753492e9b73836b4adfc7c9c0f2130a8a91
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
in python-swiftclient/swiftclient/service.py,
there is a method
def download(self, container=None, objects=None, options=None):
if container is specified but objects not, it download all
objects in specified container.
if both container and objects are specified, it download all
specified objects in the container.
when it comes to the case that, objects argument is specified,
but it turned out to be empty array [ ], the download method
download all the objects under specified container.
this may be not reasonable.
for example,
the objects was not empty when it came from command line,
but it's filtered, maybe by --prefix argument.
at last, it turned out to be empty array.
when calling download method with objects arguments
being empty array, we should download nothing instead of
all the objects under the specified container.
Change-Id: I81aab935533a50b40679c8b3575f298c285233a8
Closes-bug: #1549881
|
|/
|
|
|
|
|
|
|
|
|
| |
My understanding is that it was mainly being used so we could have sane
testing on py26. With py26 support being dropped, we no longer need it.
Also drop discover from test-requirements.txt, as we don't seem to
actually use it.
Change-Id: Iee04c42890596d3b483c1473169480a3ae19aac8
Related-Change: I37116731db11449d0c374a6a83a3a43789a19d5f
|
|\ |
|
| |
| |
| |
| | |
Change-Id: Idab172aefd8e69ca8e4d623918eba1bc1da91a42
|
|\ \
| |/ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This fix ensures creation and use of the correct default segment
container when pseudo-folder paths are passed via <container> arg.
Change-Id: I90356b041dc9dfbd55eb341271975621759476b9
Closes-Bug: 1532981
Related-Bug: 1478210
|
|\ \ |
|
| | |
| | |
| | |
| | |
| | |
| | | |
Still use it if we can, but stop throwing ValueErrors if it's garbage.
Change-Id: I2cf25b535ad62cfacb7561954a92a4a73d91000a
|
|/ /
| |
| |
| | |
Change-Id: Ic453fd990ebbf17e0eeabc39b126d2bc14234e23
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
It's crashing now (sometimes):
_StringException: Traceback (most recent call last):
File "/build/python-swiftclient-2.6.0/tests/unit/test_service.py", line 1085, in test_upload_object_job_stream
self.assertAlmostEqual(mtime, expected_mtime, delta=0.5)
File "/usr/lib/python3/dist-packages/unittest2/case.py", line 883, in assertAlmostEqual
raise self.failureException(msg)
AssertionError: 1453224313.0 != 1453224312.4944572 within 0.5 delta
Change-Id: Ib2eeb13cd07febcb7c8b4e1435b885c4339093e4
|
|/
|
|
|
|
|
| |
This patch include a test case to test unicode path.
Change-Id: I7697679f0034ce65b068791d7d5145286d992bd1
Closes-Bug: #1532096
|
|
|
|
| |
Change-Id: I29d891e2dc900eb93e703f77f4e56f68710a8955
|
|
|
|
| |
Change-Id: Iaf7f30a7ae0c2ac76fc5cdcee31ea74c08ce601e
|
|
|
|
|
|
|
|
| |
New documentation for python-swiftclient that introduces
the APIs available and gives some opinionated advice about
when to use the shell, the client API and the service API.
Change-Id: I19020f041fab2e72469979f712ffe3951c431d24
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Always use testtools.TestCase, since we're relying on testtools
* Always use mock (as opposed to unittest.mock) since we're relying on
mock
* Add note about when a missing logging handler was added
* Stop %-formatting the giant usage string that doesn't actually need
any formatting
* Prefer assertIs, assertIn, assertIsInstance over assertTrue
* Use else-self.fail instead of sentinel values
* Check resp.get('error') is None before checking resp['success'] is
True, so test failures actually tell you something useful
* Tighten some isinstance assertions
* Import MockHttpTest from correct location
* Only populate clean_os_environ once
* Use setUp for setup, not __init__
* Replace assertIn(key, dict) and assertEqual(foo, dict[key]) with
assertEqual(foo, dict.get(key)) when key is a literal and foo is not
None
* Use mock.patch.object instead of manually patching for tests
* Use six.binary_type instead of type(''.encode('utf-8'))
* Stop shadowing builtin bytes
* Reclaim some margin
* Stop checking the return-type of encode_utf8; we already know it's
bytes
Change-Id: I2138ea553378ce88810b7353147c8645a8f8c90e
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This patch adds exception logging to the swift service API. Each
operation that results in failure of any operation will now log
the exception as well as report a timestamp and full stack trace
in the results returned by the service API calls.
Change-Id: I7336b28354e7740ea7d048bdf355e3c1a1b4436c
|
|/
|
|
|
|
|
|
|
| |
Some improvements to the tests for staggered download
that were added in [1].
[1] Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
Change-Id: Ib999bc7bd198c1d9c217c57501f751e854d4c6ad
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
st_download
The current code builds a full object listing before performing either a multiple
download or delete operation (and also shuffles this complete list in the case of
a download). This patch removes the creation of the full object list and adds the
ability to turn off shuffle for files when downloading. Also added is a limit on
the number of list results that can be queued by a single call to service.list
without consuming results (reduces memory overhead for large listings).
Some tests added for service.py download and list.
Change-Id: Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This patch adds "--output-dir" and "--remove-prefix" options
to the "download" command and unit tests for it.
Example:
$ swift list example --prefix swift2.2
swift2.2/bin/swift-object-auditor
swift2.2/bin/swift-object-expirer
swift2.2/bin/swift-object-info
swift2.2/bin/swift-object-replicator
swift2.2/bin/swift-object-server
swift2.2/bin/swift-object-updater
When given "--output-dir <directory>", client downloads objects to <directory>.
$ swift download example --prefix swift2.2 \
--output-dir new/swift/dir
The folder structure:
.
└── new
└── swift
└── dir
└── swift2.2
└── bin
├── swift-object-auditor
├── swift-object-expirer
├── swift-object-info
├── swift-object-replicator
├── swift-object-server
└── swift-object-updater
When given "--remove-prefix", client downloads objects without <prefix>.
$ swift download example --prefix swift2.2 \
--remove-prefix \
--output-dir swift
The folder structure:
.
└── swift
└── bin
├── swift-object-auditor
├── swift-object-expirer
├── swift-object-info
├── swift-object-replicator
├── swift-object-server
└── swift-object-updater
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I7463fe2941cc94f9a50a4756a97c2ccdf946294d
Implements: blueprint swiftclient-download-pseudo-folder-to-specific-target
|