summaryrefslogtreecommitdiff
path: root/swiftclient/shell.py
Commit message (Collapse)AuthorAgeFilesLines
* shell: Allow uploads from stdin to specify --use-dloTim Burke2023-03-231-1/+1
| | | | Change-Id: Id6818ab416380242cc215a980a541fc59109c69c
* Merge "Use SLO by default for segmented uploads if the cluster supports it"Zuul2023-03-221-7/+13
|\
| * Use SLO by default for segmented uploads if the cluster supports itTim Burke2023-03-211-7/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've had SLO for nearly ten years, and it is preferable for most use-cases. It's time to default to SLO rather than DLO. Add a new --use-dlo option to give users a way to return to old behavior; ensure there is still a --use-slo option so we don't break existing scripts that may use it. UpgradeImpact: ============== The default segmented-upload behavior has changed; Static Large Objects are now used by default rather than Dynamic Large Objects. To revert to the old behavior: * CLI users may use the new `--use-dlo` option * Service API users may explicitly set `use_slo` to False in their options dicts Change-Id: Iebcd64ced8eab6efebb9f02a0c4be6bf6338cb3b
* | shell: Allow timeouts to have unitsTim Burke2023-03-201-2/+2
|/ | | | | Related-Change: Ibbe7e5aa8aa8e54935da76109c2ea13fb83bc7ab Change-Id: Ifeaaea790d1dadc84b157a7cf2be7590949c70f0
* Merge "Allow tempurl times to have units"4.1.0Zuul2022-08-251-8/+10
|\
| * Allow tempurl times to have unitsTim Burke2022-08-191-8/+10
| | | | | | | | | | | | | | Specifically, let users add a suffix for seconds, minutes, hours, or days. Change-Id: Ibbe7e5aa8aa8e54935da76109c2ea13fb83bc7ab
* | Merge "Add option to skip container PUT during upload"Zuul2022-08-231-3/+10
|\ \ | |/ |/|
| * Add option to skip container PUT during uploadTim Burke2022-01-111-3/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, a user with read/write access to a container (but without access to creat new containers) recieves a warning every time they upload. Now, allow them to avoid the extra request and warning by specifying --skip-container-put on the command line. This is also useful when testing: developers can HEAD a container to ensure it's in memcache, shut down all container servers, then upload and creaate a bunch of async pendings. Previously, the 503 on container PUT would prevent the object upload from even being attempted. Closes-Bug: 1317956 Related-Bug: 1204558 Change-Id: I3d9129a0b6b65c6c6187ae6af003b221afceef47 Related-Change: If1f8a02ee7459ea2158ffa6e958f67d299ec529e
* | Merge "More cleanup following py2 removal"Zuul2022-08-181-2/+0
|\ \
| * | More cleanup following py2 removalTim Burke2022-05-241-2/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Drop py2-only hacking pin from test-requirements. * Remove quote() helper; urllib.parse.quote() works fine. * Remove some useless code. Change-Id: I9ffc923f58f1d11538f83ff26f7beb53cdf134c3
* | | tempurl: Support sha256 and sha512 signatures4.0.1Tim Burke2022-06-081-1/+10
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | Up the default to sha256 since * the proxy has supported (and defaulted to allowing) it for four years now, and * Rackspace has supported it for even longer. Include a note in the --help about older clusters likely requiring sha1. Change-Id: Ibac2bb7e2e4c9946c7384f0aab8e43d0d79ba645 Related-Change: Ia9dd1a91cc3c9c946f5f029cdefc9e66bcf01046 Related-Bug: #1733634 Closes-Bug: #1977867
* | Add timeout for Swift service4.0.0Pavel Abalikhin2022-04-211-0/+3
| | | | | | | | | | | | | | Connection class has timeout parameter but SwiftService and shell don't use it. That can lead to an endless wait when network is unreachable. Change-Id: Iafa42fc2f8b56feefa2bc8ea6a1b8845717d3bab
* | Remove sixStephen Finucane2022-03-211-7/+2
| | | | | | | | | | | | | | This mostly affects tests. Nothing too complicated Signed-off-by: Stephen Finucane <stephenfin@redhat.com> Change-Id: Iabc78f651e1d48db35638280722f8019798eccd6
* | Remove __future__ importsStephen Finucane2022-03-211-2/+0
|/ | | | | | | These aren't needed in modern Python 3 versions. Signed-off-by: Stephen Finucane <stephenfin@redhat.com> Change-Id: I5e81d6fb2e2cb8e4bfae4ed746da002f44e871c4
* Merge "Include storage policy when listing account with --long"3.13.0Zuul2021-10-131-2/+3
|\
| * Include storage policy when listing account with --longTim Burke2021-09-211-2/+3
| | | | | | | | Change-Id: Ibc2f9445b5a8e80cfb73d0706e20a7e4c62eec4a
* | Correctly aggregate totals for >10k itemsTim Burke2021-09-211-15/+17
|/ | | | | | | | | | | | | | | | | Previously, we would write out totals for every page of listings, like $ swift list sync --prefix=09-21 --total -l 80000000000 80000000000 80000000000 58096000000 Now, roll those all into a single total: $ swift list sync --prefix=09-21 --total -l 298096000000 Change-Id: Icc265636815220e33e8c9eec0a3ab80e9f899038
* Improve formatting for billions of objectsTim Burke2021-09-201-2/+2
| | | | Change-Id: If8aa08c4c8c8ad6ca2c861602baf1eefa8642a8a
* Fix Typo in shell.pyjonasdlindner2021-04-291-1/+1
| | | | Change-Id: I2615e0d6b54d8cc020c24d1b5b4064e038f0934b
* Have `delete --all` imply `--versions` for the CLITim Burke2020-10-161-2/+4
| | | | Change-Id: Id5a6d4cef3d4ed76c897a099a62a4ba3ed8f8dab
* Include transaction ID in ClientException.__str__Tim Burke2020-09-091-1/+2
| | | | | | | | | | It's fairly annoying getting a traceback in swift's probe tests then only having a URL and status code to go searching for in logs. Leave the shell.py output untouched, though, since we output the transaction ID on a new line anyway. Change-Id: Idb849848ec08b6c04812b088467c9a687c2a7e27
* Application credential support follow-upTim Burke2020-05-271-4/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Following the recent v3applicationcredentials patch, if you have your environment variables set up to work with python-openstackclient using swiftclient's v1password plugin, swiftclient won't work: $ env | egrep '^(OS|ST)_' ST_KEY=testing ST_USER=test:tester OS_AUTH_URL=http://saio/auth/v1.0 ST_AUTH=http://saio/auth/v1.0 OS_USERNAME=test:tester OS_AUTH_TYPE=v1password OS_PASSWORD=testing $ openstack object store account show +------------+----------------------------+ | Field | Value | +------------+----------------------------+ | Account | AUTH_test | | Bytes | 0 | | Containers | 11 | | Objects | 0 | +------------+----------------------------+ $ swift stat Only "v3applicationcredential" is supported for --os-auth-type We don't really want to allow (and mostly ignore) arbitrary OS_AUTH_TYPE values, though -- there are a whole bunch of plugins we don't remotely support. But it seems OK to allow any of the password plugins; while we won't actually use them (currently), we provide roughly equivalent functionality. Handful of other drive-bys: * Use a None sentinel to determine whether keystoneauth1 is installed instead of trying to catch a NameError. * Clarify error state when keystoneauth1 is not installed. * Fix a typo: "sses" -> "sess". Change-Id: Id7ea9c3ea8278ae86a04d057a472a8f8a87b8eae Related-Change: I9190e5e7e24b6a741970fa0d0ac792deccf73d25
* Support v3 application credentials auth.Charles Hsu2020-04-161-5/+49
| | | | | | | | | | | | | | Use keystoneauth1 application credential plugin and session to fetch a token and endpoint catalog url. $ swift --os-auth-url http://172.16.1.2:5000/v3 --auth-version 3\ --os-application-credential-id THE_ID \ --os-application-credential-secret THE_SECRET \ --os-auth-type v3applicationcredential auth Change-Id: I9190e5e7e24b6a741970fa0d0ac792deccf73d25 Closes-Bug: 1843901 Closes-Bug: 1856635
* Improve `list --versions` outputTim Burke2020-04-101-4/+11
| | | | | | | | | | Have `--versions` imply `--long` and add a new column for version_id. Also, have version-aware listings show all versions as "null" on old Swifts that don't support object versioning (or when object versioning is not enabled). Change-Id: I0e009bce2471d1c140ac9b83700591cb355fee3f
* object versioning featuresClay Gerrard2020-04-081-2/+37
| | | | | | | | | | * add --versions to list * add --versions to delete * add --version-id to stat * add --version-id to delete * add --version-id to download Change-Id: I89802064921778fee7efe57c7d60c976cdde3a27
* Add test for bulk-delete-attempt-counter fixTim Burke2019-11-151-1/+1
| | | | | | Change-Id: Ifdeefeb4a5a3fc6895bd6cda695684de02f8c602 Related-Change: If4af9141fe4f3436a4e9e0e2dfc24c6ec7292996 Related-Bug: #1852808
* Fix printed object names on successful bulk-deleteSébastien Blaisot2019-11-151-4/+2
| | | | | | | | | Replace the 1 always concatenated to printed object names for each successfully deleted object in bulk-delete with an optional [after x attempts] if x > 1 Change-Id: If4af9141fe4f3436a4e9e0e2dfc24c6ec7292996 Closes-Bug: 1852808
* Merge "Optionally display listings in raw json"Zuul2019-07-101-1/+18
|\
| * Optionally display listings in raw jsonClay Gerrard2019-07-091-1/+18
| | | | | | | | | | | | | | | | | | | | | | | | Symlinks have recently added some new keys to container listings. It's very convenient to be able to see and reason about the extra information in container listings. Allowing raw json output is similar with what the client already does for the info command, and it's forward compatible with any listing enhancements added by future middleware development. Change-Id: I88fb38529342ac4e4198aeccd2f10c69c7396704
* | Merge "Add missing <sync-to> value in command line docs"Zuul2019-07-091-1/+1
|\ \ | |/ |/|
| * Add missing <sync-to> value in command line docsKota Tsuyuzaki2018-02-231-1/+1
| | | | | | | | | | | | Because it should take a value of either realm or full url. Change-Id: I1fe30825ef1620e256c9fd3057da6808b03d7200
* | Clean up warnings from newer flake8Tim Burke2019-06-271-0/+1
| | | | | | | | Change-Id: I18a6327b3acdd4db5ae80097080c043f7c20c353
* | Update hacking versionZhijunWei2019-01-031-2/+3
| | | | | | | | | | | | | | 1. update hacking version to latest 2. fix pep8 failed Change-Id: Ifc3bfeff4038c93d8c8cf2c9d7814c3003e73504
* | Merge "Back out some version bumps"Zuul2018-07-241-3/+8
|\ \
| * | Back out some version bumpsTim Burke2018-07-111-3/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I'm giving up on trying to back out all of the test-requirements up-revs, but let's try to stay compatibile with old requests/six. As part of that, only disable some requests warnings on new-enough requests. Note that we should now be compatible with distro packages back to Ubuntu 16.04 and CentOS 6. Our six is still too new for Trusty, but hey, there's less than a year left on that anyway, right? Change-Id: Iccb23638393616f9ec3da660dd5e39ea4ea94220 Related-Change: I2a8f465c8b08370517cbec857933b08fca94ca38
* | | Add bash_completion to swiftclientMatthew Oliver2018-07-131-99/+213
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch basically follows the bash completion model that other OpenStack clients use. It creates a new command to swiftclient called `bash_completion`. The `bash_completion` command by default will print all base flags and exsiting commands. If you pass it a command, it'll print out all base flags and any flags that command accepts. So as you type out your swift command and auto-complete, only the current available flags are offered to you. This is used by the swift.bash_completion script to allow swift commands to be bash completed. To make it work, place the swift.bash_completion file into /etc/bash_completion.d and source it: cp tools/swift.bash_completion /etc/bash_completion.d/swift source /etc/bash_completion.d/swift Because swiftclient itself is creating this flag/command output it should automatically add anything we add to the swiftclient CLI. Change-Id: I5609a19018269762b4640403daae5827bb9ad724
* | | Add ability to generate a temporary URL with anmmcardle2018-07-101-1/+10
|/ / | | | | | | | | | | IP range restriction Change-Id: I4734599886e4f4a563162390d0ff3bb1ef639db4
* | Add option for user to enter passwordAlistair Coles2018-06-111-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | Add the --prompt option for the CLI which will cause the user to be prompted to enter a password. Any password otherwise specified by --key, --os-password or an environment variable will be ignored. The swift client will exit with a warning if the password cannot be entered without its value being echoed. Closes-Bug: #1357562 Change-Id: I513647eed460007617f129691069c6fb1bfe62d7
* | Merge "show option per line"Zuul2018-06-051-4/+8
|\ \
| * | show option per lineThiago da Silva2018-03-171-4/+8
| |/ | | | | | | | | | | | | | | ading multiple options on the same line makes it easy to miss when quickly scanning the options. Change-Id: I8e324fca48cd05d9e381d5106135542274c2ff7f Signed-off-by: Thiago da Silva <thiago@redhat.com>
* | Add force auth retry mode in swiftclientKota Tsuyuzaki2018-03-131-0/+6
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch attemps to add an option to force get_auth call while retrying an operation even if it gets errors other than 401 Unauthorized. Why we need this: The main reason why we need this is current python-swiftclient requests could never get succeeded under certion situation using third party proxies/load balancers between the client and swift-proxy server. I think, it would be general situation of the use case. Specifically describing nginx case, the nginx can close the socket from the client when the response code from swift is not 2xx series. In default, nginx can wait the buffers from the client for a while (default 30s)[1] but after the time past, nginx will close the socket immediately. Unfortunately, if python-swiftclient has still been sending the data into the socket, python-swiftclient will get socket error (EPIPE, BrokenPipe). From the swiftclient perspective, this is absolutely not an auth error, so current python-swiftclient will continue to retry without re-auth. However, if the root cause is sort of 401 (i.e. nginx got 401 unauthorized from the swift-proxy because of token expiration), swiftclient will loop 401 -> EPIPE -> 401... until it consume the max retry times. In particlar, less time to live of the token and multipart object upload with large segments could not get succeeded as below: Connection Model: python-swiftclient -> nginx -> swift-proxy -> swift-backend Case: Try to create slo with large segments and the auth token expired with 1 hour 1. client create a connection to nginx with successful response from swift-proxy and its auth 2. client continue to put large segment objects (e.g. 1~5GB for each and the total would 20~30GB, i.e. 20~30 segments) 3. after some of segments uploaded, 1 hour past but client is still trying to send remaining segment objects. 4. nginx got 401 from swift-proxy for a request and wait that the connection is closed from the client but timeout past because the python-swiftclient is still sending much data into the socket before reading the 401 response. 5. client got socket error because nginx closed the connection during sending the buffer. 6. client retries a new connection to nginx without re-auth... <loop 4-6> 7. finally python-swiftclient failed with socket error (Broken Pipe) In operational perspective, setting longer timeout for lingering close would be an option but it's not complete solution because any other proxy/LB may not support the options. If we actually do THE RIGHT THING in python-swiftclient, we should send expects: 100-continue header and handle the first response to re-auth correctly. HOWEVER, the current python's httplib and requests module used by python-swiftclient doesn't support expects: 100-continue header [2] and the thread proposed a fix [3] is not super active. And we know the reason we depends on the library is to fix a security issue that existed in older python-swiftclient [4] so that we should touch around it super carefully. In the reality, as the hot fix, this patch try to mitigate the unfortunate situation described above WITHOUT 100-continue fix, just users can force to re-auth when any errors occurred during the retries that can be accepted in the upstream. 1: http://nginx.org/en/docs/http/ngx_http_core_module.html#lingering_close 2: https://github.com/requests/requests/issues/713 3: https://bugs.python.org/issue1346874 4: https://review.openstack.org/#/c/69187/ Change-Id: I3470b56e3f9cf9cdb8c2fc2a94b2c551927a3440
* Allow for object uploads > 5GB from stdin.Timur Alperovich2018-01-181-0/+8
| | | | | | | | | | | | | When uploading from standard input, swiftclient should turn the upload into an SLO in the case of large objects. This patch picks the threshold as 10MB (and uses that as the default segment size). The consumers can also supply the --segment-size option to alter that threshold and the SLO segment size. The patch does buffer one segment in memory (which is why 10MB default was chosen). (test is updated) Change-Id: Ib13e0b687bc85930c29fe9f151cf96bc53b2e594
* Merge "Allow --meta on upload"Zuul2017-12-081-2/+9
|\
| * Allow --meta on uploadTim Burke2017-07-061-2/+9
| | | | | | | | | | | | | | Previously, the --meta option was only allowed on post or copy subcommands. Change-Id: I87bf0338c34b5e89aa946505bee68dbeb37d784c Closes-Bug: #1616238
* | Allow for uploads from standard input.Timur Alperovich2017-07-261-3/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | If "-" is passed in for the source, python-swiftclient will upload the object by reading the contents of the standard input. The object name option must be set, as well, and this cannot be used in conjunction with other files. This approach stores the entire contents as one object. A follow on patch will change this behavior to upload from standard input as SLO, unless the segment size is larger than the content size. Change-Id: I1a8be6377de06f702e0f336a5a593408ed49be02
* | Option to ignore mtime metadata entry.Christopher Bartz2017-07-061-0/+9
|/ | | | | | | | | | | | | | | Currently, the swiftclient upload command passes a custom metadata header for each object (called object-meta-mtime), whose value is the current UNIX timestamp. When downloading such an object with the swiftclient, the mtime header is parsed and passed as the atime and mtime for the newly created file. There are use-cases where this is not desired, for example when using tmp or scratch directories in which files older than a specific date are deleted. This commit provides a boolean option for ignoring the mtime header. Change-Id: If60b389aa910c6f1969b999b5d3b6d0940375686
* ISO 8601 timestamps for tempurlChristopher Bartz2017-03-291-15/+53
| | | | | | | | | Client-side implementation for ISO 8601 timestamp support of tempurl middleware. Please see https://review.openstack.org/#/c/422679/ Change-Id: I76da28b48948475ec1bae5258e0b39a316553fb7
* [trivial] Be consistent in using example.comPetr Kovar2017-03-131-1/+1
| | | | | | * Use distro-agnostic example.com in all examples. Change-Id: If0a7217b7312875bb4ca8258f4ef86c52cd2e577
* Expose --prefix as an option for st_deleteJoel Wright2017-02-231-1/+3
| | | | | | | | | The SwiftService and shell support the ability to limit deletions to only those objects that match a specified prefix, so let's expose that (really useful) behaviour in the command line help as well :) Change-Id: I9ef177aa96e4829196b5200dd8e9d0d2f7f89b63
* prefix-based tempurls supportChristopher Bartz2017-01-191-5/+12
| | | | | | | | | Implements client-side functionality for prefix-based tempurls. Please see: https://review.openstack.org/#/c/274048/ Change-Id: I8d7701daee888ed1120271a96c0660b01543ca2d
View Lorry Controller Status