diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/manpages/swift.1 | 12 | ||||
| -rw-r--r-- | doc/requirements.txt | 5 | ||||
| -rw-r--r-- | doc/source/_static/.gitignore | 0 | ||||
| -rw-r--r-- | doc/source/cli.rst | 442 | ||||
| -rw-r--r-- | doc/source/cli/index.rst | 958 | ||||
| -rw-r--r-- | doc/source/conf.py | 18 | ||||
| -rw-r--r-- | doc/source/index.rst | 22 | ||||
| -rw-r--r-- | doc/source/introduction.rst | 90 | ||||
| -rw-r--r-- | doc/source/service-api.rst | 54 |
9 files changed, 1061 insertions, 540 deletions
diff --git a/doc/manpages/swift.1 b/doc/manpages/swift.1 index 1f288d6..00e1440 100644 --- a/doc/manpages/swift.1 +++ b/doc/manpages/swift.1 @@ -63,8 +63,11 @@ Uploads to the given container the files and directories specified by the remaining args. The \-c or \-\-changed is an option that will only upload files that have changed since the last upload. The \-\-object\-name <object\-name> is an option that will upload file and name object to <object\-name> or upload dir -and use <object\-name> as object prefix. The \-S <size> or \-\-segment\-size <size> -and \-\-leave\-segments and others are options as well (see swift upload \-\-help for more). +and use <object\-name> as object prefix. If the file name is "-", reads the +content from standard input. In this case, \-\-object\-name is required and no +other files may be given. The \-S <size> or \-\-segment\-size <size> and +\-\-leave\-segments and others are options as well (see swift upload \-\-help +for more). .RE \fBpost\fR [\fIcommand-options\fR] [\fIcontainer\fR] [\fIobject\fR] @@ -102,6 +105,9 @@ with \-\-no-download actually not to write anything to disk. The \-\-ignore-checksum is an option that turns off checksum validation. You can specify optional headers with the repeatable cURL-like option \-H [\-\-header]. For more details and options see swift download \-\-help. +The \-\-ignore\-mtime option ignores the x\-object\-meta\-mtime metadata entry +on the object (if present) and instead creates the downloaded files with +fresh atime and mtime values. .RE \fBdelete\fR [\fIcommand-options\fR] [\fIcontainer\fR] [\fIobject\fR] [\fIobject\fR] [...] @@ -205,4 +211,4 @@ swift \-A https://127.0.0.1:443/auth/v1.0 \-U swiftops:swiftops \-K swiftops sta .SH DOCUMENTATION .LP More in depth documentation about OpenStack Swift as a whole can be found at -.BI https://docs.openstack.org/developer/swift +.BI https://docs.openstack.org/swift/latest/ diff --git a/doc/requirements.txt b/doc/requirements.txt new file mode 100644 index 0000000..6cdad2a --- /dev/null +++ b/doc/requirements.txt @@ -0,0 +1,5 @@ +keystoneauth1>=3.4.0 # Apache-2.0 +sphinx!=1.6.6,!=1.6.7,<2.0.0,>=1.6.2;python_version=='2.7' # BSD +sphinx!=1.6.6,!=1.6.7,!=2.1.0,>=1.6.2;python_version>='3.4' # BSD +reno>=2.5.0 # Apache-2.0 +openstackdocstheme>=1.20.0 # Apache-2.0 diff --git a/doc/source/_static/.gitignore b/doc/source/_static/.gitignore new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/doc/source/_static/.gitignore diff --git a/doc/source/cli.rst b/doc/source/cli.rst deleted file mode 100644 index 8d80d1b..0000000 --- a/doc/source/cli.rst +++ /dev/null @@ -1,442 +0,0 @@ -==== -CLI -==== - -The ``swift`` tool is a command line utility for communicating with an OpenStack -Object Storage (swift) environment. It allows one to perform several types of -operations. - -Authentication -~~~~~~~~~~~~~~ - -This section covers the options for authenticating with a swift -object store. The combinations of options required for each authentication -version are detailed below, but are just a subset of those that can be used -to successfully authenticate. These are the most common and recommended -combinations. - -You should obtain the details of your authentication version and credentials -from your storage provider. These details should make it clearer which of the -authentication sections below are most likely to allow you to connect to your -storage account. - -Keystone v3 ------------ - -.. code-block:: bash - - swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \ - --os-project-name project1 --os-project-domain-name domain1 \ - --os-username user --os-user-domain-name domain1 \ - --os-password password list - - swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \ - --os-project-id 0123456789abcdef0123456789abcdef \ - --os-user-id abcdef0123456789abcdef0123456789 \ - --os-password password list - -Manually specifying the options above on the command line can be avoided by -setting the following combinations of environment variables: - -.. code-block:: bash - - ST_AUTH_VERSION=3 - OS_USERNAME=user - OS_USER_DOMAIN_NAME=domain1 - OS_PASSWORD=password - OS_PROJECT_NAME=project1 - OS_PROJECT_DOMAIN_NAME=domain1 - OS_AUTH_URL=https://api.example.com:5000/v3 - - ST_AUTH_VERSION=3 - OS_USER_ID=abcdef0123456789abcdef0123456789 - OS_PASSWORD=password - OS_PROJECT_ID=0123456789abcdef0123456789abcdef - OS_AUTH_URL=https://api.example.com:5000/v3 - -Keystone v2 ------------ - -.. code-block:: bash - - swift --os-auth-url https://api.example.com:5000/v2.0 \ - --os-tenant-name tenant \ - --os-username user --os-password password list - -Manually specifying the options above on the command line can be avoided by -setting the following environment variables: - -.. code-block:: bash - - ST_AUTH_VERSION=2.0 - OS_USERNAME=user - OS_PASSWORD=password - OS_TENANT_NAME=tenant - OS_AUTH_URL=https://api.example.com:5000/v2.0 - -Legacy auth systems -------------------- - -You can configure swift to work with any number of other authentication systems -that we will not cover in this document. If your storage provider is not using -Keystone to provide access tokens, please contact them for instructions on the -required options. It is likely that the options will need to be specified as -below: - -.. code-block:: bash - - swift -A https://api.example.com/v1.0 -U user -K api_key list - -Specifying the options above manually on the command line can be avoided by -setting the following environment variables: - -.. code-block:: bash - - ST_AUTH_VERSION=1.0 - ST_AUTH=https://api.example.com/v1.0 - ST_USER=user - ST_KEY=key - -It is also possible that you need to use a completely separate auth system, in which -case ``swiftclient`` cannot request a token for you. In this case you should make the -authentication request separately and access your storage using the token and -storage URL options shown below: - -.. code-block:: bash - - swift --os-auth-token 6ee5eb33efad4e45ab46806eac010566 \ - --os-storage-url https://10.1.5.2:8080/v1/AUTH_ced809b6a4baea7aeab61a \ - list - -.. We need the backslash below in order to indent the note -\ - - .. note:: - - Leftover environment variables are a common source of confusion when - authorization fails. - -CLI commands -~~~~~~~~~~~~ - -Stat ----- - - ``stat [container [object]]`` - - Displays information for the account, container, or object depending on - the arguments given (if any). In verbose mode, the storage URL and the - authentication token are displayed as well. - -List ----- - - ``list [command-options] [container]`` - - Lists the containers for the account or the objects for a container. - The ``-p <prefix>`` or ``--prefix <prefix>`` is an option that will only - list items beginning with that prefix. The ``-d <delimiter>`` or - ``--delimiter <delimiter>`` is an option (for container listings only) - that will roll up items with the given delimiter (see `OpenStack Swift - general documentation <http://docs.openstack.org/developer/swift/>` for - what this means). - - The ``-l`` and ``--lh`` options provide more detail, similar to ``ls -l`` - and ``ls -lh``, the latter providing sizes in human readable format - (For example: ``3K``, ``12M``, etc). The latter two switches use more - overhead to retrieve the displayed details, which is directly proportional - to the number of container or objects listed. - -Upload ------- - - ``upload [command-options] container file_or_directory [file_or_directory] [...]`` - - Uploads the files and directories specified by the remaining arguments to the - given container. The ``-c`` or ``--changed`` is an option that will only - upload files that have changed since the last upload. The - ``--object-name <object-name>`` is an option that will upload a file and - name object to ``<object-name>`` or upload a directory and use ``<object-name>`` - as object prefix. The ``-S <size>`` or ``--segment-size <size>`` and - ``--leave-segments`` are options as well (see ``--help`` for more). - -Post ----- - - ``post [command-options] [container] [object]`` - - Updates meta information for the account, container, or object depending - on the arguments given. If the container is not found, the ``swiftclient`` - will create it automatically, but this is not true for accounts and - objects. Containers also allow the ``-r <read-acl>`` (or ``--read-acl - <read-acl>``) and ``-w <write-acl>`` (or ``--write-acl <write-acl>``) options. - The ``-m`` or ``--meta`` option is allowed on accounts, containers and objects, - and is used to define the user metadata items to set in the form ``Name:Value``. - You can repeat this option. For example: ``post -m Color:Blue -m Size:Large`` - - For more information about ACL formats see the documentation: - `ACLs <http://docs.openstack.org/developer/swift/misc.html#acls/>`_. - -Download --------- - - ``download [command-options] [container] [object] [object] [...]`` - - Downloads everything in the account (with ``--all``), or everything in a - container, or a list of objects depending on the arguments given. For a - single object download, you may use the ``-o <filename>`` or ``--output <filename>`` - option to redirect the output to a specific file or ``-`` to - redirect to stdout. The ``--ignore-checksum`` is an option that turn off - checksum validation. You can specify optional headers with the repeatable - cURL-like option ``-H [--header <name:value>]``. - -Delete ------- - - ``delete [command-options] [container] [object] [object] [...]`` - - Deletes everything in the account (with ``--all``), or everything in a - container, or a list of objects depending on the arguments given. Segments - of manifest objects will be deleted as well, unless you specify the - ``--leave-segments`` option. - -Copy ----- - - ``copy [command-options] container object`` - - Copies an object to a new destination or adds user metadata to an object. Depending - on the options supplied, you can preserve existing metadata in contrast to the post - command. The ``--destination`` option sets the copy target destination in the form - ``/container/object``. If not set, the object will be copied onto itself which is useful - for adding metadata. You can use the ``-M`` or ``--fresh-metadata`` option to copy - an object without existing user meta data, and the ``-m`` or ``--meta`` option - to define user meta data items to set in the form ``Name:Value``. You can repeat - this option. For example: ``copy -m Color:Blue -m Size:Large``. - -Capabilities ------------- - - ``capabilities [proxy-url]`` - - Displays cluster capabilities. The output includes the list of the - activated Swift middlewares as well as relevant options for each ones. - Additionally the command displays relevant options for the Swift core. If - the ``proxy-url`` option is not provided, the storage URL retrieved after - authentication is used as ``proxy-url``. - -Tempurl -------- - - ``tempurl [command-options] [method] [time] [path] [key]`` - - Generates a temporary URL for a Swift object. ``method`` option sets an HTTP method to - allow for this temporary URL that is usually ``GET` or ``PUT``. ``time`` option sets - the amount of time the temporary URL will be valid for. - ``time`` can be specified as an integer, denoting the number of seconds - from now on until the URL shall be valid; or, if ``--absolute`` - is passed, the Unix timestamp when the temporary URL will expire. - But beyond that, ``time`` can also be specified as an ISO 8601 timestamp - in one of following formats: - - i) Complete date: YYYY-MM-DD (eg 1997-07-16) - - ii) Complete date plus hours, minutes and seconds: - YYYY-MM-DDThh:mm:ss - (eg 1997-07-16T19:20:30) - - iii) Complete date plus hours, minutes and seconds with UTC designator: - YYYY-MM-DDThh:mm:ssZ - (eg 1997-07-16T19:20:30Z) - - Please be aware that if you don't provide the UTC designator (i.e., Z) - the timestamp is generated using your local timezone. If only a date is - specified, the time part used will equal to ``00:00:00``. - - ``path`` option sets the full path to the Swift object. - Example: ``/v1/AUTH_account/c/o``. ``key`` option is - the secret temporary URL key set on the Swift cluster. To set a key, run - ``swift post -m "Temp-URL-Key: <your secret key>"``. To generate a prefix-based temporary - URL use the ``--prefix-based`` option. This URL will contain the path to the prefix. Do not - forget to append the desired objectname at the end of the path portion (and before the - query portion) before sharing the URL. It is possible to use ISO 8601 UTC timestamps within the - URL by using the ``--iso8601`` option. - -Auth ----- - - ``auth`` - - Display authentication variables in shell friendly format. Command to run to export storage - URL and auth token into ``OS_STORAGE_URL`` and ``OS_AUTH_TOKEN``: ``swift auth``. - Command to append to a runcom file (e.g. ``~/.bashrc``, ``/etc/profile``) for automatic - authentication: ``swift auth -v -U test:tester -K testing``. - -Examples -~~~~~~~~ - -In this section we present some example usage of the ``swift`` CLI. To keep the -examples as short as possible, these examples assume that the relevant authentication -options have been set using environment variables. You can obtain the full list of -commands and options available in the ``swift`` CLI by executing the following: - -.. code-block:: bash - - > swift --help - > swift <command> --help - -Simple examples ---------------- - -List the existing swift containers: - -.. code-block:: bash - - > swift list - - container_1 - -Create a new container: - -.. code-block:: bash - - > swift post TestContainer - -Upload an object into a container: - -.. code-block:: bash - - > swift upload TestContainer testSwift.txt - - testSwift.txt - -List the contents of a container: - -.. code-block:: bash - - > swift list TestContainer - - testSwift.txt - -Copy an object to new destination: - -.. code-block:: bash - - > swift copy -d /DestContainer/testSwift.txt SourceContainer testSwift.txt - - SourceContainer/testSwift.txt copied to /DestContainer/testSwift.txt - -Delete an object from a container: - -.. code-block:: bash - - > swift delete TestContainer testSwift.txt - - testSwift.txt - -Delete a container: - -.. code-block:: bash - - > swift delete TestContainer - - TestContainer - -Display auth related authentication variables in shell friendly format: - -.. code-block:: bash - - > swift auth - - export OS_STORAGE_URL=http://127.0.0.1:8080/v1/AUTH_bf5e63572f7a420a83fcf0aa8c72c2c7 - export OS_AUTH_TOKEN=c597015ae19943a18438b52ef3762e79 - -Download an object from a container: - -.. code-block:: bash - - > swift download TestContainer testSwift.txt - - testSwift.txt [auth 0.028s, headers 0.045s, total 0.045s, 0.002 MB/s] - -.. We need the backslash below in order to indent the note -\ - - .. note:: - - To upload an object to a container, your current working directory must be - where the file is located or you must provide the complete path to the file. - In other words, the --object-name <object-name> is an option that will upload - file and name object to <object-name> or upload directory and use <object-name> as - object prefix. In the case that you provide the complete path of the file, - that complete path will be the name of the uploaded object. - -For example: - -.. code-block:: bash - - > swift upload TestContainer /home/swift/testSwift/testSwift.txt - - home/swift/testSwift/testSwift.txt - - > swift list TestContainer - - home/swift/testSwift/testSwift.txt - -More complex examples ---------------------- - -Swift has a single object size limit of 5GiB. In order to upload files larger -than this, we must create a large object that consists of smaller segments. -The example below shows how to upload a large video file as a static large -object in 1GiB segments: - -.. code-block:: bash - - > swift upload videos --use-slo --segment-size 1G myvideo.mp4 - - myvideo.mp4 segment 8 - myvideo.mp4 segment 4 - myvideo.mp4 segment 2 - myvideo.mp4 segment 7 - myvideo.mp4 segment 0 - myvideo.mp4 segment 1 - myvideo.mp4 segment 3 - myvideo.mp4 segment 6 - myvideo.mp4 segment 5 - myvideo.mp4 - -This command will upload segments to a container named ``videos_segments``, and -create a manifest file describing the entire object in the ``videos`` container. -For more information on large objects, see the documentation `here -<http://docs.openstack.org/developer/swift/overview_large_objects.html>`_. - -.. code-block:: bash - - > swift list videos - - myvideo.mp4 - - > swift list videos_segments - - myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000000 - myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000001 - myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000002 - myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000003 - myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000004 - myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000005 - myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000006 - myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000007 - myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000008 - -Firstly, the key should be set, then generate a temporary URL for a Swift object: - -.. code-block:: bash - - > swift post -m "Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4" - - > swift tempurl GET 6000 /v1/AUTH_bf5e63572f7a420a83fcf0aa8c72c2c7\ - /firstcontainer/clean.sh b3968d0207b54ece87cccc06515a89d4 - - /v1/AUTH_/firstcontainer/clean.sh?temp_url_sig=\ - 9218fc288cc09e5edd857b6a3d43cf2122b906dc&temp_url_expires=1472203614 diff --git a/doc/source/cli/index.rst b/doc/source/cli/index.rst new file mode 100644 index 0000000..1762989 --- /dev/null +++ b/doc/source/cli/index.rst @@ -0,0 +1,958 @@ +==== +CLI +==== + +The ``swift`` tool is a command line utility for communicating with an OpenStack +Object Storage (swift) environment. It allows one to perform several types of +operations. + + +For help on a specific :command:`swift` command, enter: + +.. code-block:: console + + $ swift COMMAND --help + +.. _swift_command_usage: + +swift usage +~~~~~~~~~~~ + +.. code-block:: console + + Usage: swift [--version] [--help] [--os-help] [--snet] [--verbose] + [--debug] [--info] [--quiet] [--auth <auth_url>] + [--auth-version <auth_version> | + --os-identity-api-version <auth_version> ] + [--user <username>] + [--key <api_key>] [--retries <num_retries>] + [--os-username <auth-user-name>] [--os-password <auth-password>] + [--os-user-id <auth-user-id>] + [--os-user-domain-id <auth-user-domain-id>] + [--os-user-domain-name <auth-user-domain-name>] + [--os-tenant-id <auth-tenant-id>] + [--os-tenant-name <auth-tenant-name>] + [--os-project-id <auth-project-id>] + [--os-project-name <auth-project-name>] + [--os-project-domain-id <auth-project-domain-id>] + [--os-project-domain-name <auth-project-domain-name>] + [--os-auth-url <auth-url>] [--os-auth-token <auth-token>] + [--os-storage-url <storage-url>] [--os-region-name <region-name>] + [--os-service-type <service-type>] + [--os-endpoint-type <endpoint-type>] + [--os-cacert <ca-certificate>] [--insecure] + [--os-cert <client-certificate-file>] + [--os-key <client-certificate-key-file>] + [--no-ssl-compression] + <subcommand> [--help] [<subcommand options>] + +**Subcommands:** + +``delete`` + Delete a container or objects within a container. + +``download`` + Download objects from containers. + +``list`` + Lists the containers for the account or the objects + for a container. + +``post`` + Updates meta information for the account, container, + or object; creates containers if not present. + +``copy`` + Copies object, optionally adds meta + +``stat`` + Displays information for the account, container, + or object. + +``upload`` + Uploads files or directories to the given container. + +``capabilities`` + List cluster capabilities. + +``tempurl`` + Create a temporary URL. + +``auth`` + Display auth related environment variables. + +.. _swift_command_options: + +swift optional arguments +~~~~~~~~~~~~~~~~~~~~~~~~ + +``--version`` + show program's version number and exit + +``-h, --help`` + show this help message and exit + +``--os-help`` + Show OpenStack authentication options. + +``-s, --snet`` + Use SERVICENET internal network. + +``-v, --verbose`` + Print more info. + +``--debug`` + Show the curl commands and results of all http queries + regardless of result status. + +``--info`` + Show the curl commands and results of all http queries + which return an error. + +``-q, --quiet`` + Suppress status output. + +``-A AUTH, --auth=AUTH`` + URL for obtaining an auth token. + +``-V AUTH_VERSION, --auth-version=AUTH_VERSION, --os-identity-api-version=AUTH_VERSION`` + Specify a version for authentication. Defaults to + ``env[ST_AUTH_VERSION]``, ``env[OS_AUTH_VERSION]``, + ``env[OS_IDENTITY_API_VERSION]`` or 1.0. + +``-U USER, --user=USER`` + User name for obtaining an auth token. + +``-K KEY, --key=KEY`` + Key for obtaining an auth token. + +``-R RETRIES, --retries=RETRIES`` + The number of times to retry a failed connection. + +``--insecure`` + Allow swiftclient to access servers without having to + verify the SSL certificate. Defaults to + ``env[SWIFTCLIENT_INSECURE]`` (set to 'true' to enable). + +``--no-ssl-compression`` + This option is deprecated and not used anymore. SSL + compression should be disabled by default by the + system SSL library. + +``--prompt`` + Prompt user to enter a password which overrides any password supplied via + ``--key``, ``--os-password`` or environment variables. + +Authentication +~~~~~~~~~~~~~~ + +This section covers the options for authenticating with a swift +object store. The combinations of options required for each authentication +version are detailed below, but are just a subset of those that can be used +to successfully authenticate. These are the most common and recommended +combinations. + +You should obtain the details of your authentication version and credentials +from your storage provider. These details should make it clearer which of the +authentication sections below are most likely to allow you to connect to your +storage account. + +Keystone v3 +----------- + +.. code-block:: bash + + swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \ + --os-project-name project1 --os-project-domain-name domain1 \ + --os-username user --os-user-domain-name domain1 \ + --os-password password list + + swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \ + --os-project-id 0123456789abcdef0123456789abcdef \ + --os-user-id abcdef0123456789abcdef0123456789 \ + --os-password password list + +Manually specifying the options above on the command line can be avoided by +setting the following combinations of environment variables: + +.. code-block:: bash + + ST_AUTH_VERSION=3 + OS_USERNAME=user + OS_USER_DOMAIN_NAME=domain1 + OS_PASSWORD=password + OS_PROJECT_NAME=project1 + OS_PROJECT_DOMAIN_NAME=domain1 + OS_AUTH_URL=https://api.example.com:5000/v3 + + ST_AUTH_VERSION=3 + OS_USER_ID=abcdef0123456789abcdef0123456789 + OS_PASSWORD=password + OS_PROJECT_ID=0123456789abcdef0123456789abcdef + OS_AUTH_URL=https://api.example.com:5000/v3 + +Keystone v2 +----------- + +.. code-block:: bash + + swift --os-auth-url https://api.example.com:5000/v2.0 \ + --os-tenant-name tenant \ + --os-username user --os-password password list + +Manually specifying the options above on the command line can be avoided by +setting the following environment variables: + +.. code-block:: bash + + ST_AUTH_VERSION=2.0 + OS_USERNAME=user + OS_PASSWORD=password + OS_TENANT_NAME=tenant + OS_AUTH_URL=https://api.example.com:5000/v2.0 + +Legacy auth systems +------------------- + +You can configure swift to work with any number of other authentication systems +that we will not cover in this document. If your storage provider is not using +Keystone to provide access tokens, please contact them for instructions on the +required options. It is likely that the options will need to be specified as +below: + +.. code-block:: bash + + swift -A https://api.example.com/v1.0 -U user -K api_key list + +Specifying the options above manually on the command line can be avoided by +setting the following environment variables: + +.. code-block:: bash + + ST_AUTH_VERSION=1.0 + ST_AUTH=https://api.example.com/v1.0 + ST_USER=user + ST_KEY=key + +It is also possible that you need to use a completely separate auth system, in which +case ``swiftclient`` cannot request a token for you. In this case you should make the +authentication request separately and access your storage using the token and +storage URL options shown below: + +.. code-block:: bash + + swift --os-auth-token 6ee5eb33efad4e45ab46806eac010566 \ + --os-storage-url https://10.1.5.2:8080/v1/AUTH_ced809b6a4baea7aeab61a \ + list + +.. note:: + + Leftover environment variables are a common source of confusion when + authorization fails. + +CLI commands +~~~~~~~~~~~~ + +.. _swift_auth: + +Auth +---- + +.. code-block:: console + + Usage: swift auth + +Display authentication variables in shell friendly format. Command to run to export storage +URL and auth token into ``OS_STORAGE_URL`` and ``OS_AUTH_TOKEN``: ``swift auth``. +Command to append to a runcom file (e.g. ``~/.bashrc``, ``/etc/profile``) for automatic +authentication: ``swift auth -v -U test:tester -K testing``. + +.. _swift_stat: + +swift stat +---------- + +.. code-block:: console + + Usage: swift stat [--lh] [--header <header:value>] + [<container> [<object>]] + +Displays information for the account, container, or object depending on +the arguments given (if any). In verbose mode, the storage URL and the +authentication token are displayed as well. + +**Positional arguments:** + +``[container]`` + Name of container to stat from. + +``[object]`` + Name of object to stat. + +**Optional arguments:** + +``--lh`` + Report sizes in human readable format similar to + ls -lh. + +``-H, --header <header:value>`` + Adds a custom request header to use for stat. + +.. _swift_list: + +swift list +---------- + +.. code-block:: console + + Usage: swift list [--long] [--lh] [--totals] [--prefix <prefix>] + [--delimiter <delimiter>] [--header <header:value>] + [<container>] + +Lists the containers for the account or the objects for a container. +The ``-p <prefix>`` or ``--prefix <prefix>`` is an option that will only +list items beginning with that prefix. The ``-d <delimiter>`` or +``--delimiter <delimiter>`` is an option (for container listings only) +that will roll up items with the given delimiter (see `OpenStack Swift +general documentation <https://docs.openstack.org/swift/latest/>` for +what this means). + +The ``-l`` and ``--lh`` options provide more detail, similar to ``ls -l`` +and ``ls -lh``, the latter providing sizes in human readable format +(For example: ``3K``, ``12M``, etc). The latter two switches use more +overhead to retrieve the displayed details, which is directly proportional +to the number of container or objects listed. + +**Positional arguments:** + +``[container]`` + Name of container to list object in. + +**Optional arguments:** + +``-l, --long`` + Long listing format, similar to ls -l. + +``--lh`` + Report sizes in human readable format similar to + ls -lh. + +``-t, --totals`` + Used with -l or --lh, only report totals. + +``-p <prefix>, --prefix <prefix>`` + Only list items beginning with the prefix. + +``-d <delim>, --delimiter <delim>`` + Roll up items with the given delimiter. For containers + only. See OpenStack Swift API documentation for what + this means. + +``-H, --header <header:value>`` + Adds a custom request header to use for listing. + +.. _swift_upload: + +swift upload +------------ + +.. code-block:: console + + Usage: swift upload [--changed] [--skip-identical] [--segment-size <size>] + [--segment-container <container>] [--leave-segments] + [--object-threads <thread>] [--segment-threads <threads>] + [--header <header>] [--use-slo] [--ignore-checksum] + [--object-name <object-name>] + <container> <file_or_directory> [<file_or_directory>] [...] + +Uploads the files and directories specified by the remaining arguments to the +given container. The ``-c`` or ``--changed`` is an option that will only +upload files that have changed since the last upload. The +``--object-name <object-name>`` is an option that will upload a file and +name object to ``<object-name>`` or upload a directory and use ``<object-name>`` +as object prefix. If the file name is "-", client reads content from standard +input. In this case ``--object-name`` is required to set the name of the object +and no other files may be given. The ``-S <size>`` or ``--segment-size <size>`` +and ``--leave-segments`` are options as well (see ``--help`` for more). + +**Positional arguments:** + +``<container>`` + Name of container to upload to. + +``<file_or_directory>`` + Name of file or directory to upload. Specify multiple + times for multiple uploads. + +**Optional arguments:** + +``-c, --changed`` + Only upload files that have changed since the last + upload. + +``--skip-identical`` + Skip uploading files that are identical on both sides. + +``-S, --segment-size <size>`` + Upload files in segments no larger than <size> (in + Bytes) and then create a "manifest" file that will + download all the segments as if it were the original + file. + +``--segment-container <container>`` + Upload the segments into the specified container. If + not specified, the segments will be uploaded to a + <container>_segments container to not pollute the + main <container> listings. + +``--leave-segments`` + Indicates that you want the older segments of manifest + objects left alone (in the case of overwrites). + +``--object-threads <threads>`` + Number of threads to use for uploading full objects. + Default is 10. + +``--segment-threads <threads>`` + Number of threads to use for uploading object segments. + Default is 10. + +``-H, --header <header:value>`` + Adds a customized request header. This option may be + repeated. Example: -H "content-type:text/plain" + -H "Content-Length: 4000". + +``--use-slo`` + When used in conjunction with --segment-size it will + create a Static Large Object instead of the default + Dynamic Large Object. + +``--object-name <object-name>`` + Upload file and name object to <object-name> or upload + dir and use <object-name> as object prefix instead of + folder name. + +``--ignore-checksum`` + Turn off checksum validation for uploads. + + +.. _swift_post: + +swift post +---------- + +.. code-block:: console + + Usage: swift post [--read-acl <acl>] [--write-acl <acl>] [--sync-to <sync-to>] + [--sync-key <sync-key>] [--meta <name:value>] + [--header <header>] + [<container> [<object>]] + +Updates meta information for the account, container, or object depending +on the arguments given. If the container is not found, the ``swiftclient`` +will create it automatically, but this is not true for accounts and +objects. Containers also allow the ``-r <read-acl>`` (or ``--read-acl +<read-acl>``) and ``-w <write-acl>`` (or ``--write-acl <write-acl>``) options. +The ``-m`` or ``--meta`` option is allowed on accounts, containers and objects, +and is used to define the user metadata items to set in the form ``Name:Value``. +You can repeat this option. For example: ``post -m Color:Blue -m Size:Large`` + +For more information about ACL formats see the documentation: +`ACLs <https://docs.openstack.org/swift/latest/misc.html#acls>`_. + +**Positional arguments:** + +``[container]`` + Name of container to post to. + +``[object]`` + Name of object to post. + +**Optional arguments:** + +``-r, --read-acl <acl>`` + Read ACL for containers. Quick summary of ACL syntax: + ``.r:*``, ``.r:-.example.com``, ``.r:www.example.com``, + ``account1`` (v1.0 identity API only), + ``account1:*``, ``account2:user2`` (v2.0+ identity API). + +``-w, --write-acl <acl>`` + Write ACL for containers. Quick summary of ACL syntax: + ``account1`` (v1.0 identity API only), + ``account1:*``, ``account2:user2`` (v2.0+ identity API). + +``-t, --sync-to <sync-to>`` + Sync To for containers, for multi-cluster replication. + +``-k, --sync-key <sync-key>`` + Sync Key for containers, for multi-cluster replication. + +``-m, --meta <name:value>`` + Sets a meta data item. This option may be repeated. + + Example: -m Color:Blue -m Size:Large + +``-H, --header <header:value>`` + Adds a customized request header. + This option may be repeated. + + Example: -H "content-type:text/plain" -H "Content-Length: 4000" + +.. _swift_download: + +swift download +-------------- + +.. code-block:: console + + Usage: swift download [--all] [--marker <marker>] [--prefix <prefix>] + [--output <out_file>] [--output-dir <out_directory>] + [--object-threads <threads>] [--ignore-checksum] + [--container-threads <threads>] [--no-download] + [--skip-identical] [--remove-prefix] + [--header <header:value>] [--no-shuffle] + [<container> [<object>] [...]] + +Downloads everything in the account (with ``--all``), or everything in a +container, or a list of objects depending on the arguments given. For a +single object download, you may use the ``-o <filename>`` or ``--output <filename>`` +option to redirect the output to a specific file or ``-`` to +redirect to stdout. The ``--ignore-checksum`` is an option that turn off +checksum validation. You can specify optional headers with the repeatable +cURL-like option ``-H [--header <name:value>]``. ``--ignore-mtime`` ignores the +``x-object-meta-mtime`` metadata entry on the object (if present) and instead +creates the downloaded files with fresh atime and mtime values. + +**Positional arguments:** + +``<container>`` + Name of container to download from. To download a + whole account, omit this and specify --all. + +``<object>`` + Name of object to download. Specify multiple times + for multiple objects. Omit this to download all + objects from the container. + +**Optional arguments:** + +``-a, --all`` + Indicates that you really want to download + everything in the account. + +``-m, --marker <marker>`` + Marker to use when starting a container or account + download. + +``-p, --prefix <prefix>`` + Only download items beginning with <prefix> + +``-r, --remove-prefix`` + An optional flag for --prefix <prefix>, use this + option to download items without <prefix> + +``-o, --output <out_file>`` + For a single file download, stream the output to + <out_file>. Specifying "-" as <out_file> will + redirect to stdout. + +``-D, --output-dir <out_directory>`` + An optional directory to which to store objects. + By default, all objects are recreated in the current + directory. + +``--object-threads <threads>`` + Number of threads to use for downloading objects. + Default is 10. + +``--container-threads <threads>`` + Number of threads to use for downloading containers. + Default is 10. + +``--no-download`` + Perform download(s), but don't actually write anything + to disk. + +``-H, --header <header:value>`` + Adds a customized request header to the query, like + "Range" or "If-Match". This option may be repeated. + + Example: --header "content-type:text/plain" + +``--skip-identical`` + Skip downloading files that are identical on both + sides. + +``--ignore-checksum`` + Turn off checksum validation for downloads. + +``--no-shuffle`` + By default, when downloading a complete account or + container, download order is randomised in order to + reduce the load on individual drives when multiple + clients are executed simultaneously to download the + same set of objects (e.g. a nightly automated download + script to multiple servers). Enable this option to + submit download jobs to the thread pool in the order + they are listed in the object store. + +.. _swift_delete: + +swift delete +------------ + +.. code-block:: console + + Usage: swift delete [--all] [--leave-segments] + [--object-threads <threads>] + [--container-threads <threads>] + [--header <header:value>] + [<container> [<object>] [...]] + +Deletes everything in the account (with ``--all``), or everything in a +container, or a list of objects depending on the arguments given. Segments +of manifest objects will be deleted as well, unless you specify the +``--leave-segments`` option. + +**Positional arguments:** + +``[<container>]`` + Name of container to delete from. + +``[<object>]`` + Name of object to delete. Specify multiple times + for multiple objects. + +**Optional arguments:** + +``-a, --all`` + Delete all containers and objects. + +``--leave-segments`` + Do not delete segments of manifest objects. + +``-H, --header <header:value>`` + Adds a custom request header to use for deleting + objects or an entire container. + + +``--object-threads <threads>`` + Number of threads to use for deleting objects. + Default is 10. + +``--container-threads <threads>`` + Number of threads to use for deleting containers. + Default is 10. + +.. _swift_copy: + +swift copy +---------- + +.. code-block:: console + + Usage: swift copy [--destination </container/object>] [--fresh-metadata] + [--meta <name:value>] [--header <header>] <container> + <object> [<object>] [...] + +Copies an object to a new destination or adds user metadata to an object. Depending +on the options supplied, you can preserve existing metadata in contrast to the post +command. The ``--destination`` option sets the copy target destination in the form +``/container/object``. If not set, the object will be copied onto itself which is useful +for adding metadata. You can use the ``-M`` or ``--fresh-metadata`` option to copy +an object without existing user meta data, and the ``-m`` or ``--meta`` option +to define user meta data items to set in the form ``Name:Value``. You can repeat +this option. For example: ``copy -m Color:Blue -m Size:Large``. + +**Positional arguments:** + +``<container>`` + Name of container to copy from. + +``<object>`` + Name of object to copy. Specify multiple times for multiple objects + +**Optional arguments:** + +``-d, --destination </container[/object]>`` + The container and name of the destination object. Name + of destination object can be omitted, then will be + same as name of source object. Supplying multiple + objects and destination with object name is invalid. + +``-M, --fresh-metadata`` + Copy the object without any existing metadata, + If not set, metadata will be preserved or appended + +``-m, --meta <name:value>`` + Sets a meta data item. This option may be repeated. + + Example: -m Color:Blue -m Size:Large + +``-H, --header <header:value>`` + Adds a customized request header. This option may be repeated. + + Example: -H "content-type:text/plain" -H "Content-Length: 4000" + +.. _swift_capabilities: + +swift capabilities +------------------ + +.. code-block:: console + + Usage: swift capabilities [--json] [<proxy_url>] + +Displays cluster capabilities. The output includes the list of the +activated Swift middlewares as well as relevant options for each ones. +Additionally the command displays relevant options for the Swift core. If +the ``proxy-url`` option is not provided, the storage URL retrieved after +authentication is used as ``proxy-url``. + +**Optional positional arguments:** + +``<proxy_url>`` + Proxy URL of the cluster to retrieve capabilities. + +``--json`` + Print the cluster capabilities in JSON format. + +.. _swift_tempurl: + +swift tempurl +------------- + +.. code-block:: console + + Usage: swift tempurl [--absolute] [--prefix-based] + <method> <seconds> <path> <key> + +Generates a temporary URL for a Swift object. ``method`` option sets an HTTP method to +allow for this temporary URL that is usually ``GET`` or ``PUT``. ``time`` option sets +the amount of time the temporary URL will be valid for. +``time`` can be specified as an integer, denoting the number of seconds +from now on until the URL shall be valid; or, if ``--absolute`` +is passed, the Unix timestamp when the temporary URL will expire. +But beyond that, ``time`` can also be specified as an ISO 8601 timestamp +in one of following formats: + +i) Complete date: YYYY-MM-DD (e.g. 1997-07-16) + +ii) Complete date plus hours, minutes and seconds: + YYYY-MM-DDThh:mm:ss + (e.g. 1997-07-16T19:20:30) + +iii) Complete date plus hours, minutes and seconds with UTC designator: + YYYY-MM-DDThh:mm:ssZ + (e.g. 1997-07-16T19:20:30Z) + +Please be aware that if you don't provide the UTC designator (i.e., Z) +the timestamp is generated using your local timezone. If only a date is +specified, the time part used will equal to ``00:00:00``. + +``path`` option sets the full path to the Swift object. +Example: ``/v1/AUTH_account/c/o``. ``key`` option is +the secret temporary URL key set on the Swift cluster. To set a key, run +``swift post -m "Temp-URL-Key: <your secret key>"``. To generate a prefix-based temporary +URL use the ``--prefix-based`` option. This URL will contain the path to the prefix. Do not +forget to append the desired objectname at the end of the path portion (and before the +query portion) before sharing the URL. It is possible to use ISO 8601 UTC timestamps within the +URL by using the ``--iso8601`` option. + +**Positional arguments:** + +``<method>`` + An HTTP method to allow for this temporary URL. + Usually 'GET' or 'PUT'. + +``<seconds>`` + The amount of time in seconds the temporary URL will be + valid for; or, if --absolute is passed, the Unix + timestamp when the temporary URL will expire. + +``<path>`` + The full path to the Swift object. + + Example: /v1/AUTH_account/c/o + or: http://saio:8080/v1/AUTH_account/c/o + +``<key>`` + The secret temporary URL key set on the Swift cluster. + To set a key, run 'swift post -m + "Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4"' + +**Optional arguments:** + +``--absolute`` + Interpret the <seconds> positional argument as a Unix + timestamp rather than a number of seconds in the + future. + +``--prefix-based`` + If present, a prefix-based tempURL will be generated. + +Examples +~~~~~~~~ + +In this section we present some example usage of the ``swift`` CLI. To keep the +examples as short as possible, these examples assume that the relevant authentication +options have been set using environment variables. You can obtain the full list of +commands and options available in the ``swift`` CLI by executing the following: + +.. code-block:: bash + + > swift --help + > swift <command> --help + +Simple examples +--------------- + +List the existing swift containers: + +.. code-block:: bash + + > swift list + + container_1 + +Create a new container: + +.. code-block:: bash + + > swift post TestContainer + +Upload an object into a container: + +.. code-block:: bash + + > swift upload TestContainer testSwift.txt + + testSwift.txt + +List the contents of a container: + +.. code-block:: bash + + > swift list TestContainer + + testSwift.txt + +Copy an object to new destination: + +.. code-block:: bash + + > swift copy -d /DestContainer/testSwift.txt SourceContainer testSwift.txt + + SourceContainer/testSwift.txt copied to /DestContainer/testSwift.txt + +Delete an object from a container: + +.. code-block:: bash + + > swift delete TestContainer testSwift.txt + + testSwift.txt + +Delete a container: + +.. code-block:: bash + + > swift delete TestContainer + + TestContainer + +Display auth related authentication variables in shell friendly format: + +.. code-block:: bash + + > swift auth + + export OS_STORAGE_URL=http://127.0.0.1:8080/v1/AUTH_bf5e63572f7a420a83fcf0aa8c72c2c7 + export OS_AUTH_TOKEN=c597015ae19943a18438b52ef3762e79 + +Download an object from a container: + +.. code-block:: bash + + > swift download TestContainer testSwift.txt + + testSwift.txt [auth 0.028s, headers 0.045s, total 0.045s, 0.002 MB/s] + +.. note:: + + To upload an object to a container, your current working directory must be + where the file is located or you must provide the complete path to the file. + In other words, the --object-name <object-name> is an option that will upload + file and name object to <object-name> or upload directory and use <object-name> as + object prefix. In the case that you provide the complete path of the file, + that complete path will be the name of the uploaded object. + +For example: + +.. code-block:: bash + + > swift upload TestContainer /home/swift/testSwift/testSwift.txt + + home/swift/testSwift/testSwift.txt + + > swift list TestContainer + + home/swift/testSwift/testSwift.txt + +More complex examples +--------------------- + +Swift has a single object size limit of 5GiB. In order to upload files larger +than this, we must create a large object that consists of smaller segments. +The example below shows how to upload a large video file as a static large +object in 1GiB segments: + +.. code-block:: bash + + > swift upload videos --use-slo --segment-size 1G myvideo.mp4 + + myvideo.mp4 segment 8 + myvideo.mp4 segment 4 + myvideo.mp4 segment 2 + myvideo.mp4 segment 7 + myvideo.mp4 segment 0 + myvideo.mp4 segment 1 + myvideo.mp4 segment 3 + myvideo.mp4 segment 6 + myvideo.mp4 segment 5 + myvideo.mp4 + +This command will upload segments to a container named ``videos_segments``, and +create a manifest file describing the entire object in the ``videos`` container. +For more information on large objects, see the documentation `here +<https://docs.openstack.org/swift/latest/overview_large_objects.html>`_. + +.. code-block:: bash + + > swift list videos + + myvideo.mp4 + + > swift list videos_segments + + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000000 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000001 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000002 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000003 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000004 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000005 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000006 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000007 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000008 + +Firstly, the key should be set, then generate a temporary URL for a Swift object: + +.. code-block:: bash + + > swift post -m "Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4" + + > swift tempurl GET 6000 /v1/AUTH_bf5e63572f7a420a83fcf0aa8c72c2c7\ + /firstcontainer/clean.sh b3968d0207b54ece87cccc06515a89d4 + + /v1/AUTH_/firstcontainer/clean.sh?temp_url_sig=\ + 9218fc288cc09e5edd857b6a3d43cf2122b906dc&temp_url_expires=1472203614 diff --git a/doc/source/conf.py b/doc/source/conf.py index 3505f13..85dd81e 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -31,8 +31,11 @@ sys.path.insert(0, ROOT) # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.todo', - 'sphinx.ext.coverage', 'oslosphinx'] +extensions = ['sphinx.ext.autodoc', + 'sphinx.ext.doctest', + 'sphinx.ext.todo', + 'sphinx.ext.coverage', + 'openstackdocstheme'] autoclass_content = 'both' autodoc_default_flags = ['members', 'undoc-members', 'show-inheritance'] @@ -50,17 +53,8 @@ source_suffix = '.rst' master_doc = 'index' # General information about the project. -project = u'Swiftclient' copyright = u'2013-2016 OpenStack, LLC.' -# The version info for the project you're documenting, acts as replacement for -# |version| and |release|, also used in various other places throughout the -# built documents. -# -import swiftclient.version -release = swiftclient.version.version_string -version = swiftclient.version.version_string - # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. # language = None @@ -104,7 +98,7 @@ pygments_style = 'sphinx' # The theme to use for HTML and HTML Help pages. Major themes that come with # Sphinx are currently 'default' and 'sphinxdoc'. -#html_theme = 'nature' +html_theme = 'openstackdocs' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the diff --git a/doc/source/index.rst b/doc/source/index.rst index f123b7b..ab05c6b 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -16,7 +16,7 @@ Developer Documentation .. toctree:: :maxdepth: 2 - cli + cli/index service-api client-api @@ -39,17 +39,17 @@ Indices and tables License ~~~~~~~ - Copyright 2013 OpenStack, LLC. +Copyright 2013 OpenStack, LLC. - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 +* http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. diff --git a/doc/source/introduction.rst b/doc/source/introduction.rst index 926b1b9..d6f9819 100644 --- a/doc/source/introduction.rst +++ b/doc/source/introduction.rst @@ -16,41 +16,41 @@ Use Cases Alongside the command line tool, the ``python-swiftclient`` includes two levels of API: - * A low level client API that provides simple Python wrappers around the - various authentication mechanisms and the individual HTTP requests. - * A high level service API that provides methods for performing common - operations in parallel on a thread pool. +* A low level client API that provides simple Python wrappers around the + various authentication mechanisms and the individual HTTP requests. +* A high level service API that provides methods for performing common + operations in parallel on a thread pool. Example use cases: - * Uploading and retrieving data - Use the command line tool if you are simply uploading and downloading - files and directories to and from your filesystem. The command line tool - can be integrated into a shell script to automate tasks. - - * Integrating into an automated Python workflow - Use the ``SwiftService`` API to perform operations offered by the CLI - if your use case requires integration with a Python-based workflow. - This method offers greater control and flexibility over individual object - operations, such as the metadata set on each object. The ``SwiftService`` - class provides methods to perform multiple sets of operations against a - swift object store using a configurable shared thread pool. A single - instance of the ``SwiftService`` class can be shared between multiple - threads in your own code. - - * Developing an application in Python to access a swift object store - Use the ``SwiftService`` API to develop Python applications that use - swift to store and retrieve objects. A ``SwiftService`` instance provides - a configurable thread pool for performing all operations supported by the - CLI. - - * Fine-grained control over threading or the requests being performed - Use the ``Connection`` API if your use case requires fine grained control - over advanced features or you wish to use your own existing threading - model. Examples of advanced features requiring the use of the - ``Connection`` API include creating an SLO manifest that references - already existing objects, or fine grained control over the query strings - supplied with each HTTP request. +* Uploading and retrieving data + Use the command line tool if you are simply uploading and downloading + files and directories to and from your filesystem. The command line tool + can be integrated into a shell script to automate tasks. + +* Integrating into an automated Python workflow + Use the ``SwiftService`` API to perform operations offered by the CLI + if your use case requires integration with a Python-based workflow. + This method offers greater control and flexibility over individual object + operations, such as the metadata set on each object. The ``SwiftService`` + class provides methods to perform multiple sets of operations against a + swift object store using a configurable shared thread pool. A single + instance of the ``SwiftService`` class can be shared between multiple + threads in your own code. + +* Developing an application in Python to access a swift object store + Use the ``SwiftService`` API to develop Python applications that use + swift to store and retrieve objects. A ``SwiftService`` instance provides + a configurable thread pool for performing all operations supported by the + CLI. + +* Fine-grained control over threading or the requests being performed + Use the ``Connection`` API if your use case requires fine grained control + over advanced features or you wish to use your own existing threading + model. Examples of advanced features requiring the use of the + ``Connection`` API include creating an SLO manifest that references + already existing objects, or fine grained control over the query strings + supplied with each HTTP request. Important considerations ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -66,19 +66,19 @@ the proper use case, and not treat the storage like a traditional filesystem. There are two main restrictions to bear in mind when designing an application that uses an object store: - * You cannot rename objects. Due to fact that the name of an object is one - of the factors that determines where the object and its replicas are stored, - renaming would require multiple copies of the data to be moved between - physical storage devices. If you want to rename an object you must upload - to the new location, or make a server side copy request to the new location, - and then delete the original. - - * You cannot modify objects. Objects are stored in multiple locations and - are checked for integrity based on the MD5 sum calculated during - upload. In order to modify the contents of an object, the entire desired - contents must be re-uploaded. In certain special cases it is possible to - work around this restriction using large objects, but no general - file-like access is available to modify a stored object. +* You cannot rename objects. Due to fact that the name of an object is one + of the factors that determines where the object and its replicas are stored, + renaming would require multiple copies of the data to be moved between + physical storage devices. If you want to rename an object you must upload + to the new location, or make a server side copy request to the new location, + and then delete the original. + +* You cannot modify objects. Objects are stored in multiple locations and + are checked for integrity based on the MD5 sum calculated during + upload. In order to modify the contents of an object, the entire desired + contents must be re-uploaded. In certain special cases it is possible to + work around this restriction using large objects, but no general + file-like access is available to modify a stored object. Objects cannot be locked ------------------------ diff --git a/doc/source/service-api.rst b/doc/source/service-api.rst index 8efd43a..cc4dcc2 100644 --- a/doc/source/service-api.rst +++ b/doc/source/service-api.rst @@ -26,10 +26,10 @@ the auth version based on the combination of options specified, but supplying options from multiple different auth versions can cause unexpected behaviour. - .. note:: +.. note:: - Leftover environment variables are a common source of confusion when - authorization fails. + Leftover environment variables are a common source of confusion when + authorization fails. Keystone V3 ~~~~~~~~~~~ @@ -109,17 +109,17 @@ in this dictionary are described below, along with their defaults: Options ~~~~~~~ - ``retries``: ``5`` +``retries``: ``5`` The number of times that the library should attempt to retry HTTP actions before giving up and reporting a failure. - ``container_threads``: ``10`` +``container_threads``: ``10`` - ``object_dd_threads``: ``10`` +``object_dd_threads``: ``10`` - ``object_uu_threads``: ``10`` +``object_uu_threads``: ``10`` - ``segment_threads``: ``10`` +``segment_threads``: ``10`` The above options determine the size of the available thread pools for performing swift operations. Container operations (such as listing a container) operate in the container threads, and a similar pattern @@ -131,86 +131,86 @@ Options ``uu`` and ``dd``. This stands for "upload/update" and "download/delete", and the corresponding actions will be run on separate threads pools. - ``segment_size``: ``None`` +``segment_size``: ``None`` If specified, this option enables uploading of large objects. Should the object being uploaded be larger than 5G in size, this option is mandatory otherwise the upload will fail. This option should be specified as a size in bytes. - ``use_slo``: ``False`` +``use_slo``: ``False`` Used in combination with the above option, ``use_slo`` will upload large objects as static rather than dynamic. Only static large objects provide error checking for the downloaded object, so we recommend this option. - ``segment_container``: ``None`` +``segment_container``: ``None`` Allows the user to select the container into which large object segments will be uploaded. We do not recommend changing this value as it could make locating orphaned segments more difficult in the case of errors. - ``leave_segments``: ``False`` +``leave_segments``: ``False`` Setting this option to true means that when deleting or overwriting a large object, its segments will be left in the object store and must be cleaned up manually. This option can be useful when sharing large object segments between multiple objects in more advanced scenarios, but must be treated with care, as it could lead to ever increasing storage usage. - ``changed``: ``None`` +``changed``: ``None`` This option affects uploads and simply means that those objects which already exist in the object store will not be overwritten if the ``mtime`` and size of the source is the same as the existing object. - ``skip_identical``: ``False`` +``skip_identical``: ``False`` A slightly more thorough case of the above, but rather than ``mtime`` and size uses an object's ``MD5 sum``. - ``yes_all``: ``False`` +``yes_all``: ``False`` This options affects only download and delete, and in each case must be specified in order to download/delete the entire contents of an account. This option has no effect on any other calls. - ``no_download``: ``False`` +``no_download``: ``False`` This option only affects download and means that all operations proceed as normal with the exception that no data is written to disk. - ``header``: ``[]`` +``header``: ``[]`` Used with upload and post operations to set headers on objects. Headers are specified as colon separated strings, e.g. "content-type:text/plain". - ``meta``: ``[]`` +``meta``: ``[]`` Used to set metadata on an object similarly to headers. .. note:: Setting metadata is a destructive operation, so when updating one of many metadata values all desired metadata for an object must be re-applied. - ``long``: ``False`` +``long``: ``False`` Affects only list operations, and results in more metrics being made available in the results at the expense of lower performance. - ``fail_fast``: ``False`` +``fail_fast``: ``False`` Applies to delete and upload operations, and attempts to abort queued tasks in the event of errors. - ``prefix``: ``None`` +``prefix``: ``None`` Affects list operations; only objects with the given prefix will be returned/affected. It is not advisable to set at the service level, as those operations that call list to discover objects on which they should operate will also be affected. - ``delimiter``: ``None`` +``delimiter``: ``None`` Affects list operations, and means that listings only contain results up to the first instance of the delimiter in the object name. This is useful for working with objects containing '/' in their names to simulate folder structures. - ``dir_marker``: ``False`` +``dir_marker``: ``False`` Affects uploads, and allows empty 'pseudofolder' objects to be created when the source of an upload is ``None``. - ``checksum``: ``True`` +``checksum``: ``True`` Affects uploads and downloads. If set check md5 sum for the transfer. - ``shuffle``: ``False`` +``shuffle``: ``False`` When downloading objects, the default behaviour of the CLI is to shuffle lists of objects in order to spread the load on storage drives when multiple clients are downloading the same files to multiple locations (e.g. in the @@ -220,12 +220,12 @@ Options are downloaded in lexically-sorted order. Setting this option to ``True`` gives the same shuffling behaviour as the CLI. - ``destination``: ``None`` +``destination``: ``None`` When copying objects, this specifies the destination where the object will be copied to. The default of None means copy will be the same as source. - ``fresh_metadata``: ``None`` +``fresh_metadata``: ``None`` When copying objects, this specifies that the object metadata on the source will *not* be applied to the destination object - the destination object will have a new fresh set of metadata that includes |