diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/source/cli.rst | 361 | ||||
| -rw-r--r-- | doc/source/client-api.rst | 177 | ||||
| -rw-r--r-- | doc/source/index.rst | 24 | ||||
| -rw-r--r-- | doc/source/introduction.rst | 94 | ||||
| -rw-r--r-- | doc/source/sdk.rst | 48 | ||||
| -rw-r--r-- | doc/source/service-api.rst (renamed from doc/source/apis.rst) | 601 |
6 files changed, 976 insertions, 329 deletions
diff --git a/doc/source/cli.rst b/doc/source/cli.rst index 9527fbf..12de02f 100644 --- a/doc/source/cli.rst +++ b/doc/source/cli.rst @@ -1,29 +1,334 @@ -=== +==== CLI -=== - -Top-level commands -~~~~~~~~~~~~~~~~~~ - -.. TODO - - delete - download - list - post - stat - upload - info/capabilities - tempurl - auth - -Prescriptive examples -~~~~~~~~~~~~~~~~~~~~~ - -.. TODO - - A "Hello World" example - uploading an object - creating a tempurl - listing the contents of a container - downloading an object
\ No newline at end of file +==== + +The ``swift`` tool is a command line utility for communicating with an OpenStack +Object Storage (swift) environment. It allows one to perform several types of +operations. + +Authentication +~~~~~~~~~~~~~~ + +This section covers the options for authenticating with a swift +object store. The combinations of options required for each authentication +version are detailed below, but are just a subset of those that can be used +to successfully authenticate. These are the most common and recommended +combinations. + +You should obtain the details of your authentication version and credentials +from your storage provider. These details should make it clearer which of the +authentication sections below are most likely to allow you to connect to your +storage account. + +Keystone v3 +----------- + +.. code-block:: bash + + swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \ + --os-project-name project1 --os-project-domain-name domain1 \ + --os-username user --os-user-domain-name domain1 \ + --os-password password list + + swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \ + --os-project-id 0123456789abcdef0123456789abcdef \ + --os-user-id abcdef0123456789abcdef0123456789 \ + --os-password password list + +Manually specifying the options above on the command line can be avoided by +setting the following combinations of environment variables: + +.. code-block:: bash + + ST_AUTH_VERSION=3 + OS_USERNAME=user + OS_USER_DOMAIN_NAME=domain1 + OS_PASSWORD=password + OS_PROJECT_NAME=project1 + OS_PROJECT_DOMAIN_NAME=domain1 + OS_AUTH_URL=https://api.example.com:5000/v3 + + ST_AUTH_VERSION=3 + OS_USER_ID=abcdef0123456789abcdef0123456789 + OS_PASSWORD=password + OS_PROJECT_ID=0123456789abcdef0123456789abcdef + OS_AUTH_URL=https://api.example.com:5000/v3 + +Keystone v2 +----------- + +.. code-block:: bash + + swift --os-auth-url https://api.example.com:5000/v2.0 \ + --os-tenant-name tenant \ + --os-username user --os-password password list + +Manually specifying the options above on the command line can be avoided by +setting the following environment variables: + +.. code-block:: bash + + ST_AUTH_VERSION=2.0 + OS_USERNAME=user + OS_PASSWORD=password + OS_TENANT_NAME=tenant + OS_AUTH_URL=https://api.example.com:5000/v2.0 + +Legacy auth systems +------------------- + +You can configure swift to work with any number of other authentication systems +that we will not cover in this document. If your storage provider is not using +Keystone to provide access tokens, please contact them for instructions on the +required options. It is likely that the options will need to be specified as +below: + +.. code-block:: bash + + swift -A https://auth.api.rackspacecloud.com/v1.0 -U user -K api_key list + +Specifying the options above manually on the command line can be avoided by +setting the following environment variables: + +.. code-block:: bash + + ST_AUTH_VERSION=1.0 + ST_AUTH=https://auth.api.rackspacecloud.com/v1.0 + ST_USER=user + ST_KEY=key + +It is also possible that you need to use a completely separate auth system, in which +case ``swiftclient`` cannot request a token for you. In this case you should make the +authentication request separately and access your storage using the token and +storage URL options shown below: + +.. code-block:: bash + + swift --os-auth-token 6ee5eb33efad4e45ab46806eac010566 \ + --os-storage-url https://10.1.5.2:8080/v1/AUTH_ced809b6a4baea7aeab61a \ + list + +.. We need the backslash below in order to indent the note +\ + + .. note:: + + Leftover environment variables are a common source of confusion when + authorization fails. + +CLI commands +~~~~~~~~~~~~ + +Stat +---- + + ``stat [container [object]]`` + + Displays information for the account, container, or object depending on + the arguments given (if any). In verbose mode, the storage URL and the + authentication token are displayed as well. + +List +---- + + ``list [command-options] [container]`` + + Lists the containers for the account or the objects for a container. + The ``-p <prefix>`` or ``--prefix <prefix>`` is an option that will only + list items beginning with that prefix. The ``-d <delimiter>`` or + ``--delimiter <delimiter>`` is an option (for container listings only) + that will roll up items with the given delimiter (see `OpenStack Swift + general documentation <http://docs.openstack.org/developer/swift/>` for + what this means). + + The ``-l`` and ``--lh`` options provide more detail, similar to ``ls -l`` + and ``ls -lh``, the latter providing sizes in human readable format + (For example: ``3K``, ``12M``, etc). The latter two switches use more + overhead to retrieve the displayed details, which is directly proportional + to the number of container or objects listed. + +Upload +------ + + ``upload [command-options] container file_or_directory [file_or_directory] [...]`` + + Uploads the files and directories specified by the remaining arguments to the + given container. The ``-c`` or ``--changed`` is an option that will only + upload files that have changed since the last upload. The + ``--object-name <object-name>`` is an option that will upload a file and + name object to ``<object-name>`` or upload a directory and use ``<object-name>`` + as object prefix. The ``-S <size>`` or ``--segment-size <size>`` and + ``--leave-segments`` are options as well (see ``--help`` for more). + +Post +---- + + ``post [command-options] [container] [object]`` + + Updates meta information for the account, container, or object depending + on the arguments given. If the container is not found, the ``swiftclient`` + will create it automatically, but this is not true for accounts and + objects. Containers also allow the ``-r <read-acl>`` (or ``--read-acl + <read-acl>``) and ``-w <write-acl>`` (or ``--write-acl <write-acl>``) options. + The ``-m`` or ``--meta`` option is allowed on accounts, containers and objects, + and is used to define the user metadata items to set in the form ``Name:Value``. + You can repeat this option. For example: ``post -m Color:Blue -m Size:Large`` + + For more information about ACL formats see the documentation: + `ACLs <http://docs.openstack.org/developer/swift/misc.html#acls/>`_. + +Download +-------- + + ``download [command-options] [container] [object] [object] [...]`` + + Downloads everything in the account (with ``--all``), or everything in a + container, or a list of objects depending on the arguments given. For a + single object download, you may use the ``-o <filename>`` or ``--output <filename>`` + option to redirect the output to a specific file or ``-`` to + redirect to stdout. You can specify optional headers with the repeatable + cURL-like option ``-H [--header <name:value>]``. + +Delete +------ + + ``delete [command-options] [container] [object] [object] [...]`` + + Deletes everything in the account (with ``--all``), or everything in a + container, or a list of objects depending on the arguments given. Segments + of manifest objects will be deleted as well, unless you specify the + ``--leave-segments`` option. + +Capabilities +------------ + + ``capabilities [proxy-url]`` + + Displays cluster capabilities. The output includes the list of the + activated Swift middlewares as well as relevant options for each ones. + Additionally the command displays relevant options for the Swift core. If + the ``proxy-url`` option is not provided, the storage URL retrieved after + authentication is used as ``proxy-url``. + + +Examples +~~~~~~~~ + +In this section we present some example usage of the ``swift`` CLI. To keep the +examples as short as possible, these examples assume that the relevant authentication +options have been set using environment variables. You can obtain the full list of +commands and options available in the ``swift`` CLI by executing the following: + +.. code-block:: bash + + > swift --help + > swift <command> --help + +Simple examples +--------------- + +List the existing swift containers: + +.. code-block:: bash + + > swift list + + container_1 + +Create a new container: + +.. code-block:: bash + + > swift post TestContainer + +Upload an object into a container: + +.. code-block:: bash + + > swift upload TestContainer testSwift.txt + + testSwift.txt + +List the contents of a container: + +.. code-block:: bash + + > swift list TestContainer + + testSwift.txt + +Download an object from a container: + +.. code-block:: bash + + > swift download TestContainer testSwift.txt + + testSwift.txt [auth 0.028s, headers 0.045s, total 0.045s, 0.002 MB/s] + +.. We need the backslash below in order to indent the note +\ + + .. note:: + + To upload an object to a container, your current working directory must be + where the file is located or you must provide the complete path to the file. + In the case that you provide the complete path of the file, that complete + path will be the name of the uploaded object. + +For example: + +.. code-block:: bash + + > swift upload TestContainer /home/swift/testSwift/testSwift.txt + + home/swift/testSwift/testSwift.txt + + > swift list TestContainer + + home/swift/testSwift/testSwift.txt + +More complex examples +--------------------- + +Swift has a single object size limit of 5GiB. In order to upload files larger +than this, we must create a large object that consists of smaller segments. +The example below shows how to upload a large video file as a static large +object in 1GiB segments: + +.. code-block:: bash + + > swift upload videos --use-slo --segment-size 1G myvideo.mp4 + + myvideo.mp4 segment 8 + myvideo.mp4 segment 4 + myvideo.mp4 segment 2 + myvideo.mp4 segment 7 + myvideo.mp4 segment 0 + myvideo.mp4 segment 1 + myvideo.mp4 segment 3 + myvideo.mp4 segment 6 + myvideo.mp4 segment 5 + myvideo.mp4 + +This command will upload segments to a container named ``videos_segments``, and +create a manifest file describing the entire object in the ``videos`` container. +For more information on large objects, see the documentation `here +<http://docs.openstack.org/developer/swift/overview_large_objects.html>`_. + +.. code-block:: bash + + > swift list videos + + myvideo.mp4 + + > swift list videos_segments + + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000000 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000001 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000002 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000003 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000004 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000005 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000006 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000007 + myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000008 diff --git a/doc/source/client-api.rst b/doc/source/client-api.rst new file mode 100644 index 0000000..5677f70 --- /dev/null +++ b/doc/source/client-api.rst @@ -0,0 +1,177 @@ +============================== +The swiftclient.Connection API +============================== + +A low level API that provides methods for authentication and methods that +correspond to the individual REST API calls described in the swift +documentation. + +For usage details see the client docs: :mod:`swiftclient.client`. + +Authentication +-------------- + +This section covers the various combinations of kwargs required when creating +and instance of the ``Connection`` object for communicating with a swift +object store. The combinations of options required for each authentication +version are detailed below, but are +just a subset of those that can be used to successfully authenticate. These +are the most common and recommended combinations. + +Keystone v3 +~~~~~~~~~~~ + +.. code-block:: python + + _authurl = 'http://127.0.0.1:5000/v3/' + _auth_version = '3' + _user = 'tester' + _key = 'testing' + _os_options = { + 'user_domain_name': 'Default', + 'project_domain_name': 'Default', + 'project_name': 'Default' + } + + conn = Connection( + authurl=_authurl, + user=_user, + key=_key, + os_options=_os_options, + auth_version=_auth_version + ) + +.. code-block:: python + + _authurl = 'http://127.0.0.1:5000/v3/' + _auth_version = '3' + _user = 'tester' + _key = 'testing' + _os_options = { + 'user_domain_id': 'Default', + 'project_domain_id': 'Default', + 'project_id': 'Default' + } + + conn = Connection( + authurl=_authurl, + user=_user, + key=_key, + os_options=_os_options, + auth_version=_auth_version + ) + +Keystone v2 +~~~~~~~~~~~ + +.. code-block:: python + + _authurl = 'http://127.0.0.1:5000/v2.0/' + _auth_version = '2' + _user = 'tester' + _key = 'testing' + _tenant_name = 'test' + + conn = Connection( + authurl=_authurl, + user=_user, + key=_key, + tenant_name=_tenant_name, + auth_version=_auth_version + ) + +Legacy Auth +~~~~~~~~~~~ + +.. code-block:: python + + _authurl = 'http://127.0.0.1:8080/' + _auth_version = '1' + _user = 'tester' + _key = 'testing' + _tenant_name = 'test' + + conn = Connection( + authurl=_authurl, + user=_user, + key=_key, + tenant_name=_tenant_name, + auth_version=_auth_version + ) + +Examples +-------- + +In this section we present some simple code examples that demonstrate the usage +of the ``Connection`` API. You can find full details of the options and methods +available to the ``Connection`` API in the docstring generated documentation: +:mod:`swiftclient.client`. + +List the available containers: + +.. code-block:: python + + resp_headers, containers = conn.get_account() + print("Response headers: %s" % resp_headers) + for container in containers: + print(container) + +Create a new container: + +.. code-block:: python + + container = 'new-container' + conn.put_container(container) + resp_headers, containers = conn.get_account() + if container in containers: + print("The container was created") + +Create a new object with the contents of a local text file: + +.. code-block:: python + + container = 'new-container' + with open('local.txt', 'r') as local: + conn.put_object( + container, + 'local_object.txt', + contents=local, + content_type='text/plain' + ) + +Confirm presence of the object: + +.. code-block:: python + + obj = 'local_object.txt' + container = 'new-container' + try: + resp_headers = conn.head_object(container, obj) + print('The object was successfully created') + except ClientException as e: + if e.http_status = '404': + print('The object was not found') + else: + print('An error occurred checking for the existence of the object') + +Download the created object: + +.. code-block:: python + + obj = 'local_object.txt' + container = 'new-container' + resp_headers, obj_contents = conn.get_object(container, obj) + with open('local_copy.txt', 'w') as local: + local.write(obj_contents) + +Delete the created object: + +.. code-block:: python + + obj = 'local_object.txt' + container = 'new-container' + try: + conn.delete_object(container, obj) + print("Successfully deleted the object") + except ClientException as e: + print("Failed to delete the object with error: %s" % e) diff --git a/doc/source/index.rst b/doc/source/index.rst index da16a3c..f123b7b 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -1,18 +1,28 @@ +====================================== Welcome to the python-swiftclient Docs -************************************** +====================================== + +Introduction +~~~~~~~~~~~~ + +.. toctree:: + :maxdepth: 2 + + introduction Developer Documentation -======================= +~~~~~~~~~~~~~~~~~~~~~~~ .. toctree:: :maxdepth: 2 - apis cli - sdk + service-api + client-api + Code-Generated Documentation -============================ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. toctree:: :maxdepth: 2 @@ -20,14 +30,14 @@ Code-Generated Documentation swiftclient Indices and tables -================== +~~~~~~~~~~~~~~~~~~ * :ref:`genindex` * :ref:`modindex` * :ref:`search` License -======= +~~~~~~~ Copyright 2013 OpenStack, LLC. diff --git a/doc/source/introduction.rst b/doc/source/introduction.rst new file mode 100644 index 0000000..926b1b9 --- /dev/null +++ b/doc/source/introduction.rst @@ -0,0 +1,94 @@ +============ +Introduction +============ + +Where to Start? +~~~~~~~~~~~~~~~ + +The ``python-swiftclient`` project comprises a command line tool and two +separate APIs for accessing swift programmatically. Choosing the most +appropriate method for a given use case is the first problem a user needs to +solve. + +Use Cases +--------- + +Alongside the command line tool, the ``python-swiftclient`` includes two +levels of API: + + * A low level client API that provides simple Python wrappers around the + various authentication mechanisms and the individual HTTP requests. + * A high level service API that provides methods for performing common + operations in parallel on a thread pool. + +Example use cases: + + * Uploading and retrieving data + Use the command line tool if you are simply uploading and downloading + files and directories to and from your filesystem. The command line tool + can be integrated into a shell script to automate tasks. + + * Integrating into an automated Python workflow + Use the ``SwiftService`` API to perform operations offered by the CLI + if your use case requires integration with a Python-based workflow. + This method offers greater control and flexibility over individual object + operations, such as the metadata set on each object. The ``SwiftService`` + class provides methods to perform multiple sets of operations against a + swift object store using a configurable shared thread pool. A single + instance of the ``SwiftService`` class can be shared between multiple + threads in your own code. + + * Developing an application in Python to access a swift object store + Use the ``SwiftService`` API to develop Python applications that use + swift to store and retrieve objects. A ``SwiftService`` instance provides + a configurable thread pool for performing all operations supported by the + CLI. + + * Fine-grained control over threading or the requests being performed + Use the ``Connection`` API if your use case requires fine grained control + over advanced features or you wish to use your own existing threading + model. Examples of advanced features requiring the use of the + ``Connection`` API include creating an SLO manifest that references + already existing objects, or fine grained control over the query strings + supplied with each HTTP request. + +Important considerations +~~~~~~~~~~~~~~~~~~~~~~~~ + +This section covers some important considerations, helpful hints, and things to +avoid when integrating an object store into your workflow. + +An object store is not a filesystem +----------------------------------- + +It cannot be stressed enough that your usage of the object store should reflect +the proper use case, and not treat the storage like a traditional filesystem. +There are two main restrictions to bear in mind when designing an application +that uses an object store: + + * You cannot rename objects. Due to fact that the name of an object is one + of the factors that determines where the object and its replicas are stored, + renaming would require multiple copies of the data to be moved between + physical storage devices. If you want to rename an object you must upload + to the new location, or make a server side copy request to the new location, + and then delete the original. + + * You cannot modify objects. Objects are stored in multiple locations and + are checked for integrity based on the MD5 sum calculated during + upload. In order to modify the contents of an object, the entire desired + contents must be re-uploaded. In certain special cases it is possible to + work around this restriction using large objects, but no general + file-like access is available to modify a stored object. + +Objects cannot be locked +------------------------ + +There is no mechanism to perform a combination of reading the +data/metadata from an object and writing an update to that data/metadata in an +atomic way. Any user with access to a container could update the contents or +metadata associated with an object at any time. + +Workflows that assume that no updates have been made since the last read of an +object should be discouraged. Enabling a workflow of this type requires an +external object locking mechanism and/or cooperation between all clients +accessing the data. diff --git a/doc/source/sdk.rst b/doc/source/sdk.rst deleted file mode 100644 index aa15250..0000000 --- a/doc/source/sdk.rst +++ /dev/null @@ -1,48 +0,0 @@ -=== -SDK -=== - -Where to start? -~~~~~~~~~~~~~~~ - -.. TODO - - when to use SwiftService - when to use client.py - -SwiftService classes and methods -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. TODO - - docs for each method (autogen from docstrings?) - -Client classes and methods -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. TODO - - docs for each method (autogen from docstrings?) - -Guidelines for writing an app -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. TODO - - auth - how to use various features - when to use various features - pooling connections - concurrency - retries - -Prescriptive examples -~~~~~~~~~~~~~~~~~~~~~ - -.. TODO - - A "Hello World" example - connecting - uploading an object - uploading a directory -
\ No newline at end of file diff --git a/doc/source/apis.rst b/doc/source/service-api.rst index 935b4a4..7d65fd1 100644 --- a/doc/source/apis.rst +++ b/doc/source/service-api.rst @@ -1,63 +1,93 @@ -====================== -python-swiftclient API -====================== +================================ +The swiftclient.SwiftService API +================================ -The python-swiftclient includes two levels of API. A low level client API that -provides simple python wrappers around the various authentication mechanisms, -the individual HTTP requests, and a high level service API that provides -methods for performing common operations in parallel on a thread pool. +A higher-level API aimed at allowing developers an easy way to perform multiple +operations asynchronously using a configurable thread pool. Documentation for +each service method call can be found here: :mod:`swiftclient.service`. -This document aims to provide guidance for choosing between these APIs and -examples of usage for the service API. +Authentication +-------------- +This section covers the various options for authenticating with a swift +object store. The combinations of options required for each authentication +version are detailed below. Once again, these are just a subset of those that +can be used to successfully authenticate, but they are the most common and +recommended. -Important Considerations -~~~~~~~~~~~~~~~~~~~~~~~~ +The relevant authentication options are presented as python dictionaries that +should be added to any other options you are supplying to your ``SwiftService`` +instance. As indicated in the python code, you can also set these options as +environment variables that will be loaded automatically if the relevant option +is not specified. -This section covers some important considerations, helpful hints, and things -to avoid when integrating an object store into your workflow. +The ``SwiftService`` authentication attempts to automatically select +the auth version based on the combination of options specified, but +supplying options from multiple different auth versions can cause unexpected +behaviour. -An Object Store is not a filesystem ------------------------------------ + .. note:: -.. important:: + Leftover environment variables are a common source of confusion when + authorization fails. - It cannot be stressed enough that your usage of the object store should reflect - the use case, and not treat the storage like a filesystem. +Keystone V3 +~~~~~~~~~~~ -There are 2 main restrictions to bear in mind here when designing your use of the object -store: +.. code-block:: python -#. Objects cannot be renamed due to the way in which objects are stored and - references by the object store. This usually requires multiple copies of - the data to be moved between physical storage devices. - As a result, a move operation is not provided. If the user wants to move an - object they must re-upload to the new location and delete the - original. -#. Objects cannot be modified. Objects are stored in multiple locations and are - checked for integrity based on the ``MD5 sum`` calculated during upload. - Object creation is a 1-shot event, and in order to modify the contents of an - object the entire new contents must be re-uploaded. In certain special cases - it is possible to work around this restriction using large objects, but no - general file-like access is available to modify a stored object. + { + ... + "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '3' + "os_username": environ.get('OS_USERNAME'), + "os_password": environ.get('OS_PASSWORD'), + "os_project_name": environ.get('OS_PROJECT_NAME'), + "os_project_domain_name": environ.get('OS_PROJECT_DOMAIN_NAME'), + "os_auth_url": environ.get('OS_AUTH_URL'), + ... + } +.. code-block:: python -The swiftclient.Connection API -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + { + ... + "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '3' + "os_username": environ.get('OS_USERNAME'), + "os_password": environ.get('OS_PASSWORD'), + "os_project_id": environ.get('OS_PROJECT_ID'), + "os_project_domain_id": environ.get('OS_PROJECT_DOMAIN_ID'), + "os_auth_url": environ.get('OS_AUTH_URL'), + ... + } -A low level API that provides methods for authentication and methods that -correspond to the individual REST API calls described in the swift -documentation. +Keystone V2 +~~~~~~~~~~~ -For usage details see the client docs: :mod:`swiftclient.client`. +.. code-block:: python + { + ... + "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '2.0' + "os_username": environ.get('OS_USERNAME'), + "os_password": environ.get('OS_PASSWORD'), + "os_tenant_name": environ.get('OS_TENANT_NAME'), + "os_auth_url": environ.get('OS_AUTH_URL'), + ... + } -The swiftclient.SwiftService API -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Legacy Auth +~~~~~~~~~~~ -A higher level API aimed at allowing developers an easy way to perform multiple -operations asynchronously using a configurable thread pool. Documentation for each -service method call can be found here: :mod:`swiftclient.service`. +.. code-block:: python + + { + ... + "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '1.0' + "auth": environ.get('ST_AUTH'), + "user": environ.get('ST_USER'), + "key": environ.get('ST_KEY'), + ... + } Configuration ------------- @@ -77,7 +107,7 @@ passed to the ``SwiftService`` during initialisation. The options available in this dictionary are described below, along with their defaults: Options -^^^^^^^ +~~~~~~~ ``retries``: ``5`` The number of times that the library should attempt to retry HTTP @@ -192,51 +222,8 @@ source code for ``python-swiftclient``. Each ``SwiftService`` method also allows for an optional dictionary to override those specified at init time, and the appropriate docstrings show which options modify each method's behaviour. -Authentication -~~~~~~~~~~~~~~ - -This section covers the various options for authenticating with a swift -object store. The combinations of options required for each authentication -version are detailed below. - -Version 1.0 Auth ----------------- - - ``auth_version``: ``environ.get('ST_AUTH_VERSION')`` - - ``auth``: ``environ.get('ST_AUTH')`` - - ``user``: ``environ.get('ST_USER')`` - - ``key``: ``environ.get('ST_KEY')`` - - -Version 2.0 and 3.0 Auth ------------------------- - - ``auth_version``: ``environ.get('ST_AUTH_VERSION')`` - - ``os_username``: ``environ.get('OS_USERNAME')`` - - ``os_password``: ``environ.get('OS_PASSWORD')`` - - ``os_tenant_name``: ``environ.get('OS_TENANT_NAME')`` - - ``os_auth_url``: ``environ.get('OS_AUTH_URL')`` - -As is evident from the default values, if these options are not set explicitly -in the options dictionary, then they will default to the values of the given -environment variables. The ``SwiftService`` authentication automatically selects -the auth version based on the combination of options specified, but -having options from different auth versions can cause unexpected behaviour. - - .. note:: - - Leftover environment variables are a common source of confusion when - authorization fails. - -Operation Return Values -~~~~~~~~~~~~~~~~~~~~~~~ +Available Operations +-------------------- Each operation provided by the service API may raise a ``SwiftError`` or ``ClientException`` for any call that fails completely (or a call which @@ -371,32 +358,14 @@ operation was not successful, and will include the keys below: } Example -------- +^^^^^^^ The code below demonstrates the use of ``stat`` to retrieve the headers for a given list of objects in a container using 20 threads. The code creates a -mapping from object name to headers. - -.. code-block:: python +mapping from object name to headers which is then pretty printed to the log. - import logging - - from swiftclient.service import SwiftService - - logger = logging.getLogger() - _opts = {'object_dd_threads': 20} - with SwiftService(options=_opts) as swift: - container = 'container1' - objects = [ 'object_%s' % n for n in range(0,100) ] - header_data = {} - stats_it = swift.stat(container=container, objects=objects) - for stat_res in stats_it: - if stat_res['success']: - header_data[stat_res['object']] = stat_res['headers'] - else: - logger.error( - 'Failed to retrieve stats for %s' % stat_res['object'] - ) +.. literalinclude:: ../../examples/stat.py + :language: python List ~~~~ @@ -456,55 +425,38 @@ dictionary as described below: } Example -------- +^^^^^^^ The code below demonstrates the use of ``list`` to list all items in a container that are over 10MiB in size: -.. code-block:: python - - container = 'example_container' - minimum_size = 10*1024**2 - with SwiftService() as swift: - try: - stats_parts_gen = swift.list(container=container) - for stats in stats_parts_gen: - if stats["success"]: - for item in stats["listing"]: - i_size = int(item["bytes"]) - if i_size > minimum_size: - i_name = item["name"] - i_etag = item["hash"] - print( - "%s [size: %s] [etag: %s]" % - (i_name, i_size, i_etag) - ) - else: - raise stats["error"] - except SwiftError as e: - output_manager.error(e.value) +.. literalinclude:: ../../examples/list.py + :language: python Post ~~~~ Post can be called against an account, container or list of objects in order to -update the metadata attached to the given items. Each element of the object list -may be a plain string of the object name, or a ``SwiftPostObject`` that -allows finer control over the options applied to each of the individual post -operations. In the first two cases a single dictionary is returned containing the -results of the operation, and in the case of a list of objects being supplied, -an iterator over the results generated for each object post is returned. If the -given container or account does not exist, the ``post`` method will raise a -``SwiftError``. - -.. When a string is given for the object name, the options - -Successful metadata update results are dictionaries as described below: +update the metadata attached to the given items. In the first two cases a single +dictionary is returned containing the results of the operation, and in the case +of a list of objects being supplied, an iterator over the results generated for +each object post is returned. + +Each element of the object list may be a plain string of the object name, or a +``SwiftPostObject`` that allows finer control over the options and metadata +applied to each of the individual post operations. When a string is given for +the object name, the options and metadata applied are a combination of those +supplied to the call to ``post()`` and the defaults of the ``SwiftService`` +object. + +If the given container or account does not exist, the ``post`` method will +raise a ``SwiftError``. Successful metadata update results are dictionaries as +described below: .. code-block:: python { - 'action': <'post_account'|<'post_container'>|'post_object'>, + 'action': <'post_account'|'post_container'|'post_object'>, 'success': True, 'container': <container>, 'object': <object>, @@ -513,28 +465,87 @@ Successful metadata update results are dictionaries as described below: } .. note:: - Updating user metadata keys will not only add any specified keys, but will also remove user metadata that has previously been set. This means that each time user metadata is updated, the complete set of desired key-value pairs must be specified. +Example +^^^^^^^ +The code below demonstrates the use of ``post`` to set an archive folder in a +given container to expire after a 24 hour delay: -.. Example -.. ------- +.. literalinclude:: ../../examples/post.py + :language: python -.. TBD +Download +~~~~~~~~ + +Download can be called against an entire account, a single container, or a list +of objects in a given container. Each element of the object list is a string +detailing the full name of an object to download. + +In order to download the full contents of an entire account, you must set the +value of ``yes_all`` to ``True`` in the ``options`` dictionary supplied to +either the ``SwiftService`` instance or the call to ``download``. + +If the given container or account does not exist, the ``download`` method will +raise a ``SwiftError``, otherwise an iterator over the results generated for +each object download is returned. + +See :mod:`swiftclient.service.SwiftService.download` for docs generated from the +method docstring. + +For each successfully downloaded object, the results returned by the iterator +will be a dictionary as described below (results are not returned for completed +container or object segment downloads): + +.. code-block:: python + + { + 'action': 'download_object', + 'container': <container>, + 'object': <object name>, + 'success': True, + 'path': <local path to downloaded object>, + 'pseudodir': <if true, the download created an empty directory>, + 'start_time': <time download started>, + 'end_time': <time download completed>, + 'headers_receipt': <time the headers from the object were retrieved>, + 'auth_end_time': <time authentication completed>, + 'read_length': <bytes_read>, + 'attempts': <attempt count>, + 'response_dict': <HTTP response details> + } -.. Download -.. ~~~~~~~~ +Any failure uploading an object will return a failure dictionary as described +below: -.. TBD +.. code-block:: python -.. Example -.. ------- + { + 'action': 'download_object', + 'container': <container>, + 'object': <object name>, + 'success': False, + 'path': <local path of the failed download>, + 'pseudodir': <if true, the failed download was an empty directory>, + 'attempts': <attempt count>, + 'error': <error>, + 'traceback': <trace>, + 'error_timestamp': <timestamp>, + 'response_dict': <HTTP response details> + } -.. TBD +Example +^^^^^^^ + +The code below demonstrates the use of ``download`` to download all PNG images +from a dated archive folder in a given container: + +.. literalinclude:: ../../examples/download.py + :language: python Upload ~~~~~~ @@ -550,7 +561,7 @@ the upload are those supplied to the call to ``upload``. Constructing a ``SwiftUploadObject`` allows the user to supply an object name for the uploaded file, and modify the options used by ``upload`` at the -granularity of invidivual files. +granularity of individual files. If the given container or account does not exist, the ``upload`` method will raise a ``SwiftError``, otherwise an iterator over the results generated for @@ -622,97 +633,195 @@ below: } Example -------- +^^^^^^^ The code below demonstrates the use of ``upload`` to upload all files and -folders in ``/tmp``, and renaming each object by replacing ``/tmp`` in the -object or directory marker names with ``temporary-objects``: +folders in a given directory, and rename each object by replacing the root +directory name with 'my-<d>-objects', where <d> is the name of the uploaded +directory: + +.. literalinclude:: ../../examples/upload.py + :language: python + +Delete +~~~~~~ + +Delete can be called against an account or a container to remove the containers +or objects contained within them. Each call to ``delete`` returns an iterator +over results of each resulting sub-request. + +If the number of requested delete operations is large and the target swift +cluster is running the bulk middleware, the call to ``SwiftService.delete`` will +make use of bulk operations and the returned result iterator will return +``bulk_delete`` results rather than individual ``delete_object``, +``delete_container`` or ``delete_segment`` results. + +See :mod:`swiftclient.service.SwiftService.delete` for docs generated from the +method docstring. + +For each successfully deleted container, object or segment, the results returned +by the iterator will be a dictionary as described below: + +.. code-block:: python + + { + 'action': <'delete_object'|'delete_segment'>, + 'container': <container>, + 'object': <object name>, + 'success': True, + 'attempts': <attempt count>, + 'response_dict': <HTTP response details> + } + + { + 'action': 'delete_container', + 'container': <container>, + 'success': True, + 'response_dict': <HTTP response details>, + 'attempts': <attempt count> + } + + { + 'action': 'bulk_delete', + 'container': <container>, + 'objects': <[objects]>, + 'success': True, + 'attempts': <attempt count>, + 'response_dict': <HTTP response details> + } + +Any failure in a delete operation will return a failure dictionary as described +below: .. code-block:: python - _opts['object_uu_threads'] = 20 - with SwiftService(options=_opts) as swift, OutputManager() as out_manager: - try: - # Collect all the files and folders in '/tmp' - objs = [] - dir_markers = [] - dir = '/tmp': - for (_dir, _ds, _fs) in walk(f): - if not (_ds + _fs): - dir_markers.append(_dir) - else: - objs.extend([join(_dir, _f) for _f in _fs]) - - # Now that we've collected all the required files and dir markers - # build the ``SwiftUploadObject``s for the call to upload - objs = [ - SwiftUploadObject( - o, object_name=o.replace( - '/tmp', 'temporary-objects', 1 - ) - ) for o in objs - ] - dir_markers = [ - SwiftUploadObject( - None, object_name=d.replace( - '/tmp', 'temporary-objects', 1 - ), options={'dir_marker': True} - ) for d in dir_markers - ] - - # Schedule uploads on the SwiftService thread pool and iterate - # over the results - for r in swift.upload(container, objs + dir_markers): - if r['success']: - if 'object' in r: - out_manager.print_msg(r['object']) - elif 'for_object' in r: - out_manager.print_msg( - '%s segment %s' % (r['for_object'], - r['segment_index']) - ) - else: - error = r['error'] - if r['action'] == "create_container": - out_manager.warning( - 'Warning: failed to create container ' - "'%s'%s", container, msg - ) - elif r['action'] == "upload_object": - out_manager.error( - "Failed to upload object %s to container %s: %s" % - (container, r['object'], error) - ) - else: - out_manager.error("%s" % error) - - except SwiftError as e: - out_manager.error(e.value) - -.. Delete -.. ~~~~~~ -.. Do we want to hide this section until it is complete? - -.. TBD - -.. Example -.. ------- - -.. Do we want to hide this section until it is complete? - -.. TBD - -.. Capabilities -.. ~~~~~~~~~~~~ - -.. Do we want to hide this section until it is complete? - -.. TBD - -.. Example -.. ------- - -.. Do we want to hide this section until it is complete? - -.. TBD + { + 'action': ('delete_object'|'delete_segment'), + 'container': <container>, + 'object': <object name>, + 'success': False, + 'attempts': <attempt count>, + 'error': <error>, + 'traceback': <trace>, + 'error_timestamp': <timestamp>, + 'response_dict': <HTTP response details> + } + + { + 'action': 'delete_container', + 'container': <container>, + 'success': False, + 'error': <error>, + 'traceback': <trace>, + 'error_timestamp': <timestamp>, + 'response_dict': <HTTP response details>, + 'attempts': <attempt count> + } + + { + 'action': 'bulk_delete', + 'container': <container>, + 'objects': <[objects]>, + 'success': False, + 'attempts': <attempt count>, + 'error': <error>, + 'traceback': <trace>, + 'error_timestamp': <timestamp>, + 'response_dict': <HTTP response details> + } + +Example +^^^^^^^ + +The code below demonstrates the use of ``delete`` to remove a given list of +objects from a specified container. As the objects are deleted the transaction +id of the relevant request is printed along with the object name and number +of attempts required. By printing the transaction id, the printed operations +can be easily linked to events in the swift server logs: + +.. literalinclude:: ../../examples/delete.py + :language: python + +Capabilities +~~~~~~~~~~~~ + +Capabilities can be called against an account or a particular proxy URL in +order to determine the capabilities of the swift cluster. These capabilities +include details about configuration options and the middlewares that are +installed in the proxy pipeline. + +See :mod:`swiftclient.service.SwiftService.capabilities` for docs generated from +the method docstring. + +For each successful call to list capabilities, a result dictionary will be +returned with the contents described below: + + { + 'action': 'capabilities', + 'timestamp': <time of the call>, + 'success': True, + 'capabilities': <dictionary containing capability details> + } + +The contents of the capabilities dictionary contain the core swift capabilities +under the key ``swift``, all other keys show the configuration options for +additional middlewares deployed in the proxy pipeline. An example capabilities +dictionary is given below: + +.. code-block:: python + + { + 'account_quotas': {}, + 'bulk_delete': { + 'max_deletes_per_request': 10000, + 'max_failed_deletes': 1000 + }, + 'bulk_upload': { + 'max_containers_per_extraction': 10000, + 'max_failed_extractions': 1000 + }, + 'container_quotas': {}, + 'container_sync': {'realms': {}}, + 'formpost': {}, + 'keystoneauth': {}, + 'slo': { + 'max_manifest_segments': 1000, + 'max_manifest_size': 2097152, + 'min_segment_size': 1048576 + }, + 'swift': { + 'account_autocreate': True, + 'account_listing_limit': 10000, + 'allow_account_management': True, + 'container_listing_limit': 10000, + 'extra_header_count': 0, + 'max_account_name_length': 256, + 'max_container_name_length': 256, + 'max_file_size': 5368709122, + 'max_header_size': 8192, + 'max_meta_count': 90, + 'max_meta_name_length': 128, + 'max_meta_overall_size': 4096, + 'max_meta_value_length': 256, + 'max_object_name_length': 1024, + 'policies': [ + {'default': True, 'name': 'Policy-0'} + ], + 'strict_cors_mode': False, + 'version': '2.2.2' + }, + 'tempurl': { + 'methods': ['GET', 'HEAD', 'PUT'] + } + } + +Example +^^^^^^^ + +The code below demonstrates the us of ``capabilities`` to determine if the +Swift cluster supports static large objects, and if so, the maximum number of +segments that can be described in a single manifest file, along with the +size restrictions on those objects: +.. literalinclude:: ../../examples/capabilities.py + :language: python |