Merge "update to README"

1 files changed, 81 insertions, 234 deletions

diff --git a/README.rst b/README.rst
index 584b87d..1861163 100644
--- a/README.rst
+++ b/README.rst
@@ -2,284 +2,131 @@ PyEClib
 -------
 
 This library provides a simple Python interface for implementing erasure codes
-and is known to work with Python v2.6, 2.7 and 3.x.
+and is known to work with Python v2.6, 2.7 and 3.x. To obtain the best possible
+performance, the library utilizes liberasurecode, which is a C based erasure
+code library.
 
-To obtain the best possible performance, the library utilizes liberasurecode,
-which is a C based erasure code library.  Please let us know if you have any
-issues building or installing (email: kmgreen2@gmail.com or tusharsg@gmail.com).
+PyECLib supports a variety of Erasure Coding backends including the standard
+Reed-Solomon implementations provided by Jerasure [1], liberasurecode [3],
+Intel's ISA-L [4] and Phazr.IO's libphazr.  It also provides support for a flat
+XOR-based encoder and decoder (part of liberasurecode) - a class of HD
+Combination Codes based on "Flat XOR-based erasure codes in storage systems:
+Constructions, efficient recovery, and tradeoffs" in IEEE MSST 2010[2]).
+These codes are well-suited to archival use-cases, have a simple construction
+and require a minimum number of participating disks during single-disk
+reconstruction (think XOR-based LRC code).
 
-PyECLib supports a variety of Erasure Coding backends including the standard Reed
-Soloman implementations provided by Jerasure [1], liberasurecode [3] and Intel
-ISA-L [4].  It also provides support for a flat XOR-based encoder and decoder
-(part of liberasurecode) - a class of HD Combination Codes based on "Flat
-XOR-based erasure codes in storage systems: Constructions, efficient recovery,
-and tradeoffs" in IEEE MSST 2010[2]).  These codes are well-suited to archival
-use-cases, have a simple construction and require a minimum number of
-participating disks during single-disk reconstruction (think XOR-based LRC code).
+-----
 
-Examples of using PyECLib are provided in the "tools" directory:
-
-  Command-line encoder::
-
-      tools/pyeclib_encode.py
-
-  Command-line decoder::
-
-      tools/pyeclib_decode.py
-
-  Utility to determine what is needed to reconstruct missing fragments::
-
-      tools/pyeclib_fragments_needed.py
-
-  A configuration utility to help compare available EC schemes in terms of
-  performance and redundancy::
-
-      tools/pyeclib_conf_tool.py
-
-PyEClib initialization::
-
-    ec_driver = ECDriver(k=<num_encoded_data_fragments>,
-                         m=<num_encoded_parity_fragments>,
-                         ec_type=<ec_scheme>))
-
-Supported ``ec_type`` values:
-
-  * ``liberasurecode_rs_vand`` => Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode [3]
-  * ``jerasure_rs_vand`` => Vandermonde Reed-Solomon encoding, based on Jerasure [1]
-  * ``jerasure_rs_cauchy`` => Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [1]
-  * ``flat_xor_hd_3``, ``flat_xor_hd_4`` => Flat-XOR based HD combination codes, liberasurecode [3]
-  * ``isa_l_rs_vand`` => Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4]
-  * ``isa_l_rs_cauchy`` => Cauchy Reed-Solomon encoding (ISA-L variant) [4]
-  * ``shss`` => NTT Lab Japan's Erasure Coding Library [5]
-  * ``libphazr`` => Phazr.IO's erasure code library with built-in privacy [6]
-
-
-The Python API supports the following functions:
-
-- EC Encode
-
-  Encode N bytes of a data object into k (data) + m (parity) fragments::
-
-      def encode(self, data_bytes)
-
-      input:   data_bytes - input data object (bytes)
-      returns: list of fragments (bytes)
-      throws:
-        ECBackendInstanceNotAvailable - if the backend library cannot be found
-        ECBackendNotSupported - if the backend is not supported by PyECLib (see ec_types above)
-        ECInvalidParameter - if invalid parameters were provided
-        ECOutOfMemory - if the process has run out of memory
-        ECDriverError - if an unknown error occurs
-
-- EC Decode
-
-  Decode between k and k+m fragments into original object::
-
-      def decode(self, fragment_payloads)
-
-      input:   list of fragment_payloads (bytes)
-      returns: decoded object (bytes)
-      throws:
-        ECBackendInstanceNotAvailable - if the backend library cannot be found
-        ECBackendNotSupported - if the backend is not supported by PyECLib (see ec_types above)
-        ECInvalidParameter - if invalid parameters were provided
-        ECOutOfMemory - if the process has run out of memory
-        ECInsufficientFragments - if an insufficient set of fragments has been provided (e.g. not enough)
-        ECInvalidFragmentMetadata - if the fragment headers appear to be corrupted
-        ECDriverError - if an unknown error occurs
+Installation
+===========
 
+Install pre-requisites:
 
-*Note*: ``bytes`` is a synonym to ``str`` in Python 2.6, 2.7.
-In Python 3.x, ``bytes`` and ``str`` types are non-interchangeable and care
-needs to be taken when handling input to and output from the ``encode()`` and
-``decode()`` routines.
+* Python 2.6, 2.7 or 3.x (including development packages), argparse, setuptools
+* liberasurecode v1.3.1 or greater [3]
+* Erasure code backend libraries, gf-complete and Jerasure [1],[2], ISA-L [4], etc
 
+Install dependencies:
 
-- EC Reconstruct
+Debian/Ubuntu hosts::
 
-  Reconstruct "missing_fragment_indexes" using "available_fragment_payloads"::
+    $ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev
+    $ sudo pip install -U bindep -r test-requirements.txt
 
-      def reconstruct(self, available_fragment_payloads, missing_fragment_indexes)
+RHEL/CentOS hosts::
+    
+    $ sudo yum install -y redhat-lsb python2-pip python-devel liberasurecode-devel
+    $ sudo pip install -U bindep -r test-requirements.txt
 
-      input: available_fragment_payloads - list of fragment payloads
-      input: missing_fragment_indexes - list of indexes to reconstruct
-      output: list of reconstructed fragments corresponding to missing_fragment_indexes
-      throws:
-        ECBackendInstanceNotAvailable - if the backend library cannot be found
-        ECBackendNotSupported - if the backend is not supported by PyECLib (see ec_types above)
-        ECInvalidParameter - if invalid parameters were provided
-        ECOutOfMemory - if the process has run out of memory
-        ECInsufficientFragments - if an insufficient set of fragments has been provided (e.g. not enough)
-        ECInvalidFragmentMetadata - if the fragment headers appear to be corrupted
-        ECDriverError - if an unknown error occurs
+If you want to confirm all dependency packages installed successfully, try::
 
+    $ sudo bindep -f bindep.txt
 
-- Minimum parity fragments needed for durability gurantees::
+*Note*: Currently, for Ubuntu, liberasurecode-dev in package repo is older than v1.2.0.
+For CentOS, make sure to install the latest Openstack Cloud SIG repo
+to be able to install the latest available version of liberasurecode-devel.
 
-      def min_parity_fragments_needed(self)
+Install PyECLib::
 
-      NOTE: Currently hard-coded to 1, so this can only be trusted for MDS codes, such as
-            Reed-Solomon.
+    $ sudo python setup.py install
 
-      output: minimum number of additional fragments needed to be synchronously written to tolerate
-              the loss of any one fragment (similar guarantees to 2 out of 3 with 3x replication)
-      throws:
-        ECBackendInstanceNotAvailable - if the backend library cannot be found
-        ECBackendNotSupported - if the backend is not supported by PyECLib (see ec_types above)
-        ECInvalidParameter - if invalid parameters were provided
-        ECOutOfMemory - if the process has run out of memory
-        ECDriverError - if an unknown error occurs
+Run test suite included::
 
+    $ ./.unittests
 
-- Fragments needed for EC Reconstruct
+If the test suite fails because it cannot find any of the shared libraries,
+then you probably need to add /usr/local/lib to the path searched when loading
+libraries.  The best way to do this (on Linux) is to add '/usr/local/lib' to::
 
-  Return the indexes of fragments needed to reconstruct "missing_fragment_indexes"::
+    /etc/ld.so.conf
 
-      def fragments_needed(self, missing_fragment_indexes)
+and then make sure to run::
 
-      input: list of missing_fragment_indexes
-      output: list of fragments needed to reconstruct fragments listed in missing_fragment_indexes
-      throws:
-        ECBackendInstanceNotAvailable - if the backend library cannot be found
-        ECBackendNotSupported - if the backend is not supported by PyECLib (see ec_types above)
-        ECInvalidParameter - if invalid parameters were provided
-        ECOutOfMemory - if the process has run out of memory
-        ECDriverError - if an unknown error occurs
+    $ sudo ldconfig
 
+-----
 
-- Get EC Metadata
+Getting started
+===============
 
-  Return an opaque header known by the underlying library or a formatted header (Python dict)::
-
-      def get_metadata(self, fragment, formatted = 0)
-
-      input: raw fragment payload
-      input: boolean specifying if returned header is opaque buffer or formatted string
-      output: fragment header (opaque or formatted)
-      throws:
-        ECBackendInstanceNotAvailable - if the backend library cannot be found
-        ECBackendNotSupported - if the backend is not supported by PyECLib (see ec_types above)
-        ECInvalidParameter - if invalid parameters were provided
-        ECOutOfMemory - if the process has run out of memory
-        ECDriverError - if an unknown error occurs
-
-- Verify EC Stripe Consistency
-
-  Use opaque buffers from get_metadata() to verify a the consistency of a stripe::
-
-      def verify_stripe_metadata(self, fragment_metadata_list)
-
-      intput: list of opaque fragment headers
-      output: formatted string containing the 'status' (0 is success) and 'reason' if verification fails
-      throws:
-        ECBackendInstanceNotAvailable - if the backend library cannot be found
-        ECBackendNotSupported - if the backend is not supported by PyECLib (see ec_types above)
-        ECInvalidParameter - if invalid parameters were provided
-        ECOutOfMemory - if the process has run out of memory
-        ECDriverError - if an unknown error occurs
-
-
-- Get EC Segment Info
-
-  Return a dict with the keys - segment_size, last_segment_size, fragment_size, last_fragment_size and num_segments::
-
-      def get_segment_info(self, data_len, segment_size)
-
-      input: total data_len of the object to store
-      input: target segment size used to segment the object into multiple EC stripes
-      output: a dict with keys - segment_size, last_segment_size, fragment_size, last_fragment_size and num_segments
-      throws:
-        ECBackendInstanceNotAvailable - if the backend library cannot be found
-        ECBackendNotSupported - if the backend is not supported by PyECLib (see ec_types above)
-        ECInvalidParameter - if invalid parameters were provided
-        ECOutOfMemory - if the process has run out of memory
-        ECDriverError - if an unknown error occurs
-
-
-- Get EC Segment Info given a list of ranges, data length and segment size::
-
-      def get_segment_info_byterange(self, ranges, data_len, segment_size)
-
-      input: byte ranges
-      input: total data_len of the object to store
-      input: target segment size used to segment the object into multiple EC stripes
-      output: (see below)
-      throws:
-        ECBackendInstanceNotAvailable - if the backend library cannot be found
-        ECBackendNotSupported - if the backend is not supported by PyECLib (see ec_types above)
-        ECInvalidParameter - if invalid parameters were provided
-        ECOutOfMemory - if the process has run out of memory
-        ECDriverError - if an unknown error occurs
-
-  Assume a range request is given for an object with segment size 3K and
-  a 1 MB file::
-
-      Ranges = (0, 1), (1, 12), (10, 1000), (0, segment_size-1),
-               (1, segment_size+1), (segment_size-1, 2*segment_size)
-
-  This will return a map keyed on the ranges, where there is a recipe
-  given for each range::
-
-      {
-       (0, 1): {0: (0, 1)},
-       (10, 1000): {0: (10, 1000)},
-       (1, 12): {0: (1, 12)},
-       (0, 3071): {0: (0, 3071)},
-       (3071, 6144): {0: (3071, 3071), 1: (0, 3071), 2: (0, 0)},
-       (1, 3073): {0: (1, 3071), 1: (0,0)}
-      }
-
-
-Quick Start
-
-  Install pre-requisites::
+Examples of using PyECLib are provided in the "tools" directory:
 
-    * Python 2.6, 2.7 or 3.x (including development packages), argparse, setuptools
-    * liberasurecode v1.2.0 or greater [3]
-    * Erasure code backend libraries, gf-complete and Jerasure [1],[2], ISA-L [4] etc
+Command-line encoder::
 
-  An example for ubuntu to install dependency packages::
+    tools/pyeclib_encode.py
 
-      $ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev
-      $ sudo pip install -U bindep -r test-requirements.txt
+Command-line decoder::
 
-  If you want to confirm all dependency packages installed successfully, try::
+    tools/pyeclib_decode.py
 
-      $ sudo bindep -f bindep.txt
+Utility to determine what is needed to reconstruct missing fragments::
 
-  *Note*: currently liberasurecode-dev/liberasurecode-devel in package repo is older than v1.2.0
+    tools/pyeclib_fragments_needed.py
 
-  Install PyECLib::
+A configuration utility to help compare available EC schemes in terms of
+performance and redundancy::
 
-      $ sudo python setup.py install
+    tools/pyeclib_conf_tool.py
 
-  Run test suite included::
+PyEClib initialization::
 
-      $ ./.unittests
+    ec_driver = ECDriver(k=<num_encoded_data_fragments>,
+                         m=<num_encoded_parity_fragments>,
+                         ec_type=<ec_scheme>))
 
-  If all of this works, then you should be good to go.  If not, send us an email!
+Supported ``ec_type`` values:
 
-  If the test suite fails because it cannot find any of the shared libraries,
-  then you probably need to add /usr/local/lib to the path searched when loading
-  libraries.  The best way to do this (on Linux) is to add '/usr/local/lib' to::
+* ``liberasurecode_rs_vand`` => Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode [3]
+* ``jerasure_rs_vand`` => Vandermonde Reed-Solomon encoding, based on Jerasure [1]
+* ``jerasure_rs_cauchy`` => Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [1]
+* ``flat_xor_hd_3``, ``flat_xor_hd_4`` => Flat-XOR based HD combination codes, liberasurecode [3]
+* ``isa_l_rs_vand`` => Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4]
+* ``isa_l_rs_cauchy`` => Cauchy Reed-Solomon encoding (ISA-L variant) [4]
+* ``shss`` => NTT Lab Japan's Erasure Coding Library [5]
+* ``libphazr`` => Phazr.IO's erasure code library with built-in privacy [6]
 
-      /etc/ld.so.conf
+-----
 
-  and then make sure to run::
+Code Maintenance
+================
 
-      $ sudo ldconfig
+This library is currently mainly maintained by the Openstack Swift community.
+For questions or any other help, come ask in #openstack-swift on freenode.
 
+-----
 
 References
+==========
 
- [1] Jerasure, C library that supports erasure coding in storage applications, http://jerasure.org
+[1] Jerasure, C library that supports erasure coding in storage applications, http://jerasure.org
 
- [2] Greenan, Kevin M et al, "Flat XOR-based erasure codes in storage systems", http://www.kaymgee.com/Kevin_Greenan/Publications_files/greenan-msst10.pdf
+[2] Greenan, Kevin M et al, "Flat XOR-based erasure codes in storage systems", http://www.kaymgee.com/Kevin_Greenan/Publications_files/greenan-msst10.pdf
 
- [3] liberasurecode, C API abstraction layer for erasure coding backends, https://github.com/openstack/liberasurecode
+[3] liberasurecode, C API abstraction layer for erasure coding backends, https://github.com/openstack/liberasurecode
 
- [4] Intel(R) Storage Acceleration Library (Open Source Version), https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version
+[4] Intel(R) Storage Acceleration Library (Open Source Version), https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version
 
- [5] Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>, "NTT SHSS Erasure Coding backend"
+[5] Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>, "NTT SHSS Erasure Coding backend"
 
- [6] Jim Cheung <support@phazr.io>, "Phazr.IO libphazr erasure code backend with built-in privacy"
+[6] Jim Cheung <support@phazr.io>, "Phazr.IO libphazr erasure code backend with built-in privacy"