summaryrefslogtreecommitdiff
path: root/src/erasurecode_preprocessing.c
Commit message (Collapse)AuthorAgeFilesLines
* Fix decode realloc bitmap segfaultfix-realloc-bmKota Tsuyuzaki2015-07-151-8/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When using an instance with k + m parameter more than 32 (e.g. k=30, m=20), decoding process might free invalid fragments memory passed from an upper application and it might cause double free corruption on upper application layer. That is because some of realloc_bm calculation like as follows might make invalid handling to free memory. e.g. (for reproducing): - k=30, m=20 - available fragments 0, 1~49 (i.e. fragment 1 dropped) passed in. and then after decoding... - liberasurecode frees the realloc memory for fragment 1 *AND 33 (1 + 32)*! When realloc_bm = 2 (i.e. wants to free only data[1]), that if-condition results in as folows: The result of (realloc_bm & 1 << i): i = 0 -> 0 i = 1 -> 2 i = 2~32 -> 0 i = 33 -> 2 (overflowing!!!) This overflowing makes liberasurecode to free the memory for fragment 33, though the memory must not be freed. To prevent this corruption, this patch makes the base integer as 64bit and liberasurecode to raise an Exception when k + m > 64 at initialization. (i.e. we cannot use more than 64 fragments with current realloc_bm because it is an instance based on int64)
* This fixes a bug discovered when passing a new insufficient fragments exceptionKevin Greenan2015-07-121-1/+1
| | | | | | | | | | up through Python. The code that preprocesses decoded fragments to see if it can simply concat the data fragments instead of decodeing was not properly deduping fragments, which leads to a failed assertion. This properly dedups fragments in the fragments_to_string function.
* This is the fix for Issue #13:Kevin Greenan2015-04-271-1/+15
| | | | | | | | | | | | | | https://bitbucket.org/tsg-/liberasurecode/issue/13/decode-fails-for-many-cases-when-m-k This fix includes: 1.) Proper buffer allocation for the 'missing_idxs' structure, which was not allocating enough space when k > m. 2.) Checks to use header fields of parity fragments during decode when *no* data fragments are available. 3.) Fixed the unit tests to properly handle the case where k <= m. 4.) Extended the unit test framework to support multiple tests per backend 5.) Added tests for all RS implementations: (4,8), (4,4), (10,10)
* Fix invalid metadata handlingKota Tsuyuzaki2015-03-301-0/+3
| | | | | | | | | On the current code, get_fragment_partition might touch the invlid memory area with minus index (that means a invalid header) and it causes segmentation fault. This fixes it to handle the minus index as a EBADHEADER and then no segmentaition fault appeared on the case.
* Sanitize fragments_to_string() errorcodes, add tests for frags w/o fmetadataTushar Gohad2015-03-281-8/+8
| | | | Addresses issue#10
* Improve error code returnsTushar Gohad2015-03-081-0/+4
|
* Apply EINSUFFFRAGS to reconstructKota Tsuyuzaki2015-03-061-1/+1
|
* Rename metadata_adder on backend_commonKota Tsuyuzaki2015-03-051-1/+1
| | | | This patch renames the "metadata_adder" variable to "backend_metadata_size"
* Enable to get fragment_len includes metadata_adderKota Tsuyuzaki2015-02-271-17/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch allows to get correct fragment size includes metadata_adder. Current implementaion automatically allocates extra bytes for the metadata_adder in alloc_buffer, and then, no information about the extra bytes will be returned to the api caller side. It's too confusable because callers couldn't know how size they assumes as the fragment size. To be easy to find out the size infomation, this patch adds "frag_adder_size" variable into fragment metadata and also some functions to get fragment size. The definitions of these size infomation are here, fragment_meta: - size-> raw data size used to encode/fragment_to_string - frag_adder_size-> metadata_adder of backend specification And the definitions of functions are here, - get_fragment_size: -> return sizeof(fragument_header) + size + frag_adder_size - get_fragment_buffer_size: -> return size + frag_adder_size - get_fragment_payload_size: -> return size By using these function above, users could get the size information directly from fragments. It results in enabling to return fragment_len to the caller side easily.
* Move backend metadata adding to fragment allocationKota Tsuyuzaki2015-02-271-6/+11
| | | | | | | | | | | | | | On the first consideration[1], metadata_adder is defined as a extra byte size for "each" fragment. However, current implementation is an element affects to data_len. (i.e. aligned_data_size for original segment data) We should make metadata_adder to be a fixed value against to each fragment, otherwise the extra bytes for the fragments will have flexible length depends on "K". It will be quite complex for backend implementation to know "How large bytes the raw data size is", "How large bytes the backend could use as extra bytes for each fragment". 1: https://bitbucket.org/tsg-/liberasurecode/commits/032b57d9b1c7aadc547fccbacf88af786c9067e7?at=master
* Fix get_fragment_partition return valueKota Tsuyuzaki2015-02-271-2/+3
| | | | | | | | When num_missing is over than the num of parities (i.e. > m), get_fragment_partition should return -1 as an error code. This patch fixes it and adds a test called "test_get_fragment_partition" into liberasurecode_test.c.
* Small fix to check the return code when malloc'ing temporary buffers.Mark Storer2014-10-011-0/+8
|
* Add doxygen config. Update README, copyrights.Tushar Gohad2014-09-011-1/+1
| | | | Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
* Split metadata handling into own routine, add crc32 supportTushar Gohad2014-08-071-9/+0
| | | | Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
* In the erasurecode_preprocessing.c routines for decode/reconstruct, clearlyEric Lambert2014-07-311-13/+0
| | | | | | distinguish between the fragment and the data/parity buffers, ie maintain data, parity pointer arrays separate from the fragment pointers.
* Parity Fragment Headers now contain original data length (without themEric Lambert2014-07-241-0/+1
| | | | | | the fragements_to_string function called by decode bails because of size mismatch). Also update simple encode/decode test to pass all fragments to decode.
* Fix simple encode_decode unit test and remove debugging prints.Kevin Greenan2014-07-221-10/+10
|
* Add encode postprocessing, checksum helpersTushar Gohad2014-07-211-6/+13
| | | | Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
* API to get total fragment sizeTushar Gohad2014-07-211-7/+7
| | | | Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
* Make int types consistent across APITushar Gohad2014-07-181-1/+3
| | | | Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
* Move data/parity return buffer allocs back to mainTushar Gohad2014-07-181-16/+4
| | | | Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
* Move encode() helpers to common preprocessing codeTushar Gohad2014-07-171-16/+100
| | | | Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
* Fix build breakage...Kevin Greenan2014-07-191-1/+1
|
* Wire-in the decoding preprocessing...Kevin Greenan2014-07-191-11/+26
| | | | This compiles, but has not been tested!
* Add preprocessing for decode and reconstruct.Kevin Greenan2014-07-191-0/+270
I'll fill-in the details of decode and reconstruct in a separate commit.
View Lorry Controller Status