summaryrefslogtreecommitdiff
path: root/src/erasurecode.c
Commit message (Collapse)AuthorAgeFilesLines
* Merge "Remove get_chksum to hush warnings"Zuul2019-02-141-3/+3
|\
| * Remove get_chksum to hush warningsPete Zaitcev2019-02-131-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This popped up because Fedora mandates warning-free builds, and get_chksum triggers a warning because it returns an unaligned pointer (it is so analigned, static analysis in the compiler can detect it). The easiest fix is to remove it altogether. We think it should be safe, because: - The function is not listed in any headers - Its counterpart is called "set_checksum", not "set_chksum" - PyECLib does not use it We also hush some doxygen warnings about wrong comments. Change-Id: Ie5bc736f912706e0ffd507765abd24e7f4761233
* | Make our alt crc32 more portablePete Zaitcev2019-02-131-2/+1
|/ | | | | | | | | | | | | | | | | | | Apparently the author of our old crc32 assumed that shifting an int to the right sign-extends, which is not always the case. Result is, building and running make test on s390x fails. The fix is to force a sign-extension using the "xor 0x80; sub 0x80" trick. N.B. This does not cause a compatibility problem, because by a miracle the "broken" crc32_alt was actually computing a stock crc32, same that zlib has. Therefore, if someone, somewhere, ran a Swift cluster on s390x with erasure coding policy, the data in it is already stored with zlib checksums, as we do it now anyway. This fix only permits the tests pass, which used the bad data sample from x86. Change-Id: Ibd5e4e6c02be00540a9648cc7e0f8efda275bf3f Related-Change: Ib5ea2a830c7c23d66bf2ca404a3eb84ad00c5bc5 Related-Bug: 1666320
* Allow reading of little-endian frags on big-endianTim Burke2018-09-111-15/+45
| | | | | | | | ... and vice-versa. We'll fix up frag header values for our output parameter from liberasurecode_get_fragment_metadata but otherwise avoid manipulating the in-memory fragment much. Change-Id: Idd6833bdea60e27c9a0148ee28b4a2c1070be148
* Merge "Negative data or parity args are invalid"Jenkins2017-09-131-0/+2
|\
| * Negative data or parity args are invalidTim Burke2017-09-131-0/+2
| | | | | | | | | | | | | | While we're at it, tighten up some test_create_backend_invalid_args assertions. Change-Id: Id6c70cdb2d86580280ededc3ec6ec648c6cb7d57
* | Un-inline get/set_metatdata_chksumTim Burke2017-07-061-7/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Each was only really used in one place, they had some strange return types, and recent versions of clang on OS X would refuse to compile with erasurecode_helpers.c:531:26: error: taking address of packed member 'metadata_chksum' of class or structure 'fragment_header_s' may result in an unaligned pointer value [-Werror,-Waddress-of-packed-member] return (uint32_t *) &header->metadata_chksum; ^~~~~~~~~~~~~~~~~~~~~~~ We don't really *care* about the pointer; we just want the value! Change-Id: I8a5e42312948a75f5dd8b23b6f5ccfa7bd22eb1d
* | Use zlib for CRC-32Tim Burke2017-07-061-3/+18
|/ | | | | | | | | | | | | | | | Previously, we had our own CRC that was almost but not quite like zlib's implementation. However, * it hasn't been subjected to the same rigor with regard to error-detection properties and * it may not even get used, depending upon whether zlib happens to get loaded before or after liberasurecode. Now, we'll use zlib's CRC-32 when writing new frags, while still tolerating frags that were created with the old implementation. Change-Id: Ib5ea2a830c7c23d66bf2ca404a3eb84ad00c5bc5 Closes-Bug: 1666320
* Stop using ceill() to compute padded data sizePete Zaitcev2017-06-061-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The well-known idiom to compute a required number of data blocks of size B to contain data of length d is: (d + (B-1))/B The code we use, with ceill(), computes the same value, but does it in an unorthodox way. This makes a reviewer to doubt himself and even run tests to make sure we're really computing the obvious thing. Apropos the reviewer confusion, the code in Phazr.IO looks weird. It uses (word_size - hamming_distance) to compute the necessary number of blocks... but then returns the amount of memory needed to store blocks of a different size (word_size). We left all of it alone and return exactly the same values that the old computation returned. All these computations were the only thing in the code that used -lm, so drop that too. Coincidentially, this patch solves the crash of distro-built packages of liberasurecode (see Red Hat bug #1454543). But it's a side effect. Expect a proper patch soon. Change-Id: Ib297f6df304abf5ca8c27d3392b1107a525e0be0
* Merge "Add Phazr.IO libphazr backend to liberasurecode"Jenkins2017-03-011-3/+9
|\
| * Add Phazr.IO libphazr backend to liberasurecodeJim Cheung2017-02-281-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, there are several implementations of erasure codes that are available within OpenStack Swift. Most, if not all, of which are based on the Reed Solomon coding algorithm. Phazr.IO’s Erasure Coding technology uses a patented algorithm which are significantly more efficient and improves the speed of coding, decoding and reconstruction. In addition, Phazr.IO Erasure Code use a non-systematic algorithm which provides data protection at rest and in transport without the need to use encryption. Please contact support@phazr.io for more info on our technology. Change-Id: I4e40d02a8951e38409ad3c604c5dd6f050fa7ea0
* | Fix valgrind-check and memory leakKota Tsuyuzaki2017-02-201-0/+7
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Can you believe that we ware testing the memory leak with valgrind to just *bash scripts* instead of actual binaries on liberasurecode_test and libec_slap? That is why we cannot find such an easy memory leak[1] at the gate. Now this patch enable to run the valgrind against to the binaries. With this fix, we found various memory leak at liberasurecode_test as follows and this patch also fixes them: - If we create fake fragments, we're responsible for freeing all of the frags as well as the array holding the pointers to the frags. - If we allocate any space, we're responsible for freeing it. - If we create an EC descriptor, we're responsible for destroying it. - If we create a fragment or skip array, we're responsible for freeing it. - If that happens inside a loop, we're responsible for doing it *inside that same loop*. In addition to the test fix, this patch fixes following memory leaks at the code which is affected to other users (pyeclib, OpenStack Swift) * Refuse to decode fragments that aren't even long enough to include fragment headers. * Fix a small memory leak in the builtin rs_vand implementation. Closes-Bug: #1665242 Co-Authored-By: Tim Burke <tim@swiftstack.com> 1: https://review.openstack.org/#/c/431812 Change-Id: I96f124e4e536bbd7544208acc084de1cda5c19b2
* Merge "Add get_version functionality to liberasurecode lib"Jenkins2016-12-081-0/+10
|\
| * Add get_version functionality to liberasurecode libKota Tsuyuzaki2016-12-041-0/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | Currently, we have liberasurecode version info in the header and pyeclib is using the info to detect the version. However it's a bit painful because it requires to rebuild pyeclib c code for you to see the actual installed version. This addition for liberasurecode_get_version enables caller to get the version integer from compiled shared library file (.so) and it will rescure to re-compiled operation from pyeclib. Change-Id: I8161ea7da3b069e83c93e11cb41ce12fa60c6f32
* | Merge "ISA-L Cauchy support"Jenkins2016-12-071-0/+2
|\ \
| * | ISA-L Cauchy supportKota Tsuyuzaki2016-12-061-0/+2
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is for supporting ISA-L cauchy based matrix. The difference from isa_l_rs_vand is only the matrix to use the encode/decode calculation. As a known issue, isa_l_rs_vand backend has constraint for the combinations of the available fragment to be able to decode/reconstuct. (See related change in detail) To avoid the constraint, this patch adds another isa-l backend to use cauchy matrix and keep the backward compatibility, this is in another isa_l_rs_cauchy namespace. For implementation consieration, the code is almost same except the matrix generation fucntion so that this patch makes isa_l_common.c file for gathering common fucntions like init/encode/decode/reconstruct. And then the common init funciton takes an extra args "gen_matrix_func_name" for entry point to load the fucntion by dlsym from isa-l .so file. Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Related-Change: Icee788a0931fe692fe0de31fabc4ba450e338a87 Change-Id: I6eb150d9d0c3febf233570fa7729f9f72df2e9be
* | Merge "Fix reconstruct to return an error when memory allocation failed"Jenkins2016-11-241-0/+3
|\ \ | |/ |/|
| * Fix reconstruct to return an error when memory allocation failedKota Tsuyuzaki2016-10-111-0/+3
| | | | | | | | | | | | | | | | | | | | | | As like current code, if we do go jump to just "out" section when memory allocation failed, liberaurecode/PyECLib will report it as success. However, in fact, the returned binary is not the one reconstructed. This patch fixes liberasurecode_reconstruct to return error code (ENOMEM) if either malloc or memset for working space is failed. Change-Id: I7bfc481c7a9bbc64288760df2dc6053c57657b41
* | Merge "Fix liberasurecode skipping a bunch of invalid_args tests"Jenkins2016-11-091-0/+3
|\ \
| * | Fix liberasurecode skipping a bunch of invalid_args testsKota Tsuyuzaki2016-11-041-0/+3
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since the commit a01b1818c874a65d1d1fb8f11ea441e9d3e18771, we have been to able to test some parameters for each tests. However, instead, the NULL which means the end of the test parameters causes skipping a bunch of unit tests which doesn't take args (e.g. test_create_backend_invalid_args). That is the worse because we have not tested anymore for the tests since the commit. This patch fixes to make it tested and more, fix a minor bug for a case that corrupted header incomming to get_metadata. Closes-Bug: #1634403 Change-Id: Ib99a8aa6032f02d0c7d1ab94b8da1ebfd9047d74
* | Merge "Fix clang compile time error"Jenkins2016-11-071-1/+1
|\ \
| * | Fix clang compile time errorEric Lambert2016-06-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | Removed unnecessary check that metadata index was not negative. Check is unneeded because type is unsigned and as such can not be negative. Change-Id: I72eddf78b25b3ff9bedbd596095f04be855c3817 Closes-Bug: 1587954
* | | Merge "Fix a typo in the erasurecode file"Jenkins2016-10-181-1/+1
|\ \ \ | |_|/ |/| |
| * | Fix a typo in the erasurecode filegengchc22016-08-301-1/+1
| |/ | | | | | | | | | | there is a spelling error,"instace" should be "instance" Change-Id: Iddba9e334bf55cc1e7d5cb397db5f3dfc5aac584
* | Add get_by_desc ret val handling to get_fragment_sizeKota Tsuyuzaki2016-09-071-0/+2
|/ | | | | | | | | | As well as any other callers, libersurecode_get_fragment_size should handle the return value of liberasurecode_get_backend_instance_by_desc. Otherwise, get_by_desc can return NULL and it causes an invalid memory access in librerasurecode_get_fragment_size. Change-Id: I489f8b5d049610863b5e0b477b6ff70ead245b55
* Fragment metadata checksumming supportfrag_metadata_checksumTushar Gohad2016-03-101-27/+70
|
* Add NULL instance check to backend_open()Tushar Gohad2016-03-101-0/+2
|
* Add a method to check if a backend is present.Timur Alperovich2016-03-081-19/+32
| | | | | | | Uses dlopen to check if a backend is present. This may be used by consumers who need to check which backends are present on a system. Issue #23
* Check frag idx validity when verifying frag metadataTushar Gohad2016-02-141-0/+5
| | | | ... also add related test code
* Revert "Log to syslog and stderr by default"v1.1.1Tushar Gohad2015-12-041-1/+1
| | | | This reverts commit 21ed77fed2e31ec5183460a6bd9ff4592140b409.
* Split helpers.h include for backward compatibilityTushar Gohad2015-09-221-0/+1
| | | | | | | Users of liberasurecode <= 1.0.7 used alloc/free helpers (which they shouldn't have). This change is to make sure we are still able to those older revs of programs and they work with newer liberasurecode.
* Eliminate erasurecode_stdinc.h dependency on log.hTushar Gohad2015-09-181-0/+2
|
* Enforce the (k + m < 32) limit, add unit testsKota Tsuyuzaki2015-08-181-0/+6
|
* This will prevent the backends from reconstructing a specific index, when it issimilar_to_issue_63Kevin Greenan2015-06-231-0/+29
| | | | | | | specified as "available" by the caller. I feel that only buggy code would do this... NOTE: In the future, we should return an error when this happens.
* Rename rs_vand_internal to liberasurecode_rs_vandTushar Gohad2015-06-221-2/+2
|
* Plugging new internal RS backend into liberasurecode.Kevin Greenan2015-06-181-0/+2
|
* This is the fix for Issue #13:Kevin Greenan2015-04-271-2/+2
| | | | | | | | | | | | | | https://bitbucket.org/tsg-/liberasurecode/issue/13/decode-fails-for-many-cases-when-m-k This fix includes: 1.) Proper buffer allocation for the 'missing_idxs' structure, which was not allocating enough space when k > m. 2.) Checks to use header fields of parity fragments during decode when *no* data fragments are available. 3.) Fixed the unit tests to properly handle the case where k <= m. 4.) Extended the unit test framework to support multiple tests per backend 5.) Added tests for all RS implementations: (4,8), (4,4), (10,10)
* Fix invalid metadata handlingKota Tsuyuzaki2015-03-301-4/+1
| | | | | | | | | On the current code, get_fragment_partition might touch the invlid memory area with minus index (that means a invalid header) and it causes segmentation fault. This fixes it to handle the minus index as a EBADHEADER and then no segmentaition fault appeared on the case.
* Sanitize fragments_to_string() errorcodes, add tests for frags w/o fmetadataTushar Gohad2015-03-281-1/+4
| | | | Addresses issue#10
* Log to syslog and stderr by defaultTushar Gohad2015-03-131-1/+1
|
* Reallow 0 byte encodesTushar Gohad2015-03-111-6/+0
|
* Use enum value when handling shssKota Tsuyuzaki2015-03-091-1/+1
|
* Fix memory leak by alloc internal_payloadKota Tsuyuzaki2015-03-091-12/+14
| | | | | | shss always needs to decode but fragments_to_string will alloc internal_payload as a decoded data. It causes duplicated memory allocation and memory leak.
* Improve error code returnsTushar Gohad2015-03-081-17/+17
|
* Rename metadata_adder on backend_commonKota Tsuyuzaki2015-03-051-1/+1
| | | | This patch renames the "metadata_adder" variable to "backend_metadata_size"
* Add liberasurecode_get_fragment_size functionKota Tsuyuzaki2015-02-271-0/+10
| | | | | | | | | | | | | | | For get_segment_info function of PyECLib, liberasurecode should support get_fragment_size function because if pyeclib and liberasurecode have the calculation of fragment size each other, it might cause the size mismatch (i.e. it'll be a bug) in the future development work. This patch introduces liberasurecode_get_fragment_size function to return the fragment_size calculated at liberasurecode accoring to specified backend descriptor. It really usefull to help caller knows how large size it have to expect and all pyeclib has to do for retrieving fragment_size will be just calling the liberasurecode_get_fragment_size function on get_segment_info.
* Enable to get fragment_len includes metadata_adderKota Tsuyuzaki2015-02-271-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch allows to get correct fragment size includes metadata_adder. Current implementaion automatically allocates extra bytes for the metadata_adder in alloc_buffer, and then, no information about the extra bytes will be returned to the api caller side. It's too confusable because callers couldn't know how size they assumes as the fragment size. To be easy to find out the size infomation, this patch adds "frag_adder_size" variable into fragment metadata and also some functions to get fragment size. The definitions of these size infomation are here, fragment_meta: - size-> raw data size used to encode/fragment_to_string - frag_adder_size-> metadata_adder of backend specification And the definitions of functions are here, - get_fragment_size: -> return sizeof(fragument_header) + size + frag_adder_size - get_fragment_buffer_size: -> return size + frag_adder_size - get_fragment_payload_size: -> return size By using these function above, users could get the size information directly from fragments. It results in enabling to return fragment_len to the caller side easily.
* Move backend metadata adding to fragment allocationKota Tsuyuzaki2015-02-271-3/+3
| | | | | | | | | | | | | | On the first consideration[1], metadata_adder is defined as a extra byte size for "each" fragment. However, current implementation is an element affects to data_len. (i.e. aligned_data_size for original segment data) We should make metadata_adder to be a fixed value against to each fragment, otherwise the extra bytes for the fragments will have flexible length depends on "K". It will be quite complex for backend implementation to know "How large bytes the raw data size is", "How large bytes the backend could use as extra bytes for each fragment". 1: https://bitbucket.org/tsg-/liberasurecode/commits/032b57d9b1c7aadc547fccbacf88af786c9067e7?at=master
* Ensure fragment pointers passed to cleanupKota Tsuyuzaki2015-02-271-1/+8
| | | | | | | | | | | | | | | | | | | | | | | This patch achieves a couple of things as follows: - Undoing the liberasurecode_encode_cleanup specification to expect "fragment" pointers as its arguments. - Ensuring liberasurecode_encode to pass "fratment" pointers to liberasurecode_encode_cleanup. liberasurecode_encode_cleanup is used also in pyeclib so that it is expected that the argument pointers (i.e. encoded_data and encoded_parity) should be the collection of the heads of "fragment" pointers. However, when the backend encode fails, liberasurecode keeps "data" pointers behind of fragment_header, and then, goes to "out:" statement to cleanup its memories. It causes invalid pointer failure. This patch adds a translation function from "data" pointers to "fragment" pointers and ensure liberasurecode_encode to pass correct pointers to libersurecode_encode_cleanup.
* Add NTT backend called "shss"ntt_backendKota Tsuyuzaki2015-02-101-31/+35
| | | | | | | | | | | | | | This introduces a new plug-able backend called "shss" made by Nippon Telegraph and Telephone corporation (NTT). Note that this produces a just plug-in to shss erasure coding binary so that users have to install a shss binary (i.e. libshss.so) aside from liberasurecode when using shss. Please contact us if you are insterested in the NTT backend (welcome!): Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp> Co-Author: Ryuta Kon <kon.ryuta@po.ntts.co.jp>
View Lorry Controller Status