summaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
* Add build-time option to suffix library names sent to dlopenTim Burke2021-10-269-18/+18
| | | | | | This is useful when repacking libraries for python wheels, for example. Change-Id: Ie7b36584de5054c14a9b77d87a5c5fa5cc7a3719
* Fix underflow in flat_xor_hd codeTim Burke2021-06-281-5/+7
| | | | | Change-Id: I9102f9883e979862557bd33958b2d116795b3169 Closes-Bug: #1726816
* Be willing to write fragments with legacy crcTim Burke2020-09-302-3/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ...if users *really* want to. They opt-in at run time by setting LIBERASURECODE_WRITE_LEGACY_CRC=1 in the environment; leaving it unset, set to an empty string, or set to the string "0" continues to write zlib crcs. UpgradeImpact ============= This option is intended to allow a smooth upgrade from liberasurecode 1.5.0 and earlier in a system with multiple readers and writers: * Before upgrade, ensure the environment variable is set on all nodes. This will be ignored by earlier versions. * Upgrade liberasurecode on each node in the system, restarting any services that use it. Every node continues writing CRCs that are still usable by nodes that have not yet upgraded. * Now that every node is capable of reading zlib CRCs, remove the environment variable from each node to start writing new CRCs. If you are already using 1.6.0 or later, just upgrade normally. Closes-Bug: #1886088 Closes-Bug: #1867937 Related-Bug: #1666320 Needed-By: https://review.opendev.org/#/c/739164/ Change-Id: I9adfbe631a2dddc592fd08f8a325f3e8331b92f1
* Merge "Remove get_chksum to hush warnings"Zuul2019-02-142-18/+5
|\
| * Remove get_chksum to hush warningsPete Zaitcev2019-02-132-18/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This popped up because Fedora mandates warning-free builds, and get_chksum triggers a warning because it returns an unaligned pointer (it is so analigned, static analysis in the compiler can detect it). The easiest fix is to remove it altogether. We think it should be safe, because: - The function is not listed in any headers - Its counterpart is called "set_checksum", not "set_chksum" - PyECLib does not use it We also hush some doxygen warnings about wrong comments. Change-Id: Ie5bc736f912706e0ffd507765abd24e7f4761233
* | Make our alt crc32 more portablePete Zaitcev2019-02-132-3/+3
|/ | | | | | | | | | | | | | | | | | | Apparently the author of our old crc32 assumed that shifting an int to the right sign-extends, which is not always the case. Result is, building and running make test on s390x fails. The fix is to force a sign-extension using the "xor 0x80; sub 0x80" trick. N.B. This does not cause a compatibility problem, because by a miracle the "broken" crc32_alt was actually computing a stock crc32, same that zlib has. Therefore, if someone, somewhere, ran a Swift cluster on s390x with erasure coding policy, the data in it is already stored with zlib checksums, as we do it now anyway. This fix only permits the tests pass, which used the bad data sample from x86. Change-Id: Ibd5e4e6c02be00540a9648cc7e0f8efda275bf3f Related-Change: Ib5ea2a830c7c23d66bf2ca404a3eb84ad00c5bc5 Related-Bug: 1666320
* Allow reading of little-endian frags on big-endianTim Burke2018-09-111-15/+45
| | | | | | | | ... and vice-versa. We'll fix up frag header values for our output parameter from liberasurecode_get_fragment_metadata but otherwise avoid manipulating the in-memory fragment much. Change-Id: Idd6833bdea60e27c9a0148ee28b4a2c1070be148
* Merge "flat_xor_hd: better validate args"Jenkins2017-09-131-1/+1
|\
| * flat_xor_hd: better validate argsTim Burke2017-09-131-1/+1
| | | | | | | | | | | | Previously, we'd segfault with args like k=1, m=3, hd=3. Change-Id: I8a1d92f388c54fe4f5b357eaf85a27e6b334df11
* | Merge "Negative data or parity args are invalid"Jenkins2017-09-131-0/+2
|\ \ | |/ |/|
| * Negative data or parity args are invalidTim Burke2017-09-131-0/+2
| | | | | | | | | | | | | | While we're at it, tighten up some test_create_backend_invalid_args assertions. Change-Id: Id6c70cdb2d86580280ededc3ec6ec648c6cb7d57
* | Un-inline get/set_metatdata_chksumTim Burke2017-07-063-51/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Each was only really used in one place, they had some strange return types, and recent versions of clang on OS X would refuse to compile with erasurecode_helpers.c:531:26: error: taking address of packed member 'metadata_chksum' of class or structure 'fragment_header_s' may result in an unaligned pointer value [-Werror,-Waddress-of-packed-member] return (uint32_t *) &header->metadata_chksum; ^~~~~~~~~~~~~~~~~~~~~~~ We don't really *care* about the pointer; we just want the value! Change-Id: I8a5e42312948a75f5dd8b23b6f5ccfa7bd22eb1d
* | Use zlib for CRC-32Tim Burke2017-07-064-9/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, we had our own CRC that was almost but not quite like zlib's implementation. However, * it hasn't been subjected to the same rigor with regard to error-detection properties and * it may not even get used, depending upon whether zlib happens to get loaded before or after liberasurecode. Now, we'll use zlib's CRC-32 when writing new frags, while still tolerating frags that were created with the old implementation. Change-Id: Ib5ea2a830c7c23d66bf2ca404a3eb84ad00c5bc5 Closes-Bug: 1666320
* | Stop pretending to support SSE4-optimized CRC-32CTim Burke2017-07-061-29/+0
|/ | | | | | | | | It isn't the CRC we want, and we never used it anyway. While we may define INTEL_SSE41 or INTEL_SSE42 if CPU seems to support it, we've never defined INTEL_SSE4. Change-Id: I04e1dd6458ccde58a0a2f3f4d6947569a31e9697 Partial-Bug: 1666320
* Stop using ceill() to compute padded data sizePete Zaitcev2017-06-063-7/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The well-known idiom to compute a required number of data blocks of size B to contain data of length d is: (d + (B-1))/B The code we use, with ceill(), computes the same value, but does it in an unorthodox way. This makes a reviewer to doubt himself and even run tests to make sure we're really computing the obvious thing. Apropos the reviewer confusion, the code in Phazr.IO looks weird. It uses (word_size - hamming_distance) to compute the necessary number of blocks... but then returns the amount of memory needed to store blocks of a different size (word_size). We left all of it alone and return exactly the same values that the old computation returned. All these computations were the only thing in the code that used -lm, so drop that too. Coincidentially, this patch solves the crash of distro-built packages of liberasurecode (see Red Hat bug #1454543). But it's a side effect. Expect a proper patch soon. Change-Id: Ib297f6df304abf5ca8c27d3392b1107a525e0be0
* Merge "Fix warning on automake tool"Jenkins2017-05-101-6/+9
|\
| * Fix warning on automake toolKota Tsuyuzaki2017-02-201-6/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch fixes automake tools warnings as follows (and a my fault in previous patch [1]): - Change INCLUDES variable to APP_CPP_FLAGS and APP_C_FLAGS according to the warning message - Remove non-POSIX variable to suppress that warnings Note that the latter change may be affected to the testing so please be careful on your review what's going on your testing environment. In the past change [2], they ware designed to use the shared object binaries in the build path. However, Daniel had pointed out the good thing at [1] if we runs the test/liberasruecode_test (it's shell with linker) in the dir, we can touch the expected binaries. My fault in the previous patch is that I didn't replace ./test/.libs/liberasurecode_test to ./test/liberasurecode_test even we use --trace-children=yes option for the valgrind. Now, all tests run via ./test/<test name> syntax and IMO no matters exist even if we remove the non-POSIX variable for test settings. 1: https://review.openstack.org/#/c/434162 2: 93446db9414f311bd9fc7dc047eb4dbbeb3e6feb Change-Id: I0e79bed7755a1f286b746a70fcf56fdc972bfd5d
* | Merge "Jerasure: Handle initialization errors correctly"Jenkins2017-05-101-3/+7
|\ \
| * | Jerasure: Handle initialization errors correctlyTim Burke2017-03-161-3/+7
| | | | | | | | | | | | | | | | | | | | | Otherwise we can get backtraces where we try to free something that was never initialized. Change-Id: Iaea427b977fd20819e2da5678cc4889d3a42dd65
* | | Merge "ISA-L: Only calculate gf tables on init, not every encode"Jenkins2017-03-291-10/+18
|\ \ \
| * | | ISA-L: Only calculate gf tables on init, not every encodeDaniel Axtens2017-02-131-10/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, the Galois Field multiplication tables are recalcuated every time an encode is done. This is wasteful, as they are fixed by k and m, which is set on init. Calculate the tables only once, on init. This trades off a little bit of per-context memory and creation time for measurably faster encodes when using the same context. On powerpc64le, when repeatedly encoding a 4kB file with pyeclib, this increases the measured speed by over 10%. Change-Id: I2f025aaee2d13cb1717a331e443e179ad5a13302 Signed-off-by: Daniel Axtens <dja@axtens.net>
* | | | Merge "Add Phazr.IO libphazr backend to liberasurecode"Jenkins2017-03-0113-20/+437
|\ \ \ \ | |_|/ / |/| | |
| * | | Add Phazr.IO libphazr backend to liberasurecodeJim Cheung2017-02-2813-20/+437
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, there are several implementations of erasure codes that are available within OpenStack Swift. Most, if not all, of which are based on the Reed Solomon coding algorithm. Phazr.IO’s Erasure Coding technology uses a patented algorithm which are significantly more efficient and improves the speed of coding, decoding and reconstruction. In addition, Phazr.IO Erasure Code use a non-systematic algorithm which provides data protection at rest and in transport without the need to use encryption. Please contact support@phazr.io for more info on our technology. Change-Id: I4e40d02a8951e38409ad3c604c5dd6f050fa7ea0
* | | | jerasure: plug memory leaksDaniel Axtens2017-02-212-1/+44
| |_|/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | Jerasure inits some global variables on init. We need to free them, or we'll leak memory. Partial-Bug: #1666674 Change-Id: Ie7073738428a71910016e910a66dbd92ca98eb92 Signed-off-by: Daniel Axtens <dja@axtens.net>
* | | Fix valgrind-check and memory leakKota Tsuyuzaki2017-02-202-1/+8
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Can you believe that we ware testing the memory leak with valgrind to just *bash scripts* instead of actual binaries on liberasurecode_test and libec_slap? That is why we cannot find such an easy memory leak[1] at the gate. Now this patch enable to run the valgrind against to the binaries. With this fix, we found various memory leak at liberasurecode_test as follows and this patch also fixes them: - If we create fake fragments, we're responsible for freeing all of the frags as well as the array holding the pointers to the frags. - If we allocate any space, we're responsible for freeing it. - If we create an EC descriptor, we're responsible for destroying it. - If we create a fragment or skip array, we're responsible for freeing it. - If that happens inside a loop, we're responsible for doing it *inside that same loop*. In addition to the test fix, this patch fixes following memory leaks at the code which is affected to other users (pyeclib, OpenStack Swift) * Refuse to decode fragments that aren't even long enough to include fragment headers. * Fix a small memory leak in the builtin rs_vand implementation. Closes-Bug: #1665242 Co-Authored-By: Tim Burke <tim@swiftstack.com> 1: https://review.openstack.org/#/c/431812 Change-Id: I96f124e4e536bbd7544208acc084de1cda5c19b2
* | ISA-L: free matrix on exitDaniel Axtens2017-02-101-0/+1
|/ | | | | | | | | | | isa_l_common_init allocates desc->matrix, but this isn't freed in isa_l_exit. Instead, the entire isa_l_desc structure is freed, thus leaking the memory. Explicitly free desc->matrix. Change-Id: Ibf672d1a309498591b87d739632a90a1b3704f7e Signed-off-by: Daniel Axtens <dja@axtens.net>
* Merge "Add get_version functionality to liberasurecode lib"Jenkins2016-12-081-0/+10
|\
| * Add get_version functionality to liberasurecode libKota Tsuyuzaki2016-12-041-0/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | Currently, we have liberasurecode version info in the header and pyeclib is using the info to detect the version. However it's a bit painful because it requires to rebuild pyeclib c code for you to see the actual installed version. This addition for liberasurecode_get_version enables caller to get the version integer from compiled shared library file (.so) and it will rescure to re-compiled operation from pyeclib. Change-Id: I8161ea7da3b069e83c93e11cb41ce12fa60c6f32
* | Merge "ISA-L Cauchy support"Jenkins2016-12-075-545/+643
|\ \
| * | ISA-L Cauchy supportKota Tsuyuzaki2016-12-065-545/+643
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is for supporting ISA-L cauchy based matrix. The difference from isa_l_rs_vand is only the matrix to use the encode/decode calculation. As a known issue, isa_l_rs_vand backend has constraint for the combinations of the available fragment to be able to decode/reconstuct. (See related change in detail) To avoid the constraint, this patch adds another isa-l backend to use cauchy matrix and keep the backward compatibility, this is in another isa_l_rs_cauchy namespace. For implementation consieration, the code is almost same except the matrix generation fucntion so that this patch makes isa_l_common.c file for gathering common fucntions like init/encode/decode/reconstruct. And then the common init funciton takes an extra args "gen_matrix_func_name" for entry point to load the fucntion by dlsym from isa-l .so file. Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Related-Change: Icee788a0931fe692fe0de31fabc4ba450e338a87 Change-Id: I6eb150d9d0c3febf233570fa7729f9f72df2e9be
* | Merge "Fix posix_memalign handling"Jenkins2016-11-291-1/+1
|\ \
| * | Fix posix_memalign handlingKota Tsuyuzaki2016-07-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Docs[1] says posix_memalign will return non-zero value if it failed but currently it checked only if it's negative (or not) so that probably it can triggers something wrong (e.g. core dumped) if something like ENOTMEM returned. Change-Id: I0b8883ea60a904d4dc0efef94a33946a44ecfe01 1: http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_memalign.html
* | | Merge "Fix reconstruct to return an error when memory allocation failed"Jenkins2016-11-241-0/+3
|\ \ \ | |_|/ |/| |
| * | Fix reconstruct to return an error when memory allocation failedKota Tsuyuzaki2016-10-111-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As like current code, if we do go jump to just "out" section when memory allocation failed, liberaurecode/PyECLib will report it as success. However, in fact, the returned binary is not the one reconstructed. This patch fixes liberasurecode_reconstruct to return error code (ENOMEM) if either malloc or memset for working space is failed. Change-Id: I7bfc481c7a9bbc64288760df2dc6053c57657b41
* | | Fix error handling on gf_ivnert_matrix in isa-l backendKota Tsuyuzaki2016-11-091-2/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Current isa-l has possibility to return corrupted decoded data or corrupted reconstructed data on decode/reconstruct without error code. That is from the specification of isa-l rs vandermond matrix discussed at [1]. With many # of parities cases, we may hit the case above due to failing to get the inverse matrix from the encode matrix. The isa-l maintener gbtucker suggests a good way to detect the failing inverse matrix, that we should handle the return value gf_invert_matrix. If gf_invert_matrix returns not 0, we should stop to decode/reconstruct and return failure return code to the caller immediately. Otherwise, the caller regards the garbage data/fragment as correct one. And this patch adds the specific test case we can hit the issue (it happens not so general). 1: https://github.com/01org/isa-l/issues/10 Related-Change: I6eb150d9d0c3febf233570fa7729f9f72df2e9be Change-Id: Icee788a0931fe692fe0de31fabc4ba450e338a87
* | | Merge "Fix liberasurecode skipping a bunch of invalid_args tests"Jenkins2016-11-091-0/+3
|\ \ \
| * | | Fix liberasurecode skipping a bunch of invalid_args testsKota Tsuyuzaki2016-11-041-0/+3
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since the commit a01b1818c874a65d1d1fb8f11ea441e9d3e18771, we have been to able to test some parameters for each tests. However, instead, the NULL which means the end of the test parameters causes skipping a bunch of unit tests which doesn't take args (e.g. test_create_backend_invalid_args). That is the worse because we have not tested anymore for the tests since the commit. This patch fixes to make it tested and more, fix a minor bug for a case that corrupted header incomming to get_metadata. Closes-Bug: #1634403 Change-Id: Ib99a8aa6032f02d0c7d1ab94b8da1ebfd9047d74
* | | Merge "Fix clang compile time error"Jenkins2016-11-071-1/+1
|\ \ \
| * | | Fix clang compile time errorEric Lambert2016-06-011-1/+1
| | |/ | |/| | | | | | | | | | | | | | | | | | | | | | Removed unnecessary check that metadata index was not negative. Check is unneeded because type is unsigned and as such can not be negative. Change-Id: I72eddf78b25b3ff9bedbd596095f04be855c3817 Closes-Bug: 1587954
* | | Merge "Fix a typo in the erasurecode file"Jenkins2016-10-181-1/+1
|\ \ \ | |_|/ |/| |
| * | Fix a typo in the erasurecode filegengchc22016-08-301-1/+1
| |/ | | | | | | | | | | there is a spelling error,"instace" should be "instance" Change-Id: Iddba9e334bf55cc1e7d5cb397db5f3dfc5aac584
* | Add get_by_desc ret val handling to get_fragment_sizeKota Tsuyuzaki2016-09-071-0/+2
|/ | | | | | | | | | As well as any other callers, libersurecode_get_fragment_size should handle the return value of liberasurecode_get_backend_instance_by_desc. Otherwise, get_by_desc can return NULL and it causes an invalid memory access in librerasurecode_get_fragment_size. Change-Id: I489f8b5d049610863b5e0b477b6ff70ead245b55
* Fragment metadata checksumming supportfrag_metadata_checksumTushar Gohad2016-03-103-27/+113
|
* Add NULL instance check to backend_open()Tushar Gohad2016-03-101-0/+2
|
* Add a method to check if a backend is present.Timur Alperovich2016-03-081-19/+32
| | | | | | | Uses dlopen to check if a backend is present. This may be used by consumers who need to check which backends are present on a system. Issue #23
* Fix prototype declaration / definition mismatch (issue #25)Dirk Mueller2016-03-041-1/+1
|
* Check frag idx validity when verifying frag metadataTushar Gohad2016-02-141-0/+5
| | | | ... also add related test code
* Fix segfault when fragment index is out of rangeTushar Gohad2016-02-141-1/+1
| | | | | | ... (in a corrupt/invalid fragment scenario) Fixes issue #21
* Get version from liberasurecode version headerTushar Gohad2015-12-101-1/+1
| | | | ... (keep version in one place)
* Revert "Log to syslog and stderr by default"v1.1.1Tushar Gohad2015-12-041-1/+1
| | | | This reverts commit 21ed77fed2e31ec5183460a6bd9ff4592140b409.
View Lorry Controller Status