summaryrefslogtreecommitdiff
path: root/src/journal/test-compress.c
Commit message (Collapse)AuthorAgeFilesLines
* sd-journal: move source files for sd-journal to src/libsystemd/sd-journalYu Watanabe2021-01-191-372/+0
|
* license: LGPL-2.1+ -> LGPL-2.1-or-laterYu Watanabe2020-11-091-1/+1
|
* Rename find_binary to find_executableZbigniew Jędrzejewski-Szmek2020-09-181-1/+1
| | | | "executable" is more correct than "binary", since scripts are OK too.
* journal/compress: remove loop in decompress_startswith_zstd()Zbigniew Jędrzejewski-Szmek2020-07-211-0/+2
| | | | | This should be more efficient with no downsides. Same considerations as in the previous commit hold.
* tree-wide: add new HAVE_COMPRESSION compile time flagLennart Poettering2020-06-251-2/+2
| | | | | | | | let's simplify the checks for ZSTD/LZ4/XZ As suggested: https://github.com/systemd/systemd/pull/16096#discussion_r440705585
* journal: support zstd compression for large objects in journal filesLennart Poettering2020-06-251-0/+17
|
* coredump: add zstandard support for coredumpsNorbert Lange2020-05-041-43/+49
| | | | | this will hook libzstd into coredump, using this format as default.
* test: Fix build with !HAVE_LZ4 && HAVE_XZMichal Koutný2020-05-021-1/+2
| | | | | | | | | | HUGE_SIZE was defined inconsistently. > In file included from ../src/basic/alloc-util.h:9, > from ../src/journal/test-compress.c:9: > ../src/journal/test-compress.c: In function ‘main’: > ../src/journal/test-compress.c:280:33: error: ‘HUGE_SIZE’ undeclared (first use in this function) > 280 | assert_se(huge = malloc(HUGE_SIZE));
* tests: various small fixes for strict systemsTopi Miettinen2020-04-261-8/+11
| | | | | | | | | | | | | | | | | | | | | | Don't assume that 4MB can be allocated from stack since there could be smaller DefaultLimitSTACK= in force, so let's use malloc(). NUL terminate the huge strings by hand, also ensure termination in test_lz4_decompress_partial() and optimize the memset() for the string. Some items in /proc and /etc may not be accessible to poor unprivileged users due to e.g. SELinux, BOFH or both, so check for EACCES and EPERM. /var/tmp may be a symlink to /tmp and then path_compare() will always fail, so let's stick to /tmp like elsewhere. /tmp may be mounted with noexec option and then trying to execute scripts from there would fail. Detect and warn if seccomp is already in use, which could make seccomp test fail if the syscalls are already blocked. Unset $TMPDIR so it will not break specifier tests where %T is assumed to be /tmp and %V /var/tmp.
* headers: remove unneeded includes from util.hZbigniew Jędrzejewski-Szmek2019-03-271-0/+2
| | | | | This means we need to include many more headers in various files that simply included util.h before, but it seems cleaner to do it this way.
* util: split out memcmp()/memset() related calls into memory-util.[ch]Lennart Poettering2019-03-131-1/+1
| | | | Just some source rearranging.
* util-lib: split out all temporary file related calls into tmpfiles-util.cLennart Poettering2018-12-021-1/+1
| | | | | | | | This splits out a bunch of functions from fileio.c that have to do with temporary files. Simply to make the header files a bit shorter, and to group things more nicely. No code changes, just some rearranging of source files.
* journal: adapt for new improved LZ4_decompress_safe_partial()Zbigniew Jędrzejewski-Szmek2018-10-301-10/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With lz4 1.8.3, this function can now decompress partial results into a smaller buffer. The release news don't say anything interesting, but the test case that was previously failing now works OK. Fixes #10259. A test is added. It shows that with *older* lz4, a partial decompression can occur with the returned size smaller then the requested number of bytes _and_ smaller then the size of the compressed data: (lz4-libs-1.8.2-1.fc29.x86_64) Compressed 4194304 → 16464 Decompressed → 4194304 Decompressed partial 12/4194304 → 4194304 Decompressed partial 1/1 → -2 (bad) Decompressed partial 2/2 → -2 (bad) Decompressed partial 3/3 → -2 (bad) Decompressed partial 4/4 → -2 (bad) Decompressed partial 5/5 → -2 (bad) Decompressed partial 6/6 → 6 (good) Decompressed partial 7/7 → 6 (good) Decompressed partial 8/8 → 6 (good) Decompressed partial 9/9 → 6 (good) Decompressed partial 10/10 → 6 (good) Decompressed partial 11/11 → 6 (good) Decompressed partial 12/12 → 6 (good) Decompressed partial 13/13 → 6 (good) Decompressed partial 14/14 → 6 (good) Decompressed partial 15/15 → 6 (good) Decompressed partial 16/16 → 6 (good) Decompressed partial 17/17 → 6 (good) Decompressed partial 18/18 → -16459 (bad) (lz4-libs-1.8.3-1.fc29.x86_64) Compressed 4194304 → 16464 Decompressed → 4194304 Decompressed partial 12/4194304 → 12 Decompressed partial 1/1 → 1 (good) Decompressed partial 2/2 → 2 (good) Decompressed partial 3/3 → 3 (good) Decompressed partial 4/4 → 4 (good) ... If we got such a short "successful" decompression in decompress_startswith() as implemented before this patch, we could be confused and return a false negative result. But it turns out that this only occurs with small output buffer sizes. We use greedy_realloc() to manager the buffer, so it is always at least 64 bytes. I couldn't hit a case where decompress_startswith() would actually return a bogus result. But since the lack of proof is not conclusive, the code for *older* lz4 is changed too, just to be safe. We cannot rule out that on a different architecture or with some unlucky compressed string we could hit this corner case. The fallback code is guarded by a version check. The check uses a function not the compile-time define, because there was no soversion bump in lz4 or new symbols, and we could be compiled against a newer lz4 and linked at runtime with an older one. (This happens routinely e.g. when somebody upgrades a subset of distro packages.)
* test-compress: add test for short decompress_startswith callsZbigniew Jędrzejewski-Szmek2018-10-301-0/+32
| | | | | | I thought this might fail with lz4 < 1.8.3, but it seems that because of greedy_realloc, we always use a buffer that is large enough, and it always passes.
* Drop support for lz4 < 1.3.0Zbigniew Jędrzejewski-Szmek2018-10-291-4/+0
| | | | | | lz4-r130 was released on May 29th, 2015. Let's drop the work-around for older versions. In particular, we won't test any new code against those ancient releases, so we shouldn't pretend they are supported.
* tests: use a helper function to parse environment and open loggingZbigniew Jędrzejewski-Szmek2018-09-141-1/+2
| | | | | The advantages are that we save a few lines, and that we can override logging using environment variables in more test executables.
* test: log when skipping tests in more casesYu Watanabe2018-09-131-0/+1
| | | | Follow-up for the previous commit.
* tree-wide: drop copyright headers from frequent contributorsZbigniew Jędrzejewski-Szmek2018-06-201-3/+0
| | | | | | | | Fixes #9320. for p in Shapovalov Chevalier Rozhkov Sievers Mack Herrmann Schmidt Rudenberg Sahani Landden Andersen Watanabe; do git grep -e 'Copyright.*'$p -l|xargs perl -i -0pe 's|/([*][*])?[*]\s+([*#]\s+)?Copyright[^\n]*'$p'[^\n]*\s*[*]([*][*])?/\n*|\n|gms; s|\s+([*#]\s+)?Copyright[^\n]*'$p'[^\n]*\n*|\n|gms' done
* tree-wide: beautify remaining copyright statementsLennart Poettering2018-06-141-1/+1
| | | | | | Let's unify an beautify our remaining copyright statements, with a unicode ©. This means our copyright statements are now always formatted the same way. Yay.
* tree-wide: drop 'This file is part of systemd' blurbLennart Poettering2018-06-141-2/+0
| | | | | | | | | | | | | | | | This part of the copyright blurb stems from the GPL use recommendations: https://www.gnu.org/licenses/gpl-howto.en.html The concept appears to originate in times where version control was per file, instead of per tree, and was a way to glue the files together. Ultimately, we nowadays don't live in that world anymore, and this information is entirely useless anyway, as people are very welcome to copy these files into any projects they like, and they shouldn't have to change bits that are part of our copyright header for that. hence, let's just get rid of this old cruft, and shorten our codebase a bit.
* fs-util,test: add helper to remove tempfilesZbigniew Jędrzejewski-Szmek2018-05-181-5/+4
| | | | | | | This simplifies the use of tempfiles in tests and fixes "leaked" temporary files in test-fileio, test-catalog, test-conf-parser. Not the whole tree is converted.
* tree-wide: drop license boilerplateZbigniew Jędrzejewski-Szmek2018-04-061-13/+0
| | | | | | | | | | Files which are installed as-is (any .service and other unit files, .conf files, .policy files, etc), are left as is. My assumption is that SPDX identifiers are not yet that well known, so it's better to retain the extended header to avoid any doubt. I also kept any copyright lines. We can probably remove them, but it'd nice to obtain explicit acks from all involved authors before doing that.
* test: handle gracefully if decompressor tools are not installed (#7455)Lennart Poettering2017-11-241-1/+8
| | | Fixes: #7441
* Add SPDX license identifiers to source files under the LGPLZbigniew Jędrzejewski-Szmek2017-11-191-0/+1
| | | | | This follows what the kernel is doing, c.f. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5fd54ace4721fc5ce2bb5aef6318fcf17f421460.
* Merge pull request #6974 from keszybz/clean-up-definesLennart Poettering2017-10-041-8/+8
|\ | | | | Clean up define definitions
| * build-sys: use #if Y instead of #ifdef Y everywhereZbigniew Jędrzejewski-Szmek2017-10-041-8/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The advantage is that is the name is mispellt, cpp will warn us. $ git grep -Ee "conf.set\('(HAVE|ENABLE)_" -l|xargs sed -r -i "s/conf.set\('(HAVE|ENABLE)_/conf.set10('\1_/" $ git grep -Ee '#ifn?def (HAVE|ENABLE)' -l|xargs sed -r -i 's/#ifdef (HAVE|ENABLE)/#if \1/; s/#ifndef (HAVE|ENABLE)/#if ! \1/;' $ git grep -Ee 'if.*defined\(HAVE' -l|xargs sed -i -r 's/defined\((HAVE_[A-Z0-9_]*)\)/\1/g' $ git grep -Ee 'if.*defined\(ENABLE' -l|xargs sed -i -r 's/defined\((ENABLE_[A-Z0-9_]*)\)/\1/g' + manual changes to meson.build squash! build-sys: use #if Y instead of #ifdef Y everywhere v2: - fix incorrect setting of HAVE_LIBIDN2
* | tree-wide: use IN_SET macro (#6977)Yu Watanabe2017-10-041-1/+1
|/
* test-compress*: silence warning about unused definitions when w/o both xz ↵Zbigniew Jędrzejewski-Szmek2017-04-191-0/+6
| | | | | | | | and lz4 I think it's nice to mark the test as skipped instead of omitting it entirely, hence #ifdefs in the code instead of excluding the test in Makefile.am/meson.build.
* Fix missing space in comments (#5439)AsciiWolf2017-02-241-1/+1
|
* test-compress: fix warning about LZ4_compress_limitedOutputZbigniew Jędrzejewski-Szmek2016-12-171-0/+4
| | | | 691b90d465 fixed one spot, but missed the other one.
* test-compression: allow the file to compress to be specifiedZbigniew Jędrzejewski-Szmek2016-10-311-2/+5
| | | | | | | | | I'm seeing strange decompression errors with lz4, which might be content-dependent. Extend test-compression to allow testing specific content. (Edit: PEBKAC: lzcat and lz4cat are not the same beast. Nevertheless, the test might still be useful in the future.)
* fileio: simplify mkostemp_safe() (#4090)Topi Miettinen2016-09-131-2/+2
| | | | | | According to its manual page, flags given to mkostemp(3) shouldn't include O_RDWR, O_CREAT or O_EXCL flags as these are always included. Beyond those, the only flag that all callers (except a few tests where it probably doesn't matter) use is O_CLOEXEC, so set that unconditionally.
* journal: add the "repeating sequence" test caseZbigniew Jędrzejewski-Szmek2015-12-131-35/+54
| | | | | | | | | | | This was the case that caused various problems that were fixed in preceding patches, so it is good to add a test that uses it directly. In "may_fail" test cases try again with a bigger buffer. Instead of allocating various buffers on the stack, malloc them. This is more reliable in case of big buffers, and allows tools like valgrind and address sanitizer to find overflows more easily.
* journal: add "xfail" test for partial lz4 decompressionZbigniew Jędrzejewski-Szmek2015-12-131-0/+42
| | | | | | Add a test that LZ4_decompress_safe_partial does (not) work as expected, so that if it starts to work at some point, we'll catch this and adjust our code.
* journal: add dst_allocated_size parameter for compress_blobZbigniew Jędrzejewski-Szmek2015-12-131-7/+5
| | | | | | | | | | | | | | compress_blob took src, src_size, dst and *dst_size, but dst_size wasn't used as an input parameter with the size of dst, but only as an output parameter. dst was implicitly assumed to be at least src_size-1. This code wasn't *wrong*, because the only real caller in journal-file.c got it right. But it was misleading, and the tests in test-compress.c got it wrong, and worked only because the output buffer happened to be the same size as input buffer. So add a seperate dst_allocated_size parameter to make it explicit what the size of the buffer is, and to allow test to proceed with different output buffer sizes.
* util-lib: split out allocation calls into alloc-util.[ch]Lennart Poettering2015-10-271-0/+1
|
* util-lib: move more file I/O related calls into fileio.[ch]Lennart Poettering2015-10-271-0/+1
|
* util-lib: split out fd-related operations into fd-util.[ch]Lennart Poettering2015-10-251-1/+2
| | | | | There are more than enough to deserve their own .c file, hence move them over.
* coredump: use lz4frame api to compress coredumpsZbigniew Jędrzejewski-Szmek2015-10-101-5/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | This converts the stream compression to use the new lz4frame api, compatible with lz4cat. Previous code used custom headers, so the compressed file was not compatible with lz4 command line tools. I considered this the last blocker to using lz4 by default. Speed seems to be reasonable, although a bit (a few percent) slower than the lz4 binary, even though compression is the same. I don't consider this important. It could be caused by the overhead of library calls, but is probably caused by slightly different buffer sizes or such. The code in this patch uses mmap, since since this allows the buffer to be reused while not making the code more complicated at all. In my testing, this version is noticably faster (~20%) than a naive single-buffered version. mmap can cause the program to be killed with SIGBUS, if the underlying file is truncated or a disk error occurs. We only use this from within coredump and coredumpctl, so I don't consider this an issue. Old decompression code is retained and is used if the new code fails indicating a format error. There have been reports of various smaller distributions using previous lz4 code, i.e. the old format, and it is nice to provide backwards compatibility. We can remove the legacy code in a few versions. The way that blobs are compressed in the journal is not affected.
* tree-wide: never use the off_t unless glibc makes us use itLennart Poettering2015-09-101-2/+2
| | | | | | | | | | | off_t is a really weird type as it is usually 64bit these days (at least in sane programs), but could theoretically be 32bit. We don't support off_t as 32bit builds though, but still constantly deal with safely converting from off_t to other types and back for no point. Hence, never use the type anymore. Always use uint64_t instead. This has various benefits, including that we can expose these values directly as D-Bus properties, and also that the values parse the same in all cases.
* shared: add random-util.[ch]Ronny Chevalier2015-04-111-0/+1
|
* tests: use assert_se instead of assertRonny Chevalier2015-01-221-2/+2
| | | | Otherwise they can be optimized away with -DNDEBUG
* tests: use assert_se instead of assertRonny Chevalier2014-11-301-9/+9
| | | | Otherwise they can be optimized away with -DNDEBUG
* treewide: no need to negate errno for log_*_errno()Michal Schmidt2014-11-281-2/+2
| | | | It corrrectly handles both positive and negative errno values.
* treewide: auto-convert the simple cases to log_*_errno()Michal Schmidt2014-11-281-2/+2
| | | | | | | | | | | | | As a followup to 086891e5c1 "log: add an "error" parameter to all low-level logging calls and intrdouce log_error_errno() as log calls that take error numbers", use sed to convert the simple cases to use the new macros: find . -name '*.[ch]' | xargs sed -r -i -e \ 's/log_(debug|info|notice|warning|error|emergency)\("(.*)%s"(.*), strerror\(-([a-zA-Z_]+)\)\);/log_\1_errno(-\4, "\2%m"\3);/' Multi-line log_*() invocations are not covered. And we also should add log_unit_*_errno().
* test-compress: also test with incompressible inputsZbigniew Jędrzejewski-Szmek2014-08-301-26/+65
|
* test-compress: make sure asserts with side effects use assert_se()Filipe Brandenburger2014-08-261-2/+2
| | | | | | | | Otherwise the test fails when built with CPPFLAGS='-DNDEBUG' which disables assertions. Tested: - make check TESTS='test-compress' CPPFLAGS='-DNDEBUG'
* Fix misuse of uint64_t as size_tZbigniew Jędrzejewski-Szmek2014-08-031-10/+9
| | | | They have different size on 32 bit, so they are really not interchangable.
* fix #ifdefRonny Chevalier2014-07-081-1/+1
|
* journal: add LZ4 as optional compressorZbigniew Jędrzejewski-Szmek2014-07-061-46/+124
| | | | | | | | | | | | | | Add liblz4 as an optional dependency when requested with --enable-lz4, and use it in preference to liblzma for journal blob and coredump compression. To retain backwards compatibility, XZ is used to decompress old blobs. Things will function correctly only with lz4-119. Based on the benchmarks found on the web, lz4 seems to be the best choice for "quick" compressors atm. For pkg-config status, see http://code.google.com/p/lz4/issues/detail?id=135.
View Lorry Controller Status