summaryrefslogtreecommitdiff
path: root/CMakeLists.txt
Commit message (Collapse)AuthorAgeFilesLines
* Add support for Zstd-based compression in LevelDB.leveldb Team2023-03-281-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change implements support for Zstd-based compression in LevelDB. Building up from the Snappy compression (which has been supported since inception), this change adds Zstd as an alternate compression algorithm. We are implementing this to provide alternative options for users who might have different performance and efficiency requirements. For instance, the Zstandard website (https://facebook.github.io/zstd/) claims that the Zstd algorithm can achieve around 30% higher compression ratios than Snappy, with relatively smaller (~10%) slowdowns in de/compression speeds. Benchmarking results: $ blaze-bin/third_party/leveldb/db_bench LevelDB: version 1.23 Date: Thu Feb 2 18:50:06 2023 CPU: 56 * Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz CPUCache: 35840 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 2.613 micros/op; 42.3 MB/s fillsync : 3924.432 micros/op; 0.0 MB/s (1000 ops) fillrandom : 3.609 micros/op; 30.7 MB/s overwrite : 4.508 micros/op; 24.5 MB/s readrandom : 6.136 micros/op; (864322 of 1000000 found) readrandom : 5.446 micros/op; (864083 of 1000000 found) readseq : 0.180 micros/op; 613.3 MB/s readreverse : 0.321 micros/op; 344.7 MB/s compact : 827043.000 micros/op; readrandom : 4.603 micros/op; (864105 of 1000000 found) readseq : 0.169 micros/op; 656.3 MB/s readreverse : 0.315 micros/op; 350.8 MB/s fill100K : 854.009 micros/op; 111.7 MB/s (1000 ops) crc32c : 1.227 micros/op; 3184.0 MB/s (4K per op) snappycomp : 3.610 micros/op; 1081.9 MB/s (output: 55.2%) snappyuncomp : 0.691 micros/op; 5656.3 MB/s zstdcomp : 15.731 micros/op; 248.3 MB/s (output: 44.1%) zstduncomp : 4.218 micros/op; 926.2 MB/s PiperOrigin-RevId: 509957778
* Remove main() from most tests.Victor Costan2022-01-031-34/+54
| | | | | | | | | | | | | | | This gives some flexibility to embedders. Currently, embedders have to build a binary for each test file. After this CL, embedders can still choose to have a binary for each test file, by linking each test file with a googletest target that includes main() (usually "gtest_main"). Embedders can also choose to build a single binary for almost all test files, and link with a googletest target that includes main(). The latter is more convenient for projects that have very few test binaries, like Chromium. PiperOrigin-RevId: 419470798
* Extract benchmark from db_test.cc.Victor Costan2021-12-291-7/+7
| | | | | | | The benchmark in db/db_test.cc is extracted to its own file, benchmarks/db_bench_log.cc. PiperOrigin-RevId: 418713499
* Change version to 1.23.1.23Chris Mumford2021-02-231-1/+1
| | | | PiperOrigin-RevId: 359111035
* Merge pull request #853 from cmumford:benchmarkChris Mumford2020-11-301-1/+6
|\ | | | | | | PiperOrigin-RevId: 344909677
| * Added google/benchmark submodule.Chris Mumford2020-11-301-1/+3
|/
* Remove leveldb::port::kLittleEndian.Victor Costan2020-04-141-3/+0
| | | | | | | | | Clang 10 includes the optimizations described in https://bugs.llvm.org/show_bug.cgi?id=41761. This means that the platform-independent implementations of {Decode,Encode}Fixed{32,64}() compile to one instruction on the most recent Clang and GCC. PiperOrigin-RevId: 306330166
* Switch testing harness to googletest.Victor Costan2019-11-211-6/+23
| | | | PiperOrigin-RevId: 281815695
* Allow different C/C++ standards when this is used as a subproject.Victor Costan2019-11-101-9/+15
| | | | | | Inspired by https://github.com/google/snappy/pull/85 PiperOrigin-RevId: 279649967
* Align CMake configuration with related projects.Victor Costan2019-11-071-7/+12
| | | | PiperOrigin-RevId: 279238007
* Remove redundant PROJECT_SOURCE_DIR usage from CMake config.Victor Costan2019-11-051-142/+142
| | | | | | Inspired by https://github.com/google/crc32c/pull/32 PiperOrigin-RevId: 278718726
* Fix installed target definition.Victor Costan2019-11-051-1/+3
| | | | | | | | | | Using CMAKE_INSTALL_INCLUDEDIR before including GNUINstallDirs results in a broken installation when CMAKE_INSTALL_PREFIX is a non-standard directory. Inspired from https://github.com/google/crc32c/pull/39 PiperOrigin-RevId: 278427974
* Using CMake's check_cxx_compiler_flag to check support for -Wthread-safety.Chris Mumford2019-10-011-17/+4
| | | | | | | | | | | | | | | Previously used check_cxx_source_compiles to attempt a build to determine support for clang thread safety checks. This change is to support static analysis of the leveldb source by lgtm.com (using Semmle). It failed to build with the following error: ``` [2019-07-04 22:29:58] [build] c++: error: unrecognized command line option ‘-Wthread-safety’; did you mean ‘-fthread-jumps’? [2019-07-04 22:30:02] [build] make[2]: *** [CMakeFiles/leveldb.dir/build.make:66: CMakeFiles/leveldb.dir/db/builder.cc.o] Error 1 ``` PiperOrigin-RevId: 272275528
* Disable exceptions and RTTI in CMake configuration.Victor Costan2019-06-131-2/+24
| | | | PiperOrigin-RevId: 252842234
* Merge pull request #624 from adam-azarchs:masterChris Mumford2019-05-061-0/+1
|\ | | | | | | PiperOrigin-RevId: 246903086
| * Add O_CLOEXEC to open calls.Adam Azarchs2019-02-221-0/+1
| | | | | | | | | | | | | | | | | | | | | | This prevents file descriptors from leaking to child processes. When compiled for older (pre-2.6.23) kernels which lack support for O_CLOEXEC there is no change in behavior. With newer kernels, child processes will no longer inherit leveldb's file handles, which reduces the changes of accidentally corrupting the database. Fixes https://github.com/google/leveldb/issues/623
* | Add argument definition for void c functions.allangj2019-05-051-0/+2
| | | | | | | | | | Allow the use c.h on projects with -Wstrict-prototypes Modify CMakelist to include -Wstrict-prototypes
* | Consolidate benchmark code to benchmarks/.Victor Costan2019-05-051-3/+3
| | | | | | | | | | | | Currently, the benchmark used to assess leveldb changes lives in db/. The codebase also contains two benchmarks against other database engines in doc/bench/. Moving all the benchmarks in one place opens up the way for extracting common code. PiperOrigin-RevId: 246737541
* | Bump the version number from 1.21 to 1.22.1.22Chris Mumford2019-05-031-1/+1
| | | | | | | | PiperOrigin-RevId: 246558281
* | Merge pull request #339 from richcole-at-amazon:masterChris Mumford2019-04-121-0/+1
|\ \ | | | | | | | | | PiperOrigin-RevId: 243156105
| * | Fix snapshot compaction bugRichard Cole2019-04-011-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Closes google/leveldb#320 During compaction it was possible that records from a block b1=(l1,u1) would be pushed down from level i to level i+1. If there is a block b2=(l2,u2) at level i with k1 = user_key(u1) = user_key(l2) then a subsequent search for k1 will yield the record l2 which has a smaller sequence number than u1 because the sort order for records sorts increasing by user key but decreaing by sequence number. This change add a call to a new function AddBoundaryInputs to SetupOtherInputs. AddBoundaryInputs searches for a block b2 matching the criteria above and adds it to the set of files to be compacted. Whenever AddBoundaryInputs is called it is important that the compaction fileset in level i+1 (known as c->inputs_[1] in the code) be recomputed. Each call to AddBoundaryInputs is followed by a call to GetOverlappingInputs. SetupOtherInputs is called on both manual and automated compaction passes. It is called for both level zero and for levels greater than 0. The original change posted in https://github.com/google/leveldb/pull/339 has been modified to also include changed made by Chris Mumford<cmumford@google.com> in https://github.com/cmumford/leveldb/commit/4b72cb14f8da2aab12451c24b8e205aff686e9dc 1. Releasing snapshots during test cleanup to avoid memory leak warnings. 2. Refactored test to use testutil.h to be in line with other issue tests and to create the test database in the correct temporary location. 3. Added copyright banner. Otherwise, just minor formatting and limiting character width to 80 characters. Additionally the change was rebased on top of current master and changes previously made to the Makefile were ported to the CMakeLists.txt. Testing Done: A test program (issue320_test) was constructed that performs mutations while snapshots are active. issue320_test fails without this bug fix after 64k writes. It passes with this bug fix. It was run with 200M writes and passed. Unit tests were written for the new function that was added to the code. Make test was run and seen to pass. Signed-off-by: Richard Cole <richcole@amazon.com>
* | | Align version/soversion CMake setup closer with other repositories.Victor Costan2019-04-011-4/+4
| | | | | | | | | | | | PiperOrigin-RevId: 241432456
* | | Restore soname versioning with CMake buildleveldb Team2019-04-011-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Before: $ readelf -d build/libleveldb.so | grep soname 0x000000000000000e (SONAME) Library soname: [libleveldb.so] After: $ readelf -d build/libleveldb.so | grep soname 0x000000000000000e (SONAME) Library soname: [libleveldb.so.1] This matches the soname from v1.20. PiperOrigin-RevId: 241334113
* | | Update the version to 1.21 in preparation for a new release.1.21Chris Mumford2019-03-291-0/+1
|/ / | | | | | | PiperOrigin-RevId: 241053616
* | Deleted dangling reference to deleted atomic_pointer.h.cmumford2019-03-111-1/+0
| | | | | | | | | | | | | | | | | | Forgot one reference to atomic_pointer.h in CMakeLists.txt from prior CL. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=237870915
* | Align windows_logger with posix_logger.costan2019-03-071-10/+10
| | | | | | | | | | | | | | | | | | | | Fixes GitHub issue #657. This CL also makes the Windows CI green. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=237255887
* | Added native support for Windows.cmumford2019-03-011-6/+25
|/ | | | | | | | | | | | | | | | | | | | This change adds a native Windows port (port_windows.h) and a Windows Env (WindowsEnv). Note1: "small" is defined when including <Windows.h> so some parameters were renamed to avoid conflict. Note2: leveldb::Env defines the method: "DeleteFile" which is also a constant defined when including <Windows.h>. The solution was to ensure this macro is defined in env.h which forces the function, when compiled, to be either DeleteFileA or DeleteFileW when building for MBCS or UNICODE respectively. This resolves #519 on GitHub. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=236364778
* leveldb: Fix PosixWritableFile::Sync() on Apple systems.costan2019-01-091-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Apple doesn't follow POSIX specifications for fsync(). Instead, fsync() guarantees to flush the buffer cache to the device, which means the data will survive kernel panics, but may not survive power outages. Applications that need stronger guarantees (like databases) need to use fcntl(F_FULLFSYNC). This CL switches PosixWritableFile::Sync() to get the stronger guarantees on Apple systems. The improved implementation follows the same principles as SQLite [1] and node.js [2]. Research for the fcntl() to fsync() fallback strategy: Apple's released source code at https://opensource.apple.com/ shows at least three different error codes being returned when a filesystem does not support F_FULLFSYNC. fcntl() is implemented in xnu-4903.221.2 in bsd/kern/kern_descrip.c, where it delegates to fcntl_nocancel(). The documentation for fcntl_nocancel() mentions error codes for some operations, but does not include F_FULLFSYNC. The F_FULLSYNC branch in fcntl_nocancel() calls VNOP_IOCTL(_, F_FULLSYNC, NULL, 0, _), whose return value sets the error code. VNOP_IOCTL() is implemented in bsd/vfs/kpi_vfs.c and calls the ioctl function in the vnode's operation vector. The per-filesystem function names follow the pattern _vnop_ioctl() for all the instances in opensource code: {hfs,msdosfs,nfs,ntfs,smbfs,webdav,zfs}_vnop_ioctl(). hfs-407.30.1, msdosfs-229.200.3, and nfs in xnu-4903.221.2 handle F_FULLFSYNC. ntfs-94.200.1 and smb-759.40.1 do not handle F_FULLFSYNC, and the default branch returns ENOSUP. webdav-380.200.1 also does not handle F_FULLFSYNC, but the default branch returns EINVAL. zfs-59 also does not handle F_FULLSYNC, and its default branch returns ENOTTY. From a different angle, Apple's ntfs-94.200.1 includes utility code that uses fcntl(F_FULLFSYNC) and falls back to fsync() just like we do, supporting the hypothesis that there is no good way to detect lack of F_FULLFSYNC support. Also, Apple's fcntl() man page [3] does not mention a way to detect lack of F_FULLFSYNC support. [1] https://www.sqlite.org/src/doc/trunk/src/os_unix.c [2] https://github.com/libuv/libuv/blob/master/src/unix/fs.c [3] https://developer.apple.com/library/archive/documentatiVon/System/Conceptual/ManPages_iPhoneOS/man2/fcntl.2.html Tested: https://travis-ci.org/pwnall/leveldb/builds/477318498 TAP global presubmit ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=228593729
* Fix fdatasync() feature detection in opensource build.costan2019-01-081-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | The CMake feature-detection code used check_symbol_exists(), which invokes the C compiler. However, some glibc versions don't expose the fdatasync() declaration when compiled with -std=c11, but do expose it when compiled with -std=c++11. This most likely comes down to how _POSIX_SOURCE is defined -- it needs to be >= 201112L for <unistd.h> to expose fdatasync(). This CL switches to check_cxx_symbol_exists(), which uses the C++ compiler. Asides from fixing the problem above, this is the right thing to do, because we use <unistd.h> in env_posix.cc, which is compiled with the C++ compiler. This CL also fixes a previously introduced inconsistency, where the macro indicating the fdatasync() feature detection result was referred to as HAVE_FDATASYNC and HAVE_FUNC_FDATASYNC. The former appears to be used in other libraries, so this CL switches all our references to HAVE_FDATASYNC. Fixes https://github.com/google/leveldb/issues/629 ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=228392612
* Remove InitOnce from the port API.costan2018-09-101-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | This is not an API-breaking change, because it reduces the API that the leveldb embedder must implement. The project will build just fine against ports that still implement InitOnce. C++11 guarantees thread-safe initialization of static variables inside functions. This is a more restricted form of std::call_once or pthread_once_t (e.g., single call site), so the compiler might be able to generate better code [1]. Equally important, having less code in port_example.h makes it easier to port to other platforms. Due to the change above, this CL introduces a new approach for storing the singleton BytewiseComparatorImpl instance returned by BytewiseComparator(). The new approach avoids a dynamic memory allocation, which eliminates the false positive from LeakSanitizer reported in https://github.com/google/leveldb/issues/200 [1] https://stackoverflow.com/a/27206650/ ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=212348004
* Add move constructor to Status.costan2018-04-231-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This will result in smaller code generation when Status instances are passed around. Benchmarks don't indicate a significant change either way. CPU: 48 * Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz CPUCache: 30720 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Baseline: fillseq : 3.589 micros/op; 30.8 MB/s fillsync : 4165.299 micros/op; 0.0 MB/s (1000 ops) fillrandom : 5.864 micros/op; 18.9 MB/s overwrite : 7.830 micros/op; 14.1 MB/s readrandom : 5.534 micros/op; (1000000 of 1000000 found) readrandom : 4.292 micros/op; (1000000 of 1000000 found) readseq : 0.312 micros/op; 354.1 MB/s readreverse : 0.501 micros/op; 220.8 MB/s compact : 886211.000 micros/op; readrandom : 3.518 micros/op; (1000000 of 1000000 found) readseq : 0.251 micros/op; 441.2 MB/s readreverse : 0.456 micros/op; 242.4 MB/s fill100K : 1329.723 micros/op; 71.7 MB/s (1000 ops) crc32c : 1.976 micros/op; 1976.7 MB/s (4K per op) snappycomp : 4.705 micros/op; 830.2 MB/s (output: 55.1%) snappyuncomp : 0.958 micros/op; 4079.1 MB/s acquireload : 0.727 micros/op; (each op is 1000 loads) New: fillseq : 3.129 micros/op; 35.4 MB/s fillsync : 2748.099 micros/op; 0.0 MB/s (1000 ops) fillrandom : 5.394 micros/op; 20.5 MB/s overwrite : 7.253 micros/op; 15.3 MB/s readrandom : 5.655 micros/op; (1000000 of 1000000 found) readrandom : 4.425 micros/op; (1000000 of 1000000 found) readseq : 0.298 micros/op; 371.3 MB/s readreverse : 0.508 micros/op; 217.9 MB/s compact : 885842.000 micros/op; readrandom : 3.545 micros/op; (1000000 of 1000000 found) readseq : 0.252 micros/op; 438.2 MB/s readreverse : 0.425 micros/op; 260.2 MB/s fill100K : 1418.347 micros/op; 67.2 MB/s (1000 ops) crc32c : 1.987 micros/op; 1966.0 MB/s (4K per op) snappycomp : 4.767 micros/op; 819.4 MB/s (output: 55.1%) snappyuncomp : 0.916 micros/op; 4264.9 MB/s acquireload : 0.665 micros/op; (each op is 1000 loads) ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=194002392
* Replace port_posix with port_stdcxx.costan2018-04-171-48/+16
| | | | | | | | | | | | | | | The porting layer implements threading primitives: atomic pointers, condition variables, mutexes, thread-safe initialization. These are all specified in C++11, so the reference open source port implementation can become platform-independent. The porting layer will remain in place to allow the use of other implementations with more features, such as the built-in deadlock detection in abseil's Mutex. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=193245934
* Add tests for ConsumeDecimalNumber.costan2018-04-131-0/+1
| | | | | | | | | | ConsumeDecimalNumber has fairly non-trivial logic, and a previous version has crashed inexplicably on Android. Having some test coverage will make it easier to tweak / simplify the function later on. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=192821751
* Define LEVELDB_HAS_PORT_CONFIG_H for old compilers.costan2018-04-091-0/+33
| | | | | | | | | | | | The CMake-based build relies on __has_include, which is standardized in C++17. Unfortunately, __has_include is available without requiring --std=c++17 on all the compilers on CI, so this problem was not caught. Fixes https://github.com/google/leveldb/issues/572 ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=192208842
* Rename CMake project / targets from Leveldb to leveldb.costan2018-03-231-9/+9
| | | | | | ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=190262375
* Take <atomic> for granted in port/atomic_pointer.h.costan2018-03-211-3/+0
| | | | | | | | | C++11 requires <atomic>. This lets us remove the header detection (LEVELDB_ATOMIC_PRESENT) and simplify port/atomic_pointer.h. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=189919098
* leveldb: Require C++11.costan2018-03-211-3/+8
| | | | | | | | | | This CL switches the public headers to C++11 default and deleted constructors, and adds override to the relevant leveldb::EnvWrapper methods. This should be a good test for C++11 compiler support. Once this CL settles, the rest of the codebase can be safely modernized to C++11. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=189873212
* Add export.h to CMakeLists.txtHyzeta2018-03-201-0/+2
| | | | | | | | | | The header is missing from the file lists, resulting in incomplete installs. This was originally contributed in https://github.com/google/leveldb/pull/568. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=189822907
* Add CMake build support.costan2018-03-161-0/+412
Fixes https://github.com/google/leveldb/issues/466 ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=189425354
View Lorry Controller Status