summaryrefslogtreecommitdiff
path: root/table
Commit message (Collapse)AuthorAgeFilesLines
* leveldb: Check slice length in Footer::DecodeFrom()HEADmainleveldb Team2023-04-201-0/+4
| | | | | | | | | | | Without this check decoding the footer in Table::Open() can read uninitialized bytes from a buffer allocated on the stack if the file was unexpectedly short. In practice this is probably fine since this function validates a magic number but MSan complains about branching on uninitialized data. PiperOrigin-RevId: 525271012
* Support Zstd compression level in Leveldbleveldb Team2023-04-202-2/+3
| | | | PiperOrigin-RevId: 520556840
* Add support for Zstd-based compression in LevelDB.leveldb Team2023-03-283-8/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change implements support for Zstd-based compression in LevelDB. Building up from the Snappy compression (which has been supported since inception), this change adds Zstd as an alternate compression algorithm. We are implementing this to provide alternative options for users who might have different performance and efficiency requirements. For instance, the Zstandard website (https://facebook.github.io/zstd/) claims that the Zstd algorithm can achieve around 30% higher compression ratios than Snappy, with relatively smaller (~10%) slowdowns in de/compression speeds. Benchmarking results: $ blaze-bin/third_party/leveldb/db_bench LevelDB: version 1.23 Date: Thu Feb 2 18:50:06 2023 CPU: 56 * Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz CPUCache: 35840 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 2.613 micros/op; 42.3 MB/s fillsync : 3924.432 micros/op; 0.0 MB/s (1000 ops) fillrandom : 3.609 micros/op; 30.7 MB/s overwrite : 4.508 micros/op; 24.5 MB/s readrandom : 6.136 micros/op; (864322 of 1000000 found) readrandom : 5.446 micros/op; (864083 of 1000000 found) readseq : 0.180 micros/op; 613.3 MB/s readreverse : 0.321 micros/op; 344.7 MB/s compact : 827043.000 micros/op; readrandom : 4.603 micros/op; (864105 of 1000000 found) readseq : 0.169 micros/op; 656.3 MB/s readreverse : 0.315 micros/op; 350.8 MB/s fill100K : 854.009 micros/op; 111.7 MB/s (1000 ops) crc32c : 1.227 micros/op; 3184.0 MB/s (4K per op) snappycomp : 3.610 micros/op; 1081.9 MB/s (output: 55.2%) snappyuncomp : 0.691 micros/op; 5656.3 MB/s zstdcomp : 15.731 micros/op; 248.3 MB/s (output: 44.1%) zstduncomp : 4.218 micros/op; 926.2 MB/s PiperOrigin-RevId: 509957778
* Use GTEST_SKIP in leveldb:table_testleveldb Team2022-07-181-4/+2
| | | | | | This replaces a usage of fprintf to stderr PiperOrigin-RevId: 454620969
* Remove main() from most tests.Victor Costan2022-01-032-10/+0
| | | | | | | | | | | | | | | This gives some flexibility to embedders. Currently, embedders have to build a binary for each test file. After this CL, embedders can still choose to have a binary for each test file, by linking each test file with a googletest target that includes main() (usually "gtest_main"). Embedders can also choose to build a single binary for almost all test files, and link with a googletest target that includes main(). The latter is more convenient for projects that have very few test binaries, like Chromium. PiperOrigin-RevId: 419470798
* Optimize leveldb block seeks to utilize the current iterator location.leveldb Team2021-01-111-1/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | This is beneficial when iterators are reused and seeks are not random but increasing. It is additionally beneficial with larger block sizes and keys with common prefixes. Add a benchmark "seekordered" to db_bench that reuses iterators across increasing seeks. Add support to the benchmark to count comparisons made and to support common key prefix length. Change benchmark random seeds to be reproducible for entire benchmark suite executions but unique for threads in different benchmarks runs. This changes a benchmark suite of readrandom,seekrandom from having a 100% found ratio as previously it had the same seed used for fillrandom. ./db_bench --benchmarks=fillrandom,compact,seekordered --block_size=262144 --comparisons=1 --key_prefix=100 without this change (though with benchmark changes): seekrandom : 55.309 micros/op; (631820 of 1000000 found) Comparisons: 27001049 seekordered : 1.732 micros/op; (631882 of 1000000 found) Comparisons: 26998402 with this change: seekrandom : 55.866 micros/op; (631820 of 1000000 found) Comparisons: 26952143 seekordered : 1.686 micros/op; (631882 of 1000000 found) Comparisons: 25549369 For ordered seeking, this is a reduction of 5% comparisons and a 3% speedup. For random seeking (with single use iterators) the comparisons and speed are less than 1% and likely noise. PiperOrigin-RevId: 351149832
* Merge pull request #798 from lntotk:masterVictor Costan2020-05-041-6/+4
|\ | | | | | | PiperOrigin-RevId: 309738404
| * remove unnessary status judgelntotk2020-04-241-6/+4
| |
* | Add some std:: qualifiers to types and functions.Victor Costan2020-04-291-16/+16
| | | | | | | | PiperOrigin-RevId: 309110431
* | Switch from C headers to C++ headers.Victor Costan2020-04-296-12/+8
|/ | | | | | | | | | | | | | | This CL makes the following substitutions. * assert.h -> cassert * math.h -> cmath * stdarg.h -> cstdarg * stddef.h -> cstddef * stdint.h -> cstdint * stdio.h -> cstdio * stdlib.h -> cstdlib * string.h -> cstring PiperOrigin-RevId: 309080151
* Internal change.leveldb Team2019-12-022-2/+2
| | | | PiperOrigin-RevId: 282373286
* Switch testing harness to googletest.Victor Costan2019-11-212-26/+28
| | | | PiperOrigin-RevId: 281815695
* unsigned char -> uint8_tVictor Costan2019-05-281-3/+4
| | | | PiperOrigin-RevId: 250309603
* Switch to using C++ 11 override specifier.Chris Mumford2019-05-095-45/+45
| | | | PiperOrigin-RevId: 247491163
* Style cleanup.Victor Costan2019-05-043-46/+47
| | | | | | | | | 1) Convert iterator-based for loops to C++11 foreach loops. 2) Convert "void operator=" to "T& operator=". 3) Switch from copy operators from private to public deleted. 4) Switch from empty ctors / dtors to "= default" where appropriate. PiperOrigin-RevId: 246679195
* Merge pull request #278 from wankai:masterVictor Costan2019-05-041-1/+1
|\ | | | | | | PiperOrigin-RevId: 246591372
| * block_builder header file dependency fixedWankai Zhang2015-01-291-1/+1
| |
* | Correct class/structure declaration order.Chris Mumford2019-05-036-41/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | 1. Correct the class/struct declaration order to be IAW the Google C++ style guide[1]. 2. For non-copyable classes, switched from non-implemented private methods to explicitly deleted[2] methods. 3. Minor const and member initialization fixes. [1] https://google.github.io/styleguide/cppguide.html#Declaration_Order [2] http://eel.is/c++draft/dcl.fct.def.delete PiperOrigin-RevId: 246521844
* | Format all files IAW the Google C++ Style Guide.Chris Mumford2019-05-0218-336/+271
| | | | | | | | | | | | Use clang-format to correct formatting to be in agreement with the [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html). Doing this simplifies the process of accepting changes. Also fixed a few warnings flagged by clang-tidy. PiperOrigin-RevId: 246350737
* | Check for possibly invalid offset in test.Chris Mumford2019-04-231-1/+1
| | | | | | | | | | | | | | | | Fix a possible array bounds offset issue flagged in issue #668. Not the source of any known bug, but will silence any static analyzers. PiperOrigin-RevId: 243697659
* | Clean up Iterator.costan2018-06-041-28/+33
| | | | | | | | | | | | | | | | | | | | | | This CL renames the private struct Iterator::Cleanup -> Iterator::CleanupNode, to better reflect that it's a linked list node, and extracts duplicated code from its user in IsEmpty() and Run() methods. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=199175058
* | Replace NULL with nullptr in C++ files.costan2018-04-109-71/+71
| | | | | | | | | | | | ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=192365747
* | Remove extern from function declarations.costan2018-03-123-6/+6
| | | | | | | | | | | | | | | | | | | | | | External linkage is the default for function declarations in C++. This also fixes ClangTidy errors generated by removing the "extern" keyword as described above. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=188730416
* | Simplify Table::Open() flow and remove a delete call.costan2017-09-011-8/+3
| | | | | | | | | | | | ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=167303843
* | Convert documentation to markdown.cmumford2017-03-011-1/+1
| | | | | | | | | | | | | | | | | | Markdown is more readable in a text editor and when hosted on GitHub is more readable than HTML. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=148830423
* | A zippy change broke test assumptions about the size of compressed output.sanjay2016-07-061-6/+14
| | | | | | | | | | | | | | Fix the tests by allowing more slop in zippy's behavior. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=123432472
* | Fix LevelDB build when asserts are enabled in release builds. (#367)John Abd-El-Malek2016-04-151-2/+1
| | | | | | | | | | | | | | | | | | | | * Fix LevelDB build when asserts are enabled in release builds. BUG=https://bugs.chromium.org/p/chromium/issues/detail?id=603166 * fix * Add comment
* | Deleted redundant null ptr check prior to delete.cmumford2016-03-311-1/+1
| | | | | | | | | | | | | | Fixes issue #338. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=113439460
* | Fix size_t/int comparison/conversion issues in leveldb.pkasting2015-12-091-2/+2
|/ | | | | | | | | The create function took |num_keys| as an int, but callers and implementers wanted it to function as a size_t (e.g. passing std::vector::size() in, passing it to vector constructors as a size arg, indexing containers by it, etc.). This resulted in implicit conversions between the two types as well as warnings (found with Chromium's external copy of these sources, built with MSVC) about signed vs. unsigned comparisons. The leveldb sources were already widely using size_t elsewhere, e.g. for key and filter lengths, so using size_t here is not inconsistent with the existing code. However, it does change the public C API. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=101074871
* Clean up layering of storage/leveldb/...Chris Mumford2014-12-111-0/+3
| | | | | With these changes, this package should be properly cleaned up and not require any further changes.
* Release 1.18v1.18Chris Mumford2014-09-164-5/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
* Release LevelDB 1.15v1.15David Grogan2013-12-101-1/+1
| | | | | | | | | | | | | | | | | | | - switched from mmap based writing to simpler stdio based writing. Has a minor impact (0.5 microseconds) on microbenchmarks for asynchronous writes. Synchronous writes speed up from 30ms to 10ms on linux/ext4. Should be much more reliable on diverse platforms. - compaction errors now immediately put the database into a read-only mode (until it is re-opened). As a downside, a disk going out of space and then space being created will require a re-open to recover from, whereas previously that would happen automatically. On the plus side, many corruption possibilities go away. - force the DB to enter an error-state so that all future writes fail when a synchronous log write succeeds but the sync fails. - repair now regenerates sstables that exhibit problems - fix issue 218 - Use native memory barriers on OSX - fix issue 212 - QNX build is broken - fix build on iOS with xcode 5 - make tests compile and pass on windows
* Release leveldb 1.10v1.10David Grogan2013-05-143-7/+37
| | | | | | | | | | | | | | | Fixes issues 147 - thanks feniksgordonfreeman 153 156 166 Additionally, * Remove calls to exit(1). * Fix unused-variable warnings from clang. * Fix possible overflow error related to num_restart value >= (2^32/4). * Add leveldbutil to .gitignore. * Add better log messages when Write is stalled on a compaction.
* Update to leveldb 1.6v1.6David Grogan2012-10-121-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Highlights ---------- Mmap at most 1000 files on Posix to improve performance for large databases. Support for more architectures (thanks to Alexander K.) Building and porting -------------------- HP/UX support (issue 126) AtomicPointer for ia64 (issue 123) Sparc v9 support (issue 124) Atomic ops for powerpc Use -fno-builtin-memcmp only when using g++ Simplify IOS build rules (issue 114) Use CXXFLAGS instead of CFLAGS when invoking C++ compiler (issue 118) Fix snappy shared library problem (issue 94) Fix shared library installation path regression Endian-ness detection tweak for FreeBSD Bug fixes --------- Stop ignoring FLAGS_open_files in db_bench Make bloom test behavior agnostic to endian-ness Performance ----------- Limit number of mmapped files to 1000 to improve perf for large dbs Do not delay for 1 second on shutdown path (issue 125) Misc ---- Make InMemoryEnv return a no-op logger C binding now has a wrapper for free (issue 117) Add thread-safety annotations Added an in-process lock table (issue 120) Make RandomAccessFile and SequentialFile non-copyable
* Added bloom filter support.v1.4Sanjay Ghemawat2012-04-1710-63/+500
| | | | | | | | | | | | | | | | | | | | In particular, we add a new FilterPolicy class. An instance of this class can be supplied in Options when opening a database. If supplied, the instance is used to generate summaries of keys (e.g., a bloom filter) which are placed in sstables. These summaries are consulted by DB::Get() so we can avoid reading sstable blocks that are guaranteed to not contain the key we are looking for. This change provides one implementation of FilterPolicy based on bloom filters. Other changes: - Updated version number to 1.4. - Some build tweaks. - C binding for CompactRange. - A few more benchmarks: deleteseq, deleterandom, readmissing, seekrandom. - Minor .gitignore update.
* use mmap on 64-bit machines to speed-up reads; small build fixesSanjay Ghemawat2012-03-156-17/+35
|
* A number of fixes:Hans Wennborg2011-10-3114-19/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Replace raw slice comparison with a call to user comparator. Added test for custom comparators. - Fix end of namespace comments. - Fixed bug in picking inputs for a level-0 compaction. When finding overlapping files, the covered range may expand as files are added to the input set. We now correctly expand the range when this happens instead of continuing to use the old range. For example, suppose L0 contains files with the following ranges: F1: a .. d F2: c .. g F3: f .. j and the initial compaction target is F3. We used to search for range f..j which yielded {F2,F3}. However we now expand the range as soon as another file is added. In this case, when F2 is added, we expand the range to c..j and restart the search. That picks up file F1 as well. This change fixes a bug related to deleted keys showing up incorrectly after a compaction as described in Issue 44. (Sync with upstream @25072954)
* Speed up Snappy uncompression, new Logger interface.gabor@google.com2011-07-211-8/+12
| | | | | | | | | | | | | | | | | | | | - Removed one copy of an uncompressed block contents changing the signature of Snappy_Uncompress() so it uncompresses into a flat array instead of a std::string. Speeds up readrandom ~10%. - Instead of a combination of Env/WritableFile, we now have a Logger interface that can be easily overridden applications that want to supply their own logging. - Separated out the gcc and Sun Studio parts of atomic_pointer.h so we can use 'asm', 'volatile' keywords for Sun Studio. git-svn-id: https://leveldb.googlecode.com/svn/trunk@39 62dab493-f737-651d-591e-8d6aee1b9529
* Sun Studio support, and fix for test related memory fixes.gabor@google.com2011-07-191-2/+7
| | | | | | | | | | | | - LevelDB patch for Sun Studio Based on a patch submitted by Theo Schlossnagle - thanks! This fixes Issue 17. - Fix a couple of test related memory leaks. git-svn-id: https://leveldb.googlecode.com/svn/trunk@38 62dab493-f737-651d-591e-8d6aee1b9529
* A number of smaller fixes and performance improvements:gabor@google.com2011-06-221-5/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Implemented Get() directly instead of building on top of a full merging iterator stack. This speeds up the "readrandom" benchmark by up to 15-30%. - Fixed an opensource compilation problem. Added --db=<name> flag to control where the database is placed. - Automatically compact a file when we have done enough overlapping seeks to that file. - Fixed a performance bug where we would read from at least one file in a level even if none of the files overlapped the key being read. - Makefile fix for Mac OSX installations that have XCode 4 without XCode 3. - Unified the two occurrences of binary search in a file-list into one routine. - Found and fixed a bug where we would unnecessarily search the last file when looking for a key larger than all data in the level. - A fix to avoid the need for trivial move compactions and therefore gets rid of two out of five syncs in "fillseq". - Removed the MANIFEST file write when switching to a new memtable/log-file for a 10-20% improvement on fill speed on ext4. - Adding a SNAPPY setting in the Makefile for folks who have Snappy installed. Snappy compresses values and speeds up writes. git-svn-id: https://leveldb.googlecode.com/svn/trunk@32 62dab493-f737-651d-591e-8d6aee1b9529
* sync with upstream @21627589dgrogan@chromium.org2011-06-021-9/+9
| | | | | | | | | | | Minor changes: * Reformat the bodies of the iterator interface routines in IteratorWrapper to make them a bit easier to read * Switched the default in the leveldb makefile to be optimized mode, rather than debug mode * Fix build problem in chromium port git-svn-id: https://leveldb.googlecode.com/svn/trunk@30 62dab493-f737-651d-591e-8d6aee1b9529
* Update from upstream @21551990dgrogan@chromium.org2011-05-282-7/+5
| | | | | | | | * Patch LevelDB to build for OSX and iOS * Fix race condition in memtable iterator deletion. * Other small fixes. git-svn-id: https://leveldb.googlecode.com/svn/trunk@29 62dab493-f737-651d-591e-8d6aee1b9529
* sync with upstream @ 21409451dgrogan@chromium.org2011-05-212-6/+10
| | | | | | Check the NEWS file for details of what changed. git-svn-id: https://leveldb.googlecode.com/svn/trunk@28 62dab493-f737-651d-591e-8d6aee1b9529
* fix build on at least linuxdgrogan@chromium.org2011-04-201-1/+1
| | | | git-svn-id: https://leveldb.googlecode.com/svn/trunk@25 62dab493-f737-651d-591e-8d6aee1b9529
* @20776309dgrogan@chromium.org2011-04-203-4/+6
| | | | | | | | | * env_chromium.cc should not export symbols. * Fix MSVC warnings. * Removed large value support. * Fix broken reference to documentation file git-svn-id: https://leveldb.googlecode.com/svn/trunk@24 62dab493-f737-651d-591e-8d6aee1b9529
* reverting disastrous MOE commit, returning to r21dgrogan@chromium.org2011-04-1915-0/+2518
| | | | git-svn-id: https://leveldb.googlecode.com/svn/trunk@23 62dab493-f737-651d-591e-8d6aee1b9529
* Revision created by MOE tool push_codebase.dgrogan@chromium.org2011-04-1915-2518/+0
| | | | | | | MOE_MIGRATION= git-svn-id: https://leveldb.googlecode.com/svn/trunk@22 62dab493-f737-651d-591e-8d6aee1b9529
* chmod a-xdgrogan@chromium.org2011-04-1815-0/+0
| | | | git-svn-id: https://leveldb.googlecode.com/svn/trunk@21 62dab493-f737-651d-591e-8d6aee1b9529
* @20602303. Default file permission is now 755.dgrogan@chromium.org2011-04-1215-2/+2
| | | | git-svn-id: https://leveldb.googlecode.com/svn/trunk@20 62dab493-f737-651d-591e-8d6aee1b9529
* Move include files into a leveldb subdir.jorlow@chromium.org2011-03-3013-25/+25
| | | | git-svn-id: https://leveldb.googlecode.com/svn/trunk@18 62dab493-f737-651d-591e-8d6aee1b9529
View Lorry Controller Status