| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
| |
PiperOrigin-RevId: 520556840
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change implements support for Zstd-based compression in LevelDB. Building
up from the Snappy compression (which has been supported since inception), this
change adds Zstd as an alternate compression algorithm.
We are implementing this to provide alternative options for users who might
have different performance and efficiency requirements. For instance, the
Zstandard website (https://facebook.github.io/zstd/) claims that the Zstd
algorithm can achieve around 30% higher compression ratios than Snappy, with
relatively smaller (~10%) slowdowns in de/compression speeds.
Benchmarking results:
$ blaze-bin/third_party/leveldb/db_bench
LevelDB: version 1.23
Date: Thu Feb 2 18:50:06 2023
CPU: 56 * Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
CPUCache: 35840 KB
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
RawSize: 110.6 MB (estimated)
FileSize: 62.9 MB (estimated)
------------------------------------------------
fillseq : 2.613 micros/op; 42.3 MB/s
fillsync : 3924.432 micros/op; 0.0 MB/s (1000 ops)
fillrandom : 3.609 micros/op; 30.7 MB/s
overwrite : 4.508 micros/op; 24.5 MB/s
readrandom : 6.136 micros/op; (864322 of 1000000 found)
readrandom : 5.446 micros/op; (864083 of 1000000 found)
readseq : 0.180 micros/op; 613.3 MB/s
readreverse : 0.321 micros/op; 344.7 MB/s
compact : 827043.000 micros/op;
readrandom : 4.603 micros/op; (864105 of 1000000 found)
readseq : 0.169 micros/op; 656.3 MB/s
readreverse : 0.315 micros/op; 350.8 MB/s
fill100K : 854.009 micros/op; 111.7 MB/s (1000 ops)
crc32c : 1.227 micros/op; 3184.0 MB/s (4K per op)
snappycomp : 3.610 micros/op; 1081.9 MB/s (output: 55.2%)
snappyuncomp : 0.691 micros/op; 5656.3 MB/s
zstdcomp : 15.731 micros/op; 248.3 MB/s (output: 44.1%)
zstduncomp : 4.218 micros/op; 926.2 MB/s
PiperOrigin-RevId: 509957778
|
| |\
| |
| |
| | |
PiperOrigin-RevId: 461612590
|
| |/ |
|
| |
|
|
|
|
|
| |
The benchmark in db/db_test.cc is extracted to its own file,
benchmarks/db_bench_log.cc.
PiperOrigin-RevId: 418713499
|
| |
|
|
|
|
| |
Using %zu for size_t instead of %ld.
PiperOrigin-RevId: 357976882
|
| |
|
|
| |
PiperOrigin-RevId: 351442409
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is beneficial when iterators are reused and seeks are not random
but increasing. It is additionally beneficial with larger block sizes and keys with common prefixes.
Add a benchmark "seekordered" to db_bench that reuses iterators across
increasing seeks. Add support to the benchmark to count comparisons made and to support common key prefix length. Change benchmark random seeds to be reproducible for entire benchmark suite executions but unique for threads in different benchmarks runs. This changes a benchmark suite of readrandom,seekrandom from having a 100% found ratio as previously it had the same seed used for fillrandom.
./db_bench --benchmarks=fillrandom,compact,seekordered --block_size=262144 --comparisons=1 --key_prefix=100
without this change (though with benchmark changes):
seekrandom : 55.309 micros/op; (631820 of 1000000 found)
Comparisons: 27001049
seekordered : 1.732 micros/op; (631882 of 1000000 found)
Comparisons: 26998402
with this change:
seekrandom : 55.866 micros/op; (631820 of 1000000 found)
Comparisons: 26952143
seekordered : 1.686 micros/op; (631882 of 1000000 found)
Comparisons: 25549369
For ordered seeking, this is a reduction of 5% comparisons and a 3% speedup. For random seeking (with single use iterators) the comparisons and speed are less than 1% and likely noise.
PiperOrigin-RevId: 351149832
|
| |
|
|
| |
PiperOrigin-RevId: 309110431
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This CL makes the following substitutions.
* assert.h -> cassert
* math.h -> cmath
* stdarg.h -> cstdarg
* stddef.h -> cstddef
* stdint.h -> cstdint
* stdio.h -> cstdio
* stdlib.h -> cstdlib
* string.h -> cstring
PiperOrigin-RevId: 309080151
|
| |
|
|
|
|
|
| |
leveldb::Env::DeleteFile was replaced with leveldb::Env::RemoveFile in
all tests. This allows us to remove workarounds for windows.h #defining
DeleteFile.
PiperOrigin-RevId: 289121105
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The "DeleteFile" method name causes pain for Windows developers, because
<windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA.
Current code uses workarounds, like #undefining DeleteFile everywhere an
Env is declared, implemented, or used.
This CL removes the need for workarounds by renaming Env::DeleteFile to
Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to
Env::RemoveDir. A few internal methods are also renamed for consistency.
Software that supports Windows is expected to migrate any Env
implementations and usage to Remove{File,Dir}, and never use the name
Env::Delete{File,Dir} in its code.
The renaming is done in a backwards-compatible way, at the risk of
making it slightly more difficult to build a new correct Env
implementation. The backwards compatibility is achieved using the
following hacks:
1) Env::Remove{File,Dir} methods are added, with a default
implementation that calls into Env::Delete{File,Dir}. This makes old
Env implementations compatible with code that calls into the updated
API.
2) The Env::Delete{File,Dir} methods are no longer pure virtuals.
Instead, they gain a default implementation that calls into
Env::Remove{File,Dir}. This makes updated Env implementations
compatible with code that calls into the old API.
The cost of this approach is that it's possible to write an Env without
overriding either Rename{File,Dir} or Delete{File,Dir}, without getting
a compiler warning. However, attempting to run the test suite will
immediately fail with an infinite call stack ending in
{Remove,Delete}{File,Dir}, making developers aware of the problem.
PiperOrigin-RevId: 288710907
|
| |
|
|
|
|
| |
Env::DeleteFile.
PiperOrigin-RevId: 283607548
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The SQLite-specific schema feature is documented at
https://www.sqlite.org/withoutrowid.html and
https://www.sqlite.org/rowidtable.html.
By default, SQLite stores each table in a B-tree keyed by an integer,
called the ROWID. Any index, including the PRIMARY KEY index, is a
separate B-tree mapping index keys to ROWIDs. Tables without ROWIDs are
stored in a B-tree keyed by the primary key. Additional indexes (the
PRIMARY KEY index is implicitly built into the table) are stored as
B-trees mapping index keys to row primary keys.
This CL introduces a boolean --use-rowids flag to db_bench_sqlite. When
the flag is false (default), the schema of the test table includes
WITHOUT ROWID. The test table uses a primary key, so adding WITHOUT
ROWID to the schema reduces the number of B-trees used by the benchmark
from 2 to 1. This brings SQLite's disk usage closer to LevelDB.
When WITHOUT ROWID is used, SQLite fares better (than today) on
benchmarks with small (16-byte) keys, and worse on benchmarks with large
(100kb) keys.
Baseline results:
fillseq : 21.310 micros/op; 5.2 MB/s
fillseqsync : 146.377 micros/op; 0.8 MB/s (10000 ops)
fillseqbatch : 2.065 micros/op; 53.6 MB/s
fillrandom : 34.767 micros/op; 3.2 MB/s
fillrandsync : 159.943 micros/op; 0.7 MB/s (10000 ops)
fillrandbatch : 15.055 micros/op; 7.3 MB/s
overwrite : 43.660 micros/op; 2.5 MB/s
overwritebatch : 27.691 micros/op; 4.0 MB/s
readrandom : 12.725 micros/op;
readseq : 2.602 micros/op; 36.7 MB/s
fillrand100K : 606.333 micros/op; 157.3 MB/s (1000 ops)
fillseq100K : 657.457 micros/op; 145.1 MB/s (1000 ops)
readseq : 46.523 micros/op; 2049.9 MB/s
readrand100K : 54.943 micros/op;
Results after this CL:
fillseq : 16.231 micros/op; 6.8 MB/s
fillseqsync : 147.460 micros/op; 0.8 MB/s (10000 ops)
fillseqbatch : 2.294 micros/op; 48.2 MB/s
fillrandom : 27.871 micros/op; 4.0 MB/s
fillrandsync : 141.979 micros/op; 0.8 MB/s (10000 ops)
fillrandbatch : 16.087 micros/op; 6.9 MB/s
overwrite : 26.829 micros/op; 4.1 MB/s
overwritebatch : 19.014 micros/op; 5.8 MB/s
readrandom : 11.657 micros/op;
readseq : 0.155 micros/op; 615.0 MB/s
fillrand100K : 816.812 micros/op; 116.8 MB/s (1000 ops)
fillseq100K : 754.689 micros/op; 126.4 MB/s (1000 ops)
readseq : 47.112 micros/op; 2024.3 MB/s
readrand100K : 287.679 micros/op;
Results after this CL, with --use-rowids=1
fillseq : 20.655 micros/op; 5.4 MB/s
fillseqsync : 146.408 micros/op; 0.8 MB/s (10000 ops)
fillseqbatch : 2.045 micros/op; 54.1 MB/s
fillrandom : 34.080 micros/op; 3.2 MB/s
fillrandsync : 154.582 micros/op; 0.7 MB/s (10000 ops)
fillrandbatch : 14.404 micros/op; 7.7 MB/s
overwrite : 42.928 micros/op; 2.6 MB/s
overwritebatch : 27.829 micros/op; 4.0 MB/s
readrandom : 12.835 micros/op;
readseq : 2.483 micros/op; 38.4 MB/s
fillrand100K : 603.265 micros/op; 158.1 MB/s (1000 ops)
fillseq100K : 662.473 micros/op; 144.0 MB/s (1000 ops)
readseq : 45.478 micros/op; 2097.0 MB/s
readrand100K : 54.439 micros/op;
PiperOrigin-RevId: 283407101
|
| |
|
|
|
|
| |
Avoids a use before initialization error. This fixes issue #676.
PiperOrigin-RevId: 246855204
|
|
|
Currently, the benchmark used to assess leveldb changes lives in db/. The codebase also contains two benchmarks against other database engines in doc/bench/. Moving all the benchmarks in one place opens up the way for extracting common code.
PiperOrigin-RevId: 246737541
|
