summaryrefslogtreecommitdiff
path: root/src/third_party/wiredtiger/src/docs/compression.dox
blob: 74bed5c6f682537fa43fbf3eaea56176ea8cd843 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
/*! @m_page{{c,java},compression,Compressors}

This section explains how to configure WiredTiger's builtin support for
the lz4, snappy, zlib and zstd compression engines.

@section compression_lz4 Using LZ4 compression

To use the builtin support for Yann Collet's
<a href="https://github.com/Cyan4973/lz4">LZ4</a>
compression, first check that LZ4 is installed in include and library
directories searched by the compiler.  Once LZ4 is installed, you can
enable LZ4 using the \c --enable-lz4 option to configure.

If LZ4 is installed in a location not normally searched by the
compiler toolchain, you'll need to modify the \c CPPFLAGS and \c LDFLAGS
to indicate these locations.  For example, with the LZ4 includes and
libraries installed in \c /usr/local/include and \c /usr/local/lib, you
would run configure with the following additional arguments:

@code
--enable-lz4 CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/include"
@endcode

When opening the WiredTiger database, load the LZ4 shared library as
an extension.  For example, with the WiredTiger library installed in
\c /usr/local/lib, you would use the following extension:

@snippet ex_all.c Configure lz4 extension

Finally, when creating the WiredTiger object, set \c block_compressor
to \c lz4:

@snippet ex_all.c Create a lz4 compressed table

@section compression_snappy Using snappy compression

To use the builtin support for
<a href="http://code.google.com/p/snappy/">Google's snappy</a>
compression, first check that snappy is installed in include and library
directories searched by the compiler.  Once snappy is installed, you can
enable snappy using the \c --enable-snappy option to configure.

If snappy is installed in a location not normally searched by the
compiler toolchain, you'll need to modify the \c CPPFLAGS and \c LDFLAGS
to indicate these locations.  For example, with the snappy includes and
libraries installed in \c /usr/local/include and \c /usr/local/lib, you
would run configure with the following additional arguments:

@code
--enable-snappy CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/include"
@endcode

When opening the WiredTiger database, load the snappy shared library as
an extension.  For example, with the WiredTiger library installed in
\c /usr/local/lib, you would use the following extension:

@snippet ex_all.c Configure snappy extension

Finally, when creating the WiredTiger object, set \c block_compressor
to \c snappy:

@snippet ex_all.c Create a snappy compressed table

@section compression_zlib Using zlib compression

To use the builtin support for Greg Roelofs' and Mark Adler's
<a href="http://www.zlib.net/">zlib</a>
compression, first check that zlib is installed in include and library
directories searched by the compiler.  Once zlib is installed, you can
enable zlib using the \c --enable-zlib option to configure.

If zlib is installed in a location not normally searched by the compiler
toolchain, you'll need to modify the \c CPPFLAGS and \c LDFLAGS to
indicate these locations.  For example, with the zlib includes and
libraries installed in \c /usr/local/include and \c /usr/local/lib, you
would run configure with the following additional arguments:

@code
--enable-zlib CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/include"
@endcode

When opening the WiredTiger database, load the zlib shared library as
an extension.  For example, with the WiredTiger library installed in
\c /usr/local/lib, you would use the following extension:

@snippet ex_all.c Configure zlib extension

The default compression level for the zlib compression is
\c Z_DEFAULT_COMPRESSION (see the zlib documentation for further
information); compression can be configured to other levels using the
additional configuration argument \c compression_level.

@snippet ex_all.c Configure zlib extension with compression level

Finally, when creating the WiredTiger object, set \c block_compressor
to \c zlib:

@snippet ex_all.c Create a zlib compressed table

@section compression_zstd Using Zstd compression

To use the builtin support for Facebook's
<a href="https://github.com/facebook/zstd">Zstd</a>
compression, first check that Zstd is installed in include and library
directories searched by the compiler.  Once Zstd is installed, you can
enable Zstd using the \c --enable-zstd option to configure.

If Zstd is installed in a location not normally searched by the
compiler toolchain, you'll need to modify the \c CPPFLAGS and \c LDFLAGS
to indicate these locations.  For example, with the Zstd includes and
libraries installed in \c /usr/local/include and \c /usr/local/lib, you
would run configure with the following additional arguments:

@code
--enable-zstd CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/include"
@endcode

When opening the WiredTiger database, load the Zstd shared library as
an extension.  For example, with the WiredTiger library installed in
\c /usr/local/lib, you would use the following extension:

@snippet ex_all.c Configure zstd extension

The default compression level for the zstd compression is 3; compression
can be configured to other levels using the additional configuration
argument \c compression_level.

@snippet ex_all.c Configure zstd extension with compression level

Finally, when creating the WiredTiger object, set \c block_compressor
to \c zstd:

@snippet ex_all.c Create a zstd compressed table

@section compression_upgrading Upgrading compression engines

WiredTiger does not store information with file blocks to identify the
compression engine used to compressed the block.  Applications wanting
to upgrade to some future compression engine (without requiring a file
dump and re-load), should ensure each compressed block includes enough
information to identify the compression engine used, so its compression
code can correctly decompress old and new blocks.

@section compression_custom Custom compression engines

WiredTiger may be extended by adding custom compression engines; see
WT_COMPRESSOR for more information.

 */