WT-2552 Add public API for pluggable filesystems (#2671)

* WT-2552 Add public API for pluggable filesystems

Not yet compiling. The main parts of this change should be here,
but it involved extensive parameter re-organization. There are also
a number of layering violations between our existing file system
implementations and the WT_FH, that aren't possible with the new
structure.

There are a number of specific todo comments in the code. One of the main
issues is that the in-memory file system had a special close semantic
that relied on WiredTiger handle tracking. The in-memory file-system should
do it's own tracking of file handles, I've gone part way down that road by
adding a queue for closed handles. Need to also add in live handles, and
manage the queue as appropriate.

I haven't created an example application that uses the new API yet.

* WT-2552 Add public API for pluggable filesystems

I always forget you have to remove the already-built html files when
changing PREDEFINED, add a reminder to the complaint.

* WT-2552 Add public API for pluggable filesystems

You have to remove the .js files, too.

* WT-2552 Add public API for pluggable filesystems

Make dist/s_all run cleanly.

* WT-2552 Add public API for pluggable filesystems

Whitespace.

* WT-2552 Add public API for pluggable filesystems

Make it compile/build/lint.

* WT-2552 Add public API for pluggable filesystems

block_write.c: In function '__wt_block_extend':
block_write.c:130:71: error: missing terminating ' character [-Werror]

* WT-2552 Add public API for pluggable filesystems

os_fs_inmemory.c: In function '__im_file_truncate':
os_fs_inmemory.c:344:10: error: 'session' is used uninitialized in this
function [-Werror=uninitialized]

* WT-2552 Add public API for pluggable filesystems

os_fs.c: In function '__posix_directory_sync':
os_fs.c:92:10: error: 'session' is used uninitialized in this function
[-Werror=uninitialized]

* WT-2552 Add public API for pluggable filesystems

Go back to using bool types in the file-system API, this requires we add
<stdbool.h> to the "standard" wiredtiger.h includes.

Consistently use wt_session to represent a WT_SESSION, we were using
"wtsession" in some places.

Make a pass over the Windows code, but I'm sure it doesn't compile yet.

* WT-2552 Add public API for pluggable filesystems

Fix up another couple of bool types.

* WT-2552 Add public API for pluggable filesystems

Move the file naming work out of the underlying filesystem functions,
the calls to __wt_filename are now in the upper-level code,n os_fs.i;
that means the filesystem code is no longer responsible for figuring out
paths. This is cleaner, although the directory-sync call is a bit of a
kluge, and I've commimtted us to handling NULL filesystem methods.

With this set of changes, in-memory runs again.

More Windows naming fixes.

* WT-2552 Add public API for pluggable filesystems

os_fs.c: In function '__posix_directory_sync':
os_fs.c:96:3: error: label 'err' used but not defined

* WT-2552 Add public API for pluggable filesystems

Pull out another call to __wt_filename() from the filesystem-dependent
code.

* WT-2552 Add public API for pluggable filesystems

Consistently check for missing file-system methods when doing
file-system calls.

Other minor lint & cleanup.

* WT-2552 Add public API for pluggable filesystems

Change the in-memory code to maintain a complete list of the files it
has ever opened, and depend on that list instead of reaching up into the
common layer for the WT_FH handle list.

This means __wt_handle_search is only used by the common WT_FH handle
code, simplify it, and add a __wt_handle_is_open function that can be
called for diagnostic purposes (to check for open files that are being
renamed or removed, for example).

* Fix comiler warning and ignore the file system API in Java

* Flesh out the example file system implementation.

* Add in some plumbing for set_file_system in wiredtiger_open.

* WT-2552 Add public API for pluggable filesystems

Whitespace.

* WT-2552 Add public API for pluggable filesystems

WT_CONFIG_ITEM.val isn't a boolean, don't use boolean types in
equal/not-equal comparisons.

* WT-2552 Add public API for pluggable filesystems

Remove unused #includes.

Increment/decrement the DEMO_FILE_SYSTEM.{opened,closed}_file_count.

Allocate demo structures, they're larger than the underlying structures.
Swap the number/size calloc arguments, number comes first.

Fix a couple of statics.

* WT-2552 Add public API for pluggable filesystems

Use %u instead of casting to %d.

* WT-2552 Add public API for pluggable filesystems

Add ex_file_system.c to the list of example programs.

* WT-2552 Add public API for pluggable filesystems

Change ex_file_system.c to not require <wt_internal.h>: strip down a
copy of FreeBSD's <queue.h> for local inclusion, rewrite a few other
minor pieces of code.

* WT-2552 Add public API for pluggable filesystems

Update spell check info

* WT-2552 Add public API for pluggable filesystems

__conn_load_extensions() shouldn't set the "early" boolean to true.

* WT-2552 Add public API for pluggable filesystems

Don't indirect through a NULL pointer if "local" was set and no path was
specified, always set the name to something useful.

* WT-2552 Add public API for pluggable filesystems

Don't indirect through a NULL pointer if "local" was set and no path was
specified, always set the name to something useful.

* WT-2552 Add public API for pluggable filesystems

wt_off_t vs. size_t conversion lint.

* WT-2552 Add public API for pluggable filesystems

Add -rdynamic to the load for ex_file_system, the main executable
symbols are not exported by default.

* WT-2552 Add public API for pluggable filesystems

The underlying handle name includes the enclosing directory,
compare against the WT_FH.name field instead.

* WT-2552 Add public API for pluggable filesystems

demo_fs_rename should return 0 if successful, simplify error handling

Don't bother casting arguments to free(), it's not necessary.

* WT-2552 Add public API for pluggable filesystems

General WT_FILE_SYSTEM cleanup.

Move OS initialization into the wiredtiger_open() code (the
os_common/os_init.c file is no longer needed).

Allow early-load extensions to be part of the environment settings,
matching the "in-memory" and "readonly" configurations.

Syntax check the set of a file-system, remove tests for NULL methods in
the file-system structure unless it's legal for them to be NULL.

Windows, POSIX and in-memory file systems now set WT_FILE_SYSTEM.terminate,
call that function to cleanup when discarding a WT_CONNECTION.

Export file-type and open-flags constants for WT_FILE_SYSTEM.open_file,
sort the WT_FILE_SYSTEM methods, do an editing pass.

Change the WT_FILE_HANDLE type from (const char *) to (char *), it's
"owned" by the underlying layer, and it's simpler that way.

Minor (untested) cleanup of the Windows WT_FILE_SYSTEM.open-file method.

* WT-2552 Add public API for pluggable filesystems

Export the advise argument #defines for the WT_FILE_HANDLE.fadvise method.

Sort the WT_FILE_HANDLE methods.

* WT-2552 Add public API for pluggable filesystems

Clean up and simplify WT_FILE_SYSTEM/WT_FILE_HANDLE documentation's
description of the handles.

* WT-2552 Add public API for pluggable filesystems

WT_FILE_HANDLE.close is a required function (at the least, it
has to free the memory).

WT_FILE_HANDLE.fadvise isn't a required function, if it's not
configured, don't call it.

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_HANDLE.lock function is required.

Change the __wt_open() signature to match WT_FILE_SYSTEM.open_file().

* WT-2552 Add public API for pluggable filesystems

Rework all of the WT_FILE_HANDLE mapped region methods to be optional.

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_HANDLE.{read,size} methods are required.
The WT_FILE_HANDLE.sync method is not required.

Split the WT_FILE_HANDLE.sync method into .sync and .sync_nowait versions,
it makes the upper-level code simpler (Windows supports .sync but doesn't
support .sync_nowait).

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_HANDLE.{truncate,write} methods are required IFF the file
is not readonly.

* WT-2552 Add public API for pluggable filesystems

POSIX shouldn't declare a no-sync handle function unless the
sync_file_range system call is available.

* WT-2552 Add public API for pluggable filesystems

Typo, missing semi-colon.

* Fix a bug in ex_file_system.c

* Fix a memory leak in posix file handle implementation

* WT-2552 Use the correct flags when opening backup file.

* WT-2552 Add public API for pluggable filesystems

Simplify open-file error handling by calling the close function on the
handle, that way we won't forget to free all of the applicable memory
allocations.

* WT-2552 Add public API for pluggable filesystems

Simplify the directory-list method, don't pass in an include/exclude
file, if prefix is non-NULL, it implies we only want files matching
the prefix.

* WT-2552 Add public API for pluggable filesystems

Replace WT_FILE_HANDLE_POSIX.fallocate_{available,requires_locking} wiht
WT_FILE_HANDLE.fallocate and WT_FILE_HANDLE.fallocate_nolock.

Example code doesn't need to set WT_FILE_HANDLE methods to NULL, the
allocation does that.

Free the I/O buffer if open-handle allocation fails in the example code.

Remove snippets for WT_FILE_SYSTEM and WT_FILE_HANDLE methods, we're
not going to provide example code for them.

* WT-2552 Add public API for pluggable filesystems

Document we expect either ENOTSUP or EBUSY from optionally supported
APIs. Review/cleanups ENOTSUP/EBUSY returns from optionally supported
APIs.

Make WT_FILE_HANDLE.lock optional.

Don't configure or call the POSIX fadvise function on files configured
for direct I/O.

Rename __wt_filesize_name to __wt_size for consistency.

Update the spelling list.

* WT-2552 Add public API for pluggable filesystems

WT_FILE_HANDLE.truncate requires locking in all known implementations,
document it is not called concurrently with other operations.

* WT-2552 Add public API for pluggable filesystems

Don't terminate the filesystem unless we've actually configured one.

* WT-2552 Add public API for pluggable filesystems

Remove WT_FILE_SYSTEM and WT_FILE_HANDLE from SWIG so the test suite
can pass again.

* WT-2552 Add public API for pluggable filesystems

Merge __conn_load_early_extensions() and __conn_load_extensions().

Fix a problem where I moved the early extensions load to where it could
include the WiredTiger environment variable, but I didn't pass the built
cfg into the function.

* WT-2552 Add public API for pluggable filesystems

Linux build typo.

* WT-2552 Add public API for pluggable filesystems

Get rid of the "bool silent" argument to WT_FILE_SYSTEM.size by testing
for the file's existence before requesting the size (an extra system
call, but guaranteed to hit in the buffer cache at least).

* WT-2552 Add public API for pluggable filesystems

Naming consistency pass over the WT_FILE_SYSTEM functions.

* WT-2552 Add public API for pluggable filesystems

Fix a spin lock mismatch.

* WT-2552 Add public API for pluggable filesystems

Another spinlock mismatch.

* Update example pluggable file system.

Add a directory list implementation to the example, which uncovered
an issue with the API. The directory list API allocates memory that
is freed by WiredTiger, which I don't think is kosher.

* Change file-directory-sync to use reguar fsync.

The distinction in os_fs.i doesn't work with the filesystem API.

Also add directory_sync application to the example application.

* WT-2552 Add public API for pluggable filesystems

Whitespace.

* WT-2552 Add public API for pluggable filesystems

Rewrite __wt_free to not evaluate macro arguments multiple times.

* WT-2552 Add public API for pluggable filesystems

Simplify the directory-list functions: __wt_realloc_def() already
handles scaling the size of the allocations, there's no need to
involve a separate constant that increments the allocation size.

* WT-2552 Add public API for pluggable filesystems

Fix a grouping problem in a realloc call, we need to multiple the size
times the previously allocated slots + 10.

Fix buffer overrun, if "count" has already been incremented, the memset
would skip clearing the first slot and clear one slot past the end of
the buffer.

Remove a comment, realloc requires clearing allocated memory, it's not
paranoia.

* WT-2552 Add public API for pluggable filesystems

Add the mapping-cookie argument to the map-preload and map-discard
functions.

Change page-discard to stop reaching down through the block manager,
instead, provide a block-manager map-discard function that does the
work.

* WT-2552 Add public API for pluggable filesystems

Require a directory-list function.

Implement a directory-list function for the in-memory filesystem.

Consistency pass, make all the directory-list functions look the same.

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_SYSTEM.{directory_sync, remove, rename} methods are not
required for read-only systems.

* WT-2552 Add public API for pluggable filesystems

Change the WT_FILE_SYSTEM.open_file file_type argument from a set of
constants to an enum.

This requires changing how we store connection direct I/O configuration
(the constants used to be flags stored in the WT_CONNECTION_IMPL), and
requiring all callers of __wt_open() do their own work to figure out if
WT_OPEN_DIRECTIO should be specified.

* WT-2552 Add public API for pluggable filesystems

Make no guarantees WT_FILE_SYSTEM and WT_FILE_HANDLE methods are
not called concurrently (except for WT_FILE_HANDLE::fallocate and
WT_FILE_HANDLE::fallocate_nolock).

Rewrite the in-memory FS code to lock across all methods (for example,
WT_FILE_HANDLE.close), that means including a reference to the enclosing
WT_FILE_SYSTEM in the WT_FILE_HANDLE structure so we can find a lock
without using the WT_CONNECTION_IMPL structure.

* WT-2552 Add public API for pluggable filesystems

Remove __wt_directory_sync_fh, it's no longer useful.

* WT-2552 Add public API for pluggable filesystems

Rename WT_INMEMORY_FILE_SYSTEM to WT_FILE_SYSTEM_INMEM, matching
WT_FILE_HANDLE_INMEM.

* WT-2552 Add public API for pluggable filesystems

Add WT_FILE_SYSTEM.directory_list_free, to free memory allocated
by WT_FILE_SYSTEM.direct_list.

Fix a memory leak in __log_archive_once (if __wt_readlock failed,
we leaked the directory-list memory).

* WT-2552 Add public API for pluggable filesystems

Typo, check WT_DIRECT_IO_LOG, not WT_DIRECT_IO_CHECKPOINT.

* WT-2552 Add public API for pluggable filesystems

Typo, unreachable code.

* WT-2552 Add public API for pluggable filesystems

We don't require WT_FILE_SYSTEM.{remove,rename} if the system is
read-only.

* Fix Windows build with pluggable file system.

Involved removing u_int from the public API.

* Fix line wrapping.

* Fix Windows terminate function.

* Forgot something in my last commit.

* Fix Windows munmap bug.

* Add new example to Windows build. Extend example to be more complete.

* Fix example loading on Windows

* Update documentation

* Add missing spell words

* Remove old comment.

1 files changed, 1 insertions, 0 deletions

diff --git a/build_posix/aclocal/types.m4 b/build_posix/aclocal/types.m4
index 439034c89d2..089058f5611 100644
--- a/build_posix/aclocal/types.m4
+++ b/build_posix/aclocal/types.m4
@@ -7,6 +7,7 @@ AC_DEFUN([AM_TYPES], [
 #include <sys/types.h>
 #include <inttypes.h>
 #include <stdarg.h>
+#include <stdbool.h>
 #include <stdint.h>
 #include <stdio.h>"
 	AC_SUBST(wiredtiger_includes_decl)