WT-2552 Add public API for pluggable filesystems (#2671)

* WT-2552 Add public API for pluggable filesystems

Not yet compiling. The main parts of this change should be here,
but it involved extensive parameter re-organization. There are also
a number of layering violations between our existing file system
implementations and the WT_FH, that aren't possible with the new
structure.

There are a number of specific todo comments in the code. One of the main
issues is that the in-memory file system had a special close semantic
that relied on WiredTiger handle tracking. The in-memory file-system should
do it's own tracking of file handles, I've gone part way down that road by
adding a queue for closed handles. Need to also add in live handles, and
manage the queue as appropriate.

I haven't created an example application that uses the new API yet.

* WT-2552 Add public API for pluggable filesystems

I always forget you have to remove the already-built html files when
changing PREDEFINED, add a reminder to the complaint.

* WT-2552 Add public API for pluggable filesystems

You have to remove the .js files, too.

* WT-2552 Add public API for pluggable filesystems

Make dist/s_all run cleanly.

* WT-2552 Add public API for pluggable filesystems

Whitespace.

* WT-2552 Add public API for pluggable filesystems

Make it compile/build/lint.

* WT-2552 Add public API for pluggable filesystems

block_write.c: In function '__wt_block_extend':
block_write.c:130:71: error: missing terminating ' character [-Werror]

* WT-2552 Add public API for pluggable filesystems

os_fs_inmemory.c: In function '__im_file_truncate':
os_fs_inmemory.c:344:10: error: 'session' is used uninitialized in this
function [-Werror=uninitialized]

* WT-2552 Add public API for pluggable filesystems

os_fs.c: In function '__posix_directory_sync':
os_fs.c:92:10: error: 'session' is used uninitialized in this function
[-Werror=uninitialized]

* WT-2552 Add public API for pluggable filesystems

Go back to using bool types in the file-system API, this requires we add
<stdbool.h> to the "standard" wiredtiger.h includes.

Consistently use wt_session to represent a WT_SESSION, we were using
"wtsession" in some places.

Make a pass over the Windows code, but I'm sure it doesn't compile yet.

* WT-2552 Add public API for pluggable filesystems

Fix up another couple of bool types.

* WT-2552 Add public API for pluggable filesystems

Move the file naming work out of the underlying filesystem functions,
the calls to __wt_filename are now in the upper-level code,n os_fs.i;
that means the filesystem code is no longer responsible for figuring out
paths. This is cleaner, although the directory-sync call is a bit of a
kluge, and I've commimtted us to handling NULL filesystem methods.

With this set of changes, in-memory runs again.

More Windows naming fixes.

* WT-2552 Add public API for pluggable filesystems

os_fs.c: In function '__posix_directory_sync':
os_fs.c:96:3: error: label 'err' used but not defined

* WT-2552 Add public API for pluggable filesystems

Pull out another call to __wt_filename() from the filesystem-dependent
code.

* WT-2552 Add public API for pluggable filesystems

Consistently check for missing file-system methods when doing
file-system calls.

Other minor lint & cleanup.

* WT-2552 Add public API for pluggable filesystems

Change the in-memory code to maintain a complete list of the files it
has ever opened, and depend on that list instead of reaching up into the
common layer for the WT_FH handle list.

This means __wt_handle_search is only used by the common WT_FH handle
code, simplify it, and add a __wt_handle_is_open function that can be
called for diagnostic purposes (to check for open files that are being
renamed or removed, for example).

* Fix comiler warning and ignore the file system API in Java

* Flesh out the example file system implementation.

* Add in some plumbing for set_file_system in wiredtiger_open.

* WT-2552 Add public API for pluggable filesystems

Whitespace.

* WT-2552 Add public API for pluggable filesystems

WT_CONFIG_ITEM.val isn't a boolean, don't use boolean types in
equal/not-equal comparisons.

* WT-2552 Add public API for pluggable filesystems

Remove unused #includes.

Increment/decrement the DEMO_FILE_SYSTEM.{opened,closed}_file_count.

Allocate demo structures, they're larger than the underlying structures.
Swap the number/size calloc arguments, number comes first.

Fix a couple of statics.

* WT-2552 Add public API for pluggable filesystems

Use %u instead of casting to %d.

* WT-2552 Add public API for pluggable filesystems

Add ex_file_system.c to the list of example programs.

* WT-2552 Add public API for pluggable filesystems

Change ex_file_system.c to not require <wt_internal.h>: strip down a
copy of FreeBSD's <queue.h> for local inclusion, rewrite a few other
minor pieces of code.

* WT-2552 Add public API for pluggable filesystems

Update spell check info

* WT-2552 Add public API for pluggable filesystems

__conn_load_extensions() shouldn't set the "early" boolean to true.

* WT-2552 Add public API for pluggable filesystems

Don't indirect through a NULL pointer if "local" was set and no path was
specified, always set the name to something useful.

* WT-2552 Add public API for pluggable filesystems

Don't indirect through a NULL pointer if "local" was set and no path was
specified, always set the name to something useful.

* WT-2552 Add public API for pluggable filesystems

wt_off_t vs. size_t conversion lint.

* WT-2552 Add public API for pluggable filesystems

Add -rdynamic to the load for ex_file_system, the main executable
symbols are not exported by default.

* WT-2552 Add public API for pluggable filesystems

The underlying handle name includes the enclosing directory,
compare against the WT_FH.name field instead.

* WT-2552 Add public API for pluggable filesystems

demo_fs_rename should return 0 if successful, simplify error handling

Don't bother casting arguments to free(), it's not necessary.

* WT-2552 Add public API for pluggable filesystems

General WT_FILE_SYSTEM cleanup.

Move OS initialization into the wiredtiger_open() code (the
os_common/os_init.c file is no longer needed).

Allow early-load extensions to be part of the environment settings,
matching the "in-memory" and "readonly" configurations.

Syntax check the set of a file-system, remove tests for NULL methods in
the file-system structure unless it's legal for them to be NULL.

Windows, POSIX and in-memory file systems now set WT_FILE_SYSTEM.terminate,
call that function to cleanup when discarding a WT_CONNECTION.

Export file-type and open-flags constants for WT_FILE_SYSTEM.open_file,
sort the WT_FILE_SYSTEM methods, do an editing pass.

Change the WT_FILE_HANDLE type from (const char *) to (char *), it's
"owned" by the underlying layer, and it's simpler that way.

Minor (untested) cleanup of the Windows WT_FILE_SYSTEM.open-file method.

* WT-2552 Add public API for pluggable filesystems

Export the advise argument #defines for the WT_FILE_HANDLE.fadvise method.

Sort the WT_FILE_HANDLE methods.

* WT-2552 Add public API for pluggable filesystems

Clean up and simplify WT_FILE_SYSTEM/WT_FILE_HANDLE documentation's
description of the handles.

* WT-2552 Add public API for pluggable filesystems

WT_FILE_HANDLE.close is a required function (at the least, it
has to free the memory).

WT_FILE_HANDLE.fadvise isn't a required function, if it's not
configured, don't call it.

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_HANDLE.lock function is required.

Change the __wt_open() signature to match WT_FILE_SYSTEM.open_file().

* WT-2552 Add public API for pluggable filesystems

Rework all of the WT_FILE_HANDLE mapped region methods to be optional.

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_HANDLE.{read,size} methods are required.
The WT_FILE_HANDLE.sync method is not required.

Split the WT_FILE_HANDLE.sync method into .sync and .sync_nowait versions,
it makes the upper-level code simpler (Windows supports .sync but doesn't
support .sync_nowait).

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_HANDLE.{truncate,write} methods are required IFF the file
is not readonly.

* WT-2552 Add public API for pluggable filesystems

POSIX shouldn't declare a no-sync handle function unless the
sync_file_range system call is available.

* WT-2552 Add public API for pluggable filesystems

Typo, missing semi-colon.

* Fix a bug in ex_file_system.c

* Fix a memory leak in posix file handle implementation

* WT-2552 Use the correct flags when opening backup file.

* WT-2552 Add public API for pluggable filesystems

Simplify open-file error handling by calling the close function on the
handle, that way we won't forget to free all of the applicable memory
allocations.

* WT-2552 Add public API for pluggable filesystems

Simplify the directory-list method, don't pass in an include/exclude
file, if prefix is non-NULL, it implies we only want files matching
the prefix.

* WT-2552 Add public API for pluggable filesystems

Replace WT_FILE_HANDLE_POSIX.fallocate_{available,requires_locking} wiht
WT_FILE_HANDLE.fallocate and WT_FILE_HANDLE.fallocate_nolock.

Example code doesn't need to set WT_FILE_HANDLE methods to NULL, the
allocation does that.

Free the I/O buffer if open-handle allocation fails in the example code.

Remove snippets for WT_FILE_SYSTEM and WT_FILE_HANDLE methods, we're
not going to provide example code for them.

* WT-2552 Add public API for pluggable filesystems

Document we expect either ENOTSUP or EBUSY from optionally supported
APIs. Review/cleanups ENOTSUP/EBUSY returns from optionally supported
APIs.

Make WT_FILE_HANDLE.lock optional.

Don't configure or call the POSIX fadvise function on files configured
for direct I/O.

Rename __wt_filesize_name to __wt_size for consistency.

Update the spelling list.

* WT-2552 Add public API for pluggable filesystems

WT_FILE_HANDLE.truncate requires locking in all known implementations,
document it is not called concurrently with other operations.

* WT-2552 Add public API for pluggable filesystems

Don't terminate the filesystem unless we've actually configured one.

* WT-2552 Add public API for pluggable filesystems

Remove WT_FILE_SYSTEM and WT_FILE_HANDLE from SWIG so the test suite
can pass again.

* WT-2552 Add public API for pluggable filesystems

Merge __conn_load_early_extensions() and __conn_load_extensions().

Fix a problem where I moved the early extensions load to where it could
include the WiredTiger environment variable, but I didn't pass the built
cfg into the function.

* WT-2552 Add public API for pluggable filesystems

Linux build typo.

* WT-2552 Add public API for pluggable filesystems

Get rid of the "bool silent" argument to WT_FILE_SYSTEM.size by testing
for the file's existence before requesting the size (an extra system
call, but guaranteed to hit in the buffer cache at least).

* WT-2552 Add public API for pluggable filesystems

Naming consistency pass over the WT_FILE_SYSTEM functions.

* WT-2552 Add public API for pluggable filesystems

Fix a spin lock mismatch.

* WT-2552 Add public API for pluggable filesystems

Another spinlock mismatch.

* Update example pluggable file system.

Add a directory list implementation to the example, which uncovered
an issue with the API. The directory list API allocates memory that
is freed by WiredTiger, which I don't think is kosher.

* Change file-directory-sync to use reguar fsync.

The distinction in os_fs.i doesn't work with the filesystem API.

Also add directory_sync application to the example application.

* WT-2552 Add public API for pluggable filesystems

Whitespace.

* WT-2552 Add public API for pluggable filesystems

Rewrite __wt_free to not evaluate macro arguments multiple times.

* WT-2552 Add public API for pluggable filesystems

Simplify the directory-list functions: __wt_realloc_def() already
handles scaling the size of the allocations, there's no need to
involve a separate constant that increments the allocation size.

* WT-2552 Add public API for pluggable filesystems

Fix a grouping problem in a realloc call, we need to multiple the size
times the previously allocated slots + 10.

Fix buffer overrun, if "count" has already been incremented, the memset
would skip clearing the first slot and clear one slot past the end of
the buffer.

Remove a comment, realloc requires clearing allocated memory, it's not
paranoia.

* WT-2552 Add public API for pluggable filesystems

Add the mapping-cookie argument to the map-preload and map-discard
functions.

Change page-discard to stop reaching down through the block manager,
instead, provide a block-manager map-discard function that does the
work.

* WT-2552 Add public API for pluggable filesystems

Require a directory-list function.

Implement a directory-list function for the in-memory filesystem.

Consistency pass, make all the directory-list functions look the same.

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_SYSTEM.{directory_sync, remove, rename} methods are not
required for read-only systems.

* WT-2552 Add public API for pluggable filesystems

Change the WT_FILE_SYSTEM.open_file file_type argument from a set of
constants to an enum.

This requires changing how we store connection direct I/O configuration
(the constants used to be flags stored in the WT_CONNECTION_IMPL), and
requiring all callers of __wt_open() do their own work to figure out if
WT_OPEN_DIRECTIO should be specified.

* WT-2552 Add public API for pluggable filesystems

Make no guarantees WT_FILE_SYSTEM and WT_FILE_HANDLE methods are
not called concurrently (except for WT_FILE_HANDLE::fallocate and
WT_FILE_HANDLE::fallocate_nolock).

Rewrite the in-memory FS code to lock across all methods (for example,
WT_FILE_HANDLE.close), that means including a reference to the enclosing
WT_FILE_SYSTEM in the WT_FILE_HANDLE structure so we can find a lock
without using the WT_CONNECTION_IMPL structure.

* WT-2552 Add public API for pluggable filesystems

Remove __wt_directory_sync_fh, it's no longer useful.

* WT-2552 Add public API for pluggable filesystems

Rename WT_INMEMORY_FILE_SYSTEM to WT_FILE_SYSTEM_INMEM, matching
WT_FILE_HANDLE_INMEM.

* WT-2552 Add public API for pluggable filesystems

Add WT_FILE_SYSTEM.directory_list_free, to free memory allocated
by WT_FILE_SYSTEM.direct_list.

Fix a memory leak in __log_archive_once (if __wt_readlock failed,
we leaked the directory-list memory).

* WT-2552 Add public API for pluggable filesystems

Typo, check WT_DIRECT_IO_LOG, not WT_DIRECT_IO_CHECKPOINT.

* WT-2552 Add public API for pluggable filesystems

Typo, unreachable code.

* WT-2552 Add public API for pluggable filesystems

We don't require WT_FILE_SYSTEM.{remove,rename} if the system is
read-only.

* Fix Windows build with pluggable file system.

Involved removing u_int from the public API.

* Fix line wrapping.

* Fix Windows terminate function.

* Forgot something in my last commit.

* Fix Windows munmap bug.

* Add new example to Windows build. Extend example to be more complete.

* Fix example loading on Windows

* Update documentation

* Add missing spell words

* Remove old comment.

3 files changed, 150 insertions, 63 deletions

diff --git a/src/conn/conn_api.c b/src/conn/conn_api.c
index 4efa853851e..18ad383ec74 100644
--- a/src/conn/conn_api.c
+++ b/src/conn/conn_api.c
@@ -806,6 +806,7 @@ static int
 __conn_load_default_extensions(WT_CONNECTION_IMPL *conn)
 {
 	WT_UNUSED(conn);
+
 #ifdef HAVE_BUILTIN_EXTENSION_SNAPPY
 	WT_RET(snappy_extension_init(&conn->iface, NULL));
 #endif
@@ -819,18 +820,16 @@ __conn_load_default_extensions(WT_CONNECTION_IMPL *conn)
 }
 
 /*
- * __conn_load_extension --
- *	WT_CONNECTION->load_extension method.
+ * __conn_load_extension_int --
+ *	Internal extension load interface
  */
 static int
-__conn_load_extension(
-    WT_CONNECTION *wt_conn, const char *path, const char *config)
+__conn_load_extension_int(WT_SESSION_IMPL *session,
+    const char *path, const char *cfg[], bool early_load)
 {
 	WT_CONFIG_ITEM cval;
-	WT_CONNECTION_IMPL *conn;
 	WT_DECL_RET;
 	WT_DLH *dlh;
-	WT_SESSION_IMPL *session;
 	int (*load)(WT_CONNECTION *, WT_CONFIG_ARG *);
 	bool is_local;
 	const char *init_name, *terminate_name;
@@ -839,8 +838,10 @@ __conn_load_extension(
 	init_name = terminate_name = NULL;
 	is_local = strcmp(path, "local") == 0;
 
-	conn = (WT_CONNECTION_IMPL *)wt_conn;
-	CONNECTION_API_CALL(conn, session, load_extension, config, cfg);
+	/* Ensure that the load matches the phase of startup we are in. */
+	WT_ERR(__wt_config_gets(session, cfg, "early_load", &cval));
+	if ((cval.val == 0 && early_load) || (cval.val != 0 && !early_load))
+		return (0);
 
 	/*
 	 * This assumes the underlying shared libraries are reference counted,
@@ -865,20 +866,39 @@ __conn_load_extension(
 	    __wt_dlsym(session, dlh, terminate_name, false, &dlh->terminate));
 
 	/* Call the load function last, it simplifies error handling. */
-	WT_ERR(load(wt_conn, (WT_CONFIG_ARG *)cfg));
+	WT_ERR(load(&S2C(session)->iface, (WT_CONFIG_ARG *)cfg));
 
 	/* Link onto the environment's list of open libraries. */
-	__wt_spin_lock(session, &conn->api_lock);
-	TAILQ_INSERT_TAIL(&conn->dlhqh, dlh, q);
-	__wt_spin_unlock(session, &conn->api_lock);
+	__wt_spin_lock(session, &S2C(session)->api_lock);
+	TAILQ_INSERT_TAIL(&S2C(session)->dlhqh, dlh, q);
+	__wt_spin_unlock(session, &S2C(session)->api_lock);
 	dlh = NULL;
 
 err:	if (dlh != NULL)
 		WT_TRET(__wt_dlclose(session, dlh));
 	__wt_free(session, init_name);
 	__wt_free(session, terminate_name);
+	return (ret);
+}
 
-	API_END_RET_NOTFOUND_MAP(session, ret);
+/*
+ * __conn_load_extension --
+ *	WT_CONNECTION->load_extension method.
+ */
+static int
+__conn_load_extension(
+    WT_CONNECTION *wt_conn, const char *path, const char *config)
+{
+	WT_CONNECTION_IMPL *conn;
+	WT_DECL_RET;
+	WT_SESSION_IMPL *session;
+
+	conn = (WT_CONNECTION_IMPL *)wt_conn;
+	CONNECTION_API_CALL(conn, session, load_extension, config, cfg);
+
+	ret = __conn_load_extension_int(session, path, cfg, false);
+
+err:	API_END_RET_NOTFOUND_MAP(session, ret);
 }
 
 /*
@@ -886,18 +906,16 @@ err:	if (dlh != NULL)
  *	Load the list of application-configured extensions.
  */
 static int
-__conn_load_extensions(WT_SESSION_IMPL *session, const char *cfg[])
+__conn_load_extensions(
+    WT_SESSION_IMPL *session, const char *cfg[], bool early_load)
 {
 	WT_CONFIG subconfig;
 	WT_CONFIG_ITEM cval, skey, sval;
-	WT_CONNECTION_IMPL *conn;
 	WT_DECL_ITEM(exconfig);
 	WT_DECL_ITEM(expath);
 	WT_DECL_RET;
-
-	conn = S2C(session);
-
-	WT_ERR(__conn_load_default_extensions(conn));
+	const char *sub_cfg[] = {
+	    WT_CONFIG_BASE(session, WT_CONNECTION_load_extension), NULL, NULL };
 
 	WT_ERR(__wt_config_gets(session, cfg, "extensions", &cval));
 	WT_ERR(__wt_config_subinit(session, &subconfig, &cval));
@@ -912,8 +930,9 @@ __conn_load_extensions(WT_SESSION_IMPL *session, const char *cfg[])
 			WT_ERR(__wt_buf_fmt(session,
 			    exconfig, "%.*s", (int)sval.len, sval.str));
 		}
-		WT_ERR(conn->iface.load_extension(&conn->iface,
-		    expath->data, (sval.len > 0) ? exconfig->data : NULL));
+		sub_cfg[1] = sval.len > 0 ? exconfig->data : NULL;
+		WT_ERR(__conn_load_extension_int(
+		    session, expath->data, sub_cfg, early_load));
 	}
 	WT_ERR_NOTFOUND_OK(ret);
 
@@ -1192,12 +1211,12 @@ __conn_config_file(WT_SESSION_IMPL *session,
 	fh = NULL;
 
 	/* Configuration files are always optional. */
-	WT_RET(__wt_exist(session, filename, &exist));
+	WT_RET(__wt_fs_exist(session, filename, &exist));
 	if (!exist)
 		return (0);
 
 	/* Open the configuration file. */
-	WT_RET(__wt_open(session, filename, WT_FILE_TYPE_REGULAR, 0, &fh));
+	WT_RET(__wt_open(session, filename, WT_OPEN_FILE_TYPE_REGULAR, 0, &fh));
 	WT_ERR(__wt_filesize(session, fh, &size));
 	if (size == 0)
 		goto err;
@@ -1488,8 +1507,8 @@ __conn_single(WT_SESSION_IMPL *session, const char *cfg[])
 	 */
 	exist = false;
 	if (!is_create)
-		WT_ERR(__wt_exist(session, WT_WIREDTIGER, &exist));
-	ret = __wt_open(session, WT_SINGLETHREAD, WT_FILE_TYPE_REGULAR,
+		WT_ERR(__wt_fs_exist(session, WT_WIREDTIGER, &exist));
+	ret = __wt_open(session, WT_SINGLETHREAD, WT_OPEN_FILE_TYPE_REGULAR,
 	    is_create || exist ? WT_OPEN_CREATE : 0, &conn->lock_fh);
 
 	/*
@@ -1545,7 +1564,7 @@ __conn_single(WT_SESSION_IMPL *session, const char *cfg[])
 
 	/* We own the lock file, optionally create the WiredTiger file. */
 	ret = __wt_open(session, WT_WIREDTIGER,
-	    WT_FILE_TYPE_REGULAR, is_create ? WT_OPEN_CREATE : 0, &fh);
+	    WT_OPEN_FILE_TYPE_REGULAR, is_create ? WT_OPEN_CREATE : 0, &fh);
 
 	/*
 	 * If we're read-only, check for success as well as handled errors.
@@ -1582,7 +1601,7 @@ __conn_single(WT_SESSION_IMPL *session, const char *cfg[])
 	 * and there's never a database home after that point without a turtle
 	 * file. If the turtle file doesn't exist, it's a create.
 	 */
-	WT_ERR(__wt_exist(session, WT_METADATA_TURTLE, &exist));
+	WT_ERR(__wt_fs_exist(session, WT_METADATA_TURTLE, &exist));
 	conn->is_new = exist ? 0 : 1;
 
 	if (conn->is_new) {
@@ -1789,7 +1808,7 @@ __conn_write_base_config(WT_SESSION_IMPL *session, const char *cfg[])
 	 * only NOT exist if we crashed before it was created; in other words,
 	 * if the base configuration file exists, we're done.
 	 */
-	WT_RET(__wt_exist(session, WT_BASECONFIG, &exist));
+	WT_RET(__wt_fs_exist(session, WT_BASECONFIG, &exist));
 	if (exist)
 		return (0);
 
@@ -1864,6 +1883,57 @@ err:		WT_TRET(__wt_fclose(session, &fs));
 }
 
 /*
+ * __conn_set_file_system --
+ *	Configure a custom file system implementation on database open.
+ */
+static int
+__conn_set_file_system(
+    WT_CONNECTION *wt_conn, WT_FILE_SYSTEM *file_system, const char *config)
+{
+	WT_CONNECTION_IMPL *conn;
+	WT_DECL_RET;
+	WT_SESSION_IMPL *session;
+
+	conn = (WT_CONNECTION_IMPL *)wt_conn;
+	CONNECTION_API_CALL(conn, session, set_file_system, config, cfg);
+	WT_UNUSED(cfg);
+
+	conn->file_system = file_system;
+
+err:	API_END_RET(session, ret);
+}
+
+/*
+ * __conn_chk_file_system --
+ *	Check the configured file system.
+ */
+static int
+__conn_chk_file_system(WT_SESSION_IMPL *session, bool readonly)
+{
+	WT_CONNECTION_IMPL *conn;
+
+	conn = S2C(session);
+
+#define	WT_CONN_SET_FILE_SYSTEM_REQ(name)				\
+	if (conn->file_system->name == NULL)				\
+		WT_RET_MSG(session, EINVAL,				\
+		    "a WT_FILE_SYSTEM.%s method must be configured", #name)
+
+	WT_CONN_SET_FILE_SYSTEM_REQ(directory_list);
+	WT_CONN_SET_FILE_SYSTEM_REQ(directory_list_free);
+	/* not required: directory_sync */
+	WT_CONN_SET_FILE_SYSTEM_REQ(exist);
+	WT_CONN_SET_FILE_SYSTEM_REQ(open_file);
+	if (!readonly) {
+		WT_CONN_SET_FILE_SYSTEM_REQ(remove);
+		WT_CONN_SET_FILE_SYSTEM_REQ(rename);
+	}
+	WT_CONN_SET_FILE_SYSTEM_REQ(size);
+
+	return (0);
+}
+
+/*
  * wiredtiger_open --
  *	Main library entry point: open a new connection to a WiredTiger
  *	database.
@@ -1887,12 +1957,13 @@ wiredtiger_open(const char *home, WT_EVENT_HANDLER *event_handler,
 		__conn_add_compressor,
 		__conn_add_encryptor,
 		__conn_add_extractor,
+		__conn_set_file_system,
 		__conn_get_extension_api
 	};
 	static const WT_NAME_FLAG file_types[] = {
-		{ "checkpoint",	WT_FILE_TYPE_CHECKPOINT },
-		{ "data",	WT_FILE_TYPE_DATA },
-		{ "log",	WT_FILE_TYPE_LOG },
+		{ "checkpoint",	WT_DIRECT_IO_CHECKPOINT },
+		{ "data",	WT_DIRECT_IO_DATA },
+		{ "log",	WT_DIRECT_IO_LOG },
 		{ NULL, 0 }
 	};
 
@@ -1982,10 +2053,27 @@ wiredtiger_open(const char *home, WT_EVENT_HANDLER *event_handler,
 		F_SET(conn, WT_CONN_READONLY);
 
 	/*
-	 * After checking readonly and in-memory, but before we do anything that
-	 * touches the filesystem, configure the OS layer.
+	 * Load early extensions before doing further initialization (one early
+	 * extension is to configure a file system).
 	 */
-	WT_ERR(__wt_os_init(session));
+	WT_ERR(__conn_load_extensions(session, cfg, true));
+
+	/*
+	 * If the application didn't configure its own file system, configure
+	 * one of ours. Check to ensure we have a valid file system.
+	 */
+	if (conn->file_system == NULL) {
+		if (F_ISSET(conn, WT_CONN_IN_MEMORY))
+			WT_ERR(__wt_os_inmemory(session));
+		else
+#if defined(_MSC_VER)
+			WT_ERR(__wt_os_win(session));
+#else
+			WT_ERR(__wt_os_posix(session));
+#endif
+	}
+	WT_ERR(
+	    __conn_chk_file_system(session, F_ISSET(conn, WT_CONN_READONLY)));
 
 	/*
 	 * Capture the config_base setting file for later use. Again, if the
@@ -2118,8 +2206,8 @@ wiredtiger_open(const char *home, WT_EVENT_HANDLER *event_handler,
 		if (ret == 0) {
 			if (sval.val)
 				FLD_SET(conn->direct_io, ft->flag);
-		} else if (ret != WT_NOTFOUND)
-			goto err;
+		} else
+			WT_ERR_NOTFOUND_OK(ret);
 	}
 
 	WT_ERR(__wt_config_gets(session, cfg, "write_through", &cval));
@@ -2128,8 +2216,8 @@ wiredtiger_open(const char *home, WT_EVENT_HANDLER *event_handler,
 		if (ret == 0) {
 			if (sval.val)
 				FLD_SET(conn->write_through, ft->flag);
-		} else if (ret != WT_NOTFOUND)
-			goto err;
+		} else
+			WT_ERR_NOTFOUND_OK(ret);
 	}
 
 	/*
@@ -2153,15 +2241,15 @@ wiredtiger_open(const char *home, WT_EVENT_HANDLER *event_handler,
 		ret = __wt_config_subgets(session, &cval, ft->name, &sval);
 		if (ret == 0) {
 			switch (ft->flag) {
-			case WT_FILE_TYPE_DATA:
+			case WT_DIRECT_IO_DATA:
 				conn->data_extend_len = sval.val;
 				break;
-			case WT_FILE_TYPE_LOG:
+			case WT_DIRECT_IO_LOG:
 				conn->log_extend_len = sval.val;
 				break;
 			}
-		} else if (ret != WT_NOTFOUND)
-			goto err;
+		} else
+			WT_ERR_NOTFOUND_OK(ret);
 	}
 
 	WT_ERR(__wt_config_gets(session, cfg, "mmap", &cval));
@@ -2190,7 +2278,8 @@ wiredtiger_open(const char *home, WT_EVENT_HANDLER *event_handler,
 	 * everything else to be in place, and the extensions call back into the
 	 * library.
 	 */
-	WT_ERR(__conn_load_extensions(session, cfg));
+	WT_ERR(__conn_load_default_extensions(conn));
+	WT_ERR(__conn_load_extensions(session, cfg, false));
 
 	/*
 	 * The metadata/log encryptor is configured after extensions, since
diff --git a/src/conn/conn_handle.c b/src/conn/conn_handle.c
index 5f4c38e7361..509966793e5 100644
--- a/src/conn/conn_handle.c
+++ b/src/conn/conn_handle.c
@@ -149,15 +149,17 @@ __wt_connection_destroy(WT_CONNECTION_IMPL *conn)
 		__wt_spin_destroy(session, &conn->page_lock[i]);
 	__wt_free(session, conn->page_lock);
 
+	/* Destroy the file-system configuration. */
+	if (conn->file_system != NULL && conn->file_system->terminate != NULL)
+		WT_TRET(conn->file_system->terminate(
+		    conn->file_system, (WT_SESSION *)session));
+
 	/* Free allocated memory. */
 	__wt_free(session, conn->cfg);
 	__wt_free(session, conn->home);
 	__wt_free(session, conn->error_prefix);
 	__wt_free(session, conn->sessions);
 
-	/* Destroy the OS configuration. */
-	WT_TRET(__wt_os_cleanup(session));
-
 	__wt_free(NULL, conn);
 	return (ret);
 }
diff --git a/src/conn/conn_log.c b/src/conn/conn_log.c
index 672071b59bf..394378b65fc 100644
--- a/src/conn/conn_log.c
+++ b/src/conn/conn_log.c
@@ -178,6 +178,7 @@ __log_archive_once(WT_SESSION_IMPL *session, uint32_t backup_file)
 	conn = S2C(session);
 	log = conn->log;
 	logcount = 0;
+	locked = false;
 	logfiles = NULL;
 
 	/*
@@ -198,14 +199,14 @@ __log_archive_once(WT_SESSION_IMPL *session, uint32_t backup_file)
 	 * Main archive code.  Get the list of all log files and
 	 * remove any earlier than the minimum log number.
 	 */
-	WT_RET(__wt_dirlist(session, conn->log_path,
-	    WT_LOG_FILENAME, WT_DIRLIST_INCLUDE, &logfiles, &logcount));
+	WT_ERR(__wt_fs_directory_list(
+	    session, conn->log_path, WT_LOG_FILENAME, &logfiles, &logcount));
 
 	/*
 	 * We can only archive files if a hot backup is not in progress or
 	 * if we are the backup.
 	 */
-	WT_RET(__wt_readlock(session, conn->hot_backup_lock));
+	WT_ERR(__wt_readlock(session, conn->hot_backup_lock));
 	locked = true;
 	if (!conn->hot_backup || backup_file != 0) {
 		for (i = 0; i < logcount; i++) {
@@ -218,9 +219,6 @@ __log_archive_once(WT_SESSION_IMPL *session, uint32_t backup_file)
 	}
 	WT_ERR(__wt_readunlock(session, conn->hot_backup_lock));
 	locked = false;
-	__wt_log_files_free(session, logfiles, logcount);
-	logfiles = NULL;
-	logcount = 0;
 
 	/*
 	 * Indicate what is our new earliest LSN.  It is the start
@@ -232,8 +230,7 @@ __log_archive_once(WT_SESSION_IMPL *session, uint32_t backup_file)
 err:		__wt_err(session, ret, "log archive server error");
 	if (locked)
 		WT_TRET(__wt_readunlock(session, conn->hot_backup_lock));
-	if (logfiles != NULL)
-		__wt_log_files_free(session, logfiles, logcount);
+	WT_TRET(__wt_fs_directory_list_free(session, &logfiles, &logcount));
 	return (ret);
 }
 
@@ -259,10 +256,10 @@ __log_prealloc_once(WT_SESSION_IMPL *session)
 	 * Allocate up to the maximum number, accounting for any existing
 	 * files that may not have been used yet.
 	 */
-	WT_ERR(__wt_dirlist(session, conn->log_path,
-	    WT_LOG_PREPNAME, WT_DIRLIST_INCLUDE, &recfiles, &reccount));
-	__wt_log_files_free(session, recfiles, reccount);
-	recfiles = NULL;
+	WT_ERR(__wt_fs_directory_list(
+	    session, conn->log_path, WT_LOG_PREPNAME, &recfiles, &reccount));
+	WT_ERR(__wt_fs_directory_list_free(session, &recfiles, &reccount));
+
 	/*
 	 * Adjust the number of files to pre-allocate if we find that
 	 * the critical path had to allocate them since we last ran.
@@ -292,8 +289,7 @@ __log_prealloc_once(WT_SESSION_IMPL *session)
 
 	if (0)
 err:		__wt_err(session, ret, "log pre-alloc server error");
-	if (recfiles != NULL)
-		__wt_log_files_free(session, recfiles, reccount);
+	WT_TRET(__wt_fs_directory_list_free(session, &recfiles, &reccount));
 	return (ret);
 }
 
@@ -868,9 +864,9 @@ __wt_logmgr_create(WT_SESSION_IMPL *session, const char *cfg[])
 	    "log write LSN"));
 	WT_RET(__wt_rwlock_alloc(session,
 	    &log->log_archive_lock, "log archive lock"));
-	if (FLD_ISSET(conn->direct_io, WT_FILE_TYPE_LOG))
-		log->allocsize =
-		    WT_MAX((uint32_t)conn->buffer_alignment, WT_LOG_ALIGN);
+	if (FLD_ISSET(conn->direct_io, WT_DIRECT_IO_LOG))
+		log->allocsize = (uint32_t)
+		    WT_MAX(conn->buffer_alignment, WT_LOG_ALIGN);
 	else
 		log->allocsize = WT_LOG_ALIGN;
 	WT_INIT_LSN(&log->alloc_lsn);