WT-3120 Fix ordering problem in connection_close for filesystem loaded in an extension (#3261)

This commit represents fixes for Coverity errors, LeakSanitizer errors, and additional cleanup:
* pread/pwrite return value is -1 on error, but the error is in errno.

* Convert size_t and off_t to uintmax_t/PRIuMAX, not uint64_t/PRIu64.

* Coverity ID 1369085 (#1 of 1): Extra sizeof expression (SIZEOF_MISMATCH)
suspicious_pointer_arithmetic: Adding allocated * 8UL /* sizeof (char
*) */ to pointer entries of type char ** is suspicious because adding
an integral value to this pointer automatically scales that value by the
size, 8 bytes, of the pointed-to type, char *. Most likely, the
multiplication by sizeof (char *) in this expression is extraneous and
should be eliminated.

* CID 1369084 (#1 of 1): Resource leak (RESOURCE_LEAK) 9. overwrite_var:
Overwriting handle ret in ret = 12 leaks the handle.

* CID 1369083 (#1 of 1): Logically dead code (DEADCODE) dead_error_line: Execution cannot reach this statement: while (count > 0U) null:
At condition entries != NULL, the value of entries must be NULL. dead_error_condition: The condition entries != NULL cannot be true.

* Custom filesystems have to configure early-load, otherwise we'll have already configured a default filesystem by the time the extension is loaded.

* Add early-load configuration to the wt3120_filesys test.

* Add code to WiredTiger that fails if a custom filesystem is configured after we've already configured a default filesystem.

1 files changed, 8 insertions, 5 deletions

diff --git a/examples/c/ex_file_system.c b/examples/c/ex_file_system.c
index 56869171558..e807ac54d3b 100644
--- a/examples/c/ex_file_system.c
+++ b/examples/c/ex_file_system.c
@@ -399,6 +399,7 @@ demo_fs_directory_list(WT_FILE_SYSTEM *file_system,
 	uint32_t allocated, count;
 	int ret = 0;
 	char *name, **entries;
+	void *p;
 
 	(void)session;						/* Unused */
 
@@ -424,14 +425,16 @@ demo_fs_directory_list(WT_FILE_SYSTEM *file_system,
 		 * matter if the list is a bit longer than necessary.
 		 */
 		if (count >= allocated) {
-			entries = realloc(
-			    entries, (allocated + 10) * sizeof(char *));
-			if (entries == NULL) {
+			p = realloc(
+			    entries, (allocated + 10) * sizeof(*entries));
+			if (p == NULL) {
 				ret = ENOMEM;
 				goto err;
 			}
-			memset(entries + allocated * sizeof(char *),
-			    0, 10 * sizeof(char *));
+
+			entries = p;
+			memset(entries + allocated * sizeof(*entries),
+			    0, 10 * sizeof(*entries));
 			allocated += 10;
 		}
 		entries[count++] = strdup(name);