summaryrefslogtreecommitdiff
path: root/dist
Commit message (Collapse)AuthorAgeFilesLines
* WT-2737 Scrub dirty pages rather than evicting them (#2889)Keith Bostic2016-07-212-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Link/include test/testutils with wtperf so we can use the test utility macros and functions in wtperf. * Minor changes to wtperf variables, sort some options, make some things boolean. * Merge the stub memory allocation functions used by the test code and wtperf. * Remove the WT_EVICT_CLEAN flag, it's never used. * __wt_page_can_evict() doesn't need to set WT_EVICTING, the only place that cares is __evict_review(), and it already sets that flag. * Add a new reconciliation flag WT_EVICT_SCRUB; it causes reconciliation to save disk images it creates. Change the eviction of multi-block pages to optionally re-instantiate pages in memory instead of evicting them. * When instantiating pages in-memory, set the read-generation to the same value as the read-generation of the original page (unless it's set to WT_READGEN_OLDEST, in which case leave the page's read-generation unset. * Simplify the in-memory tests some. * Turn on fixed-length store tests, they're no longer slow (no clue what changed). * Sue suggested removing Helium support from wtperf. * Replace dcalloc() calls before snprintf() calls with dmalloc. * Cosmetic cleanup: set the boundary structure's entries when initializing at the start of __rec_split_write(), don't set it at some random spot in the code. * After writing a block during checkpoint, we potentially re-use that block during eviction, and during eviction we'll want a disk image for potential re-instantiation in memory. If re-using a block but the boundary structure doesn't already have a disk image, create one. I don't think the boundary structure will ever already have a disk image in the current code, but future versions of reconciliation might, and the test is the same as the one we have to do for raw compression, which has already created the disk image. * If we're closing, don't instantiate any disk images, additionally, free any disk images we don't use. Both changes should fix the same problem where we have a disk image when we're truly discarding the page during a close, one should prevent us from ever having a disk image at that point, the other should prevent use from using any disk image we have. * Fixes for wtperf directory create changes: ensure the directory exists before the configuration file is dumped and create the right directories in multi-database mode.
* WT-2769 Update documentation to reflect correct limits of memory_page_max ↵Sulabh Mahajan2016-07-191-2/+2
| | | | (#2884)
* WT-2738 Remove the ability to change the default checkpoint name (#2877)Keith Bostic2016-07-151-2/+0
| | | We used to provide the ability to specify a custom checkpoint name that the checkpoint server would use. That feature increased code complexity and led to inconsistent checkpoint behavior.
* WT-2760 Fix a bug in backup related to directory sync. Change the filesystem ↵Keith Bostic2016-07-151-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | API to make durable the default (#2867) Change the default remove/rename calls to flush the enclosing directory. Simplify the pluggable file system API by replacing the directory-sync method with "durable" boolean argument to the remove, rename and open-file methods. * Add "durable" arguments to relevant functions so that each remove or rename call specifies its durability requirements. * Switch the WT_FILE_SYSTEM::fs_open_file type enum from WT_OPEN_FILE_TYPE, with WT_OPEN_XXX names, to the WT_FS_OPEN_FILE_TYPE, with WT_FS_OPEN_XXX names. Switch the WT_FILE_SYSTEM::fs_open_file flags from WT_OPEN_XXX names to WT_FS_OPEN_XXX names. * Replace the "bool durable" argument to WT_FILE_SYSTEM.fs_remove and WT_FILE_SYSTEM.fs_rename with a "uint32_t flags" argument, and the WT_FS_DURABLE flag. * Remove a stray bracket.
* WT-2711 Remove posix expanded strftime values and use older C89 values (#2865)David Hows2016-07-111-0/+1
| | | | | | | | | | * WT-2711 Remove posix expanded strftime values and use older C89 values * Fix issues with s_string * Add a comment so nobody rewrites the strftime format and reintroduces the bug. * Fix strings sort order.
* WT-2711 Change statistics log configuration options (#2834)Keith Bostic2016-07-082-68/+110
| | | | | | | | | No longer support setting the statistics_log path in WT_CONNECTION::reconfigure. No longer support setting a custom name for statistics files, only allow a destination directory. Be more explicit about which logging configuration options are allowed in WT_CONNECTION::reconfigure. The aim of these changes is to avoid situations where applications that embed WiredTiger allow their users to overwrite unexpected files on a file system. This potentially requires an upgrade step for applications that were specifying a non-standard file name component for statistics log file names, it's not backward compatible.
* WT-2746 Add a new statistic tracking I/O for checkpoints (#2853)Keith Bostic2016-07-082-1/+3
| | | Separate cache write statistics into checkpoint and non-checkpoint I/O.
* WT-2743 Fixup new I/O thread count statistics. (#2850)Alex Gorrod2016-07-061-8/+8
| | | | The increment and decrement were backward, so they never reported positive values. While I was there make the statistics macros consistent.
* WT-2739 pluggable file systems documentation cleanups (#2846)Keith Bostic2016-07-041-0/+1
| | | | | | | Document that readonly pluggable file systems may fail attempts to open files read/write with EACCES. Document that pluggable file systems should fail attempts to open non-existent files with ENOENT.
* WT-2380 Add option to s_all to return an error code when tests fail (#2843)Keith Bostic2016-07-011-5/+8
| | | | Replace bash arrays with Bourne shell constructs, we want this script to be portable and we don't know what shell is installed as /bin/sh.
* WT-2380 Add option to s_all to return an error code when tests fail (#2839)David Hows2016-07-011-0/+13
|
* WT-2665 Limit allocator fragmentation from the WiredTiger cache (#2799)Michael Cahill2016-06-303-16/+20
| | | | | | | | | | | | | | | | | | | | | | * Add a statistic for the amount of cache used for page images vs other objects * Reset the "maximum page size" statistic each checkpoint * Add a separate queue for urgent eviction, replaces existing WOULD_BLOCK eviction mode Until now, the eviction server has had to walk the cache to find pages that would otherwise block application threads. This change allows threads to put those pages directly on a queue that is checked before ordinary LRU eviction. * Don't queue pages for urgent eviction when eviction is disabled in a tree. Decouple evict_queue_lock and the individual queue's evict_lock in __evict_get_ref. Don't block the eviction server if there are many application threads waiting on a queue. * Take care with the urgent queue. Entries in the urgent queue cannot be skipped or they will not be considered again.
* WT-2729 Focus eviction on the largest trees in cache. (#2832)Michael Cahill2016-06-281-0/+1
| | | | | | | | | | | | | | | | | | Randomize visits to trees that use a tiny fraction of the cache. Other general eviction optimizations. Now that we are queuing more entries (potentially), make sure enough of them become candidates. Previously, a skewed distribution of read generations could mean that only 10% of queue entries were considered. Improve the efficiency of sorting the queue by calculating the score once when pages are added to the queue. Add a workload to exercise differential eviction from trees. In order to do that, add range partitioning to wtperf. Don't override icount in wtperf workloads with random_range set.
* SERVER-24580 Add more eviction stats to track efficiency. (#2830)Michael Cahill2016-06-231-0/+6
|
* WT-2408 Windows error translation layer (#2819)Keith Bostic2016-06-237-29/+41
| | | | | | | | | | | | Build a Windows-to-POSIX/ANSI error translation layer. Replace the read-only error mapping to WT_NOTFOUND and WT_PERM_DENIED with EACCES and ENOENT. Windows no longer needs its own version of __wt_strerror(), move the POSIX implementation from os_posix/os_errno.c to os_common/os_errno.c. Rename os_win/os_errno.c to os_win/os_winerr.c to avoid a collision. Windows now has DWORD types in prototypes, split the Windows/POSIX extern.h files. (This actually cleans up some noise, previously we had to sort the OS prototypes to remove duplicates, which wasn't trivial.) Add the WT_EXTENSION_API.map_windows_error method to map Windows system codes to POSIX/ANSI system codes.
* WT-2724 Pass in session address, not pointer to it. (#2827)sueloverso2016-06-231-0/+1
| | | Also assert that eviction workers don't exit earlier than expected.
* SERVER-24580 Update oldest txn ID with "strict, nowait" flags. (#2829)Michael Cahill2016-06-231-0/+4
| | | Add more options for callers when updating the oldest ID to control how much they care about the ID being updated.
* WT-2722 escape regular expression meta-character so egrep works on all ↵Don Anderson2016-06-221-1/+1
| | | | systems. (#2821)
* WT-2691 Use wrappers for ctype functions to avoid sign extension errors (#2818)Don Anderson2016-06-212-0/+27
| | | | | | | | | | | | | | * Cast arguments for ctype functions to avoid sign extension errors. Create __wt_* versions of all ctype functions, and use them whenever wt_internal.h is available. Add check to prevent direct use of ctype functions from core source. * Change wrappers to use u_char arguments, and return bool. Remove unused wrappers. * Use u_char in preference to unsigned char. * Examples do not have u_char defined on Windows.
* WT-2541 Add stats showing active readers and writers (#2793)David Hows2016-06-202-2/+28
| | | | | | | | | | | | | | | | | | | | * WT-2541 Add stats showing active readers and writers * Fix use of RET_MSG instead of ERR_MSG * Change where we increment the active r/w stats * Add "active fsync" stat * Add new "Thread State" stat section, add fsync_io stat * KNF * Make thread stats not backwards breaking. Add atomic stat functions * Add ThreadState stats to system and evict groups * KNF
* WT-2707 stricter checks in dist/s_label, and add some WT_ERR calls (#2800)Don Anderson2016-06-172-1/+37
| | | | | | | | | * WT-2707 Stricter tests for label tests: match variants of WT_ERR/WT_RET, and do not allow mixing of WT_ERR/WT_RET in a loop. * Change WT_RET to WT_ERR to get proper cleanup. * Minor simplification of the error case.
* WT-2605 Add C tests used in testing joins (#2787)Don Anderson2016-06-171-0/+1
|
* WT-2319 Add units to stats descriptions, change variable names (#2795)Sulabh Mahajan2016-06-101-4/+4
|
* WT-2671 dump more information about the file layout in verify debug mode (#2755)Keith Bostic2016-06-094-5/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * WT-2671: dump more information about the file layout in verify debug mode Dump the checkpoint final and correct sizes when loading checkpoints in verify dump_XXX mode. Dump the checkpoint alloc, discard and avail lists when loading checkpoints in verify dump_XXX mode. Fix a bug, convert the count of tree pages from uint32_t to uint64_t, there's no reason pages counts can't overflow 4B. Fix a bug, reset the internal/leaf page counts at the end of every checkpoint, we were doing cumulative counts. Fix a bug, where we were using "%03zu" for a size_t type, fix the dist/s_style script to catch that case. Add a divider line between checkpoints to make it easier to read the output. Add a new __wt_buf_set_size() function that displays byte values in human-readable format. Move __wt_buf_set_printable() from btree/bt_misc.c to support/scratch.c, it's a general-purpose function. Rename verify debug option from "dump_shape" to "dump_layout", we're dumping more than the tree shape now. * Merge the two versions of the "dump extent list" functions, and stop dumping all of the elements, they're far too many for that to ever be useful again. Instead, display a list by power-of-two bucket size. * Minor changes to make bucket sizes human readable. * block_ext.c:1477:3: error: format '%s' expects argument of type 'char *', but argument 3 has type 'const void *' [-Werror=format]
* WT-2514 Fix logging to not use an empty path name. (#2789)sueloverso2016-06-081-1/+1
|
* WT-2319 Add stats to measure count and duration of the sync calls made for ↵Sulabh Mahajan2016-06-081-0/+6
| | | | log and checkpoint modules (#2784)
* WT-2682 add option to configure WiredTiger with strict compiler flags (#2773)Keith Bostic2016-06-081-2/+13
| | | | | | Add the --enable-strict option that uses our current best guess at the right flags for strict compilation. While adding the option, expand the set of flags we can compile cleanly with.
* WT-2652 Remove unnecessary wt_ftruncate call. (#2786)sueloverso2016-06-081-0/+1
| | | Make file length the actual maximum set, not the maximum plus the header record.
* WT-2668 Create join statistics that are useful and are easy to understand. ↵Don Anderson2016-06-071-2/+4
| | | | (#2767)
* WT-2677 Fix JSON output so only printable ASCII is produced. (#2771)Don Anderson2016-06-031-1/+1
| | | | | | | | | | * WT-2677 Fix JSON output so only printable ASCII is produced. The root cause is a sign extension error that shows up on systems that have a signed 'char'. * WT-2677 Fix compile warnings for sign conversions * Include changes to auto-generated files.
* WT-2664 Add ability for eviction workers to populate eviction queues. (#2651)Michael Cahill2016-06-031-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add ability for eviction workers to populate eviction queues. * Investigating performance differences with develop. For a more fair comparison, added the ability to conditionally compile the code where the server dynamically decides whether or not to evict. Not ready for review/merge. * Fixed code to properly track internal cache state values used by the server to decide when to help evict. * Check if session is null. It will be on error path. * Fix incorrect boolean that we hold the lock. * With multiple threads able to populate the queue, don't sleep. * Remove unused ifdef. * Threads call into a top level wrapper instead of specialized functions. * Whitespace * Fix function name in comment. * WT_RET should be WT_ERR. * Whitespace. Put evict_lru_pages back in the loop. * Whitespace * Restore original evict_pass. Changes moved to higher level. * Whitespace * KNF * Fix the arg to the new common thread function. * Move eviction sleep outside lock. * Fix WT_RET/WT_ERR usage error. * Have callers of evict_exclusive_on clear the dhandle directly. * Have callers of evict_exclusive_on clear the dhandle lock directly. * Rename flag and add locking macro.
* WT-2672 handle system calls that don't set errno (#2769)Keith Bostic2016-06-031-0/+2
| | | | | | | | | | | | | | | | | | | * WT-2672 handle system calls that don't set errno Split WT_SYSCALL_RETRY into two parts: WT_SYSCALL which does the error return and errno mapping, and WT_SYSCALL_RETRY which actually retries calls. Quit retrying: clock_gettime, close, fcntl(F_SETLK), fstat gettimeofday, posix_fadvise, posix_madvise, pthread_create pthread_join, rename, stat, and unlink. Check for errors when closing directory filehandles, both POSIX and Windows. * Don't bother retrying the close of a directory file descriptor. * Review comment, pthread_create should retry to deal with EAGAIN.
* WT-2658 Only include PPC-specific files in PPC builds (#2758)Keith Bostic2016-06-038-70/+48
| | | | | | | * WT-2658 Only include PPC-specific files in PPC builds * Add support for conditional file inclusion by adding a second argument on lines in dist/filelist. Add a new automake conditional, POWERPC_HOST, set for the various PPC host CPUs. Change dist/filelist to only compile the checksum/power8 files if POWERPC_HOST is set. * Merge the Windows and POSIX file lists, use POSIX_HOST and WINDOWS_HOST to mark source files needed for builds on those systems.
* WT-2672 handle system calls that don't set errno (#2765)Keith Bostic2016-06-021-0/+1
| | | | | | | | Define system call success as a 0 return, and split error handling into two parts: if the call returns -1, use errno, otherwise expect the failing return to be an error value. Replace calls to remove with unlink, so we know errno will be set. Do the best we can with rename, there's no easy workaround. POSIX requires posix_madvise return an errno value, but some OS versions return a -1/errno pair instead (at least FreeBSD and OS X). I don't care about retrying posix_madvise calls on failure, but since WT_SYSCALL_RETRY includes the necessary error handling magic, wrap the posix_madvise calls in WT_SYSCALL_RETRY.
* WT-2662 For spell check, strip out double quote literals, they confuse ↵Don Anderson2016-05-251-1/+3
| | | | aspell. (#2748)
* WT-2653: the custom file-system example should show device configuration (#2741)Keith Bostic2016-05-251-0/+1
| | | | | | | Add examples of passing in per-device configuration information to the demo file system initialization function. Use the WiredTiger extension API to handle errors in functions called by WiredTiger. Set the default session's strerror method. If one of the extensions being loaded reports an error via the WT_EXTENSION_API strerror method, but doesn't supply that method a WT_SESSION handle, we'll use the WT_CONNECTION_IMPL's default session and its strerror method.
* WT-2646 Add checkpoint_wait configuration option to drop (#2736)David Hows2016-05-241-3/+7
| | | | | * Default checkpoint_wait is true. This change is useful because it means concurrent create/drop calls don't generate EBUSY returns. * Mark lock_wait and checkpoint_wait as undoc
* WT-2554 Add initial C test framework. (#2655)Alex Gorrod2016-05-232-0/+107
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * WT-2554 Add initial C test framework. With a few sample applications. Need to do further work to cleanup the same applications and share as much code/style as possible. * WT-2554 Fixup build errors. * Fix compiler warning. * Cleanup * Ensure each test runs in a different directory. * Update C test suite names and add copyright. * Move test code into subdirectories. * Replace test_util.i with a real utility library now that we have one. Nothing needs to be an inline function. Mostly cherry picked from 7c18420. * Use WiredTiger getopt in C test suite for platform portability. * Add a header comment to each test case. * Add a C test suite entry auto generator * Style, KNF * Implement review feedback. * Build test/utility library on Windows * Add comment to script. Fix a printf.
* WT-2611 Modify wtperf to handle escaped quotes. (#2714)Don Anderson2016-05-191-0/+1
| | | | | | | | * Modify wtperf to handle escaped quotes. * Made local function static. Added backslash in quote (normally redundant) needed to keep spell check sane. Fixed C++ style comment.
* WT-2610 Reduce hazard pointer array size. (#2731)sueloverso2016-05-171-0/+3
| | | | | | | | | | | | | | | | * WT-2610 Reduce hazard pointer array size. * Add session_count_idle and workload to wtperf. * Allocate session array outside the loop. * KNF * Remove diagnostic call to check hazard pointers. * Add a few hazard related statistics. * Revert change. Reinstate diagnostic hazard page check.
* WT-2612: The dist/s_prototypes script is creating a debugging file xxx. (#2715)Keith Bostic2016-05-031-1/+1
|
* WT-2599: split out the checksum code from the support directory (#2707)Keith Bostic2016-05-025-9/+9
| | | | | * WT-2599: split out the checksum code from the support directory Also fix a lint complaint about missing enum options in a case statement and remove unnecessary WT_FILE_SYSTEM/WT_FILE_HANDLE method initialization.
* WT-2583 Detect incremental file left in source directory (#2696)sueloverso2016-04-291-0/+1
| | | | * Remove unused metadata backup file for incremental backups. * Write backup information to a temp file and then rename.
* WT-2446 Add stat for number of calls to get a page. (#2697)sueloverso2016-04-281-0/+2
|
* WT-2552 Add public API for pluggable filesystems (#2671)Alex Gorrod2016-04-285-9/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * WT-2552 Add public API for pluggable filesystems Not yet compiling. The main parts of this change should be here, but it involved extensive parameter re-organization. There are also a number of layering violations between our existing file system implementations and the WT_FH, that aren't possible with the new structure. There are a number of specific todo comments in the code. One of the main issues is that the in-memory file system had a special close semantic that relied on WiredTiger handle tracking. The in-memory file-system should do it's own tracking of file handles, I've gone part way down that road by adding a queue for closed handles. Need to also add in live handles, and manage the queue as appropriate. I haven't created an example application that uses the new API yet. * WT-2552 Add public API for pluggable filesystems I always forget you have to remove the already-built html files when changing PREDEFINED, add a reminder to the complaint. * WT-2552 Add public API for pluggable filesystems You have to remove the .js files, too. * WT-2552 Add public API for pluggable filesystems Make dist/s_all run cleanly. * WT-2552 Add public API for pluggable filesystems Whitespace. * WT-2552 Add public API for pluggable filesystems Make it compile/build/lint. * WT-2552 Add public API for pluggable filesystems block_write.c: In function '__wt_block_extend': block_write.c:130:71: error: missing terminating ' character [-Werror] * WT-2552 Add public API for pluggable filesystems os_fs_inmemory.c: In function '__im_file_truncate': os_fs_inmemory.c:344:10: error: 'session' is used uninitialized in this function [-Werror=uninitialized] * WT-2552 Add public API for pluggable filesystems os_fs.c: In function '__posix_directory_sync': os_fs.c:92:10: error: 'session' is used uninitialized in this function [-Werror=uninitialized] * WT-2552 Add public API for pluggable filesystems Go back to using bool types in the file-system API, this requires we add <stdbool.h> to the "standard" wiredtiger.h includes. Consistently use wt_session to represent a WT_SESSION, we were using "wtsession" in some places. Make a pass over the Windows code, but I'm sure it doesn't compile yet. * WT-2552 Add public API for pluggable filesystems Fix up another couple of bool types. * WT-2552 Add public API for pluggable filesystems Move the file naming work out of the underlying filesystem functions, the calls to __wt_filename are now in the upper-level code,n os_fs.i; that means the filesystem code is no longer responsible for figuring out paths. This is cleaner, although the directory-sync call is a bit of a kluge, and I've commimtted us to handling NULL filesystem methods. With this set of changes, in-memory runs again. More Windows naming fixes. * WT-2552 Add public API for pluggable filesystems os_fs.c: In function '__posix_directory_sync': os_fs.c:96:3: error: label 'err' used but not defined * WT-2552 Add public API for pluggable filesystems Pull out another call to __wt_filename() from the filesystem-dependent code. * WT-2552 Add public API for pluggable filesystems Consistently check for missing file-system methods when doing file-system calls. Other minor lint & cleanup. * WT-2552 Add public API for pluggable filesystems Change the in-memory code to maintain a complete list of the files it has ever opened, and depend on that list instead of reaching up into the common layer for the WT_FH handle list. This means __wt_handle_search is only used by the common WT_FH handle code, simplify it, and add a __wt_handle_is_open function that can be called for diagnostic purposes (to check for open files that are being renamed or removed, for example). * Fix comiler warning and ignore the file system API in Java * Flesh out the example file system implementation. * Add in some plumbing for set_file_system in wiredtiger_open. * WT-2552 Add public API for pluggable filesystems Whitespace. * WT-2552 Add public API for pluggable filesystems WT_CONFIG_ITEM.val isn't a boolean, don't use boolean types in equal/not-equal comparisons. * WT-2552 Add public API for pluggable filesystems Remove unused #includes. Increment/decrement the DEMO_FILE_SYSTEM.{opened,closed}_file_count. Allocate demo structures, they're larger than the underlying structures. Swap the number/size calloc arguments, number comes first. Fix a couple of statics. * WT-2552 Add public API for pluggable filesystems Use %u instead of casting to %d. * WT-2552 Add public API for pluggable filesystems Add ex_file_system.c to the list of example programs. * WT-2552 Add public API for pluggable filesystems Change ex_file_system.c to not require <wt_internal.h>: strip down a copy of FreeBSD's <queue.h> for local inclusion, rewrite a few other minor pieces of code. * WT-2552 Add public API for pluggable filesystems Update spell check info * WT-2552 Add public API for pluggable filesystems __conn_load_extensions() shouldn't set the "early" boolean to true. * WT-2552 Add public API for pluggable filesystems Don't indirect through a NULL pointer if "local" was set and no path was specified, always set the name to something useful. * WT-2552 Add public API for pluggable filesystems Don't indirect through a NULL pointer if "local" was set and no path was specified, always set the name to something useful. * WT-2552 Add public API for pluggable filesystems wt_off_t vs. size_t conversion lint. * WT-2552 Add public API for pluggable filesystems Add -rdynamic to the load for ex_file_system, the main executable symbols are not exported by default. * WT-2552 Add public API for pluggable filesystems The underlying handle name includes the enclosing directory, compare against the WT_FH.name field instead. * WT-2552 Add public API for pluggable filesystems demo_fs_rename should return 0 if successful, simplify error handling Don't bother casting arguments to free(), it's not necessary. * WT-2552 Add public API for pluggable filesystems General WT_FILE_SYSTEM cleanup. Move OS initialization into the wiredtiger_open() code (the os_common/os_init.c file is no longer needed). Allow early-load extensions to be part of the environment settings, matching the "in-memory" and "readonly" configurations. Syntax check the set of a file-system, remove tests for NULL methods in the file-system structure unless it's legal for them to be NULL. Windows, POSIX and in-memory file systems now set WT_FILE_SYSTEM.terminate, call that function to cleanup when discarding a WT_CONNECTION. Export file-type and open-flags constants for WT_FILE_SYSTEM.open_file, sort the WT_FILE_SYSTEM methods, do an editing pass. Change the WT_FILE_HANDLE type from (const char *) to (char *), it's "owned" by the underlying layer, and it's simpler that way. Minor (untested) cleanup of the Windows WT_FILE_SYSTEM.open-file method. * WT-2552 Add public API for pluggable filesystems Export the advise argument #defines for the WT_FILE_HANDLE.fadvise method. Sort the WT_FILE_HANDLE methods. * WT-2552 Add public API for pluggable filesystems Clean up and simplify WT_FILE_SYSTEM/WT_FILE_HANDLE documentation's description of the handles. * WT-2552 Add public API for pluggable filesystems WT_FILE_HANDLE.close is a required function (at the least, it has to free the memory). WT_FILE_HANDLE.fadvise isn't a required function, if it's not configured, don't call it. * WT-2552 Add public API for pluggable filesystems The WT_FILE_HANDLE.lock function is required. Change the __wt_open() signature to match WT_FILE_SYSTEM.open_file(). * WT-2552 Add public API for pluggable filesystems Rework all of the WT_FILE_HANDLE mapped region methods to be optional. * WT-2552 Add public API for pluggable filesystems The WT_FILE_HANDLE.{read,size} methods are required. The WT_FILE_HANDLE.sync method is not required. Split the WT_FILE_HANDLE.sync method into .sync and .sync_nowait versions, it makes the upper-level code simpler (Windows supports .sync but doesn't support .sync_nowait). * WT-2552 Add public API for pluggable filesystems The WT_FILE_HANDLE.{truncate,write} methods are required IFF the file is not readonly. * WT-2552 Add public API for pluggable filesystems POSIX shouldn't declare a no-sync handle function unless the sync_file_range system call is available. * WT-2552 Add public API for pluggable filesystems Typo, missing semi-colon. * Fix a bug in ex_file_system.c * Fix a memory leak in posix file handle implementation * WT-2552 Use the correct flags when opening backup file. * WT-2552 Add public API for pluggable filesystems Simplify open-file error handling by calling the close function on the handle, that way we won't forget to free all of the applicable memory allocations. * WT-2552 Add public API for pluggable filesystems Simplify the directory-list method, don't pass in an include/exclude file, if prefix is non-NULL, it implies we only want files matching the prefix. * WT-2552 Add public API for pluggable filesystems Replace WT_FILE_HANDLE_POSIX.fallocate_{available,requires_locking} wiht WT_FILE_HANDLE.fallocate and WT_FILE_HANDLE.fallocate_nolock. Example code doesn't need to set WT_FILE_HANDLE methods to NULL, the allocation does that. Free the I/O buffer if open-handle allocation fails in the example code. Remove snippets for WT_FILE_SYSTEM and WT_FILE_HANDLE methods, we're not going to provide example code for them. * WT-2552 Add public API for pluggable filesystems Document we expect either ENOTSUP or EBUSY from optionally supported APIs. Review/cleanups ENOTSUP/EBUSY returns from optionally supported APIs. Make WT_FILE_HANDLE.lock optional. Don't configure or call the POSIX fadvise function on files configured for direct I/O. Rename __wt_filesize_name to __wt_size for consistency. Update the spelling list. * WT-2552 Add public API for pluggable filesystems WT_FILE_HANDLE.truncate requires locking in all known implementations, document it is not called concurrently with other operations. * WT-2552 Add public API for pluggable filesystems Don't terminate the filesystem unless we've actually configured one. * WT-2552 Add public API for pluggable filesystems Remove WT_FILE_SYSTEM and WT_FILE_HANDLE from SWIG so the test suite can pass again. * WT-2552 Add public API for pluggable filesystems Merge __conn_load_early_extensions() and __conn_load_extensions(). Fix a problem where I moved the early extensions load to where it could include the WiredTiger environment variable, but I didn't pass the built cfg into the function. * WT-2552 Add public API for pluggable filesystems Linux build typo. * WT-2552 Add public API for pluggable filesystems Get rid of the "bool silent" argument to WT_FILE_SYSTEM.size by testing for the file's existence before requesting the size (an extra system call, but guaranteed to hit in the buffer cache at least). * WT-2552 Add public API for pluggable filesystems Naming consistency pass over the WT_FILE_SYSTEM functions. * WT-2552 Add public API for pluggable filesystems Fix a spin lock mismatch. * WT-2552 Add public API for pluggable filesystems Another spinlock mismatch. * Update example pluggable file system. Add a directory list implementation to the example, which uncovered an issue with the API. The directory list API allocates memory that is freed by WiredTiger, which I don't think is kosher. * Change file-directory-sync to use reguar fsync. The distinction in os_fs.i doesn't work with the filesystem API. Also add directory_sync application to the example application. * WT-2552 Add public API for pluggable filesystems Whitespace. * WT-2552 Add public API for pluggable filesystems Rewrite __wt_free to not evaluate macro arguments multiple times. * WT-2552 Add public API for pluggable filesystems Simplify the directory-list functions: __wt_realloc_def() already handles scaling the size of the allocations, there's no need to involve a separate constant that increments the allocation size. * WT-2552 Add public API for pluggable filesystems Fix a grouping problem in a realloc call, we need to multiple the size times the previously allocated slots + 10. Fix buffer overrun, if "count" has already been incremented, the memset would skip clearing the first slot and clear one slot past the end of the buffer. Remove a comment, realloc requires clearing allocated memory, it's not paranoia. * WT-2552 Add public API for pluggable filesystems Add the mapping-cookie argument to the map-preload and map-discard functions. Change page-discard to stop reaching down through the block manager, instead, provide a block-manager map-discard function that does the work. * WT-2552 Add public API for pluggable filesystems Require a directory-list function. Implement a directory-list function for the in-memory filesystem. Consistency pass, make all the directory-list functions look the same. * WT-2552 Add public API for pluggable filesystems The WT_FILE_SYSTEM.{directory_sync, remove, rename} methods are not required for read-only systems. * WT-2552 Add public API for pluggable filesystems Change the WT_FILE_SYSTEM.open_file file_type argument from a set of constants to an enum. This requires changing how we store connection direct I/O configuration (the constants used to be flags stored in the WT_CONNECTION_IMPL), and requiring all callers of __wt_open() do their own work to figure out if WT_OPEN_DIRECTIO should be specified. * WT-2552 Add public API for pluggable filesystems Make no guarantees WT_FILE_SYSTEM and WT_FILE_HANDLE methods are not called concurrently (except for WT_FILE_HANDLE::fallocate and WT_FILE_HANDLE::fallocate_nolock). Rewrite the in-memory FS code to lock across all methods (for example, WT_FILE_HANDLE.close), that means including a reference to the enclosing WT_FILE_SYSTEM in the WT_FILE_HANDLE structure so we can find a lock without using the WT_CONNECTION_IMPL structure. * WT-2552 Add public API for pluggable filesystems Remove __wt_directory_sync_fh, it's no longer useful. * WT-2552 Add public API for pluggable filesystems Rename WT_INMEMORY_FILE_SYSTEM to WT_FILE_SYSTEM_INMEM, matching WT_FILE_HANDLE_INMEM. * WT-2552 Add public API for pluggable filesystems Add WT_FILE_SYSTEM.directory_list_free, to free memory allocated by WT_FILE_SYSTEM.direct_list. Fix a memory leak in __log_archive_once (if __wt_readlock failed, we leaked the directory-list memory). * WT-2552 Add public API for pluggable filesystems Typo, check WT_DIRECT_IO_LOG, not WT_DIRECT_IO_CHECKPOINT. * WT-2552 Add public API for pluggable filesystems Typo, unreachable code. * WT-2552 Add public API for pluggable filesystems We don't require WT_FILE_SYSTEM.{remove,rename} if the system is read-only. * Fix Windows build with pluggable file system. Involved removing u_int from the public API. * Fix line wrapping. * Fix Windows terminate function. * Forgot something in my last commit. * Fix Windows munmap bug. * Add new example to Windows build. Extend example to be more complete. * Fix example loading on Windows * Update documentation * Add missing spell words * Remove old comment.
* Merge pull request #2654 from wiredtiger/wt-2360-join-disjunctionMichael Cahill2016-04-211-0/+7
|\ | | | | WT-2360 Add nesting and disjunctions to cursor joins
| * Merge branch 'develop' into wt-2360-join-disjunctionMichael Cahill2016-04-192-0/+3
| |\
| * \ Merge branch 'develop' into wt-2360-join-disjunctionDon Anderson2016-04-1412-75/+126
| |\ \
| * | | WT-2360 A large set of changes to implement disjunctive and nested joins.Don Anderson2016-04-141-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | The join iterator now manages iteration at 'one level' of the join, as subjoins are iterated, new iterators are created, forming a stack. Added internal documentation in cursor.h.
* | | | Merge pull request #2656 from wiredtiger/wt-2223Michael Cahill2016-04-212-5/+7
|\ \ \ \ | | | | | | | | | | WT-2223 WT-2553 in_memory documentation and stress testing
View Lorry Controller Status