| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Cached overflow records are large value items that are being written
after being updated in a committed transaction (which implies freeing
and the potential reuse of their backing disk blocks), but which also
must be kept available because an older reader in the system may want
to read the previous value.
Historically, we maintained a separate cache of these overflow values
in the WT_PAGE_MODIFY.WT_OVFL_TXNC structure, with a set of functions
to maintain that list. The reason was because we wanted to be able to
remove overflow items from the cache based on updated transactional
information, that is, once any older readers exited the system, we could
discard the overflow values from the cache. Since that code was written,
we've added support for removing items from key/value WT_UPDATE lists
when they're no longer needed by earlier readers in the system, making
the WT_UPDATE lists sufficient to replace the separate overflow values
cache.
Rather than enter overflow values into the separate cache, append the
value to the WT_UPDATE list, with an impossibly low transaction ID to
ensure global visibility.
* Remove a diagnostic printf.
* sort statistics lists.
* Delete the data-source and connection statistic cache_overflow_value.
It doesn't tell us anything useful other than "there were older readers
in the system", and there are better sources of that information.
* The cached remove information is a WT_CELL, no reason to hide that.
* Increment the page footprint based on __wt_update_alloc's return of the
size, don't roll our own.
* There's special-case code in reconciliation to handle an overflow value
being removed, the overflow value being cached because of an older
reader in the system, and then the page going through an update/restore
reconciliation.
The idea is that if we cache an overflow value for an older reader in
the system, then do a page update/restore eviction, re-instantiating the
page in-memory means we'll no longer find the removed overflow value in
the cache because the index for the cache is the on-page disk address
of the overflow value, which doesn't appear in a re-instantiated page.
The fix was to take a copy of the original overflow value from the cache
and append it to the key's WT_UPDATE list.
With the move of the overflow cache from a separate list to the key's
WT_UPDATE list, that special-case code is no longer needed. Now, if an
overflow value is removed and cached, the cache is the key's WT_UPDATE
list, and subsequent reconciliations will always see a globally visible
entry.
* Document the removal of the cache_overflow_value statistic.
* Add WT_UPDATE_DATA_VALUE macro, it flags a WT_UPDATE structure that includes
a standalone chunk of data, in other words, "standard" or "deleted", but not
"reserved", or in the future, "modified".
* Use "no transaction ID" instead of "impossibly low transaction ID".
|
| |\
| |
| |
| | |
Pulling in the 2.9.2 release notes
|
| | | |
|
| | |
| |
| | |
Including removing a few redundant statistics, adding some new statistics, and improving the descriptions of others.
|
| | | |
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* WT-2898. NOT ready for review. Initial implementation of dynamically tuning the number of eviction workers.
* Not ready for review. All the code is there. Still need to test/tune on different machines.
* Remove debugging prints.
* Style police
* Spelling.
* Fixup merge issue and compiler warning
* Sulab and David do a review!
* Fix compiler warning. Not ready for review. There is a performance regression after merging with develop. I'm on it.
* Conversion to signed values in percent calculation to make sure that we always correctly compute percent difference, which can be negative, regardless of how the complier performs sign extension. Change thresholds so we have less churn.
* Fix more compiler warnnings. Sorry about the churn, I don't see the same failures locally as on the autobuild even though I compile with -Werror.
* Replace 1/0 with true/false
* More compiler warning and style fixes. Configured with --enable-strict, so hopefully I have caught everything this time.
* Minor nit/pick, init a variable
* Rename free to free_thread, otherwise hides a global
* Fix indentation
* Fixes to the changes. Percent difference must be signed as it can be negative if the number of pages evicted per second decreased since the last period.
* Added stats and log messages to reflect eviction worker tuning activity. Fixed a bug in the code that checks the group bounds when stopping the thread.
* Removed verbose message, because we already have a statistic tracking evictions per second, so this is probably redundant.
* whitespace
* KNF
* More aggressive addition/removal of eviction workers. We used to add/remove them one at a time; it's difficult to see the effects of extra workers with such an incremental change, because eviction throughput is affected by other noise, such as what happens in the kernel and in the I/O system. Now we add and remove eviction workers in batches.
* Style fixes.
* Fix compiler warning.
* Simplified the tuning logic. Addressed Sulabh's comments.
* A tuning parameter change
* Fixed a bug where we needed a random value, but were not getting it via the random number generator, so it was not random and the code did not have the right behaviour. Added stats.
* Move the call to tune evict workers into __evict_pass, so we can begin the tuning earlier.
* NOT READY FOR REVIEW. Changed defaults for the number of eviction workers, so I can experiment with larger values.
* NOT READY FOR REVIEW. A parameter to put a cap on how many threads we are adding at a time.
* Reverse the changes of the last commit. That change hurt performance.
* Changed all wtperf runners that set eviction thread maximum to 30, so we could evaluate the effects of the dynamic branch.
* Updated the number reserved for internal sessions to 40, since we can now create up to 30 eviction worker threads by default.
* Fix spellchecker complaints
* KNF
* NOT READY FOR REVIEW. Revised the algorithm to settle on a good value of evict workers once we sufficiently explore the configuration space using the gradient descent approach with random adjustments. The algorithm successfully finds the best static number of workers, but performs works. I suspect that there is an issue with how threads are removed. Suspect a bug in thread support code. Have not chased it yet.
* Remove prints, add stats.
* Fix a copy-paste bug where a code line was inadvertently eliminated.
* Reduce the maximum for eviction workers to 30. Prevent dereferencing a NULL pointer if we dynamically grow a thread group after we've shrunk it and freed the associated memory.
* Cleaned up and simplified the code.
* NOT READY FOR REVIEW. A new version of the tuning algorithm that fixes a memory issue when we try to pre-allocate a large eviction thread group. Still need to tune and clean up the code.
* Clean up the code.
* Get rid of s_label warnings. Remove unused code.
* Fix various style errors.
* Fixed the logic in figuring out the maximum value for eviction threads upon cache creation or reconfiguration, which had caused a crash in one of the tests.
* Changed default max for the number of eviction threads to eight.
* Fix ranges for the minimum number of eviction threads
* Fix eviction thread ranges to make the csuite happy
* Commit automatic changes by s_all
* Review: KNF, whitespace and renamed a few things.
* Fix lock usage
* KNF
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* WT-283 Add WT_SESSION:alter method
* Add new method to ex_all.c
* Typo
* Add testing and fixes for LSM, column groups and indexes for alter.
* Fix WITH_*_LOCK macro usage.
* Whitespace
* Change table alter to not walk down into column groups and indexes.
* Whitespace
* Use base file meta config and compare strings.
* Add skip stat
* Add alter support to wt command.
* Add util_alter.c to SConstruct
* Fix subsection in doc.
* Add alter thread to test/format
* Swap the value we send into alter in test/format.
* Add data-source alter support
* Only alter access pattern hint in test/format.
* Make LSM alter not exclusive.
* Whitespace
* Use access_pattern_hint in example.
* Include base lsm metadata configuration.
* Fix text/comment.
* Minor changes from review comments.
* Unused variable warning.
|
| | |
|
| |
|
| |
Exposed via a new 'cache_walk' statistics configuration option.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
high level locks (#3086)
* WT-2955 Add statistics tracking the amount of time threads spend waiting for high level locks
Sort the statistics categories so it's easier to find stuff, no real change.
* Add counters and usec wait times to long-term locks (currently the
checkpoint, handle-list, metadata, schema and table locks).
* mutex.i:295:26: error: conversion to int64_t {aka long int} from long
unsigned int may change the sign of the result [-Werror=sign-conversion]
[t->slot_usecs] += WT_TIMEDIFF_US(leave, enter);
* Rename the lock statistics so they group together.
Split lock wait times between internal and application threads.
* Separate the connection's dummy session initialization out into its own
function, that way it's clear what we're doing.
* The session's connection statistics are fixed when the session ID is
allocated, so we can cache a pointer to them and avoid u_int divisions
(which are currently about the slowest thing you can do on a modern
architecture).
* A slightly different change: instead of caching a reference to the
connection statistics, cache the offset into the array of statistics
pointers, that way we can avoid the integer division when updating
almost all statistics.
* Review comments:
Add comments describing the use of statistics array offsets in lock
tracking.
Rename WT_STATS_FIELD_TO_SLOT to WT_STATS_FIELD_TO_OFFSET.
Whitespace cleanup.
* __wt_cache_create() doesn't need to call __wt_cache_destroy() explicitly,
if the connection open fails at any point, __wt_cache_destroy() will be
called as part of that cleanup.
* Append the suffix "_off" to the spinlock structure statistics field
names, clarify they're offsets into the statistics structures.
|
| |
|
|
|
|
|
|
|
| |
Some workloads create "deserts" within a tree where no good eviction
candidates can be found (e.g., zipfian distributions, append-only
workloads, etc.). Rather than walking all the way through such
regions, give up and try in the opposite direction next time.
Also add a statistic for the eviction empty score.
|
| |
|
|
|
|
|
|
|
|
| |
A set of changes to the eviction algorithm including:
* Fix a bug in how many items can be added to the urgent queue
* Have the eviction server sleep less so it recovers from disruptions faster.
* Only have application threads evict dirty pages if they are blocked on the dirty trigger.
* Swap eviction queues when one becomes empty.
* Have the eviction server populate the "other" queue whenever it notices that it isn't full.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead, get multi-threaded writes by dialling down the eviction dirty trigger.
Change eviction_dirty_* defaults (to 20/5).
Make sure all files are available for eviction before starting fsync.
Take more care excluding clean files from checkpoints.
Only use scrubbing mode when there is minimal cache pressure.
Avoid penalizing read-only operations with dirty eviction targets.
Enable scrubbing when eviction is keeping space available in cache.
Add stats for checkpoint scrubbing phase.
Improve cache scrubbing with big caches and slow I/O.
Use the rate of bytes written from cache to decide how long to wait for
the dirty bytes to come down.
|
| |
|
|
| |
Schema operations for this purpose are table level methods on WT_SESSION handles.
|
| |
|
|
|
|
|
|
| |
* WT-2793 Remove very long running config. Rename the one we run.
* Fix overflow stats. Make 130K overflow test use btree.
* Add line to upgrading doc stating stat field removed.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Add a statistic for the amount of cache used for page images vs other objects
* Reset the "maximum page size" statistic each checkpoint
* Add a separate queue for urgent eviction, replaces existing WOULD_BLOCK eviction mode
Until now, the eviction server has had to walk the cache to find pages that
would otherwise block application threads. This change allows threads to put
those pages directly on a queue that is checked before ordinary LRU eviction.
* Don't queue pages for urgent eviction when eviction is disabled in a tree.
Decouple evict_queue_lock and the individual queue's evict_lock in
__evict_get_ref. Don't block the eviction server if there are many
application threads waiting on a queue.
* Take care with the urgent queue.
Entries in the urgent queue cannot be skipped or they will not be considered again.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Randomize visits to trees that use a tiny fraction of the cache.
Other general eviction optimizations.
Now that we are queuing more entries (potentially), make sure enough of
them become candidates. Previously, a skewed distribution of read
generations could mean that only 10% of queue entries were considered.
Improve the efficiency of sorting the queue by calculating the score
once when pages are added to the queue.
Add a workload to exercise differential eviction from trees.
In order to do that, add range partitioning to wtperf.
Don't override icount in wtperf workloads with random_range set.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* WT-2541 Add stats showing active readers and writers
* Fix use of RET_MSG instead of ERR_MSG
* Change where we increment the active r/w stats
* Add "active fsync" stat
* Add new "Thread State" stat section, add fsync_io stat
* KNF
* Make thread stats not backwards breaking. Add atomic stat functions
* Add ThreadState stats to system and evict groups
* KNF
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* WT-2610 Reduce hazard pointer array size.
* Add session_count_idle and workload to wtperf.
* Allocate session array outside the loop.
* KNF
* Remove diagnostic call to check hazard pointers.
* Add a few hazard related statistics.
* Revert change. Reinstate diagnostic hazard page check.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
cookie. Similar to 'wt list -c', but usable in situations where it may be
difficult to run 'wt' (e.g. during a debugging session).
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
New statistics are:
* pinned by named snapshots
* number of named snapshots created
* number of named snapshots dropped
|
| |
|
|
| |
possible to tell how many values are being compressed by RLE encoding.
|
| | |
|
| | |
|
| |\
| |
| |
| |
| | |
Conflicts:
README
|
| | | |
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
clear.
cache_eviction_maximum_page_size: collection statistic, aggregate
doesn't apply, don't clear.
various txn_checkpoint_XXX statistics: collection statistic, aggregate
doesn't apply
various Btree maximum size statistics: aggregate the maximum value,
rather than doing no aggregation at all.
lsm_chunk_count: aggregate the count of LSM chunks.
Add syntax checking code to stat.py to disallow aggregation flags
for connection-level statistics.
Reference #1742.
|
| |
|
|
|
|
| |
It is often useful to know if a cache is full of internal pages. I
could have added page count tracking, but byte count tracking seems
more useful (enough to justify the extra performance overhead).
|
| |
|
|
| |
Refs #1746.
|
| |
|
|
|
|
|
| |
Not yet used, but will be used to allow more eviction during
checkpoints.
Refs #1745
|
| |
|
|
|
|
|
|
| |
Specifically cache dirty tracking and log preallocation.
This helps wtstats graph generator.
Refs #1742
|
| | |
|
| |
|
|
| |
or subdirectory.
|
| |
|
|
|
|
|
|
| |
- convert wtperf microsec to millisec
- don't skip monitor* files when parsing directory
- parsing code reorganization
- renamed wtperf stats fixture
- added tests
|
| | |
|
| |
|
|
| |
they go into a separate stats section named "wtperf".
|
| |
|
|
|
| |
Add the wtstats template file (wtstats.html.template) to the tree.
Strip the wtstats/{test,template} subdirectories from the release.
|
| | |
|
| | |
|
| | |
|
| | |
|
