summaryrefslogtreecommitdiff
path: root/crawler.h
Commit message (Collapse)AuthorAgeFilesLines
* restartable cachedormando2019-09-171-1/+3
| | | | | | | | | | | | | | | "-e /path/to/tmpfsmnt/file" SIGUSR1 for graceful stop restart requires the same memory limit, slab sizes, and some other infrequently changed details. Most other options and features can change between restarts. Binary can be upgraded between restarts. Restart does some fixup work on start for every item in cache. Can take over a minute with more than a few hundred million items in cache. Keep in mind when a cache is down it may be missing invalidations, updates, and so on.
* limit crawls for metadumperdormando2018-02-121-1/+4
| | | | | | | | | | | | | | | LRU crawler metadumper is used for getting snapshot-y looks at the LRU's. Since there's no default limit, it'll get any new items added or bumped since the roll started. with this change it limits the number of items dumped to the number that existed in that LRU when the roll was kicked off. You still end up with an approximation, but not a terrible one: - items bumped after the crawler passes them likely won't be revisited - items bumped before the crawler passes them will likely be visited toward the end, or mixed with new items. - deletes are somewhere in the middle.
* interface code for flash branchdormando2017-09-261-2/+1
| | | | | removes a few ifdef's and upstreams small internal interface tweaks for easy rebase.
* LRU crawler scheduling improvementsdormando2017-05-291-1/+1
| | | | | | | | | | | | | | | | | | | when trying to manually run a crawl, the internal autocrawler is now blocked from restarting for 60 seconds. the internal autocrawl now independently schedules LRU's, and can re-schedule sub-LRU's while others are still running. should allow much better memory control when some sub-lru's (such as TEMP or WARM) are small, or slab classes are differently sized. this also makes the crawler drop its lock frequently.. this fixes an issue where a long crawl happening at the same time as a hash table expansion could hang the server until the crawl finished. to improve still: - elapsed time can be wrong in the logger entry - need to cap number of entries scanned. enough set pressure and a crawl may never finish.
* pull LRU crawler out into its own file.dormando2016-08-191-0/+39
~600 lines gone from items.c makes it a lot more manageable. this change is almost purely moving code around and renaming functions. very little logic has changed.