LRU crawler scheduling improvements

when trying to manually run a crawl, the internal autocrawler is now blocked
from restarting for 60 seconds.

the internal autocrawl now independently schedules LRU's, and can re-schedule
sub-LRU's while others are still running. should allow much better memory
control when some sub-lru's (such as TEMP or WARM) are small, or slab classes
are differently sized.

this also makes the crawler drop its lock frequently.. this fixes an issue
where a long crawl happening at the same time as a hash table expansion could
hang the server until the crawl finished.

to improve still:
 - elapsed time can be wrong in the logger entry
 - need to cap number of entries scanned. enough set pressure and a crawl may
   never finish.

1 files changed, 1 insertions, 1 deletions

diff --git a/crawler.h b/crawler.h
index 7c7c1d2..ad30e66 100644
--- a/crawler.h
+++ b/crawler.h
@@ -14,7 +14,7 @@ typedef struct {
 
 struct crawler_expired_data {
     pthread_mutex_t lock;
-    crawlerstats_t crawlerstats[MAX_NUMBER_OF_SLAB_CLASSES];
+    crawlerstats_t crawlerstats[POWER_LARGEST];
     /* redundant with crawlerstats_t so we can get overall start/stop/done */
     rel_time_t start_time;
     rel_time_t end_time;