crawler: don't hold lock while writing to network

LRU crawler in per-LRU mode only flushes the write buffer to the client
socket while not holding important locks. The hash table iterator
version of the crawler was accidentally holding an item lock while
flushing to the network. Item locks must NOT be held for long periods of
time as they will cause the daemon to lag.

Originally the code used a circular buffer for writing to the
network; this allowed it to easily do partial write flushes to the
socket and continue filling the other half of the buffer.

Fixing this requires the buffer be resizeable, so we instead use a
straight buffer allocation. The write buffer must be large enough to
handle all items within a hash table bucket.

Hash table buckets are _supposed_ to max out at an average depth of 1.5
items, so in theory it should never resize. However it's possible to go
higher if a user clamps the hash table size. There could also be larger
than average buckets naturally due to the hash algorithm and luck.

0 files changed, 0 insertions, 0 deletions