summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* core: cache.c cleanups, use queue.h freelistdormando2021-08-094-145/+31
| | | | | | | | | cache constructors/destructors were never used, which just ended up being wasted branches. Since there's no constructor/destructor for cache objects, we can use the memory itself for the freelist. This removes a doubling realloc for the freelist of cache objects and simplifies the code a bunch.
* core: add queue.h to replace handrolled queues.dormando2021-08-092-0/+873
| | | | | FreeBSD version as there seems to be portability issues if you just use sys/queue, plus some extra features I like in this one.
* logger: simplify logging codedormando2021-08-092-311/+256
| | | | | | | | | | Adding a new type of log entry required updating code in a bunch of places. Now we use callbacks in the entry structure instead, cutting down the code and some awkwardness. Should also make it easier for plugins/extra systems to add log entries. Moving the writer/parsers to their own section makes the actual logging code easier to follow, as an added bonus.
* logger: avoid polling without watchersdormando2021-08-092-1/+11
| | | | | | | | | Inspired by #768 - logger thread would wake up once per second looking for watchers or log entries, but without any watchers. Now sleeps on a condition. Note that if I ever actually get logger to syslog/stdout/stderr/etc working properly this will still always poll.
* Implement LOG_CONNEVENTS watcher flag for connection state transitionsKevin Lin2021-08-078-2/+183
| | | | | | | | Add support for `watch connevents` to report opened (`conn_new`) and closed (`conn_close`) client connections. Event log lines indicate the connection's remote IP, remote port, and transport type. `conn_close` events additionally supply a reason for the closing the connection.
* Report item sizes for fetch, mutation, and eviction watchersKevin Lin2021-08-065-14/+50
| | | | | | This adds a new field `size` to logger entry lines for item_get, item_store, and eviction events indicating the size of the associated item in bytes.
* Fix typos in doc/code comments (tem->item, etc)Tyson Andre2021-08-055-7/+7
| | | | | Note: Do not fix typos in crc32.c because it's copied from an upstream source
* Fix minor severity heap buffer overflow reading `--auth-file`1.6.10Tyson Andre2021-07-251-8/+14
| | | | | | | | | | | Fixes #805 Allocate an extra byte for reading the last entry when there is no `\n` at the end of the file. Also, check if the user contains null bytes when reading the last entry. Unrelatedly, add handling in case the auth file size changes while it is being read.
* stats_prefix.c: Check for NDEBUG before using total_written variableKhem Raj2021-07-251-2/+6
| | | | | | | | | | | | | When using NDEBUG assert macro is ineffective which is caught by latest clang and reports that total_written is set but unused. Therefore check for NDEBUG to make sure assert is used only when its effective Fixes error: variable 'total_written' set but not used [-Werror,-Wunused-but-set-variable] size_t size = 0, written = 0, total_written = 0; ^ Signed-off-by: Khem Raj <raj.khem@gmail.com>
* .gitignore updatedormando2021-07-251-0/+1
| | | | my clang vim plugin whatevers creates a .cache directory.
* Add settings stat for shutdown_command enabledKevin Lin2021-07-253-1/+5
|
* [docker] Add user and expose the memcached portHervé Beraud2021-07-254-1/+13
| | | | | | | | | | | | | | | These changes aim to improve our memcached devtools images to allow us to reuse them during development stages. These changes create a memcached user for all the images, switch to this user by default once memcached have been configured for building. That allow to run the memcached server directly when the containers are started, else, without the user the run will fail as we try to execute memcached as root user. Also we should notice that the alpine image has been fixed by adding git to the list of the requirements to install. Else the build will fails.
* meta: remove EXPERIMENTAL mark + doc fixesdormando2021-07-251-12/+12
|
* meta: response code OK -> HDdormando2021-06-105-44/+74
| | | | | | | | | | | | | | I had the response code as "HD" in the past, but standardized on OK while merging a number of "OK-like" rescodes together. This was a mistake; as many "generic" memcached response codes use "OK". Most of these are management or specialized uncommon commands. With this, a client response parser can know for sure if a response is to a meta command, or some other command. `-o meta_response_old` starttime option has been added, valid for the next 3 months, which switches the response code back from HD to OK. In case any existing users depended on this and need time to migrate.
* meta: fix metaset syntaxdormando2021-06-103-56/+60
| | | | | | | payload length is now part of the main command rather than a flag/token pair. This allows byte swallow to be more reliable in the case of a flag parsing error, and makes sets easier to parse and buffer for servers implementing the protocol.
* meta: repairs to mset commanddormando2021-06-102-83/+127
| | | | | | | allows returning cas value of the new item on set. also fixes an extra space that was tagged onto the response line. which I had not noticed because all of the tests were looking for a space after the response line :( Bad copypasta from the metaget tests.
* hash: add XXH3 to list of hash algorithms.dormando2021-06-095-4/+5341
| | | | | | | | | | | | needs more testing. in a "pure miss" test to help show hashing overhead, with four worker threads and large ascii multigets there is a measurable performance boost for longer keys. Very short keys (10-15 bytes) are about even. for 150ish byte keys 12m rps -> 13m rps roughly. Not too bad! need to test hash table distribution for large sets of keys before merging. need to consider if worth switching the defaults now or later.
* fix arm64 crc32 on old glibc/gcc.dormando2021-06-081-0/+6
| | | | | patch by @sergey-dryabzhinsky on github, but he deleted the repo so I rewrote the commit.
* Update AUTHORSBrian Aker2021-06-071-0/+1
|
* extstore: fix crash on 'stats extstore'dormando2021-06-071-0/+3
| | | | if extstore wasn't enabled, crashes. Reported by @zer0e on github.
* seccomp: extend allowed rules for extended usageLevente Polyak2021-06-071-1/+11
| | | | | This puts syscalls into the allow list that were triggered on latest systems including extended usage tests, signal and systemd handling.
* The total number of UDP datagrams required for the message is calculated ↵tom2021-06-072-2/+3
| | | | | | | | | | | | | | | incorrectly. UDP_MAX_PAYLOAD_SIZE actually contains the length of the private UDP header, but resp->tosend only contains the length of the data part. The number of required UDP packets calculated by the original code will be less than the actual need. E.g: 1000000/1400 = 714.2 ceil 715 1000000/1392 = 718.3 ceil 719 Actually 719 datagrams are needed, and 715 is wrong. Signed-off-by: AK Deng <ttttabcd@protonmail.com>
* meta: more realistic buffer limit for binary keys.dormando2021-06-071-2/+5
|
* meta: protect cachedump from bin keys and add docsdormando2021-06-072-2/+18
| | | | | cachedump was the only place in the codebase I can find which copied the key verbatim. wonder when I can finally remove the command :)
* meta: 'b' flag for ME to return base64 keysdormando2021-06-071-4/+21
|
* meta: allow base64'ed binary keys with 'b' flagdormando2021-06-076-29/+300
| | | | | ie: ms [key] b if 'k' flag is given and key is binary, returns as binary encoded.
* small improvements to readmeLSmithx22021-06-071-4/+4
| | | small improvements to readme
* Fix leaking file descriptorsTomas Korbar2021-06-071-8/+10
| | | | Another issue found by static code analysers
* Fix expiration test to use debugtime commandminkikim892021-06-073-11/+13
|
* Added debugtime commandminkikim892021-06-073-0/+51
|
* core: support malloc'ed blobs for body readdormando2020-12-112-3/+15
| | | | | | | conn_nread state handles c->item like an item, but allow it to be a temporary malloced blob via setting c->item_malloced = true. to be used for buffering value reads in the proxy code.
* improve shutdown tests1.6.9dormando2020-11-202-4/+23
| | | | | | | new tests were flaky for slow systems; for some reason writing to the closet socket would exit prove instead of throw any kind of error. Instead we check that the child pid actually exits and loop a bit to avoid the race.
* crawler: remove bad mutex unlock during errordormando2020-11-201-1/+0
| | | | Detail in: https://github.com/memcached/memcached/issues/741
* 'shutdown graceful' command for raising SIGUSR1Kevin Lin2020-11-203-4/+67
|
* Expose memory_file path in stats settingsKevin Lin2020-11-204-6/+15
|
* item crawler hash table walk modedormando2020-11-205-15/+208
| | | | | | | specifying 'hash' instead of 'all' will make the LRU crawler iterate over ever bucket in the hash table once, instead of what's in the LRU. This also doesn't suffer from missing items because of LRU reordering or high lock contention.
* idle_timeout: avoid long hangs during shutdowndormando2020-11-121-1/+15
| | | | | | | | | | | | | the idle thread would sleep for up to the max timeout given before checking for active connections again. the new shutdown code waits on threads to be joined after a shutdown signal, which could end up hanging by the idle_timeout. this now instead uses a cond_timedwait call to allow a condition signal to wake up the thread to exit. this should be done for a few other threads that sleep without conditions (lru_maintainer?)
* extstore: use fcntl locking on disk filedormando2020-11-112-2/+24
| | | | | | | | | if you accidentally start memcached with the same options twice, extstore is initiated before the listener sockets and will happily truncate its own file. So this avoids that. Keep in mind any other process can still wipe the file clean!
* portability fix for getsuboptdormando2020-11-111-3/+11
| | | | | | | | | | in the deferred IO patch I'd cribbed some code from another PR which turned out to use a non-portable return state from getsubopt. On BSD systems the subopts_value string is simply empty on a non-match. Fixing this portably looks gross but I couldn't come up with a better option... and frankly if we're going to pass options through DSO submodules in the future this is better anyway.
* net: fix compile failures when missing NAPI definedormando2020-11-112-4/+5
| | | | was defined in memcached.c, but also used in thread.c.
* Introduce NAPI ID based worker thread selectionSridhar Samudrala2020-11-026-6/+162
| | | | | | | | | | | | | | | | | | | | | | | | | | By default memcached assigns connections to worker threads in a round-robin manner. This patch introduces an option to select a worker thread based on the incoming connection's NAPI ID if SO_INCOMING_NAPI_ID socket option is supported by the OS. This allows a memcached worker thread to be associated with a NIC HW receive queue and service all the connection requests received on a specific RX queue. This mapping between a memcached thread and a HW NIC queue streamlines the flow of data from the NIC to the application. In addition, an optimal path with reduced context switches is possible, if epoll based busy polling (sysctl -w net.core.busy_poll = <non-zero value>) is also enabled. This feature is enabled via a new command line parameter -N <num> or "--napi_ids=<num>", where <num> is the number of available/assigned NIC hardware RX queues through which the connections can be received. The number of napi_ids specified cannot be greater than the number of worker threads specified using -t/--threads option. If the option is not specified, or the conditions not met, the code defaults to round robin thread selection. Signed-off-by: Kiran Patil <kiran.patil@intel.com> Signed-off-by: Sridhar Samudrala <sridhar.samudrala@intel.com>
* illumos build fixes + require libevent2David Carlier2020-11-022-1/+17
| | | | | | | libevent 1 doesn't have the socket id wrapper. since version 2 is 10 years old we should fail to build on version 1. explicit cast to avoid possible pedantic build flags complains (ie can be the old iovec interface)
* queue: replace c->io_pending to avoid a mutexdormando2020-10-303-35/+50
| | | | | | since multiple queues can be sent to different sidethreads, we need a new mechanism for knowing when to return everything. In the common case only one queue will be active, so adding a mutex would be excessive.
* core: restructure IO queue callbacksdormando2020-10-305-61/+103
| | | | | | | | | | | | | | mc_resp is the proper owner of a pending IO once it's been initialized; release it during resp_finish(). Also adds a completion callback which runs on the submitted stack after returning to the worker thread but before the response is transmitted. allows re-queueing for pending IO if processing a response generates another pending IO. also allows a further refactor to run more extstore code on the worker thread instead of the IO threads. uses proper conn_io_queue state to describe connections waiting for pending IO's.
* core: io_pending_t is an embeddable structdormando2020-10-302-37/+53
| | | | | | reserve space in an io_pending_t. users cast it to a more specific structure, avoiding extra allocations for local data. In this case what might require 3 allocs stays as just 1.
* core: compile io_queue code by defaultdormando2020-10-304-24/+7
| | | | | don't gate on EXTSTORE for the deferred io_queue code. removes a number of ifdef's and allows more clean usage of the interface.
* core: move more storage functions to storage.cdormando2020-10-3010-572/+671
| | | | | | | extstore.h is now only used from storage.c. starting a path towards getting the storage interface to be more generalized. should be no functional changes.
* core: generalize extstore's defered IO queuedormando2020-10-305-138/+217
| | | | | | | | | | | | want to reuse the deferred IO system for extstore for something else. Should allow evolving into a more plugin-centric system. step one of three(?) - replace in place and tests pass with extstore enabled. step two should move more extstore code into storage.c step three should build the IO queue code without ifdef gating.
* fix connection limit testspkarumanchi92020-10-303-24/+33
| | | | Also changes the maxconns_fast test to check against the highest fd
* logger: fix spurious watcher hangupsdormando2020-10-292-2/+1
| | | | | | | | | of course the previous commit didn't actually fix the bug: during poll() call we weren't filtering for the right watcher when we were polling a specific one. This could lead to reading a POLLHUP on the wrong socket and other issues. hopefully that solves some mysteries with the watcher system.
View Lorry Controller Status