meta text protocol commands

- we get asked a lot to provide a "metaget" command, for various uses
(debugging, etc)
- we also get asked for random one-off commands for various use cases.
- I really hate both of these situations and have been wanting to
experiment with a slight tuning of how get commands work for a long
time.

Assuming that if I offer a metaget command which gives people the
information they're curious about in an inefficient format, plus data
they don't need, we'll just end up with a slow command with
compatibility issues. No matter how you wrap warnings around a command,
people will put it into production under high load. Then I'm stuck with
it forever.

Behold, the meta commands!

See doc/protocol.txt and the wiki for a full explanation and examples.

The intent of the meta commands is to support any features the binary
protocol had over the text protocol. Though this is missing some
commands still, it is close and surpasses the binary protocol in many
ways.

1 files changed, 405 insertions, 1 deletions

diff --git a/doc/protocol.txt b/doc/protocol.txt
index 962c8dc..abe70b2 100644
--- a/doc/protocol.txt
+++ b/doc/protocol.txt
@@ -59,7 +59,7 @@ key. The client sends a command line, and then a data block; after
 that the client expects one line of response, which will indicate
 success or failure.
 
-Retrieval commands (there are two: "get" and "gets") ask the server to
+Retrieval commands ("get", "gets", "gat", and "gats") ask the server to
 retrieve data corresponding to a set of keys (one or more keys in one
 request). The client sends a command line, which includes all the
 requested keys; after that for each item the server finds it sends to
@@ -76,6 +76,50 @@ A command line always starts with the name of the command, followed by
 parameters (if any) delimited by whitespace. Command names are
 lower-case and are case-sensitive.
 
+Meta Commands [EXPERIMENTAL: MAY CHANGE]
+-------------
+
+Meta commands are a set of new ASCII-based commands. They follow the same
+structure of the basic commands but have a new flexible feature set.
+Meta commands incorporate most features available in the binary protocol, as
+well as a flag system to make the commands flexible rather than having a
+large number of high level commands. These commands completely replace the
+usage of basic Storage and Retrieval commands.
+
+Meta commands are EXPERIMENTAL and may change syntax as of this writing. They
+are documented below the basic commands.
+
+These work mixed with normal protocol commands on the same connection. All
+existing commands continue to work. The meta commands will not replace
+specialized commands that do not sit in the Storage or Retrieval categories
+noted in the 'Commands' section above.
+
+All meta commands follow a basic syntax:
+
+cm <key> <flags> <token1> <token2> <...>\r\n
+
+Where 'cm' is a 2 character command code. The number of tokens supplied is
+based off of the flags used.
+
+Responses look like:
+
+RC <flags> <token1> <token2> <...>\r\n
+
+Where 'rc' is a 2 character return code. The number of tokens returned are
+based off of the flags used.
+
+Flags are single character codes, ie 'q' or 'k' or 'I', which adjust the
+behavior of the command. The flags are reflected in the response. The order of
+which tokens are consumed or returned depend on the order of the flags given.
+For example, a metaget with flags of 'st' would return tokens for "size" and
+"TTL remaining" in that order. 'ts' would return "TTL remaining" then "size".
+
+Syntax errors are handled the same as noted under 'Error strings' section
+below.
+
+For usage examples beyond basic syntax, please see the wiki:
+https://github.com/memcached/memcached/wiki/MetaCommands
+
 Expiration times
 ----------------
 
@@ -400,6 +444,366 @@ VALUE <key> <flags> <bytes> [<cas unique>]\r\n
 
 - <data block> is the data for this item.
 
+Meta Debug
+----------
+
+The meta debug command is a human readable dump of all available internal
+metadata of an item, minus the value.
+
+me <key>\r\n
+
+- <key> means one key string.
+
+The response looks like:
+
+ME <key> <k>=<v>*\r\nEN\r\n
+
+Each of the keys and values are the internal data for the item.
+
+exp   = expiration time
+la    = time in seconds since last access
+cas   = CAS ID
+fetch = whether an item has been fetched before
+cls   = slab class id
+size  = total size in bytes
+
+Others may be added.
+
+Meta Get
+--------
+
+The meta get command is the generic command for retrieving key data from
+memcached. Based on the flags supplied, it can replace all of the commands:
+"get", "gets", "gat", "gats", "touch", as well as adding new options.
+
+mg <key> <flags> <tokens>*\r\n
+
+- <key> means one key string. Unlike "get" metaget can only take a single key.
+
+- <flags> are a set of single character codes ended with a space or newline.
+
+- <tokens>* means zero or more tokens, based on the flags supplied. They are
+  consumed in order specified by the flags.
+
+After this command, the client expects an item to be returned, received as a
+text line followed by an optional data block. After the item is transferred,
+the server sends the string
+
+"EN\r\n"
+
+to indicate the end of the response.
+
+If a response line appearing in a retrieval request is not sent back
+by the server this means that the server does not
+have the item (because it was never stored, or stored
+but deleted to make space for more items, or expired, or explicitly
+deleted by a client).
+
+An item sent by the server looks like:
+
+VA <flags> <tokens>*\r\n
+<data block>\r\n
+
+- <flags> are a reflection of the flags sent to the server. Extra flags can be
+  added to the response based on the execution of the command.
+
+- <tokens>* are tokens returned by the server, based on the flags supplied.
+  They are added in order specified by the flags sent.
+
+- <data block> is the data for this item. Note that the data black is
+  optional, requiring the 'v' flag to be supplied.
+
+The flags used by the 'mg' command are:
+
+- c: return item cas token
+- f: return client flags token
+- h: return whether item has been hit before as a 0 or 1
+- k: return key as a token
+- l: return time since item was last accessed in seconds
+- O (token): opaque value, consumes a token and copies back with response
+- q: use noreply semantics for return codes.
+- s: return item size token
+- t: return item TTL remaining in seconds (-1 for unlimited)
+- u: don't bump the item in the LRU
+- v: return item value in <data block>
+
+These flags can modify the item:
+- N (token): vivify on miss, takes TTL as a argument
+- R (token): if token is less than remaining TTL win for recache
+- T (token): update remaining TTL
+
+These extra flags can be added to the response:
+- W: client has "won" the recache flag
+- X: item is stale
+- Z: tem has already sent a winning flag
+
+The flags are now repeated with detailed information where useful:
+
+- h: return whether item has been hit before as a 0 or 1
+- l: return time since item was last accessed in seconds
+
+The above two flags return the value of "hit before?" and "last access time"
+before the command was processed. Otherwise this would always show a 1 for
+hit or always show an access time of "0" unless combined with the "u" flag.
+
+- O (token): opaque value, consumes a token and copies back with response
+
+The O (opaque) token is used by this and other commands to allow easier
+pipelining of requests while saving bytes on the wire for responses. For
+example: if pipelining three get commands together, you may not know which
+response belongs to which without also retrieving the key. If the key is very
+long this can generate a lot of traffic, especially if the data block is very
+small. Instead, you can supply an "O" flag for each mg with tokens of "1" "2"
+and "3", to match up responses to the request.
+
+Opaque tokens may be up to 32 bytes in length, and are a string similar to
+keys.
+
+- q: use noreply semantics for return codes.
+
+Noreply is a method of reducing the amount of data sent back by memcached to
+the client for normal responses. In the case of metaget, if an item is not
+available the "VA" line is normally not sent, and the response is terminated by
+"EN\r\n".
+
+With noreply enabled, the "EN\r\n" marker is suppressed. This allows you to
+pipeline several mg's together and read all of the responses without the
+"EN\r\n" lines in the middle.
+
+Errors are always returned.
+
+- u: don't bump the item in the LRU
+
+It is possible to access an item without causing it to be "bumped" to the head
+of the LRU. This also avoids marking an item as being hit or updating its last
+access time.
+
+- v: return item value in <data block>
+
+The data block for a metaget response is optional, requiring this flag to be
+passed in.
+
+These flags can modify the item:
+- N (token): vivify on miss, takes TTL as a argument
+
+Used to help with so called "dog piling" problems with recaching of popular
+items. If supplied, and metaget does not find the item in cache, it will
+create a stub item with the key and TTL as supplied. If such an item is
+created a 'W' flag is added to the response to indicate to a client that they
+have "won" the right to recache an item.
+
+The automatically created item has 0 bytes of data.
+
+Further requests will see a 'Z' flag to indicate that another client has
+already received the win flag.
+
+Can be combined with CAS flags to gate the update further.
+
+- R (token): if token is less than remaining TTL win for recache
+
+Similar to and can be combined with 'N'. If the remaining TTL of an item is
+below the supplied token, return a 'W' flag to indicate the client has "won"
+the right to recache an item. This allows refreshing an item before it leads to
+a miss.
+
+- T (token): update remaining TTL
+
+Similar to "touch" and "gat" commands, updates the remaining TTL of an item if
+hit.
+
+These extra flags can be added to the response:
+- W: client has "won" the recache flag
+
+When combined with N or R flags, a client may be informed they have "won"
+ownership of a cache item. This allows a single client to be atomically
+notified that it should cache or update an item. Further clients requesting
+the item can use the existing data block (if valid or stale), retry, wait, or
+take some other action while the item is missing.
+
+This is used when the act of recaching an item can cause undue load on another
+system (CPU, database accesses, time, and so on).
+
+- X: item is stale
+
+Items can be marked as stale by the metadelete command. This indicates to the
+client that an object needs to be updated, and that it should make a decision
+o if potentially stale data is safe to use.
+
+This is used when you want to convince clients to recache an item, but it's
+safe to use pre-existing data while the recache happens.
+
+- Z: item has already sent a winning flag
+
+When combined with the X flag, or the client N or R flags, this extra response
+flag indicates to a client that a different client is responsible for
+recaching this item. If there is data supplied it may use it, or the client
+may decide to retry later or take some other action.
+
+Meta Set
+--------
+
+The meta set command a generic command for storing data to memcached. Based on
+the flags supplied, it can replace the commands: "set", "cas". as well as
+adding new options.
+
+ms <key> <flags> <tokens>*\r\n
+
+- <key> means one key string.
+
+- <flags> are a set of single character codes ended with a space or newline.
+
+- <tokens>* means zero or more tokens, based on the flags supplied. They are
+  consumed in order specified by the flags.
+
+After this line, the client sends the data block:
+
+<data block>\r\n
+
+- <data block> is a chunk of arbitrary 8-bit data of length supplied by an 'S'
+  flag and token from the previous line. If no 'S' flag is supplied the data
+  is assumed to be 0 length.
+
+After sending the command line and the data block the client awaits
+the reply, which is of the format:
+
+CD <flags> <tokens>*\r\n
+
+Where CD is one of:
+
+- "ST" (STORED), to indicate success.
+
+- "NS" (NOT_STORED), to indicate the data was not stored, but not
+because of an error.
+
+- "EX" (EXISTS), to indicate that the item you are trying to store with
+CAS semantics has been modified since you last fetched it.
+
+- "NF" (NOT_FOUND), to indicate that the item you are trying to store
+with CAS semantics did not exist.
+
+The flags used by the 'ms' command are:
+
+- C (token): compare CAS value when storing item
+- F (token): set client flags to token (32 bit unsigned numeric)
+- I: invalid. set-to-invalid if CAS is older than it should be.
+- k: return key as a token
+- O (token): opaque value, consumes a token and copies back with response
+- q: use noreply semantics for return codes
+- S (token): size of <data block> to store
+- T (token): Time-To-Live for item, see "Expiration" above.
+
+The flags are now repeated with detailed information where useful:
+
+- C (token): compare CAS value when storing item
+
+Similar to the basic "cas" command, only store item if the supplied token
+matches the current CAS value of the item. When combined with the 'I' flag, a
+CAS value that is _lower_ than the current value may be accepted, but the item
+will be marked as "stale", returning the X flag with mget requests.
+
+- F (token): set client flags to token (32 bit unsigned numeric)
+
+Sets flags to 0 if not supplied.
+
+- I: invalid. set-to-invalid if CAS is older than it should be.
+
+Functional when combined with 'C' flag above.
+
+- O (token): opaque value, consumes a token and copies back with response
+
+See description under 'Meta Get'
+
+- q: use noreply semantics for return codes
+
+Noreply is a method of reducing the amount of data sent back by memcached to
+the client for normal responses. In the case of metaset, a response that would
+start with "ST" (STORED) will not be sent. Any other code, such as "EX"
+(EXISTS) will still be returned.
+
+Errors are always returned.
+
+Meta Delete
+-----------
+
+The meta delete command allows for explicit deletion of items, as well as
+marking items as "stale" to allow serving items as stale during revalidation.
+
+md <key> <flags> <tokens>*\r\n
+
+- <key> means one key string.
+
+- <flags> are a set of single character codes ended with a space or newline.
+
+- <tokens>* means zero or more tokens, based on the flags supplied. They are
+  consumed in order specified by the flags.
+
+The response is in the format:
+
+CD <flags> <tokens>*\r\n
+
+Where CD is one of:
+
+- "DE" (DELETED), to indicate success
+
+- "NF" (NOT_FOUND), to indicate that the item with this key was not found.
+
+- "EX" (EXISTS), to indicate that the supplied CAS token does not match the
+  stored item.
+
+The flags used by the 'md' command are:
+
+- C (token): compare CAS value
+- I: invalidate. mark as stale, bumps CAS.
+- k: return key
+- O: opaque to copy back.
+- q: noreply
+- T (token): updates TTL, only when paired with the 'I' flag
+
+The flags are now repeated with detailed information where useful:
+
+- C (token): compare CAS value
+
+Can be used to only delete or mark-stale if a supplied CAS value matches.
+
+- I: invalidate. mark as stale, bumps CAS.
+
+Instead of removing an item, this will give the item a new CAS value and mark
+it as stale. This means when it is later fetched by metaget, the client will
+be supplied an 'X' flag to show the data is stale and needs to be recached.
+
+- O (token): opaque to copy back.
+
+See description under 'Meta Get'
+
+- q: noreply
+
+See description under 'Meta Set'. In the case of meta delete, this will hide
+response lines with the code "DE". It will still return any other responses.
+
+- T (token): updates TTL, only when paired with the 'I' flag
+
+When marking an item as stale with 'I', the 'T' flag can be used to update the
+TTL as well; limiting the amount of time an item will live while stale and
+waiting to be recached.
+
+Meta No-Op
+----------
+
+The meta no-op command exists solely to return a static response code. It
+takes no flags, no arguments.
+
+This returns the static response:
+
+"EN\r\n"
+
+Similar to a meta get command.
+
+This command is useful when used with the 'q' flag and pipelining commands.
+For example, with 'mg' the response lines are blank on miss, and the 'q' flag
+will hide the "EN\r\n" response code. If pipelining several 'mg's together
+with noreply semantics, an "mn\r\n" command can be tagged to the end of the
+chain, which will return an "EN\r\n", signalling to a client that all previous
+commands have been processed.
 
 Slabs Reassign
 --------------