Memcache Binary Protocol Six Apart, Ltd.548 4th StreetSan FranciscoCA94107USAaaron@serendipity.palo-alto.ca.usSun Microsystems, INCHaakon VII g. 7BTrondheimNO-7485 TrondheimNorwaytrond.norbye@sun.com
Applications
memcache memcached cache
This memo explains the memcache binary protocol for informational
purposes.
Memcache is a high performance key-value cache. It is intentionally a
dumb cache, optimized for speed only. Applications using memcache do
not rely on it for data -- a persistent database with guaranteed
reliability is strongly recommended -- but applications can run much
faster when cached data is available in memcache.
Memcache is a high performance key-value cache. It is intentionally a
dumb cache, optimized for speed only. Applications using memcache should
not rely on it for data -- a persistent database with guaranteed
reliability is strongly recommended -- but applications can run much
faster when cached data is available in memcache.
Memcache was originally written to make
LiveJournal faster. It now powers all of
the fastest web sites that you love.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in
.
Header fields:
Magic number.Command code.Length in bytes of the text key that follows the command extras.Status of the response (non-zero on error).Length in bytes of the command extras.Reserved for future use (Sean is using this soon).Really reserved for future use (up for grabs).Length in bytes of extra + key + value.Will be copied back to you in the response.Data version check.Request packet for this protocol versionResponse packet for this protocol version
Magic byte / version. For each version of the protocol, we'll use a
different request/response value pair. This is useful for protocol
analyzers to distinguish the nature of the packet from the direction
which it is moving. Note, it is common to run a memcached instance on
a host that also runs an application server. Such a host will both
send and receive memcache packets.
The version should hopefully correspond only to different meanings of
the command byte. In an ideal world, we will not change the header
format. As reserved bytes are given defined meaning, the protocol
version / magic byte values should be incremented.
Traffic analysis tools are encouraged to identify memcache packets
and provide detailed interpretation if the magic bytes are recognized
and otherwise to provide a generic breakdown of the packet. Note, that
the key and value positions can always be identified even if the magic
byte or command opcode are not recognized.
Possible values of this two-byte field:
No errorKey not foundKey existsValue too largeInvalid argumentsItem not storedIncr/Decr on non-numeric value.Unknown commandOut of memory
Possible values of the one-byte field:
GetSetAddReplaceDeleteIncrementDecrementQuitFlushGetQNo-opVersionGetKGetKQAppendPrependStatSetQAddQReplaceQDeleteQIncrementQDecrementQQuitQFlushQAppendQPrependQ
As a convention all of the commands ending with "Q" for
Quiet. A quiet version of a command will omit responses
that are considered uninteresting. Whether a given response
is interesting is dependent upon the command. See the
descriptions of the
set commands
and set commands for
examples of commands that include quiet variants.
Possible values of the one-byte field:
Raw bytes
All communication is initiated by a request from the client,
and the server will respond to each request with zero or
multiple packets for each request. If the status code of a response
packet is non-nil, the body of the packet will contain a textual error
message. If the status code is nil, the command opcode will define the
layout of the body of the message.
The following figure illustrates the packet layout for
a packet with an error message.
Request:
MUST NOT have extras.MUST have key.MUST NOT have value.
Response (if found):
MUST have extras.MAY have key.MAY have value.4 byte flags
The get command gets a single key. The getq command is both mum
on cache miss and quiet, holding its response until a non-quiet
command is issued. Getk and getkq differs from get and getq by
adding the key into the response packet.
You're not guaranteed a response to a getq/getkq cache hit until
you send a non-getq/getkq command later, which uncorks the
server and bundles up IOs to send to the client in one go.
Clients should implement multi-get (still important for
reducing network roundtrips!) as n pipelined requests, the
first n-1 being getq/getkq, the last being a regular
get/getk. That way you're guaranteed to get a response, and
you know when the server's done. You can also do the naive
thing and send n pipelined get/getks, but then you could potentially
get back a lot of "NOT_FOUND" error code packets.
Alternatively, you can send 'n' getq/getkqs, followed by a
'noop' command.
To request the data associated with the key "Hello" the
following fields must be specified in the packet.
If the item exist on the server the following packet is returned,
otherwise a packet with status code != 0 will be returned (see
Introduction)
MUST have extras.MUST have key.MUST have value.4 byte flags4 byte expiration time
If the Data Version Check (CAS) is nonzero, the requested
operation MUST only succeed if the item exists and has a CAS value
identical to the provided value.
Add MUST fail if the item already exist.
Replace MUST fail if the item doesn't exist.
Set should store the data unconditionally if the item exists
or not.
Quiet mutations only return responses on failure. Success
is considered the general case and is suppressed when in
quiet mode, but errors should not be allowed to go
unnoticed.
The following figure shows an add-command for
Key: "Hello"Value: "World"Flags: 0xdeadbeefExpiry: in two hours
The response-packet contains no extra data, and the result of the
operation is signaled through the status code. If the command
succeeds, the CAS value for the item is returned in the CAS-field
of the packet.
MUST NOT have extras.MUST have key.MUST NOT have value.
Delete the item with the specific key.
The following figure shows a delete message for the
item "Hello".
The response-packet contains no extra data, and the result of the
operation is signaled through the status code.
MUST have extras.MUST have key.MUST NOT have value.8 byte value to add / subtract8 byte initial value (unsigned)4 byte expiration time
These commands will either add or remove the specified
amount to the requested counter.
If the counter does not exist, one of two things may happen:
If the expiration value is all one-bits (0xffffffff), the
operation will fail with NOT_FOUND.For all other expiration values, the operation will succeed
by seeding the value for this key with the provided initial
value to expire with the provided expiration time. The flags
will be set to zero.The following figure shows an incr-command for
Key: "counter"Delta: 0x01Initial: 0x00Expiry: in two hours
If the key doesn't exist, the server will respond with the
initial value. If not the incremented value will be returned.
Let's assume that the key didn't exist, so the initial value
is returned.
MUST NOT have extras.MUST NOT have key.MUST NOT have value.
Close the connection to the server.
The response-packet contains no extra data, and the result of the
operation is signaled through the status code. The server will
then close the connection.
MAY have extras.MUST NOT have key.MUST NOT have value.4 byte expiration time
Flush the items in the cache now or some time in the future as
specified by the expiration field. See the documentation of the
textual protocol for the full description on how to specify the
expiration time.
To flush the cache (delete all items) in two hours, the set
the following values in the request
The response-packet contains no extra data, and the result of the
operation is signaled through the status code.
MUST NOT have extras.MUST NOT have key.MUST NOT have value.
Used as a keep alive. Flushes outstanding getq/getkq's.
The response-packet contains no extra data, and the result of the
operation is signaled through the status code.
MUST NOT have extras.MUST NOT have key.MUST NOT have value.
Request the server version.
The server responds with a packet containing the version string
in the body with the following format: "x.y.z"
MUST NOT have extras.MUST have key.MUST have value.
These commands will either append or prepend the specified
value to the requested key.
The following example appends '!' to the 'Hello' key.
The response-packet contains no extra data, and the result of the
operation is signaled through the status code.
MUST NOT have extras.MAY have key.MUST NOT have value.
Request server statistics. Without a key specified the server will
respond with a "default" set of statistics information. Each piece
of statistical information is returned in its own packet (key
contains the name of the statistical item and the body contains the
value in ASCII format). The sequence of return packets is terminated
with a packet that contains no key and no value.
The following example requests all statistics from the server
The server will send each value in a separate packet with
an "empty" packet (no key / no value) to terminate the sequence.
Each of the response packets look like the following example:
Memcache has few authentication and no security layers whatsoever. It is
RECOMMENDED that memcache be deployed strictly on closed, protected,
back-end networks within a single data center, within a single cluster of
servers, or even on a single host, providing shared caching for multiple
applications. Memcache MUST NOT be made available on a public network.
SASL is supported as an authentication mechanism. See the wiki for more
information.
LJ NEEDS MOAR SPEEDDanga Interactivehttp://www.livejournal.com/Key words for use in RFCs to Indicate Requirement LevelsHarvard University1350 Mass. Ave.CambridgeMA 02138- +1 617 495 3864sob@harvard.edu
General
keyword
In many standards track documents several words are used to signify
the requirements in the specification. These words are often
capitalized. This document defines these words as they should be
interpreted in IETF documents. Authors who follow these guidelines
should incorporate this phrase near the beginning of their document:
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
RFC 2119.
Note that the force of these words is modified by the requirement
level of the document in which they are used.
Thanks to Brad Fitzpatrick, Anatoly Vorobey, Steven Grimm, and Dustin
Sallings, for their work on the memcached server.
Thanks to Sean Chittenden, Jonathan Steinert, Brian Aker, Evan Martin,
Nathan Neulinger, Eric Hodel, Michael Johnson, Paul Querna, Jamie
McCarthy, Philip Neustrom, Andrew O'Brien, Josh Rotenberg, Robin H.
Johnson, Tim Yardley, Paolo Borelli, Eli Bingham, Jean-Francois
Bustarret, Paul G, Paul Lindner, Alan Kasindorf, Chris Goffinet, Tomash
Brechko, and others for their work reporting bugs and maintaining
memcached client libraries and bindings in many languages.