Add overlay2 description to overlayfs-driver.md

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>

1 files changed, 146 insertions, 57 deletions

diff --git a/docs/userguide/storagedriver/overlayfs-driver.md b/docs/userguide/storagedriver/overlayfs-driver.md
index 442789b91a..17e0c1adbe 100644
--- a/docs/userguide/storagedriver/overlayfs-driver.md
+++ b/docs/userguide/storagedriver/overlayfs-driver.md
@@ -25,13 +25,20 @@ using it in production Docker environments.
 Docker's `overlay` storage driver leverages several OverlayFS features to build
  and manage the on-disk structures of images and containers.
 
+Since version 1.12, Docker also provides `overlay2` storage driver which is much
+more efficient than `overlay` in terms of inode utilization. The `overlay2`
+driver is only compatible with Linux kernel 4.0 and later.
+
+For comparison between `overlay` vs `overlay2`, please also refer to [Select a
+storage driver](selectadriver.md#overlay-vs-overlay2).
+
 >**Note**: Since it was merged into the mainline kernel, the OverlayFS *kernel 
 >module* was renamed from "overlayfs" to "overlay". As a result you may see the
 > two terms used interchangeably in some documentation. However, this document 
-> uses  "OverlayFS" to refer to the overall filesystem, and `overlay` to refer 
-> to Docker's storage-driver.
+> uses  "OverlayFS" to refer to the overall filesystem, and `overlay`/`overlay2`
+> to refer to Docker's storage-drivers.
 
-## Image layering and sharing with OverlayFS
+## Image layering and sharing with OverlayFS (`overlay`)
 
 OverlayFS takes two directories on a single Linux host, layers one on top of 
 the other, and provides a single unified view. These directories are often 
@@ -53,52 +60,60 @@ Notice how the image layer and container layer can contain the same files. When
 obscure the existence of the same files in the image layer ("lowerdir"). The 
 container mount ("merged") presents the unified view.
 
-OverlayFS only works with two layers. This means that multi-layered images 
-cannot be implemented as multiple OverlayFS layers. Instead, each image layer 
-is implemented as its own directory under `/var/lib/docker/overlay`. 
-Hard links are then used as a space-efficient way to reference data shared with
- lower layers. As of Docker 1.10, image layer IDs no longer correspond to 
-directory names in `/var/lib/docker/`
+The `overlay` driver only works with two layers. This means that multi-layered
+images cannot be implemented as multiple OverlayFS layers. Instead, each image
+layer is implemented as its own directory under `/var/lib/docker/overlay`.  Hard
+links are then used as a space-efficient way to reference data shared with lower
+layers. As of Docker 1.10, image layer IDs no longer correspond to directory
+names in `/var/lib/docker/`
 
 To create a container, the `overlay` driver combines the directory representing
  the image's top layer plus a new directory for the container. The image's top 
 layer is the "lowerdir" in the overlay and read-only. The new directory for the
  container is the "upperdir" and is writable.
 
-## Example: Image and container on-disk constructs
+### Example: Image and container on-disk constructs (`overlay`)
 
 The following `docker pull` command shows a Docker host with downloading a 
-Docker image comprising four layers.
+Docker image comprising five layers.
 
     $ sudo docker pull ubuntu
     Using default tag: latest
     latest: Pulling from library/ubuntu
-    8387d9ff0016: Pull complete
-    3b52deaaf0ed: Pull complete
-    4bd501fad6de: Pull complete
+
+    5ba4f30e5bea: Pull complete
+    9d7d19c9dc56: Pull complete
+    ac6ad7efd0f9: Pull complete
+    e7491a747824: Pull complete
     a3ed95caeb02: Pull complete
-    Digest: sha256:457b05828bdb5dcc044d93d042863fba3f2158ae249a6db5ae3934307c757c54
+    Digest: sha256:46fb5d001b88ad904c5c732b086b596b92cfb4a4840a3abd0e35dbb6870585e4
     Status: Downloaded newer image for ubuntu:latest
 
 Each image layer has it's own directory under `/var/lib/docker/overlay/`. This 
 is where the contents of each image layer are stored. 
 
-The output of the command below shows the four directories that store the 
+The output of the command below shows the five directories that store the 
 contents of each image layer just pulled. However, as can be seen, the image 
 layer IDs do not match the directory names in `/var/lib/docker/overlay`. This 
 is normal behavior in Docker 1.10 and later.
 
     $ ls -l /var/lib/docker/overlay/
-    total 24
-    drwx------ 3 root root 4096 Oct 28 11:02 1d073211c498fd5022699b46a936b4e4bdacb04f637ad64d3475f558783f5c3e
-    drwx------ 3 root root 4096 Oct 28 11:02 5a4526e952f0aa24f3fcc1b6971f7744eb5465d572a48d47c492cb6bbf9cbcda
-    drwx------ 5 root root 4096 Oct 28 11:06 99fcaefe76ef1aa4077b90a413af57fd17d19dce4e50d7964a273aae67055235
-    drwx------ 3 root root 4096 Oct 28 11:01 c63fb41c2213f511f12f294dd729b9903a64d88f098c20d2350905ac1fdbcbba
+    total 20
+    drwx------ 3 root root 4096 Jun 20 16:11 38f3ed2eac129654acef11c32670b534670c3a06e483fce313d72e3e0a15baa8
+    drwx------ 3 root root 4096 Jun 20 16:11 55f1e14c361b90570df46371b20ce6d480c434981cbda5fd68c6ff61aa0a5358
+    drwx------ 3 root root 4096 Jun 20 16:11 824c8a961a4f5e8fe4f4243dab57c5be798e7fd195f6d88ab06aea92ba931654
+    drwx------ 3 root root 4096 Jun 20 16:11 ad0fe55125ebf599da124da175174a4b8c1878afe6907bf7c78570341f308461
+    drwx------ 3 root root 4096 Jun 20 16:11 edab9b5e5bf73f2997524eebeac1de4cf9c8b904fa8ad3ec43b3504196aa3801
 
 The image layer directories contain the files unique to that layer as well as 
 hard links to the data that is shared with lower layers. This allows for 
 efficient use of disk space.
 
+    $ ls -i /var/lib/docker/overlay/38f3ed2eac129654acef11c32670b534670c3a06e483fce313d72e3e0a15baa8/root/bin/ls
+    19793696 /var/lib/docker/overlay/38f3ed2eac129654acef11c32670b534670c3a06e483fce313d72e3e0a15baa8/root/bin/ls
+    $ ls -i /var/lib/docker/overlay/55f1e14c361b90570df46371b20ce6d480c434981cbda5fd68c6ff61aa0a5358/root/bin/ls
+    19793696 /var/lib/docker/overlay/55f1e14c361b90570df46371b20ce6d480c434981cbda5fd68c6ff61aa0a5358/root/bin/ls
+
 Containers also exist on-disk in the Docker host's filesystem under 
 `/var/lib/docker/overlay/`. If you inspect the directory relating to a running 
 container using the `ls -l` command, you find the following file and 
@@ -106,17 +121,17 @@ directories.
 
     $ ls -l /var/lib/docker/overlay/<directory-of-running-container>
     total 16
-    -rw-r--r-- 1 root root   64 Oct 28 11:06 lower-id
-    drwxr-xr-x 1 root root 4096 Oct 28 11:06 merged
-    drwxr-xr-x 4 root root 4096 Oct 28 11:06 upper
-    drwx------ 3 root root 4096 Oct 28 11:06 work
+    -rw-r--r-- 1 root root   64 Jun 20 16:39 lower-id
+    drwxr-xr-x 1 root root 4096 Jun 20 16:39 merged
+    drwxr-xr-x 4 root root 4096 Jun 20 16:39 upper
+    drwx------ 3 root root 4096 Jun 20 16:39 work
 
 These four filesystem objects are all artifacts of OverlayFS. The "lower-id" 
 file contains the ID of the top layer of the image the container is based on. 
 This is used by OverlayFS as the "lowerdir".
 
-    $ cat /var/lib/docker/overlay/73de7176c223a6c82fd46c48c5f152f2c8a7e49ecb795a7197c3bb795c4d879e/lower-id
-    1d073211c498fd5022699b46a936b4e4bdacb04f637ad64d3475f558783f5c3e
+    $ cat /var/lib/docker/overlay/ec444863a55a9f1ca2df72223d459c5d940a721b2288ff86a3f27be28b53be6c/lower-id
+    55f1e14c361b90570df46371b20ce6d480c434981cbda5fd68c6ff61aa0a5358
 
 The "upper" directory is the containers read-write layer. Any changes made to 
 the container are written to this directory.
@@ -133,13 +148,85 @@ You can verify all of these constructs from the output of the `mount` command.
 (Ellipses and line breaks are used in the output below to enhance readability.)
 
     $ mount | grep overlay
-    overlay on /var/lib/docker/overlay/73de7176c223.../merged
-    type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay/1d073211c498.../root,
-    upperdir=/var/lib/docker/overlay/73de7176c223.../upper,
-    workdir=/var/lib/docker/overlay/73de7176c223.../work)
+    overlay on /var/lib/docker/overlay/ec444863a55a.../merged
+    type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay/55f1e14c361b.../root,
+    upperdir=/var/lib/docker/overlay/ec444863a55a.../upper,
+    workdir=/var/lib/docker/overlay/ec444863a55a.../work)
 
 The output reflects that the overlay is mounted as read-write ("rw").
 
+
+## Image layering and sharing with OverlayFS (`overlay2`)
+
+While the `overlay` driver only works with a single lower OverlayFS layer and
+hence requires hard links for implementation of multi-layered images, the
+`overlay2` driver natively supports multiple lower OverlayFS layers (up to 128).
+
+Hence the `overlay2` driver offers better performance for layer-related docker commands (e.g. `docker build` and `docker commit`), and consumes fewer inodes than the `overlay` driver.
+
+### Example: Image and container on-disk constructs (`overlay2`)
+
+After downloading a five-layer image using `docker pull ubuntu`, you can see
+six directories under `/var/lib/docker/overlay2`.
+
+    $ ls -l /var/lib/docker/overlay2
+    total 24
+    drwx------ 5 root root 4096 Jun 20 07:36 223c2864175491657d238e2664251df13b63adb8d050924fd1bfcdb278b866f7
+    drwx------ 3 root root 4096 Jun 20 07:36 3a36935c9df35472229c57f4a27105a136f5e4dbef0f87905b2e506e494e348b
+    drwx------ 5 root root 4096 Jun 20 07:36 4e9fa83caff3e8f4cc83693fa407a4a9fac9573deaf481506c102d484dd1e6a1
+    drwx------ 5 root root 4096 Jun 20 07:36 e8876a226237217ec61c4baf238a32992291d059fdac95ed6303bdff3f59cff5
+    drwx------ 5 root root 4096 Jun 20 07:36 eca1e4e1694283e001f200a667bb3cb40853cf2d1b12c29feda7422fed78afed
+    drwx------ 2 root root 4096 Jun 20 07:36 l
+
+The "l" directory contains shortened layer identifiers as symbolic links.  These
+shortened identifiers are used for avoid hitting the page size limitation on
+mount arguments.
+
+    $ ls -l /var/lib/docker/overlay2/l
+    total 20
+    lrwxrwxrwx 1 root root 72 Jun 20 07:36 6Y5IM2XC7TSNIJZZFLJCS6I4I4 -> ../3a36935c9df35472229c57f4a27105a136f5e4dbef0f87905b2e506e494e348b/diff
+    lrwxrwxrwx 1 root root 72 Jun 20 07:36 B3WWEFKBG3PLLV737KZFIASSW7 -> ../4e9fa83caff3e8f4cc83693fa407a4a9fac9573deaf481506c102d484dd1e6a1/diff
+    lrwxrwxrwx 1 root root 72 Jun 20 07:36 JEYMODZYFCZFYSDABYXD5MF6YO -> ../eca1e4e1694283e001f200a667bb3cb40853cf2d1b12c29feda7422fed78afed/diff
+    lrwxrwxrwx 1 root root 72 Jun 20 07:36 NFYKDW6APBCCUCTOUSYDH4DXAT -> ../223c2864175491657d238e2664251df13b63adb8d050924fd1bfcdb278b866f7/diff
+    lrwxrwxrwx 1 root root 72 Jun 20 07:36 UL2MW33MSE3Q5VYIKBRN4ZAGQP -> ../e8876a226237217ec61c4baf238a32992291d059fdac95ed6303bdff3f59cff5/diff
+
+The lowerest layer contains the "link" file which contains the name of the shortened
+identifier, and the "diff" directory which contains the contents.
+
+    $ /var/lib/docker/overlay2/3a36935c9df35472229c57f4a27105a136f5e4dbef0f87905b2e506e494e348b/
+    diff  link
+    $ cat /var/lib/docker/overlay2/3a36935c9df35472229c57f4a27105a136f5e4dbef0f87905b2e506e494e348b/link
+    6Y5IM2XC7TSNIJZZFLJCS6I4I4
+    $ ls  /var/lib/docker/overlay2/3a36935c9df35472229c57f4a27105a136f5e4dbef0f87905b2e506e494e348b/diff
+    bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
+
+The second layer contains the "lower" file for denoting the layer composition,
+and the "diff" directory for the layer contents.  It also contains the "merged" and
+the "work" directories.
+
+    $ ls /var/lib/docker/overlay2/223c2864175491657d238e2664251df13b63adb8d050924fd1bfcdb278b866f7
+    diff  link  lower  merged  work
+    $ cat /var/lib/docker/overlay2/223c2864175491657d238e2664251df13b63adb8d050924fd1bfcdb278b866f7/lower
+    l/6Y5IM2XC7TSNIJZZFLJCS6I4I4
+    $ ls /var/lib/docker/overlay2/223c2864175491657d238e2664251df13b63adb8d050924fd1bfcdb278b866f7/diff/
+    etc  sbin  usr  var
+
+A directory for running container have similar files and directories as well.
+Note that the lower list is separated by ':', and ordered from highest layer to lower.
+
+    $ ls -l /var/lib/docker/overlay/<directory-of-running-container>
+    $ cat /var/lib/docker/overlay/<directory-of-running-container>/lower
+    l/DJA75GUWHWG7EWICFYX54FIOVT:l/B3WWEFKBG3PLLV737KZFIASSW7:l/JEYMODZYFCZFYSDABYXD5MF6YO:l/UL2MW33MSE3Q5VYIKBRN4ZAGQP:l/NFYKDW6APBCCUCTOUSYDH4DXAT:l/6Y5IM2XC7TSNIJZZFLJCS6I4I4
+
+The result of `mount` is as follows:
+
+    $ mount | grep overlay
+    overlay on /var/lib/docker/overlay2/9186877cdf386d0a3b016149cf30c208f326dca307529e646afce5b3f83f5304/merged
+    type overlay (rw,relatime,
+    lowerdir=l/DJA75GUWHWG7EWICFYX54FIOVT:l/B3WWEFKBG3PLLV737KZFIASSW7:l/JEYMODZYFCZFYSDABYXD5MF6YO:l/UL2MW33MSE3Q5VYIKBRN4ZAGQP:l/NFYKDW6APBCCUCTOUSYDH4DXAT:l/6Y5IM2XC7TSNIJZZFLJCS6I4I4,
+    upperdir=9186877cdf386d0a3b016149cf30c208f326dca307529e646afce5b3f83f5304/diff,
+    workdir=9186877cdf386d0a3b016149cf30c208f326dca307529e646afce5b3f83f5304/work)
+
 ## Container reads and writes with overlay
 
 Consider three scenarios where a container opens a file for read access with 
@@ -164,9 +251,9 @@ Consider some scenarios where files in a container are modified.
 
 - **Writing to a file for the first time**. The first time a container writes 
 to an existing file, that file does not exist in the container ("upperdir"). 
-The `overlay` driver performs a *copy_up* operation to copy the file from the 
-image ("lowerdir") to the container ("upperdir"). The container then writes the
- changes to the new copy of the file in the container layer.
+The `overlay`/`overlay2` driver performs a *copy_up* operation to copy the file
+from the image ("lowerdir") to the container ("upperdir"). The container then
+writes the changes to the new copy of the file in the container layer.
 
     However, OverlayFS works at the file level not the block level. This means 
 that all OverlayFS copy-up operations copy entire files, even if the file is 
@@ -191,13 +278,13 @@ file in the image layer ("lowerdir") is not deleted. However, the whiteout file
 created in the "upperdir". This has the same effect as a whiteout file and 
 effectively masks the existence of the directory in the image's "lowerdir".
 
-## Configure Docker with the overlay storage driver
+## Configure Docker with the `overlay`/`overlay2` storage driver
 
-To configure Docker to use the overlay storage driver your Docker host must be 
+To configure Docker to use the `overlay` storage driver your Docker host must be 
 running version 3.18 of the Linux kernel (preferably newer) with the overlay 
-kernel module loaded. OverlayFS can operate on top of most supported Linux 
-filesystems. However, ext4 is currently recommended for use in production 
-environments.
+kernel module loaded. For the `overlay2` driver, the version of your kernel must
+be 4.0 or newer. OverlayFS can operate on top of most supported Linux filesystems.
+However, ext4 is currently recommended for use in production environments.
 
 The following procedure shows you how to configure your Docker host to use 
 OverlayFS. The procedure assumes that the Docker daemon is in a stopped state.
@@ -216,7 +303,7 @@ OverlayFS. The procedure assumes that the Docker daemon is in a stopped state.
         $ lsmod | grep overlay
         overlay
 
-3. Start the Docker daemon with the `overlay` storage driver.
+3. Start the Docker daemon with the `overlay`/`overlay2` storage driver.
 
         $ dockerd --storage-driver=overlay &
         [1] 29403
@@ -226,12 +313,12 @@ OverlayFS. The procedure assumes that the Docker daemon is in a stopped state.
         <output truncated>
 
     Alternatively, you can force the Docker daemon to automatically start with
-    the `overlay` driver by editing the Docker config file and adding the
-    `--storage-driver=overlay` flag to the `DOCKER_OPTS` line. Once this option
+    the `overlay`/`overlay2` driver by editing the Docker config file and adding
+    the `--storage-driver=overlay` flag to the `DOCKER_OPTS` line. Once this option
     is set you can start the daemon using normal startup scripts without having
     to manually pass in the `--storage-driver` flag.
 
-4. Verify that the daemon is using the `overlay` storage driver
+4. Verify that the daemon is using the `overlay`/`overlay2` storage driver
 
         $ docker info
         Containers: 0
@@ -244,22 +331,23 @@ OverlayFS. The procedure assumes that the Docker daemon is in a stopped state.
 `extfs`. Multiple backing filesystems are supported but `extfs` (ext4) is 
 recommended for production use cases.
 
-Your Docker host is now using the `overlay` storage driver. If you run the 
-`mount` command, you'll find Docker has automatically created the `overlay` 
-mount with the required "lowerdir", "upperdir", "merged" and "workdir" 
+Your Docker host is now using the `overlay`/`overlay2` storage driver. If you
+run the `mount` command, you'll find Docker has automatically created the
+`overlay` mount with the required "lowerdir", "upperdir", "merged" and "workdir"
 constructs.
 
 ## OverlayFS and Docker Performance
 
-As a general rule, the `overlay` driver should be fast. Almost certainly faster
- than `aufs` and `devicemapper`. In certain circumstances it may also be faster
- than `btrfs`. That said, there are a few things to be aware of relative to the
- performance of Docker using the `overlay` storage driver.
+As a general rule, the `overlay`/`overlay2` drivers should be fast. Almost
+certainly faster than `aufs` and `devicemapper`. In certain circumstances it may
+also be faster than `btrfs`. That said, there are a few things to be aware of
+relative to the performance of Docker using the `overlay`/`overlay2` storage
+drivers.
 
-- **Page Caching**. OverlayFS supports page cache sharing. This means multiple 
-containers accessing the same file can share a single page cache entry (or 
-entries). This makes the `overlay` driver efficient with memory and a good 
-option for PaaS and other high density use cases.
+- **Page Caching**. OverlayFS supports page cache sharing. This means multiple
+containers accessing the same file can share a single page cache entry (or
+entries). This makes the `overlay`/`overlay2` drivers efficient with memory and
+a good option for PaaS and other high density use cases.
 
 - **copy_up**. As with AUFS, OverlayFS has to perform copy-up operations any 
 time a container writes to a file for the first time. This can insert latency 
@@ -274,13 +362,14 @@ possible to incur far larger latencies if searching through many AUFS layers.
 - **RPMs and Yum**. OverlayFS only implements a subset of the POSIX standards. 
 This can result in certain OverlayFS operations breaking POSIX standards. One 
 such operation is the *copy-up* operation. Therefore, using `yum` inside of a 
-container on a Docker host using the `overlay` storage driver is unlikely to 
-work without implementing workarounds.
+container on a Docker host using the `overlay`/`overlay2` storage drivers is
+unlikely to work without implementing workarounds.
 
 - **Inode limits**. Use of the `overlay` storage driver can cause excessive 
 inode consumption. This is especially so as the number of images and containers
  on the Docker host grows. A Docker host with a large number of images and lots
- of started and stopped containers can quickly run out of inodes.
+ of started and stopped containers can quickly run out of inodes. The `overlay2`
+ does not have such an issue.
 
 Unfortunately you can only specify the number of inodes in a filesystem at the 
 time of creation. For this reason, you may wish to consider putting