lvmlockd: use dlock_gl to acquire the global lockdev-dct-lvmlockd6-lockgl

The global lock (gl) is a simple read/write lock that protects
global lvm metadata, i.e. metadata or information that is not
isolated to a single VG.  This global information includes:

A) The VG name space.
B) PVs or devices that do not belong to a VG.

The function to acquire/release the global lock is dlock_gl():

. To acquire the gl in exclusive mode for writing/changing A or B:
  dlock_gl(cmd, "ex", 0);

. To acquire the gl in shared mode for reading/listing A or B:
  dlock_gl(cmd, "sh", 0);

. To release the gl:
  dlock_gl(cmd, "un", 0);

The dlock_gl() function sends a message to lvmlockd, asking for
the lock in the specified mode.  lvmlockd acquires the lock from
the underlying lock manager, and sends the result back to the command.
When the command exits, or calls dlock_gl("un"), lvmlockd releases
the lock in the underlying lock manager.

When lvm is compiled without lvmlockd support, all the dlock_gl() calls
simply compile to success (1).

Using the global lock in commands is simple:

. A command that wants to get an accurate list of all VG names
  should acquire the gl in the sh mode, then read the list.

. A command that wants to add or remove a VG name should acquire
  the gl in the ex mode, then add/remove the name.

. A command that wants to get an accurate list of all PVs/devices
  should acquire the gl in the sh mode, then read the list.

. A command that wants to change a device to/from a PV or add/remove
  a PV to/from a VG should acquire the gl in the ex mode, then make
  the change.

. A command that wants to read the properties of an orphan PV
  should acquire the gl in the sh mode, then read the properties.

. A command that wants to change the properties of an orphan PV
  should acquire the gl in the ex mode, then change the properties.

. The gl is acquired at the start of a command before any processing.
  This is necessary so that the cached information used by the command
  is valid and up to date (see caching below).

. A command generally knows at the outset which of the things above
  it is going to do, so it knows which lock mode to acquire.

. If a command is given a tag, the tag matching requires a complete
  and accurate search of all VGs, and therefore implies that the global
  shared lock is needed.

Locking conflicts:

When a command calls dlock_gl(), the lock request is passed to lvmlockd.
lvmlockd makes the corresponding lock request in the lock manager using a
non-blocking request.  If another command on another host holds the gl in
a conflicting mode, the lock request fails, and lvmlockd returns the
failure to the command.  The command reports the lock conflict and fails.

If a reporting command (sh lock), conflicts with another command using an
ex lock (like vgextend), then the reporting command can simply be rerun.
A future option may enable retrying sh lock requests within lvmlockd,
making simple, incidental conflicts invisible.  (These retries could be
disabled or limited to a certain number via a command or config option.)

If a command is changing global state (using an ex lock), the conflict
could be with another sh lock (e.g. reporting command) or another ex lock
(another command changing global state.)  The lock manager does not say
which type of conflict it was.  In the case of ex/sh conflict, a retry of
the ex request could be automatic, but an ex/ex conflict would generally
want inspection before retrying.  Uncoordinated commands concurrently
changing the same global state would be uncommon, and a warning of the
conflict with failure is probably preferred, so the state can be
inspected.  Still, the same retry options as above could be applied if
needed.

Caching:

In addition to ex/sh mutual exclusion, the lock manager allows an
application to stash a chunk of its own data within a lock.  This chunk of
data is called the "lock value block" (lvb).  This opaque data does not
affect the locking behavior, but offers a convenient way for applications
to pass around extra information related to the lock.  The lvb can be read
and written by any host using the lock (the application can read the lvb
content when acquiring the lock, and write it when releasing an ex lock.)
(The dlm lvb is only 32 bytes.  The sanlock lvb is up to 512 bytes.)

An application can use the lvb for anything it wishes, and it's often used
to help manage the cache associated with the lock.  lvmlockd stores a
counter/version number in the lvb.  It's incremented when anything
protected by the gl (A or B above) is changed.  When other hosts see the
version number has increased, they know that A or B have changed.
lvmlockd refines this further by using a second version number that is
incremented only when the VG namespace (A) changes.  The dlock_gl() flag
UPDATE_NAMES tells lvmlockd that the VG namespace is being changed.

When lvmlockd acquires the gl and sees that the counters in the lvb have
been incremented, it knows that the objects protected by the gl have been
changed by another host.  This implies that the local host's cache of this
global information is likely out of date, and needs to be refreshed from
disk.  When this happens, lvmlockd sends a message to lvmetad to
invalidate the cached global information in lvmetad.  When a command sees
that the data from lvmetad is stale, it reads from disk and updates
lvmetad.

All users of the global lock want to use valid information from lvmetad,
so they always check that the lvmetad cache is valid before using it, and
refresh it if needed.  This check and refresh is done by
lvmetad_validate_global_cache().  Instead of always calling
dlock_gl()+lvmetad_validate_global_cache() back to back, dlock_gl() calls
lvmetad_validate_global_cache() as the final step before returning.

In the future, more optimizations can be made related to global cache
updating.  Similar to the UPDATE_NAMES method, commands can tell lvmlockd
more details about what they are chaning under the global lock.  lvmlockd
can propagate these details to others using the gl lvb.  Knowing these
details, other hosts can limit their rescanning to only what's necessary
given the specific changes.

vgcreate with sanlock:

With the sanlock lock_type, the sanlock locks are stored on disk, on a
hidden LV within the VG.  The first sanlock VG created will hold the
global lock.  Creating the first sanlock VG is a special case because no
global lock will exist until after the VG is created on disk.  vgcreate
calls dlock_gl_create() to handle this special case.  The comments in that
function explain the details of how it works.

command gl requirements:

Some commands are listed twice if they have two different
behaviors (depending on args) that need different gl usage.
As listed above (in A,B), the reasons for using the gl are:

A) reading or changing the VG name space
B) reading or changing PV orphans
   (orphan properties or assignment to VGs)

command:   gl mode used, reason(s) gl is needed

vgsplit:   ex, add vg name
vgrename:  ex, add vg name
vgcreate:  ex, add vg name, rem pv orphan
vgremove:  ex, rem vg name, add pv orphan
vgmerge:   ex, rem vg name
vgextend:  ex, rem pv orphan
vgreduce:  ex, add pv orphan
vgscan:    sh, get vg names
vgs:       sh, get vg names (only if tags used or no args)
vgchange:  sh, get vg names (only if tags used or no args)
vgchange:  ex, change vg system_id/uuid/lock_type (equivalent to name)

pvcreate:  ex, add pv orphan
pvremove:  ex, rem pv orphan
pvdisplay: sh, get vg names
pvscan:    sh, get vg names
pvresize:  sh, get vg names
pvresize:  ex, change pv orphan (only if pv is an orphan)
pvchange:  sh, get vg names
pvchange:  ex, change pv orphan (only if pv is an orphan)

lvchange:  sh, get vg names (only if tags used)
lvscan:    sh, get vg names

28 files changed, 530 insertions, 1 deletions

diff --git a/daemons/lvmlockd/lvmlockd-client.h b/daemons/lvmlockd/lvmlockd-client.h
index bde0191ea..dcae38468 100644
--- a/daemons/lvmlockd/lvmlockd-client.h
+++ b/daemons/lvmlockd/lvmlockd-client.h
@@ -34,6 +34,14 @@ static inline void lvmlockd_close(daemon_handle h)
 }
 
 /*
+ * Errors returned as the lvmlockd result value.
+ */
+#define ENOLS     210 /* lockspace not found */
+#define ELOCALVG  211 /* vg is local */
+#define EOTHERVG  212 /* vg sysid specifies other host */
+#define ESTARTING 213 /* lockspace is starting */
+
+/*
  * Also see lvmlockd-sanlock GL_LOCK_BEGIN, VG_LOCK_BEGIN, LV_LOCK_BEGIN.
  * gl lock at sanlock lease area 65
  * vg lock at sanlock lease area 66
diff --git a/lib/config/config_settings.h b/lib/config/config_settings.h
index 39e4b0e5a..7438143ba 100644
--- a/lib/config/config_settings.h
+++ b/lib/config/config_settings.h
@@ -194,6 +194,8 @@ cfg(global_cache_repair_executable_CFG, "cache_repair_executable", global_CFG_SE
 cfg_array(global_cache_repair_options_CFG, "cache_repair_options", global_CFG_SECTION, 0, CFG_TYPE_STRING, "#S" DEFAULT_CACHE_REPAIR_OPTIONS, vsn(2, 2, 108), NULL)
 cfg(global_system_id_source_CFG, "system_id_source", global_CFG_SECTION, CFG_DEFAULT_UNDEFINED, CFG_TYPE_STRING, NULL, vsn(2, 2, 112), NULL)
 cfg(global_system_id_file_CFG, "system_id_file", global_CFG_SECTION, CFG_DEFAULT_UNDEFINED, CFG_TYPE_STRING, NULL, vsn(2, 2, 112), NULL)
+cfg(global_allow_override_lock_modes_CFG, "allow_override_lock_modes", global_CFG_SECTION, 0, CFG_TYPE_BOOL, 0, vsn(2, 2, 112), NULL)
+cfg(global_read_only_lock_modes_CFG, "read_only_lock_modes", global_CFG_SECTION, 0, CFG_TYPE_BOOL, 0, vsn(2, 2, 112), NULL)
 
 cfg(activation_checks_CFG, "checks", activation_CFG_SECTION, 0, CFG_TYPE_BOOL, DEFAULT_ACTIVATION_CHECKS, vsn(2, 2, 86), NULL)
 cfg(activation_udev_sync_CFG, "udev_sync", activation_CFG_SECTION, 0, CFG_TYPE_BOOL, DEFAULT_UDEV_SYNC, vsn(2, 2, 51), NULL)
diff --git a/lib/locking/lvmlockd.c b/lib/locking/lvmlockd.c
index 29b72390f..ec8f1c40e 100644
--- a/lib/locking/lvmlockd.c
+++ b/lib/locking/lvmlockd.c
@@ -105,8 +105,26 @@ void lvmlockd_set_socket(const char *sock)
 	_lvmlockd_socket = sock;
 }
 
+/* Translate the result strings from lvmlockd to bit flags. */
 static void _result_str_to_flags(const char *str, uint32_t *flags)
 {
+	if (strstr(str, "NO_LOCKSPACES"))
+		*flags |= LD_RF_NO_LOCKSPACES;
+
+	if (strstr(str, "NO_GL_LS"))
+		*flags |= LD_RF_NO_GL_LS;
+
+	if (strstr(str, "LOCAL_LS"))
+		*flags |= LD_RF_LOCAL_LS;
+
+	if (strstr(str, "DUP_GL_LS"))
+		*flags |= LD_RF_DUP_GL_LS;
+
+	if (strstr(str, "INACTIVE_LS"))
+		*flags |= LD_RF_INACTIVE_LS;
+
+	if (strstr(str, "ADD_LS_ERROR"))
+		*flags |= LD_RF_ADD_LS_ERROR;
 }
 
 /*
diff --git a/lib/locking/lvmlockd.h b/lib/locking/lvmlockd.h
index 2aeaaee7d..87b49ccfe 100644
--- a/lib/locking/lvmlockd.h
+++ b/lib/locking/lvmlockd.h
@@ -19,6 +19,14 @@
 #define LOCK_TYPE_DLM     2
 #define LOCK_TYPE_SANLOCK 3
 
+/* lvmlockd_send result flags */
+#define LD_RF_NO_LOCKSPACES     0x00000001
+#define LD_RF_NO_GL_LS          0x00000002
+#define LD_RF_LOCAL_LS          0x00000004
+#define LD_RF_DUP_GL_LS         0x00000008
+#define LD_RF_INACTIVE_LS       0x00000010
+#define LD_RF_ADD_LS_ERROR      0x00000020
+
 /*
  * lock_type    lock_type_num
  * "none"    -> LOCK_TYPE_NONE
diff --git a/tools/Makefile.in b/tools/Makefile.in
index 30f5fba43..d6a4de1eb 100644
--- a/tools/Makefile.in
+++ b/tools/Makefile.in
@@ -17,7 +17,6 @@ top_srcdir = @top_srcdir@
 top_builddir = @top_builddir@
 
 SOURCES =\
-	dlock.c \
 	dumpconfig.c \
 	formats.c \
 	lvchange.c \
@@ -64,6 +63,10 @@ SOURCES =\
 	vgscan.c \
 	vgsplit.c
 
+ifeq ("@BUILD_LVMLOCKD@", "yes")
+  SOURCES += dlock.c
+endif
+
 SOURCES2 =\
 	dmsetup.c \
 	lvm.c \
diff --git a/tools/args.h b/tools/args.h
index afa4ce6f5..72f592dd5 100644
--- a/tools/args.h
+++ b/tools/args.h
@@ -47,6 +47,7 @@ arg(ignoremonitoring_ARG, '\0', "ignoremonitoring", NULL, 0)
 arg(ignoreskippedcluster_ARG, '\0', "ignoreskippedcluster", NULL, 0)
 arg(ignoreunsupported_ARG, '\0', "ignoreunsupported", NULL, 0)
 arg(labelsector_ARG, '\0', "labelsector", int_arg, 0)
+arg(lockgl_ARG, '\0', "lock-gl", string_arg, 0)
 arg(lockstart_ARG, '\0', "lock-start", string_arg, 0)
 arg(lockstop_ARG, '\0', "lock-stop", string_arg, 0)
 arg(locktype_ARG, '\0', "lock-type", string_arg, 0)
diff --git a/tools/dlock.c b/tools/dlock.c
index a962b4c6a..cefa54d7c 100644
--- a/tools/dlock.c
+++ b/tools/dlock.c
@@ -15,3 +15,366 @@
 #include "lvmcache.h"
 #include "lvmlockd-client.h"
 
+static int _mode_num(const char *mode)
+{
+	if (!strcmp(mode, "na"))
+		return -2;
+	if (!strcmp(mode, "un"))
+		return -1;
+	if (!strcmp(mode, "nl"))
+		return 0;
+	if (!strcmp(mode, "sh"))
+		return 1;
+	if (!strcmp(mode, "ex"))
+		return 2;
+	return -3;
+}
+
+/* same rules as strcmp */
+static int _mode_compare(const char *m1, const char *m2)
+{
+	int n1 = _mode_num(m1);
+	int n2 = _mode_num(m2);
+
+	if (n1 < n2)
+		return -1;
+	if (n1 == n2)
+		return 0;
+	if (n1 > n2)
+		return 1;
+	return -2;
+}
+
+/*
+ * Mode is selected by:
+ * 1. mode from command line option (only taken if allow_override is set)
+ * 2. the function arg passed by the calling command (def_mode)
+ * 3. look up a default mode for the command
+ *    (cases where the caller doesn't know a default)
+ *
+ * MODE_NOARG: don't use mode from command line option
+ */
+
+/*
+ * dlock_gl_create() is used by vgcreate to acquire and/or create the
+ * global lock.  vgcreate will have a lock_type for the new vg which
+ * dlock_gl_create() can provide in the lock-gl call.
+ *
+ * dlock_gl() and dlock_gl_create() differ in the specific cases where
+ * ENOLS (no lockspace found) is overriden.  In the vgcreate case, the
+ * override cases are related to sanlock bootstrap, and the lock_type of
+ * the vg being created is needed.
+ *
+ * 1. vgcreate of the first dlock-type vg calls dlock_gl_create()
+ *    to acquire the global lock.
+ *
+ * 2. vgcreate/dlock_gl_create passes gl lock request to lvmlockd,
+ *    along with lock_type of the new vg.
+ *
+ * 3. lvmlockd finds no global lockspace/lock.
+ *
+ * 4. dlm:
+ *    If the lock_type from vgcreate is dlm, lvmlockd creates the
+ *    dlm global lockspace, and queues the global lock request
+ *    for vgcreate.  dlock_gl_create returns sucess with the gl held.
+ *
+ *    sanlock:
+ *    If the lock_type from vgcreate is sanlock, lvmlockd returns -ENOLS
+ *    with the NO_GL_LS flag.  lvmlockd cannot create or acquire a sanlock
+ *    global lock until the VG exists on disk (the locks live within the VG).
+ *
+ *    dlock_gl_create sees sanlock/ENOLS/NO_GL_LS (and optionally the
+ *    "enable" lock-gl arg), determines that this is the sanlock
+ *    bootstrap special case, and returns success without the global lock.
+ *   
+ *    vgcreate creates the VG on disk, and calls lvmlockd_init_vg() which
+ *    initializes/enables a global lock on the new VG's internal sanlock lv.
+ *    Future dlock_gl/dlock_gl_create calls will acquire the existing gl.
+ */
+
+int dlock_gl_create(struct cmd_context *cmd, const char *def_mode, uint32_t flags,
+		    const char *vg_lock_type)
+{
+	const char *mode = NULL;
+	uint32_t result_flags;
+	int result;
+	int ret;
+
+	if (!(flags & DL_GL_MODE_NOARG)) {
+		mode = arg_str_value(cmd, lockgl_ARG, NULL);
+		if (mode && def_mode && strcmp(mode, "enable") &&
+		    (_mode_compare(mode, def_mode) < 0) &&
+		    !find_config_tree_bool(cmd, global_allow_override_lock_modes_CFG, NULL)) {
+			log_error("Disallowed lock-gl mode \"%s\"", mode);
+			return 0;
+		}
+	}
+
+	if (!mode)
+		mode = def_mode;
+
+	if (!mode) {
+		log_error("Unknown lock-gl mode");
+		return 0;
+	}
+
+	if (!strcmp(mode, "ex") && find_config_tree_bool(cmd, global_read_only_lock_modes_CFG, NULL)) {
+		log_error("Disallow lock-gl ex with read_only_lock_modes");
+		return 0;
+	}
+
+	ret = lvmlockd_send(cmd, command_name(cmd), "lock_gl",
+			    NULL, vg_lock_type, NULL, NULL, NULL, mode, "update_names",
+			    &result, &result_flags);
+	if (!ret) {
+		/* no result from lvmlockd */
+		log_error("Locking failed for global lock");
+		return 0;
+	}
+
+	/*
+	 * result and result_flags were returned from lvmlockd.
+	 * In lvmlockd, result 0 is success, and error is < 0.
+	 */
+
+	/*
+	 * No lockspace was found with a global lock.
+	 */
+	if (result == -ENOLS) {
+		/*
+		 * It's not a problem if an unlock happens after the
+		 * lockspace has been removed; ignore it.
+		 */
+		if (!strcmp(mode, "un"))
+			return 1;
+
+		/*
+		 * This is the explicit sanlock bootstrap condition for
+		 * proceding without the global lock: a chicken/egg case
+		 * for the first sanlock VG that is created.
+		 *
+		 * When creating the first sanlock VG, there is no global
+		 * lock to acquire because the gl will exist in the VG
+		 * being created.  The "enable" option makes explicit that
+		 * this is expected:
+		 *
+		 * vgcreate --lock-type sanlock --lock-gl enable
+		 *
+		 * There are three indications that this is the unique
+		 * first-sanlock-vg bootstrap case:
+		 *
+		 * - result from lvmlockd is -ENOLS because lvmlockd found
+		 *   no lockspace for this VG; expected because it's being
+		 *   created here.
+		 *
+		 * - result flag LD_RF_NO_GL_LS from lvmlockd means that
+		 *   lvmlockd has seen no other lockspace with a global lock.
+		 *   This implies that this is probably the first sanlock vg
+		 *   to be created.  If other sanlock vgs exist, the global
+		 *   lock should be available from one of them.
+		 *
+		 * - command line lock-gl arg is "enable" which means the
+		 *   user expects this to be the first sanlock vg, and the
+		 *   global lock should be enabled in it.
+		 */
+
+		if ((result_flags & LD_RF_NO_GL_LS) &&
+		    !strcmp(vg_lock_type, "sanlock") &&
+		    !strcmp(mode, "enable")) {
+			log_debug("Enabling sanlock global lock");
+			lvmetad_validate_global_cache(cmd, 1);
+			return 1;
+		}
+
+		/*
+		 * This is an implicit sanlock bootstrap condition for
+		 * proceeding without the global lock.  The command line does
+		 * not indicate explicitly that this is a bootstrap situation
+		 * (via "enable"), but it seems likely to be because lvmlockd
+		 * has seen no dlock-type vgs.  It is possible that a global
+		 * lock does exist in a vg that has not yet been seen.  If that
+		 * vg appears after this creates a new vg with a new enabled
+		 * gl, then there will be two enabled global locks, and one
+		 * will need to be disabled.  (We could instead return an error
+		 * here and insist with an error message that the --lock-gl
+		 * enable option be used to exercise the explicit case above.)
+		 */
+
+		if ((result_flags & LD_RF_NO_GL_LS) &&
+		    (result_flags & LD_RF_NO_LOCKSPACES) &&
+		    !strcmp(vg_lock_type, "sanlock")) {
+			log_print_unless_silent("Enabling sanlock global lock");
+			lvmetad_validate_global_cache(cmd, 1);
+			return 1;
+		}
+
+		/*
+		 * Allow non-dlock-type vgs to be created even when the global
+		 * lock is not available.  Once created, these vgs will only be
+		 * accessible to the local system_id, and not protected by
+		 * locks, so allowing the creation without a lock is a very
+		 * minor exception to normal locking.
+		 */
+
+		if ((result_flags & LD_RF_NO_GL_LS) &&
+		    (!strcmp(vg_lock_type, "none"))) {
+			lvmetad_validate_global_cache(cmd, 1);
+			return 1;
+		}
+
+		log_error("Global lock %s error %d", mode, result);
+		return 0;
+	}
+
+	if (result < 0) {
+		log_error("Global lock %s error %d", mode, result);
+		return 0;
+	}
+
+	if (!(flags & DL_GL_SKIP_CACHE_VALIDATE))
+		lvmetad_validate_global_cache(cmd, 0);
+
+	return 1;
+}
+
+int dlock_gl(struct cmd_context *cmd, const char *def_mode, uint32_t flags)
+{
+	const char *mode = NULL;
+	const char *opts = NULL;
+	uint32_t result_flags;
+	int result;
+	int ret;
+
+	if (!(flags & DL_GL_MODE_NOARG)) {
+		mode = arg_str_value(cmd, lockgl_ARG, NULL);
+		if (mode && def_mode &&
+		    (_mode_compare(mode, def_mode) < 0) &&
+		    !find_config_tree_bool(cmd, global_allow_override_lock_modes_CFG, NULL)) {
+			log_error("Disallowed lock-gl mode \"%s\"", mode);
+			return 0;
+		}
+	}
+
+	if (!mode)
+		mode = def_mode;
+
+	if (!mode) {
+		log_error("Unknown lock-gl mode");
+		return 0;
+	}
+
+	if (!strcmp(mode, "ex") && find_config_tree_bool(cmd, global_read_only_lock_modes_CFG, NULL)) {
+		log_error("Disallow lock-gl ex with read_only_lock_modes");
+		return 0;
+	}
+
+	/*
+	 * The dlock_gl() caller uses this flag when it is going to change the
+	 * VG namesapce.  lvmlockd uses this to encode extra information in the
+	 * global lock data (a separate version number in the lvb) about what
+	 * was changed.  Other hosts will see this extra information in the gl
+	 * data and know that the VG namespace changed, which determines the
+	 * kind of cache refresh they need to do.
+	 */
+	if (flags & DL_GL_UPDATE_NAMES)
+		opts = "update_names";
+
+	ret = lvmlockd_send(cmd, command_name(cmd), "lock_gl",
+			    NULL, NULL, NULL, NULL, NULL, mode, opts,
+			    &result, &result_flags);
+	if (!ret) {
+		/* No result from lvmlockd, it is probably not running. */
+
+		/*
+		 * These options historically mean that locking is expected
+		 * to fail and the command should be allowed to proceed anyway.
+		 */
+		if (arg_is_set(cmd, sysinit_ARG) || arg_is_set(cmd, ignorelockingfailure_ARG)) {
+			log_debug("Ignore failed locking for global lock: option %s",
+				  arg_is_set(cmd, sysinit_ARG) ? "sysinit" : "ignorelockingfailure");
+			return 1;
+		}
+		
+		/*
+		 * We don't care if an unlock operation fails in this case, and
+		 * we can allow a shared lock request to succeed without any
+		 * serious harm.  To disallow basic reading/reporting when
+		 * lvmlockd is stopped is too strict, unnecessary, and
+		 * inconvenient.  We force a global cache validation in this
+		 * case.
+		 */
+
+		if (!strcmp(mode, "un"))
+			return 1;
+
+		if (!strcmp(mode, "sh")) {
+			log_warn("Reading without shared global lock.");
+			lvmetad_validate_global_cache(cmd, 1);
+			return 1;
+		}
+
+		log_error("Locking failed for global lock");
+		return 0;
+	}
+
+	/*
+	 * result and result_flags were returned from lvmlockd.
+	 * in lvmlockd, result 0 is success, and error is < 0.
+	 */
+
+	if (result == -ENOLS || result == -ESTARTING) {
+
+		/* It doesn't matter if an unlock operation fails. */
+		if (!strcmp(mode, "un"))
+			return 1;
+
+		/*
+		 * This is a general condition for allowing the command to
+		 * proceed without a shared global lock when the global lock is
+		 * not found or ready.  This should not be a persistent
+		 * condition.  The VG containing the global lock should appear
+		 * on the system, or the global lock should be enabled in
+		 * another VG, or the the lockspace with the gl should finish
+		 * starting.
+		 *
+		 * Same reasons as above for allowing the command to proceed
+		 * with the shared gl.  We force a global cache validation and
+		 * print a warning.
+		 */
+
+		if (strcmp(mode, "sh")) {
+			log_error("Global lock %s error %d", mode, result);
+			return 0;
+		}
+
+		if (result == -ESTARTING) {
+			log_warn("Skipping global lock: lockspace is starting");
+			lvmetad_validate_global_cache(cmd, 1);
+			return 1;
+		}
+
+		if ((result_flags & LD_RF_NO_GL_LS) ||
+		    (result_flags & LD_RF_NO_LOCKSPACES)) {
+			log_warn("Skipping global lock: not found");
+			lvmetad_validate_global_cache(cmd, 1);
+			return 1;
+		}
+
+		log_error("Global lock %s error %d", mode, result);
+		return 0;
+	}
+
+	if ((result_flags & LD_RF_DUP_GL_LS) && strcmp(mode, "un"))
+		log_warn("Duplicate sanlock global locks should be corrected");
+
+	if (result < 0) {
+		log_error("Global lock %s error %d", mode, result);
+		return 0;
+	}
+
+	if (!(flags & DL_GL_SKIP_CACHE_VALIDATE))
+		lvmetad_validate_global_cache(cmd, 0);
+
+	return 1;
+}
+
diff --git a/tools/dlock.h b/tools/dlock.h
index 3c852a844..255c90080 100644
--- a/tools/dlock.h
+++ b/tools/dlock.h
@@ -11,4 +11,21 @@
 #ifndef _DLOCK_H
 #define _DLOCK_H
 
+#define DL_GL_MODE_NOARG           0x00000001
+#define DL_GL_SKIP_CACHE_VALIDATE  0x00000002
+#define DL_GL_UPDATE_NAMES         0x00000004
+
+#ifdef LVMLOCKD_SUPPORT
+
+int dlock_gl_create(struct cmd_context *cmd, const char *def_mode, uint32_t flags,
+                    const char *vg_lock_type);
+int dlock_gl(struct cmd_context *cmd, const char *def_mode, uint32_t flags);
+
+#else /* LVMLOCKD_SUPPORT */
+
+#define dlock_gl_create(cmd, def_mode, flags, vg_lock_type) (1)
+#define dlock_gl(cmd, def_mode, flags) (1)
+
+#endif /* LVMLOCKD_SUPPORT */
+
 #endif
diff --git a/tools/lvchange.c b/tools/lvchange.c
index e0bb4b1e1..c2c4256e0 100644
--- a/tools/lvchange.c
+++ b/tools/lvchange.c
@@ -1138,6 +1138,9 @@ int lvchange(struct cmd_context *cmd, int argc, char **argv)
 		}
 	}
 
+	if (arg_tag_count(argc, argv) && !dlock_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	return process_each_lv(cmd, argc, argv,
 			       update ? READ_FOR_UPDATE : 0, NULL,
 			       &_lvchange_single);
diff --git a/tools/lvmcmdline.c b/tools/lvmcmdline.c
index 9f9891fea..07427f941 100644
--- a/tools/lvmcmdline.c
+++ b/tools/lvmcmdline.c
@@ -569,6 +569,20 @@ int string_arg(struct cmd_context *cmd __attribute__((unused)),
 	return 1;
 }
 
+int arg_tag_count(int argc, char **argv)
+{
+	const char *name;
+	int count = 0;
+	int i;
+
+	for (i = 0; i < argc; i++) {
+		name = argv[i];
+		if (*name == '@')
+			count++;
+	}
+	return count;
+}
+
 int tag_arg(struct cmd_context *cmd __attribute__((unused)), struct arg_values *av)
 {
 	char *pos = av->value;
@@ -747,6 +761,7 @@ void lvm_register_commands(void)
 					    yes_ARG, \
 					    quiet_ARG, config_ARG, \
 					    commandprofile_ARG, \
+					    lockgl_ARG, \
 					    profile_ARG, -1);
 #include "commands.h"
 #undef xx
diff --git a/tools/lvscan.c b/tools/lvscan.c
index 2d7be074b..9106ab3dc 100644
--- a/tools/lvscan.c
+++ b/tools/lvscan.c
@@ -98,6 +98,9 @@ int lvscan(struct cmd_context *cmd, int argc, char **argv)
 		return EINVALID_CMD_LINE;
 	}
 
+	if (!dlock_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	return process_each_lv(cmd, argc, argv, 0, NULL,
 			       &lvscan_single);
 }
diff --git a/tools/pvchange.c b/tools/pvchange.c
index f616c0925..466ecfe2d 100644
--- a/tools/pvchange.c
+++ b/tools/pvchange.c
@@ -90,6 +90,10 @@ static int _pvchange_single(struct cmd_context *cmd, struct volume_group *vg,
 		}
 	}
 
+	/* Convert sh to ex.  gl only needed for orphans. */
+	if (is_orphan(pv) && !dlock_gl(cmd, "ex", 0))
+		return_ECMD_FAILED;
+
 	if (tagargs) {
 		/* tag or deltag */
 		if (arg_count(cmd, addtag_ARG) && !change_tag(cmd, NULL, NULL, pv, addtag_ARG))
@@ -176,6 +180,9 @@ int pvchange(struct cmd_context *cmd, int argc, char **argv)
 		return EINVALID_CMD_LINE;
 	}
 
+	if (!dlock_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	params.done = 0;
 	params.total = 0;
 
diff --git a/tools/pvcreate.c b/tools/pvcreate.c
index 139819883..8aa05c163 100644
--- a/tools/pvcreate.c
+++ b/tools/pvcreate.c
@@ -96,6 +96,9 @@ int pvcreate(struct cmd_context *cmd, int argc, char **argv)
 	int ret = ECMD_PROCESSED;
 	struct pvcreate_params pp;
 
+	if (!dlock_gl(cmd, "ex", 0))
+		return_ECMD_FAILED;
+
 	pvcreate_params_set_defaults(&pp);
 
 	if (!pvcreate_restore_params_validate(cmd, argc, argv, &pp)) {
diff --git a/tools/pvdisplay.c b/tools/pvdisplay.c
index d4cb51680..ad2260c8c 100644
--- a/tools/pvdisplay.c
+++ b/tools/pvdisplay.c
@@ -106,6 +106,9 @@ int pvdisplay(struct cmd_context *cmd, int argc, char **argv)
 		}
 	}
 
+	if (!dlock_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	ret = process_each_pv(cmd, argc, argv, NULL, 0, NULL,
 			      _pvdisplay_single);
 
diff --git a/tools/pvremove.c b/tools/pvremove.c
index 035d4fd4e..a2b71f302 100644
--- a/tools/pvremove.c
+++ b/tools/pvremove.c
@@ -30,6 +30,9 @@ int pvremove(struct cmd_context *cmd, int argc, char **argv)
 	force_count = arg_count(cmd, force_ARG);
 	prompt = arg_count(cmd, yes_ARG);
 
+	if (!dlock_gl(cmd, "ex", 0))
+		return_ECMD_FAILED;
+
 	for (i = 0; i < argc; i++) {
 		dm_unescape_colons_and_at_signs(argv[i], NULL, NULL);
 		if (!pvremove_single(cmd, argv[i], NULL, force_count, prompt)) {
diff --git a/tools/pvresize.c b/tools/pvresize.c
index 70b564dc2..66186f9b5 100644
--- a/tools/pvresize.c
+++ b/tools/pvresize.c
@@ -32,6 +32,10 @@ static int _pvresize_single(struct cmd_context *cmd,
 
 	params->total++;
 
+	/* Convert sh to ex.  gl is only needed for orphans. */
+	if (is_orphan(pv) && !dlock_gl(cmd, "ex", 0))
+		return_ECMD_FAILED;
+
 	if (!pv_resize_single(cmd, vg, pv, params->new_size))
 		return_ECMD_FAILED;
 
@@ -61,6 +65,9 @@ int pvresize(struct cmd_context *cmd, int argc, char **argv)
 	params.done = 0;
 	params.total = 0;
 
+	if (!dlock_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	ret = process_each_pv(cmd, argc, argv, NULL, READ_FOR_UPDATE, &params,
 			      _pvresize_single);
 
diff --git a/tools/pvscan.c b/tools/pvscan.c
index 3eaf0bbcb..0de8cd3dc 100644
--- a/tools/pvscan.c
+++ b/tools/pvscan.c
@@ -224,6 +224,9 @@ static int _pvscan_lvmetad(struct cmd_context *cmd, int argc, char **argv)
 		return ECMD_FAILED;
 	}
 
+	if (!dlock_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	/* Scan everything? */
 	if (!argc && !devno_args) {
 		if (!lvmetad_pvscan_all_devs(cmd, handler))
diff --git a/tools/reporter.c b/tools/reporter.c
index a3b860676..4ba9ac261 100644
--- a/tools/reporter.c
+++ b/tools/reporter.c
@@ -269,6 +269,9 @@ static int _report(struct cmd_context *cmd, int argc, char **argv,
 	if (args_are_pvs && argc)
 		cmd->filter->wipe(cmd->filter);
 
+	if ((!argc || arg_tag_count(argc, argv)) && !dlock_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	switch (report_type) {
 	case DEVTYPES:
 		keys = find_config_tree_str(cmd, report_devtypes_sort_CFG, NULL);
diff --git a/tools/tools.h b/tools/tools.h
index 11a88afe0..0f0495e0c 100644
--- a/tools/tools.h
+++ b/tools/tools.h
@@ -174,6 +174,8 @@ unsigned grouped_arg_is_set(const struct arg_values *av, int a);
 const char *grouped_arg_str_value(const struct arg_values *av, int a, const char *def);
 int32_t grouped_arg_int_value(const struct arg_values *av, int a, const int32_t def); 
 
+int arg_tag_count(int argc, char **argv);
+
 const char *command_name(struct cmd_context *cmd);
 
 int pvmove_poll(struct cmd_context *cmd, const char *pv, unsigned background);
diff --git a/tools/vgchange.c b/tools/vgchange.c
index f3e7aaa5a..80d91c647 100644
--- a/tools/vgchange.c
+++ b/tools/vgchange.c
@@ -665,6 +665,36 @@ static int vgchange_single(struct cmd_context *cmd, const char *vg_name,
         return ret;
 }
 
+static int dlock_vgchange(struct cmd_context *cmd, int argc, char **argv)
+{
+	if (!argc || arg_tag_count(argc, argv) || arg_is_set(cmd, lockstart_ARG)) {
+		/*
+		 * The first two standard conditions want the current
+		 * list of all vg names.  The lockstart condition takes
+		 * the gl to serialize with any other host that tries to
+		 * remove the VG while this tries to start it.
+		 */
+		if (!dlock_gl(cmd, "sh", 0))
+			return_ECMD_FAILED;
+
+	} else if (arg_is_set(cmd, systemid_ARG) ||
+		   arg_is_set(cmd, uuid_ARG) ||
+		   arg_is_set(cmd, locktype_ARG)) {
+		/*
+		 * VG names, uuids and system_ids are the three things that
+		 * other hosts cache related to local vg's, so we use the
+		 * name-change counter in the global lock to indicate that
+		 * one of these global VG identifiers has changed so other
+		 * hosts will update these cached values in VG's that they
+		 * otherwise ignore (because they have foreign system_ids).
+		 */
+		if (!dlock_gl(cmd, "ex", DL_GL_UPDATE_NAMES))
+			return_ECMD_FAILED;
+	}
+
+	return 1;
+}
+
 int vgchange(struct cmd_context *cmd, int argc, char **argv)
 {
 	int noupdate =
@@ -787,6 +817,9 @@ int vgchange(struct cmd_context *cmd, int argc, char **argv)
 	if (!update || !update_partial_unsafe)
 		cmd->handles_missing_pvs = 1;
 
+	if (!dlock_vgchange(cmd, argc, argv))
+		return_ECMD_FAILED;
+
 	return process_each_vg(cmd, argc, argv, update ? READ_FOR_UPDATE : 0,
 			       NULL, &vgchange_single);
 }
diff --git a/tools/vgcreate.c b/tools/vgcreate.c
index 5afffd857..1e9bf9629 100644
--- a/tools/vgcreate.c
+++ b/tools/vgcreate.c
@@ -50,6 +50,9 @@ int vgcreate(struct cmd_context *cmd, int argc, char **argv)
 	if (!vgcreate_params_validate(cmd, &vp_new))
 	    return EINVALID_CMD_LINE;
 
+	if (!dlock_gl_create(cmd, "ex", 0, vp_new.lock_type))
+		return ECMD_FAILED;
+
 	lvmcache_seed_infos_from_lvmetad(cmd);
 
 	/* Create the new VG */
diff --git a/tools/vgextend.c b/tools/vgextend.c
index 716a2ceb6..4031ea026 100644
--- a/tools/vgextend.c
+++ b/tools/vgextend.c
@@ -156,6 +156,9 @@ int vgextend(struct cmd_context *cmd, int argc, char **argv)
 	 */
 	cmd->handles_missing_pvs = 1;
 
+	if (!dlock_gl(cmd, "ex", 0))
+		return_ECMD_FAILED;
+
 	return process_each_vg(cmd, argc, argv,
 			       READ_FOR_UPDATE | ONLY_FIRST_NAME, &vp,
 			       restore ? &vgextend_restore : &vgextend_single);
diff --git a/tools/vgmerge.c b/tools/vgmerge.c
index a17a636c5..41658af9a 100644
--- a/tools/vgmerge.c
+++ b/tools/vgmerge.c
@@ -194,6 +194,9 @@ int vgmerge(struct cmd_context *cmd, int argc, char **argv)
 		return EINVALID_CMD_LINE;
 	}
 
+	if (!dlock_gl(cmd, "ex", DL_GL_UPDATE_NAMES))
+		return ECMD_FAILED;
+
 	vg_name_to = skip_dev_dir(cmd, argv[0], NULL);
 	argc--;
 	argv++;
diff --git a/tools/vgreduce.c b/tools/vgreduce.c
index 7af5a7683..95d0cca50 100644
--- a/tools/vgreduce.c
+++ b/tools/vgreduce.c
@@ -195,6 +195,9 @@ int vgreduce(struct cmd_context *cmd, int argc, char **argv)
 	init_ignore_suspended_devices(1);
 	cmd->handles_missing_pvs = 1;
 
+	if (!dlock_gl(cmd, "ex", 0))
+		return_ECMD_FAILED;
+
 	vg = vg_read_for_update(cmd, vg_name, NULL, READ_ALLOW_EXPORTED);
 	if (vg_read_error(vg) == FAILED_ALLOCATION ||
 	    vg_read_error(vg) == FAILED_NOTFOUND)
diff --git a/tools/vgremove.c b/tools/vgremove.c
index 98336362d..6c31c9a3e 100644
--- a/tools/vgremove.c
+++ b/tools/vgremove.c
@@ -79,6 +79,9 @@ int vgremove(struct cmd_context *cmd, int argc, char **argv)
 		return EINVALID_CMD_LINE;
 	}
 
+	if (!dlock_gl(cmd, "ex", DL_GL_UPDATE_NAMES))
+		return ECMD_FAILED;
+
 	cmd->handles_missing_pvs = 1;
 	ret = process_each_vg(cmd, argc, argv,
 			      READ_FOR_UPDATE,
diff --git a/tools/vgrename.c b/tools/vgrename.c
index 860ccf196..8dc413920 100644
--- a/tools/vgrename.c
+++ b/tools/vgrename.c
@@ -207,6 +207,9 @@ int vgrename(struct cmd_context *cmd, int argc, char **argv)
 		return EINVALID_CMD_LINE;
 	}
 
+	if (!dlock_gl(cmd, "ex", DL_GL_UPDATE_NAMES))
+		return_ECMD_FAILED;
+
 	if (!vg_rename_path(cmd, argv[0], argv[1]))
 		return_ECMD_FAILED;
 
diff --git a/tools/vgscan.c b/tools/vgscan.c
index bca98cc6e..f30850dc1 100644
--- a/tools/vgscan.c
+++ b/tools/vgscan.c
@@ -42,6 +42,9 @@ int vgscan(struct cmd_context *cmd, int argc, char **argv)
 		return ECMD_FAILED;
 	}
 
+	if (!dlock_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	if (cmd->filter->wipe)
 		cmd->filter->wipe(cmd->filter);
 	lvmcache_destroy(cmd, 1, 0);
diff --git a/tools/vgsplit.c b/tools/vgsplit.c
index 362f85410..339bb52aa 100644
--- a/tools/vgsplit.c
+++ b/tools/vgsplit.c
@@ -492,6 +492,9 @@ int vgsplit(struct cmd_context *cmd, int argc, char **argv)
 		return ECMD_FAILED;
 	}
 
+	if (!dlock_gl(cmd, "ex", DL_GL_UPDATE_NAMES))
+		return_ECMD_FAILED;
+
 	if (arg_count(cmd, name_ARG))
 		lv_name = arg_value(cmd, name_ARG);
 	else