lvmlockd: Global locking

The global lock (gl) is a simple read/write lock that protects
global lvm metadata, i.e. metadata or information that is not
isolated to a single VG.  This global information includes:

A) The VG name space.
B) PVs or devices that do not belong to a VG.

The function to acquire/release the global lock is lockd_gl():

. To acquire the gl in exclusive mode for writing/changing A or B:
  lockd_gl(cmd, "ex", 0);

. To acquire the gl in shared mode for reading/listing A or B:
  lockd_gl(cmd, "sh", 0);

. To release the gl:
  lockd_gl(cmd, "un", 0);

The lockd_gl() function sends a message to lvmlockd, asking for
the lock in the specified mode.  lvmlockd acquires the lock from
the underlying lock manager, and sends the result back to the command.
When the command exits, or calls lockd_gl("un"), lvmlockd releases
the lock in the underlying lock manager.

When lvm is compiled without lvmlockd support, all the lockd_gl() calls
simply compile to success (1).

Using the global lock in commands is simple:

. A command that wants to get an accurate list of all VG names
  should acquire the gl in the sh mode, then read the list.

. A command that wants to add or remove a VG name should acquire
  the gl in the ex mode, then add/remove the name.

. A command that wants to get an accurate list of all PVs/devices
  should acquire the gl in the sh mode, then read the list.

. A command that wants to change a device to/from a PV or add/remove
  a PV to/from a VG should acquire the gl in the ex mode, then make
  the change.

. A command that wants to read the properties of an orphan PV
  should acquire the gl in the sh mode, then read the properties.

. A command that wants to change the properties of an orphan PV
  should acquire the gl in the ex mode, then change the properties.

. The gl is acquired at the start of a command before any processing.
  This is necessary so that the cached information used by the command
  is valid and up to date (see caching below).

. A command generally knows at the outset which of the things above
  it is going to do, so it knows which lock mode to acquire.

. If a command is given a tag, the tag matching requires a complete
  and accurate search of all VGs, and therefore implies that the global
  shared lock is needed.

Locking conflicts:

When a command calls lockd_gl(), the lock request is passed to lvmlockd.
lvmlockd makes the corresponding lock request in the lock manager using a
non-blocking request.  If another command on another host holds the gl in
a conflicting mode, the lock request fails, and lvmlockd returns the
failure to the command.  The command reports the lock conflict and fails.

If a reporting command (sh lock), conflicts with another command using an
ex lock (like vgextend), then the reporting command can simply be rerun.
A future option may enable retrying sh lock requests within lvmlockd,
making simple, incidental conflicts invisible.  (These retries could be
disabled or limited to a certain number via a command or config option.)

If a command is changing global state (using an ex lock), the conflict
could be with another sh lock (e.g. reporting command) or another ex lock
(another command changing global state.)  The lock manager does not say
which type of conflict it was.  In the case of ex/sh conflict, a retry of
the ex request could be automatic, but an ex/ex conflict would generally
want inspection before retrying.  Uncoordinated commands concurrently
changing the same global state would be uncommon, and a warning of the
conflict with failure is probably preferred, so the state can be
inspected.  Still, the same retry options as above could be applied if
needed.

Caching:

In addition to ex/sh mutual exclusion, the lock manager allows an
application to stash a chunk of its own data within a lock.  This chunk of
data is called the "lock value block" (lvb).  This opaque data does not
affect the locking behavior, but offers a convenient way for applications
to pass around extra information related to the lock.  The lvb can be read
and written by any host using the lock (the application can read the lvb
content when acquiring the lock, and write it when releasing an ex lock.)
(The dlm lvb is only 32 bytes.  The sanlock lvb is up to 512 bytes.)

An application can use the lvb for anything it wishes, and it's often used
to help manage the cache associated with the lock.  lvmlockd stores a
counter/version number in the lvb.  It's incremented when anything
protected by the gl (A or B above) is changed.  When other hosts see the
version number has increased, they know that A or B have changed.
lvmlockd refines this further by using a second version number that is
incremented only when the VG namespace (A) changes.  The lockd_gl() flag
UPDATE_NAMES tells lvmlockd that the VG namespace is being changed.

When lvmlockd acquires the gl and sees that the counters in the lvb have
been incremented, it knows that the objects protected by the gl have been
changed by another host.  This implies that the local host's cache of this
global information is likely out of date, and needs to be refreshed from
disk.  When this happens, lvmlockd sends a message to lvmetad to
invalidate the cached global information in lvmetad.  When a command sees
that the data from lvmetad is stale, it reads from disk and updates
lvmetad.

All users of the global lock want to use valid information from lvmetad,
so they always check that the lvmetad cache is valid before using it, and
refresh it if needed.  This check and refresh is done by
lvmetad_validate_global_cache().  Instead of always calling
lockd_gl()+lvmetad_validate_global_cache() back to back, lockd_gl() calls
lvmetad_validate_global_cache() as the final step before returning.

In the future, more optimizations can be made related to global cache
updating.  Similar to the UPDATE_NAMES method, commands can tell lvmlockd
more details about what they are chaning under the global lock.  lvmlockd
can propagate these details to others using the gl lvb.  Knowing these
details, other hosts can limit their rescanning to only what's necessary
given the specific changes.

vgcreate with sanlock:

With the sanlock lock_type, the sanlock locks are stored on disk, on a
hidden LV within the VG.  The first sanlock VG created will hold the
global lock.  Creating the first sanlock VG is a special case because no
global lock will exist until after the VG is created on disk.  vgcreate
calls lockd_gl_create() to handle this special case.  The comments in that
function explain the details of how it works.

command gl requirements:

Some commands are listed twice if they have two different
behaviors (depending on args) that need different gl usage.
As listed above (in A,B), the reasons for using the gl are:

A) reading or changing the VG name space
B) reading or changing PV orphans
   (orphan properties or assignment to VGs)

command:   gl mode used, reason(s) gl is needed

vgsplit:   ex, add vg name
vgrename:  ex, add vg name
vgcreate:  ex, add vg name, rem pv orphan
vgremove:  ex, rem vg name, add pv orphan
vgmerge:   ex, rem vg name
vgextend:  ex, rem pv orphan
vgreduce:  ex, add pv orphan
vgscan:    sh, get vg names
vgs:       sh, get vg names (only if tags used or no args)
vgchange:  sh, get vg names (only if tags used or no args)
vgchange:  ex, change vg system_id/uuid/lock_type (equivalent to name)

pvcreate:  ex, add pv orphan
pvremove:  ex, rem pv orphan
pvdisplay: sh, get vg names
pvscan:    sh, get vg names
pvresize:  sh, get vg names
pvresize:  ex, change pv orphan (only if pv is an orphan)
pvchange:  sh, get vg names
pvchange:  ex, change pv orphan (only if pv is an orphan)

lvchange:  sh, get vg names (only if tags used)
lvscan:    sh, get vg names

26 files changed, 526 insertions, 3 deletions

diff --git a/daemons/lvmlockd/lvmlockd-client.h b/daemons/lvmlockd/lvmlockd-client.h
index bde0191ea..dcae38468 100644
--- a/daemons/lvmlockd/lvmlockd-client.h
+++ b/daemons/lvmlockd/lvmlockd-client.h
@@ -34,6 +34,14 @@ static inline void lvmlockd_close(daemon_handle h)
 }
 
 /*
+ * Errors returned as the lvmlockd result value.
+ */
+#define ENOLS     210 /* lockspace not found */
+#define ELOCALVG  211 /* vg is local */
+#define EOTHERVG  212 /* vg sysid specifies other host */
+#define ESTARTING 213 /* lockspace is starting */
+
+/*
  * Also see lvmlockd-sanlock GL_LOCK_BEGIN, VG_LOCK_BEGIN, LV_LOCK_BEGIN.
  * gl lock at sanlock lease area 65
  * vg lock at sanlock lease area 66
diff --git a/lib/commands/toolcontext.h b/lib/commands/toolcontext.h
index a3aaf7c28..6fc95e711 100644
--- a/lib/commands/toolcontext.h
+++ b/lib/commands/toolcontext.h
@@ -140,6 +140,9 @@ struct cmd_context {
 	const char *report_list_item_separator;
 	int hosttags;
 
+	/* Locking */
+	const char *lock_gl_mode; /* gl mode, from --lock-gl */
+
 	const char *lib_dir;		/* Cache value global/library_dir */
 	char system_dir[PATH_MAX];
 	char dev_dir[PATH_MAX];
diff --git a/lib/config/config_settings.h b/lib/config/config_settings.h
index 9c3274d8e..3cc230eab 100644
--- a/lib/config/config_settings.h
+++ b/lib/config/config_settings.h
@@ -194,6 +194,8 @@ cfg(global_cache_repair_executable_CFG, "cache_repair_executable", global_CFG_SE
 cfg_array(global_cache_repair_options_CFG, "cache_repair_options", global_CFG_SECTION, 0, CFG_TYPE_STRING, "#S" DEFAULT_CACHE_REPAIR_OPTIONS, vsn(2, 2, 108), NULL)
 cfg(global_system_id_source_CFG, "system_id_source", global_CFG_SECTION, CFG_DEFAULT_UNDEFINED, CFG_TYPE_STRING, NULL, vsn(2, 2, 112), NULL)
 cfg(global_system_id_file_CFG, "system_id_file", global_CFG_SECTION, CFG_DEFAULT_UNDEFINED, CFG_TYPE_STRING, NULL, vsn(2, 2, 112), NULL)
+cfg(global_allow_override_lock_modes_CFG, "allow_override_lock_modes", global_CFG_SECTION, 0, CFG_TYPE_BOOL, 0, vsn(2, 2, 112), NULL)
+cfg(global_read_only_lock_modes_CFG, "read_only_lock_modes", global_CFG_SECTION, 0, CFG_TYPE_BOOL, 0, vsn(2, 2, 112), NULL)
 
 cfg(activation_checks_CFG, "checks", activation_CFG_SECTION, 0, CFG_TYPE_BOOL, DEFAULT_ACTIVATION_CHECKS, vsn(2, 2, 86), NULL)
 cfg(activation_udev_sync_CFG, "udev_sync", activation_CFG_SECTION, 0, CFG_TYPE_BOOL, DEFAULT_UDEV_SYNC, vsn(2, 2, 51), NULL)
diff --git a/lib/locking/lvmlockd.c b/lib/locking/lvmlockd.c
index c4650bfeb..a1e6928d6 100644
--- a/lib/locking/lvmlockd.c
+++ b/lib/locking/lvmlockd.c
@@ -105,8 +105,26 @@ void lvmlockd_set_socket(const char *sock)
 	_lvmlockd_socket = sock;
 }
 
+/* Translate the result strings from lvmlockd to bit flags. */
 static void _result_str_to_flags(const char *str, uint32_t *flags)
 {
+	if (strstr(str, "NO_LOCKSPACES"))
+		*flags |= LD_RF_NO_LOCKSPACES;
+
+	if (strstr(str, "NO_GL_LS"))
+		*flags |= LD_RF_NO_GL_LS;
+
+	if (strstr(str, "LOCAL_LS"))
+		*flags |= LD_RF_LOCAL_LS;
+
+	if (strstr(str, "DUP_GL_LS"))
+		*flags |= LD_RF_DUP_GL_LS;
+
+	if (strstr(str, "INACTIVE_LS"))
+		*flags |= LD_RF_INACTIVE_LS;
+
+	if (strstr(str, "ADD_LS_ERROR"))
+		*flags |= LD_RF_ADD_LS_ERROR;
 }
 
 /*
@@ -197,7 +215,6 @@ static daemon_reply _lockd_send(const char *req_name, ...)
  */
 
 static int _lockd_request(struct cmd_context *cmd,
-		          const char *cmd_name,
 		          const char *req_name,
 		          const char *vg_name,
 		          const char *vg_lock_type,
@@ -209,6 +226,7 @@ static int _lockd_request(struct cmd_context *cmd,
 		          int *result,
 		          uint32_t *result_flags)
 {
+	const char *cmd_name = "unknown"; /* FIXME: setting this would help debugging */
 	daemon_reply reply;
 	int pid = getpid();
 
@@ -530,8 +548,8 @@ static int _free_vg_dlm(struct cmd_context *cmd, struct volume_group *vg)
 	 */
 
 	/* Equivalent to a standard unlock. */
-	ret = _lockd_request(cmd, "vgremove", "lock_vg", vg->name,
-			     NULL, NULL, NULL, NULL, "un", NULL,
+	ret = _lockd_request(cmd, "lock_vg",
+			     vg->name, NULL, NULL, NULL, NULL, "un", NULL,
 			     &result, &result_flags);
 
 	if (!ret || result < 0) {
@@ -858,3 +876,361 @@ out:
 	return ret;
 }
 
+static int _mode_num(const char *mode)
+{
+	if (!strcmp(mode, "na"))
+		return -2;
+	if (!strcmp(mode, "un"))
+		return -1;
+	if (!strcmp(mode, "nl"))
+		return 0;
+	if (!strcmp(mode, "sh"))
+		return 1;
+	if (!strcmp(mode, "ex"))
+		return 2;
+	return -3;
+}
+
+/* same rules as strcmp */
+static int _mode_compare(const char *m1, const char *m2)
+{
+	int n1 = _mode_num(m1);
+	int n2 = _mode_num(m2);
+
+	if (n1 < n2)
+		return -1;
+	if (n1 == n2)
+		return 0;
+	if (n1 > n2)
+		return 1;
+	return -2;
+}
+
+/*
+ * Mode is selected by:
+ * 1. mode from command line option (only taken if allow_override is set)
+ * 2. the function arg passed by the calling command (def_mode)
+ * 3. look up a default mode for the command
+ *    (cases where the caller doesn't know a default)
+ *
+ * MODE_NOARG: don't use mode from command line option
+ */
+
+/*
+ * lockd_gl_create() is used by vgcreate to acquire and/or create the
+ * global lock.  vgcreate will have a lock_type for the new vg which
+ * lockd_gl_create() can provide in the lock-gl call.
+ *
+ * lockd_gl() and lockd_gl_create() differ in the specific cases where
+ * ENOLS (no lockspace found) is overriden.  In the vgcreate case, the
+ * override cases are related to sanlock bootstrap, and the lock_type of
+ * the vg being created is needed.
+ *
+ * 1. vgcreate of the first lockd-type vg calls lockd_gl_create()
+ *    to acquire the global lock.
+ *
+ * 2. vgcreate/lockd_gl_create passes gl lock request to lvmlockd,
+ *    along with lock_type of the new vg.
+ *
+ * 3. lvmlockd finds no global lockspace/lock.
+ *
+ * 4. dlm:
+ *    If the lock_type from vgcreate is dlm, lvmlockd creates the
+ *    dlm global lockspace, and queues the global lock request
+ *    for vgcreate.  lockd_gl_create returns sucess with the gl held.
+ *
+ *    sanlock:
+ *    If the lock_type from vgcreate is sanlock, lvmlockd returns -ENOLS
+ *    with the NO_GL_LS flag.  lvmlockd cannot create or acquire a sanlock
+ *    global lock until the VG exists on disk (the locks live within the VG).
+ *
+ *    lockd_gl_create sees sanlock/ENOLS/NO_GL_LS (and optionally the
+ *    "enable" lock-gl arg), determines that this is the sanlock
+ *    bootstrap special case, and returns success without the global lock.
+ *   
+ *    vgcreate creates the VG on disk, and calls lockd_init_vg() which
+ *    initializes/enables a global lock on the new VG's internal sanlock lv.
+ *    Future lockd_gl/lockd_gl_create calls will acquire the existing gl.
+ */
+
+int lockd_gl_create(struct cmd_context *cmd, const char *def_mode, const char *vg_lock_type)
+{
+	const char *mode = NULL;
+	uint32_t result_flags;
+	int result;
+
+	if (cmd->lock_gl_mode) {
+		mode = cmd->lock_gl_mode;
+		if (mode && def_mode && strcmp(mode, "enable") &&
+		    (_mode_compare(mode, def_mode) < 0) &&
+		    !find_config_tree_bool(cmd, global_allow_override_lock_modes_CFG, NULL)) {
+			log_error("Disallowed lock-gl mode \"%s\"", mode);
+			return 0;
+		}
+	}
+
+	if (!mode)
+		mode = def_mode;
+	if (!mode) {
+		log_error("Unknown lock-gl mode");
+		return 0;
+	}
+
+	if (!strcmp(mode, "ex") && find_config_tree_bool(cmd, global_read_only_lock_modes_CFG, NULL)) {
+		log_error("Disallow lock-gl ex with read_only_lock_modes");
+		return 0;
+	}
+
+	if (!_lockd_request(cmd, "lock_gl",
+			      NULL, vg_lock_type, NULL, NULL, NULL, mode, "update_names",
+			      &result, &result_flags)) {
+		/* No result from lvmlockd, it is probably not running. */
+		log_error("Locking failed for global lock");
+		return 0;
+	}
+
+	/*
+	 * result and result_flags were returned from lvmlockd.
+	 *
+	 * ENOLS: no lockspace was found with a global lock.
+	 * It may not exist (perhaps this command is creating the first),
+	 * or it may not be visible or started on the system yet.
+	 */
+
+	if (result == -ENOLS) {
+		if (!strcmp(mode, "un"))
+			return 1;
+
+		/*
+		 * This is the explicit sanlock bootstrap condition for
+		 * proceding without the global lock: a chicken/egg case
+		 * for the first sanlock VG that is created.
+		 *
+		 * When creating the first sanlock VG, there is no global
+		 * lock to acquire because the gl will exist in the VG
+		 * being created.  The "enable" option makes explicit that
+		 * this is expected:
+		 *
+		 * vgcreate --lock-type sanlock --lock-gl enable
+		 *
+		 * There are three indications that this is the unique
+		 * first-sanlock-vg bootstrap case:
+		 *
+		 * - result from lvmlockd is -ENOLS because lvmlockd found
+		 *   no lockspace for this VG; expected because it's being
+		 *   created here.
+		 *
+		 * - result flag LD_RF_NO_GL_LS from lvmlockd means that
+		 *   lvmlockd has seen no other lockspace with a global lock.
+		 *   This implies that this is probably the first sanlock vg
+		 *   to be created.  If other sanlock vgs exist, the global
+		 *   lock should be available from one of them.
+		 *
+		 * - command line lock-gl arg is "enable" which means the
+		 *   user expects this to be the first sanlock vg, and the
+		 *   global lock should be enabled in it.
+		 */
+
+		if ((result_flags & LD_RF_NO_GL_LS) &&
+		    !strcmp(vg_lock_type, "sanlock") &&
+		    !strcmp(mode, "enable")) {
+			log_debug("Enabling sanlock global lock");
+			lvmetad_validate_global_cache(cmd, 1);
+			return 1;
+		}
+
+		/*
+		 * This is an implicit sanlock bootstrap condition for
+		 * proceeding without the global lock.  The command line does
+		 * not indicate explicitly that this is a bootstrap situation
+		 * (via "enable"), but it seems likely to be because lvmlockd
+		 * has seen no lockd-type vgs.  It is possible that a global
+		 * lock does exist in a vg that has not yet been seen.  If that
+		 * vg appears after this creates a new vg with a new enabled
+		 * gl, then there will be two enabled global locks, and one
+		 * will need to be disabled.  (We could instead return an error
+		 * here and insist with an error message that the --lock-gl
+		 * enable option be used to exercise the explicit case above.)
+		 */
+
+		if ((result_flags & LD_RF_NO_GL_LS) &&
+		    (result_flags & LD_RF_NO_LOCKSPACES) &&
+		    !strcmp(vg_lock_type, "sanlock")) {
+			log_print_unless_silent("Enabling sanlock global lock");
+			lvmetad_validate_global_cache(cmd, 1);
+			return 1;
+		}
+
+		/*
+		 * Allow non-lockd-type vgs to be created even when the global
+		 * lock is not available.  Once created, these vgs will only be
+		 * accessible to the local system_id, and not protected by
+		 * locks, so allowing the creation without a lock is a very
+		 * minor exception to normal locking.
+		 */
+
+		if ((result_flags & LD_RF_NO_GL_LS) &&
+		    (!strcmp(vg_lock_type, "none"))) {
+			lvmetad_validate_global_cache(cmd, 1);
+			return 1;
+		}
+
+		log_error("Global lock %s error %d", mode, result);
+		return 0;
+	}
+
+	if (result < 0) {
+		if (result == -ESTARTING)
+			log_error("Global lock %s error: lockspace is starting", mode);
+		else
+			log_error("Global lock %s error %d", mode, result);
+		return 0;
+	}
+
+	lvmetad_validate_global_cache(cmd, 1);
+
+	return 1;
+}
+
+int lockd_gl(struct cmd_context *cmd, const char *def_mode, uint32_t flags)
+{
+	const char *mode = NULL;
+	const char *opts = NULL;
+	uint32_t result_flags;
+	int result;
+
+	if (!(flags & LDGL_MODE_NOARG) && cmd->lock_gl_mode) {
+		mode = cmd->lock_gl_mode;
+		if (mode && def_mode &&
+		    (_mode_compare(mode, def_mode) < 0) &&
+		    !find_config_tree_bool(cmd, global_allow_override_lock_modes_CFG, NULL)) {
+			log_error("Disallowed lock-gl mode \"%s\"", mode);
+			return 0;
+		}
+	}
+
+	if (!mode)
+		mode = def_mode;
+	if (!mode) {
+		log_error("Unknown lock-gl mode");
+		return 0;
+	}
+
+	if (!strcmp(mode, "ex") && find_config_tree_bool(cmd, global_read_only_lock_modes_CFG, NULL)) {
+		log_error("Disallow lock-gl ex with read_only_lock_modes");
+		return 0;
+	}
+
+	/*
+	 * The lockd_gl() caller uses this flag when it is going to change the
+	 * VG namesapce.  lvmlockd uses this to encode extra information in the
+	 * global lock data (a separate version number in the lvb) about what
+	 * was changed.  Other hosts will see this extra information in the gl
+	 * data and know that the VG namespace changed, which determines the
+	 * kind of cache refresh they need to do.
+	 */
+	if (flags & LDGL_UPDATE_NAMES)
+		opts = "update_names";
+
+	if (!_lockd_request(cmd, "lock_gl",
+			    NULL, NULL, NULL, NULL, NULL, mode, opts,
+			    &result, &result_flags)) {
+		/* No result from lvmlockd, it is probably not running. */
+
+		/*
+		 * We don't care if an unlock operation fails in this case, and
+		 * we can allow a shared lock request to succeed without any
+		 * serious harm.  To disallow basic reading/reporting when
+		 * lvmlockd is stopped is too strict, unnecessary, and
+		 * inconvenient.  We force a global cache validation in this
+		 * case.
+		 */
+
+		if (!strcmp(mode, "un"))
+			return 1;
+
+		if (!strcmp(mode, "sh")) {
+			log_warn("Reading without shared global lock.");
+			lvmetad_validate_global_cache(cmd, 1);
+			return 1;
+		}
+
+		log_error("Locking failed for global lock");
+		return 0;
+	}
+
+	/*
+	 * result and result_flags were returned from lvmlockd.
+	 *
+	 * ENOLS: no lockspace was found with a global lock.
+	 * The VG with the global lock may not be visible or started yet,
+	 * this should be a temporary condition.
+	 *
+	 * ESTARTING: the lockspace with the gl is starting.
+	 * The VG with the global lock is starting and should finish shortly.
+	 */
+
+	if (result == -ENOLS || result == -ESTARTING) {
+		if (!strcmp(mode, "un"))
+			return 1;
+
+		/*
+		 * This is a general condition for allowing the command to
+		 * proceed without a shared global lock when the global lock is
+		 * not found or ready.  This should not be a persistent
+		 * condition.  The VG containing the global lock should appear
+		 * on the system, or the global lock should be enabled in
+		 * another VG, or the the lockspace with the gl should finish
+		 * starting.
+		 *
+		 * Same reasons as above for allowing the command to proceed
+		 * with the shared gl.  We force a global cache validation and
+		 * print a warning.
+		 */
+
+		if (strcmp(mode, "sh")) {
+			if (result == -ESTARTING)
+				log_error("Global lock %s error: lockspace is starting", mode);
+			else
+				log_error("Global lock %s error %d", mode, result);
+			return 0;
+		}
+
+		if (result == -ESTARTING) {
+			log_warn("Skipping global lock: lockspace is starting");
+			lvmetad_validate_global_cache(cmd, 1);
+			return 1;
+		}
+
+		if ((result_flags & LD_RF_NO_GL_LS) ||
+		    (result_flags & LD_RF_NO_LOCKSPACES)) {
+			log_warn("Skipping global lock: not found");
+			lvmetad_validate_global_cache(cmd, 1);
+			return 1;
+		}
+
+		log_error("Global lock %s error %d", mode, result);
+		return 0;
+	}
+
+	if ((result_flags & LD_RF_DUP_GL_LS) && strcmp(mode, "un"))
+		log_warn("Duplicate sanlock global locks should be corrected");
+
+	if (result < 0) {
+		if (ignorelockingfailure()) {
+			log_debug("Ignore failed locking for global lock");
+			lvmetad_validate_global_cache(cmd, 1);
+			return 1;
+		} else {
+			log_error("Global lock %s error %d", mode, result);
+			return 0;
+		}
+	}
+
+	if (!(flags & LDGL_SKIP_CACHE_VALIDATE))
+		lvmetad_validate_global_cache(cmd, 0);
+
+	return 1;
+}
+
+
diff --git a/lib/locking/lvmlockd.h b/lib/locking/lvmlockd.h
index e35ec0d1e..86b66cf82 100644
--- a/lib/locking/lvmlockd.h
+++ b/lib/locking/lvmlockd.h
@@ -19,6 +19,19 @@
 #define LOCK_TYPE_DLM     2
 #define LOCK_TYPE_SANLOCK 3
 
+/* lockd_gl flags */
+#define LDGL_MODE_NOARG           0x00000001
+#define LDGL_SKIP_CACHE_VALIDATE  0x00000002
+#define LDGL_UPDATE_NAMES         0x00000004
+
+/* lvmlockd result flags */
+#define LD_RF_NO_LOCKSPACES     0x00000001
+#define LD_RF_NO_GL_LS          0x00000002
+#define LD_RF_LOCAL_LS          0x00000004
+#define LD_RF_DUP_GL_LS         0x00000008
+#define LD_RF_INACTIVE_LS       0x00000010
+#define LD_RF_ADD_LS_ERROR      0x00000020
+
 /*
  * lock_type    lock_type_num
  * "none"    -> LOCK_TYPE_NONE
@@ -83,6 +96,11 @@ void lockd_free_vg_final(struct cmd_context *cmd, struct volume_group *vg);
 int lockd_start_vg(struct cmd_context *cmd, struct volume_group *vg);
 int lockd_stop_vg(struct cmd_context *cmd, struct volume_group *vg);
 
+/* locking */
+
+int lockd_gl_create(struct cmd_context *cmd, const char *def_mode, const char *vg_lock_type);
+int lockd_gl(struct cmd_context *cmd, const char *def_mode, uint32_t flags);
+
 #else /* LVMLOCKD_SUPPORT */
 
 #define lvmlockd_init(cmd)          do { } while (0)
@@ -99,6 +117,9 @@ int lockd_stop_vg(struct cmd_context *cmd, struct volume_group *vg);
 #define lockd_start_vg(cmd, vg) (1)
 #define lockd_stop_vg(cmd, vg)  (1)
 
+#define lockd_gl_create(cmd, def_mode, vg_lock_type) (1)
+#define lockd_gl(cmd, def_mode, flags) (1)
+
 #endif /* LVMLOCKD_SUPPORT */
 
 #endif
diff --git a/tools/args.h b/tools/args.h
index 17656b073..8dc6c58f0 100644
--- a/tools/args.h
+++ b/tools/args.h
@@ -48,6 +48,7 @@ arg(ignoremonitoring_ARG, '\0', "ignoremonitoring", NULL, 0)
 arg(ignoreskippedcluster_ARG, '\0', "ignoreskippedcluster", NULL, 0)
 arg(ignoreunsupported_ARG, '\0', "ignoreunsupported", NULL, 0)
 arg(labelsector_ARG, '\0', "labelsector", int_arg, 0)
+arg(lockgl_ARG, '\0', "lock-gl", string_arg, 0)
 arg(lockstart_ARG, '\0', "lock-start", string_arg, 0)
 arg(lockstop_ARG, '\0', "lock-stop", string_arg, 0)
 arg(locktype_ARG, '\0', "lock-type", string_arg, 0)
diff --git a/tools/lvchange.c b/tools/lvchange.c
index 8a175e033..5c5606709 100644
--- a/tools/lvchange.c
+++ b/tools/lvchange.c
@@ -1207,6 +1207,9 @@ int lvchange(struct cmd_context *cmd, int argc, char **argv)
 		}
 	}
 
+	if (arg_tag_count(argc, argv) && !lockd_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	return process_each_lv(cmd, argc, argv,
 			       update ? READ_FOR_UPDATE : 0, NULL,
 			       &_lvchange_single);
diff --git a/tools/lvmcmdline.c b/tools/lvmcmdline.c
index 8f1376cc3..0ffd8b446 100644
--- a/tools/lvmcmdline.c
+++ b/tools/lvmcmdline.c
@@ -569,6 +569,20 @@ int string_arg(struct cmd_context *cmd __attribute__((unused)),
 	return 1;
 }
 
+int arg_tag_count(int argc, char **argv)
+{
+	const char *name;
+	int count = 0;
+	int i;
+
+	for (i = 0; i < argc; i++) {
+		name = argv[i];
+		if (*name == '@')
+			count++;
+	}
+	return count;
+}
+
 int tag_arg(struct cmd_context *cmd __attribute__((unused)), struct arg_values *av)
 {
 	char *pos = av->value;
@@ -747,6 +761,7 @@ void lvm_register_commands(void)
 					    yes_ARG, \
 					    quiet_ARG, config_ARG, \
 					    commandprofile_ARG, \
+					    lockgl_ARG, \
 					    profile_ARG, -1);
 #include "commands.h"
 #undef xx
@@ -1026,6 +1041,9 @@ static int _get_settings(struct cmd_context *cmd)
 		cmd->current_settings.backup = 0;
 	}
 
+	if (arg_is_set(cmd, lockgl_ARG))
+		cmd->lock_gl_mode = arg_str_value(cmd, lockgl_ARG, NULL);
+
 	cmd->partial_activation = 0;
 	cmd->degraded_activation = 0;
 	activation_mode = find_config_tree_str(cmd, activation_mode_CFG, NULL);
diff --git a/tools/lvscan.c b/tools/lvscan.c
index 2d7be074b..9d33eec1a 100644
--- a/tools/lvscan.c
+++ b/tools/lvscan.c
@@ -98,6 +98,9 @@ int lvscan(struct cmd_context *cmd, int argc, char **argv)
 		return EINVALID_CMD_LINE;
 	}
 
+	if (!lockd_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	return process_each_lv(cmd, argc, argv, 0, NULL,
 			       &lvscan_single);
 }
diff --git a/tools/pvchange.c b/tools/pvchange.c
index f616c0925..996da757f 100644
--- a/tools/pvchange.c
+++ b/tools/pvchange.c
@@ -90,6 +90,10 @@ static int _pvchange_single(struct cmd_context *cmd, struct volume_group *vg,
 		}
 	}
 
+	/* Convert sh to ex.  gl only needed for orphans. */
+	if (is_orphan(pv) && !lockd_gl(cmd, "ex", 0))
+		return_ECMD_FAILED;
+
 	if (tagargs) {
 		/* tag or deltag */
 		if (arg_count(cmd, addtag_ARG) && !change_tag(cmd, NULL, NULL, pv, addtag_ARG))
@@ -176,6 +180,9 @@ int pvchange(struct cmd_context *cmd, int argc, char **argv)
 		return EINVALID_CMD_LINE;
 	}
 
+	if (!lockd_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	params.done = 0;
 	params.total = 0;
 
diff --git a/tools/pvcreate.c b/tools/pvcreate.c
index 139819883..8b4730103 100644
--- a/tools/pvcreate.c
+++ b/tools/pvcreate.c
@@ -96,6 +96,9 @@ int pvcreate(struct cmd_context *cmd, int argc, char **argv)
 	int ret = ECMD_PROCESSED;
 	struct pvcreate_params pp;
 
+	if (!lockd_gl(cmd, "ex", 0))
+		return_ECMD_FAILED;
+
 	pvcreate_params_set_defaults(&pp);
 
 	if (!pvcreate_restore_params_validate(cmd, argc, argv, &pp)) {
diff --git a/tools/pvdisplay.c b/tools/pvdisplay.c
index 50522d9c2..1ee572542 100644
--- a/tools/pvdisplay.c
+++ b/tools/pvdisplay.c
@@ -107,6 +107,9 @@ int pvdisplay(struct cmd_context *cmd, int argc, char **argv)
 		}
 	}
 
+	if (!lockd_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	ret = process_each_pv(cmd, argc, argv, NULL, 0, NULL,
 			      _pvdisplay_single);
 
diff --git a/tools/pvremove.c b/tools/pvremove.c
index b40ff794a..1baf1ac5e 100644
--- a/tools/pvremove.c
+++ b/tools/pvremove.c
@@ -32,6 +32,9 @@ int pvremove(struct cmd_context *cmd, int argc, char **argv)
 
 	dm_list_init(&pv_names);
 
+	if (!lockd_gl(cmd, "ex", 0))
+		return_ECMD_FAILED;
+
 	for (i = 0; i < argc; i++) {
 		dm_unescape_colons_and_at_signs(argv[i], NULL, NULL);
 		if (!str_list_add(cmd->mem, &pv_names, argv[i]))
diff --git a/tools/pvresize.c b/tools/pvresize.c
index 631e63e0b..04246c77a 100644
--- a/tools/pvresize.c
+++ b/tools/pvresize.c
@@ -36,6 +36,10 @@ static int _pvresize_single(struct cmd_context *cmd,
 	}
 	params->total++;
 
+	/* Convert sh to ex.  gl is only needed for orphans. */
+	if (is_orphan(pv) && !lockd_gl(cmd, "ex", 0))
+		return_ECMD_FAILED;
+
 	if (!pv_resize_single(cmd, vg, pv, params->new_size))
 		return_ECMD_FAILED;
 
@@ -65,6 +69,9 @@ int pvresize(struct cmd_context *cmd, int argc, char **argv)
 	params.done = 0;
 	params.total = 0;
 
+	if (!lockd_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	ret = process_each_pv(cmd, argc, argv, NULL, READ_FOR_UPDATE, &params,
 			      _pvresize_single);
 
diff --git a/tools/pvscan.c b/tools/pvscan.c
index 4ab2ff796..f6599f339 100644
--- a/tools/pvscan.c
+++ b/tools/pvscan.c
@@ -224,6 +224,9 @@ static int _pvscan_lvmetad(struct cmd_context *cmd, int argc, char **argv)
 		return ECMD_FAILED;
 	}
 
+	if (!lockd_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	/* Scan everything? */
 	if (!argc && !devno_args) {
 		if (!lvmetad_pvscan_all_devs(cmd, handler))
diff --git a/tools/reporter.c b/tools/reporter.c
index 236addc1e..a4d968e8c 100644
--- a/tools/reporter.c
+++ b/tools/reporter.c
@@ -419,6 +419,9 @@ static int _report(struct cmd_context *cmd, int argc, char **argv,
 	if (args_are_pvs && argc)
 		cmd->filter->wipe(cmd->filter);
 
+	if ((!argc || arg_tag_count(argc, argv)) && !lockd_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	switch (report_type) {
 	case DEVTYPES:
 		keys = find_config_tree_str(cmd, report_devtypes_sort_CFG, NULL);
diff --git a/tools/tools.h b/tools/tools.h
index 48a3d14d4..9f5982d12 100644
--- a/tools/tools.h
+++ b/tools/tools.h
@@ -173,6 +173,8 @@ unsigned grouped_arg_is_set(const struct arg_values *av, int a);
 const char *grouped_arg_str_value(const struct arg_values *av, int a, const char *def);
 int32_t grouped_arg_int_value(const struct arg_values *av, int a, const int32_t def); 
 
+int arg_tag_count(int argc, char **argv);
+
 const char *command_name(struct cmd_context *cmd);
 
 int pvmove_poll(struct cmd_context *cmd, const char *pv, unsigned background);
diff --git a/tools/vgchange.c b/tools/vgchange.c
index df29af8cf..2ca0f3629 100644
--- a/tools/vgchange.c
+++ b/tools/vgchange.c
@@ -677,6 +677,36 @@ static int vgchange_single(struct cmd_context *cmd, const char *vg_name,
         return ret;
 }
 
+static int lockd_vgchange(struct cmd_context *cmd, int argc, char **argv)
+{
+	if (!argc || arg_tag_count(argc, argv) || arg_is_set(cmd, lockstart_ARG)) {
+		/*
+		 * The first two standard conditions want the current
+		 * list of all vg names.  The lockstart condition takes
+		 * the gl to serialize with any other host that tries to
+		 * remove the VG while this tries to start it.
+		 */
+		if (!lockd_gl(cmd, "sh", 0))
+			return_ECMD_FAILED;
+
+	} else if (arg_is_set(cmd, systemid_ARG) ||
+		   arg_is_set(cmd, uuid_ARG) ||
+		   arg_is_set(cmd, locktype_ARG)) {
+		/*
+		 * VG names, uuids and system_ids are the three things that
+		 * other hosts cache related to local vg's, so we use the
+		 * name-change counter in the global lock to indicate that
+		 * one of these global VG identifiers has changed so other
+		 * hosts will update these cached values in VG's that they
+		 * otherwise ignore (because they have foreign system_ids).
+		 */
+		if (!lockd_gl(cmd, "ex", LDGL_UPDATE_NAMES))
+			return_ECMD_FAILED;
+	}
+
+	return 1;
+}
+
 int vgchange(struct cmd_context *cmd, int argc, char **argv)
 {
 	int noupdate =
@@ -799,6 +829,9 @@ int vgchange(struct cmd_context *cmd, int argc, char **argv)
 	if (!update || !update_partial_unsafe)
 		cmd->handles_missing_pvs = 1;
 
+	if (!lockd_vgchange(cmd, argc, argv))
+		return_ECMD_FAILED;
+
 	return process_each_vg(cmd, argc, argv, update ? READ_FOR_UPDATE : 0,
 			       NULL, &vgchange_single);
 }
diff --git a/tools/vgcreate.c b/tools/vgcreate.c
index 84acf8668..42d49c9cb 100644
--- a/tools/vgcreate.c
+++ b/tools/vgcreate.c
@@ -50,6 +50,9 @@ int vgcreate(struct cmd_context *cmd, int argc, char **argv)
 	if (!vgcreate_params_validate(cmd, &vp_new))
 	    return EINVALID_CMD_LINE;
 
+	if (!lockd_gl_create(cmd, "ex", vp_new.lock_type))
+		return ECMD_FAILED;
+
 	lvmcache_seed_infos_from_lvmetad(cmd);
 
 	/* Create the new VG */
diff --git a/tools/vgextend.c b/tools/vgextend.c
index 716a2ceb6..149f9a68f 100644
--- a/tools/vgextend.c
+++ b/tools/vgextend.c
@@ -156,6 +156,9 @@ int vgextend(struct cmd_context *cmd, int argc, char **argv)
 	 */
 	cmd->handles_missing_pvs = 1;
 
+	if (!lockd_gl(cmd, "ex", 0))
+		return_ECMD_FAILED;
+
 	return process_each_vg(cmd, argc, argv,
 			       READ_FOR_UPDATE | ONLY_FIRST_NAME, &vp,
 			       restore ? &vgextend_restore : &vgextend_single);
diff --git a/tools/vgmerge.c b/tools/vgmerge.c
index a17a636c5..4f5121232 100644
--- a/tools/vgmerge.c
+++ b/tools/vgmerge.c
@@ -194,6 +194,9 @@ int vgmerge(struct cmd_context *cmd, int argc, char **argv)
 		return EINVALID_CMD_LINE;
 	}
 
+	if (!lockd_gl(cmd, "ex", LDGL_UPDATE_NAMES))
+		return ECMD_FAILED;
+
 	vg_name_to = skip_dev_dir(cmd, argv[0], NULL);
 	argc--;
 	argv++;
diff --git a/tools/vgreduce.c b/tools/vgreduce.c
index 7af5a7683..e3b0910cf 100644
--- a/tools/vgreduce.c
+++ b/tools/vgreduce.c
@@ -195,6 +195,9 @@ int vgreduce(struct cmd_context *cmd, int argc, char **argv)
 	init_ignore_suspended_devices(1);
 	cmd->handles_missing_pvs = 1;
 
+	if (!lockd_gl(cmd, "ex", 0))
+		return_ECMD_FAILED;
+
 	vg = vg_read_for_update(cmd, vg_name, NULL, READ_ALLOW_EXPORTED);
 	if (vg_read_error(vg) == FAILED_ALLOCATION ||
 	    vg_read_error(vg) == FAILED_NOTFOUND)
diff --git a/tools/vgremove.c b/tools/vgremove.c
index 4514cb51e..2c36d5881 100644
--- a/tools/vgremove.c
+++ b/tools/vgremove.c
@@ -79,6 +79,9 @@ int vgremove(struct cmd_context *cmd, int argc, char **argv)
 		return EINVALID_CMD_LINE;
 	}
 
+	if (!lockd_gl(cmd, "ex", LDGL_UPDATE_NAMES))
+		return ECMD_FAILED;
+
 	cmd->handles_missing_pvs = 1;
 	ret = process_each_vg(cmd, argc, argv,
 			      READ_FOR_UPDATE,
diff --git a/tools/vgrename.c b/tools/vgrename.c
index 860ccf196..835c7acee 100644
--- a/tools/vgrename.c
+++ b/tools/vgrename.c
@@ -207,6 +207,9 @@ int vgrename(struct cmd_context *cmd, int argc, char **argv)
 		return EINVALID_CMD_LINE;
 	}
 
+	if (!lockd_gl(cmd, "ex", LDGL_UPDATE_NAMES))
+		return_ECMD_FAILED;
+
 	if (!vg_rename_path(cmd, argv[0], argv[1]))
 		return_ECMD_FAILED;
 
diff --git a/tools/vgscan.c b/tools/vgscan.c
index bca98cc6e..82fb01c75 100644
--- a/tools/vgscan.c
+++ b/tools/vgscan.c
@@ -42,6 +42,9 @@ int vgscan(struct cmd_context *cmd, int argc, char **argv)
 		return ECMD_FAILED;
 	}
 
+	if (!lockd_gl(cmd, "sh", 0))
+		return_ECMD_FAILED;
+
 	if (cmd->filter->wipe)
 		cmd->filter->wipe(cmd->filter);
 	lvmcache_destroy(cmd, 1, 0);
diff --git a/tools/vgsplit.c b/tools/vgsplit.c
index 362f85410..67c9fae2e 100644
--- a/tools/vgsplit.c
+++ b/tools/vgsplit.c
@@ -492,6 +492,9 @@ int vgsplit(struct cmd_context *cmd, int argc, char **argv)
 		return ECMD_FAILED;
 	}
 
+	if (!lockd_gl(cmd, "ex", LDGL_UPDATE_NAMES))
+		return_ECMD_FAILED;
+
 	if (arg_count(cmd, name_ARG))
 		lv_name = arg_value(cmd, name_ARG);
 	else