lvmetad: refresh content at command startdev-dct-lvmetad6

At the start of the command, check if the lvmetad
content needs updating (token not matching), and
if so do the scan and update it.  This gets the
updating done right at the beginning, before
command processing is started. The early scan
already exists for the foreign VGs case, so this
is added to it.

Previously, the command would be part way through
its own processing before discovering via lvmetad_send
that lvmetad needed updating.  It's not good to
interrupt a command, insert a 'pvscan --cache', then
continue on with the original command.  The pvscan
is likely to interfere with the original command.

7 files changed, 257 insertions, 14 deletions

diff --git a/daemons/lvmetad/lvmetad-core.c b/daemons/lvmetad/lvmetad-core.c
index a0bdfa050..6ecaa8ffb 100644
--- a/daemons/lvmetad/lvmetad-core.c
+++ b/daemons/lvmetad/lvmetad-core.c
@@ -30,6 +30,69 @@
 #define LVMETAD_SOCKET DEFAULT_RUN_DIR "/lvmetad.socket"
 
 /*
+ * Device scanning and populating lvmetad.
+ *
+ * lvmetad caches lvm metadata to reduce the amount of device
+ * scanning done by lvm commands.  Device scanning is still
+ * necessary in a number of cases.
+ *
+ * When lvmetad is started, it's "empty", it has no metadata
+ * in its cache.  It relies on lvm commands to scan devices and
+ * populate its cache by sending it information (via pv_found messages).
+ *
+ * 'pvscan --cache' is meant to scan all devices and fully populate the
+ * lvmetad cache.  No devs are specified in this command.
+ *
+ * 'pvscan --cache dev' is meant to scan only the specified device
+ * and add information about it (and from it) to the lvmetad cache.
+ *
+ * When a device appears, a uevent triggers 'pvscan --cache dev',
+ * which scans dev and adds metadata about it and from it to lvmetad.
+ * This is not meant to scan any devices beyond those specified.
+ *
+ * The sum of all individual 'pvscan --cache dev' commands is not
+ * sufficient to fully populate lvmetad because some devices may
+ * exist before lvmetad is started.  So, if 'pvscan --cache dev'
+ * finds that lvmetad is empty, it will perform a full scan
+ * (equivalent to pvscan --cache) to populate lvmetad.  This is
+ * an exception to the rule of only scanning the specified device.
+ *
+ * When any other lvm command connects to lvmetad, it first checks
+ * if lvmetad is empty, and if so, performs a full scan (equivalent
+ * to 'pvscan --cache') to populate lvmetad.  This is beneficial when
+ * lvmetad is restarted and no pvscan --cache command was run afterward.
+ *
+ * When a command connects to lvmetad, it not only checks if lvmetad
+ * is empty, it also checks if lvmetad has been populated using a
+ * different global_filter than its own.  If so, then the lvmetad
+ * cache is invalid, and the command performs a rescan just as if
+ * lvmetad were empty (and using the new global filter).
+ * This can happen if the global_filter is changed in lvm.conf,
+ * or if a global_filter is specified on the command line.
+ *
+ * Concurrent commands using different global filters can cause
+ * problems because will each want a different set of metadata in
+ * lvmetad.  If one command sees that another is updating the
+ * lvmetad cache, it will delay for a while, then retry, and
+ * hopefully the first command will be finished.
+ *
+ * If a report/display command is asked to show information
+ * about foreign VGs, then the command performs a rescan
+ * to repopulate lvmetad, just as if lvmetad were empty.
+ * This is beneficial because the version of the foreign VG
+ * cached in lvmetad could easily be out of date given changes
+ * another host may have made to its VG.
+ *
+ * When using shared VGs with lvmlockd, the cached VG in lvmetad
+ * will become out of date when another host changes the VG.
+ * This difference is detected by lvmlockd, which then invalidates
+ * the cached VG in lvmetad, or all the cached data in lvmetad.
+ * If an lvm command sees that all the lvmetad metadata has been
+ * invalidated by lvmlockd, it will perform a full scan to
+ * update the lvmetad cache.
+ */
+
+/*
  * valid/invalid state of cached metadata
  *
  * Normally when using lvmetad, the state is kept up-to-date through a
@@ -2506,9 +2569,10 @@ static response set_global_info(lvmetad_state *s, request r)
 
 static response get_global_info(lvmetad_state *s, request r)
 {
-	return daemon_reply_simple("OK", "global_invalid = %d",
-					 (s->flags & GLFL_INVALID) ? 1 : 0,
-					 NULL);
+	return daemon_reply_simple("OK",
+				   "global_invalid = %d", (s->flags & GLFL_INVALID) ? 1 : 0,
+				   "token = %s", s->token[0] ? s->token : "none",
+				   NULL);
 }
 
 static response set_vg_info(lvmetad_state *s, request r)
diff --git a/lib/cache/lvmetad.c b/lib/cache/lvmetad.c
index 090f0d722..4c6a9fb6d 100644
--- a/lib/cache/lvmetad.c
+++ b/lib/cache/lvmetad.c
@@ -206,6 +206,108 @@ void lvmetad_set_socket(const char *sock)
 	_lvmetad_socket = sock;
 }
 
+/*
+ * Check if lvmetad's token matches our token.  The token is a hash
+ * of the global filter used to populate lvmetad.  The lvmetad token
+ * was set by the last command to populate lvmetad, and it was set to
+ * the hash of the global filter that command used when scanning to
+ * populate lvmetad.
+ *
+ * Our token is a hash of the global filter this command is using.
+ *
+ * If the lvmetad token is not set (or "none"), then lvmetad has not
+ * been populated.  If the lvmetad token is "update in progress", then
+ * lvmetad is currently being populated (this should be temporary).
+ * If the lvmetad token otherwise differs from ours, then lvmetad was
+ * populated using a different global filter that we are using.
+ *
+ * Return 1 if the lvmetad token matches ours.  We can use it as is.
+ *
+ * Return 0 if the lvmetad token does not match ours (lvmetad is
+ * empty or populated using a different global filter).
+ * We cannot use the lvmetad cache until we repopulate it
+ * (and set lvmetad's token to match ours.)
+ *
+ * Return an error if lvmetad is stuck being updated.
+ * We can't use it.  This shouldn't happen, but could if
+ * the command updating lvmetad gets stuck, e.g. trying to
+ * read a bad device.  We could fail this command, or adjust
+ * this command to not use lvmetad (depending on which command
+ * it is?)
+ */
+
+int lvmetad_token_matches(struct cmd_context *cmd)
+{
+	daemon_reply reply;
+	const char *daemon_token;
+	int ret = 1;
+
+retry:
+	log_debug_lvmetad("lvmetad send get_global_info");
+
+	reply = daemon_send_simple(_lvmetad, "get_global_info",
+				   "token = %s", "skip",
+				   NULL);
+	if (reply.error) {
+		log_error("lvmetad_token_matches get_global_info error %d", reply.error);
+		ret = 0;
+		goto out;
+	}
+
+	if (strcmp(daemon_reply_str(reply, "response", ""), "OK")) {
+		log_error("lvmetad_token_matches get_global_info not ok");
+		ret = 0;
+		goto out;
+	}
+
+	daemon_token = daemon_reply_str(reply, "token", NULL);
+
+	if (!daemon_token) {
+		log_error("lvmetad_token_matches no token returned");
+		ret = 0;
+		goto out;
+	}
+
+	/*
+	 * If lvmetad is being updated by another command, then sleep and retry
+	 * until the token shows the update is done, and go on to the token
+	 * comparison.  FIXME: after retrying enough, quit and disable the use
+	 * of lvmetad for this command.
+	 */
+	if (!strcmp(daemon_token, "update in progress")) {
+		if (retries > 120) {
+			/* FIXME: disable lvmetad for this command. */
+			log_error("Not using lvmetad which is busy.");
+			ret = 0;
+			goto out;
+		}
+		log_warn("lvmetad is being updated, retrying...");
+		usleep(500000);
+		retries++;
+		goto retry;
+	}
+
+	/*
+	 * lvmetad is empty, not yet populated.
+	 */
+	if (!strcmp(daemon_token, "none")) {
+		ret = 0;
+		goto out;
+	}
+
+	/*
+	 * lvmetad has an unmatching token; it was last populated using
+	 * a different global filter.
+	 */
+	if (strcmp(daemon_token, _lvmetad_token)) {
+		ret = 0;
+		goto out;
+	}
+out:
+	daemon_reply_destroy(reply);
+	return ret;
+}
+
 static int _lvmetad_pvscan_all_devs(struct cmd_context *cmd, activation_handler handler,
 				    int ignore_obsolete);
 
@@ -219,6 +321,8 @@ static daemon_reply _lvmetad_send(const char *id, ...)
 	unsigned max_remaining_sleep_times = 1;
 	unsigned wait_usecs;
 
+	log_debug_lvmetad("lvmetad_send %s", id);
+
 retry:
 	req = daemon_request_make(id);
 
@@ -246,26 +350,44 @@ retry:
 	 * we re-scan immediately, but if we lose the potential race for
 	 * the update, we back off for a short while (0.05-0.5 seconds) and
 	 * try again.
+	 *
+	 * FIXME: remove rescanning and just fail the command.
+	 * Running pvscan here is likely to mess up the state of
+	 * the original command.  This should be very unlikely
+	 * since the token is verified to match at the start
+	 * of the command now.  Also there's no recursion check to
+	 * prevent _lvmetad_pvscan_all_devs->lvmetad_send->_lvmetad_pvscan_all_devs...
 	 */
 	if (!repl.error && !strcmp(daemon_reply_str(repl, "response", ""), "token_mismatch") &&
 	    num_rescans < MAX_RESCANS && total_usecs_waited < (SCAN_TIMEOUT_SECONDS * 1000000) && !test_mode()) {
-		if (!strcmp(daemon_reply_str(repl, "expected", ""), "update in progress") ||
-		    max_remaining_sleep_times) {
+
+		/*
+		 * The other command should finish updating lvmetad soon.
+		 * Sleep to give it a chance to finish, then retry.
+		 */
+		if (!strcmp(daemon_reply_str(repl, "expected", ""), "update in progress") || max_remaining_sleep_times) {
+			log_debug_lvmetad("lvmetad is not ready, retrying...");
 			wait_usecs = 50000 + lvm_even_rand(&_lvmetad_cmd->rand_seed, 450000); /* between 0.05s and 0.5s */
 			(void) usleep(wait_usecs);
 			total_usecs_waited += wait_usecs;
 			if (max_remaining_sleep_times)
 				max_remaining_sleep_times--;	/* Sleep once before rescanning the first time, then 5 times each time after that. */
 		} else {
+			log_error("lvmetad cache is not usable, update lvmetad and retry command.");
+			goto out;
+		}
+#if 0
+	       	else {
 			/* If the re-scan fails here, we try again later. */
 			(void) _lvmetad_pvscan_all_devs(_lvmetad_cmd, NULL, 0);
 			num_rescans++;
 			max_remaining_sleep_times = 5;
 		}
+#endif
 		daemon_reply_destroy(repl);
 		goto retry;
 	}
-
+out:
 	return repl;
 }
 
@@ -1396,6 +1518,8 @@ static int _lvmetad_pvscan_all_devs(struct cmd_context *cmd, activation_handler
 		return 0;
 	}
 
+	log_debug_lvmetad("Scanning all devices to update lvmetad.");
+
 	if (!(iter = dev_iter_create(cmd->lvmetad_filter, 1))) {
 		log_error("dev_iter creation failed");
 		return 0;
@@ -1712,6 +1836,8 @@ void lvmetad_validate_global_cache(struct cmd_context *cmd, int force)
 	if (force)
 		goto do_scan;
 
+	log_debug_lvmetad("lvmetad validate send get_global_info");
+
 	reply = daemon_send_simple(_lvmetad, "get_global_info",
 				   "token = %s", "skip",
 				   NULL);
@@ -1744,7 +1870,12 @@ void lvmetad_validate_global_cache(struct cmd_context *cmd, int force)
 
 	/*
 	 * Update the local lvmetad cache so it correctly reflects any
-	 * changes made on remote hosts.
+	 * changes made on remote hosts.  (It's possible that this command
+	 * already refreshed the local lvmetad because of a token change,
+	 * but we need to do it again here since we now hold the global
+	 * lock.  Another host may have changed things between the time
+	 * we rescanned for the token, and the time we acquired the global
+	 * lock.)
 	 */
 	lvmetad_pvscan_all_devs(cmd, NULL);
 
@@ -1754,6 +1885,8 @@ void lvmetad_validate_global_cache(struct cmd_context *cmd, int force)
 	 * from lvmetad will not see global_invalid until
 	 * another host makes another global change.
 	 */
+	log_debug_lvmetad("lvmetad validate send set_global_info");
+
 	reply = daemon_send_simple(_lvmetad, "set_global_info",
 				   "token = %s", "skip",
 				   "global_invalid = %d", 0,
diff --git a/lib/cache/lvmetad.h b/lib/cache/lvmetad.h
index af0d562fe..1c94308df 100644
--- a/lib/cache/lvmetad.h
+++ b/lib/cache/lvmetad.h
@@ -168,6 +168,7 @@ int lvmetad_pvscan_foreign_vgs(struct cmd_context *cmd, activation_handler handl
 
 int lvmetad_vg_clear_outdated_pvs(struct volume_group *vg);
 void lvmetad_validate_global_cache(struct cmd_context *cmd, int force);
+int lvmetad_token_matches(struct cmd_context *cmd);
 
 #  else		/* LVMETAD_SUPPORT */
 
@@ -197,6 +198,7 @@ void lvmetad_validate_global_cache(struct cmd_context *cmd, int force);
 #    define lvmetad_pvscan_foreign_vgs(cmd, handler)	(0)
 #    define lvmetad_vg_clear_outdated_pvs(vg)           (1)
 #    define lvmetad_validate_global_cache(cmd, force)	do { } while (0)
+#    define lvmetad_token_matches(cmd) (1)
 
 #  endif	/* LVMETAD_SUPPORT */
 
diff --git a/tools/commands.h b/tools/commands.h
index 92d13fdf0..fb41d9e4c 100644
--- a/tools/commands.h
+++ b/tools/commands.h
@@ -962,7 +962,7 @@ xx(pvs,
 
 xx(pvscan,
    "List all physical volumes",
-   PERMITTED_READ_ONLY | LOCKD_VG_SH,
+   PERMITTED_READ_ONLY | LOCKD_VG_SH | SKIP_AUTO_PVSCAN,
    "pvscan\n"
    "\t[-b|--background]\n"
    "\t[--cache [-a|--activate ay] [ DevicePath | -j|--major major --minor minor]...]\n"
diff --git a/tools/lvmcmdline.c b/tools/lvmcmdline.c
index 657797765..858b0360f 100644
--- a/tools/lvmcmdline.c
+++ b/tools/lvmcmdline.c
@@ -1640,13 +1640,25 @@ int lvm_run_command(struct cmd_context *cmd, int argc, char **argv)
 	}
 
 	/*
-	 * Other hosts might have changed foreign VGs so enforce a rescan
-	 * before processing any command using them.
+	 * The lvmetad cache may need to be repopulated before we use it because:
+	 * - We are reading foreign VGs which others hosts may have changed
+	 *   which our lvmetad would not have seen.
+	 * - lvmetad may have just been started and no command has been run
+	 *   to populate it yet (e.g. no pvscan --cache was run).
+	 * - Another local command may have run with a different global filter
+	 *   which changed the content of lvmetad from what we want (recognized
+	 *   by differnet token values.)
+	 *
+	 * The pvscan command does rescanning and lvmetad upates itself,
+	 * so it doesn't want this automatic rescanning.
 	 */
-	if (cmd->include_foreign_vgs && lvmetad_used() &&
-	    !lvmetad_pvscan_foreign_vgs(cmd, NULL)) {
-		log_error("Failed to scan devices.");
-		return ECMD_FAILED;
+	if (lvmetad_active() && !(cmd->command->flags & SKIP_AUTO_PVSCAN)) {
+		if (!lvmetad_token_matches(cmd) || cmd->include_foreign_vgs) {
+			if (!lvmetad_pvscan_all_devs(cmd, NULL)) {
+				log_error("Failed to scan devices");
+				return ECMD_FAILED;
+			}
+		}
 	}
 
 	/*
diff --git a/tools/pvscan.c b/tools/pvscan.c
index 10bf28e52..5b565033c 100644
--- a/tools/pvscan.c
+++ b/tools/pvscan.c
@@ -264,6 +264,21 @@ static int _pvscan_lvmetad(struct cmd_context *cmd, int argc, char **argv)
 		goto out;
 	}
 
+	/*
+	 * FIXME: when specific devs are named, we generally don't
+	 * want to scan any other devs, but if lvmetad is not yet
+	 * populated, the first 'pvscan --cache dev' does need to
+	 * do a full scan.  We want to remove the need for this
+	 * case so that 'pvscan --cache dev' is guaranteed to never
+	 * scan any devices other than those specified.
+	 */
+	if (lvmetad_active() && !lvmetad_token_matches(cmd)) {
+		if (!lvmetad_pvscan_all_devs(cmd, NULL)) {
+			log_error("Failed to scan devices");
+			return ECMD_FAILED;
+		}
+	}
+
 	log_verbose("Using physical volume(s) on command line");
 
 	/* Process any command line PVs first. */
@@ -387,6 +402,21 @@ int pvscan(struct cmd_context *cmd, int argc, char **argv)
 			  arg_count(cmd, exported_ARG) ?
 			  "of exported volume group(s)" : "in no volume group");
 
+	/*
+	 * All pvscan commands skip the automatic repopulating of
+	 * lvmetad when the token doesn't match, because pvscan_lvmetad
+	 * above (for pvscan --cache) needs to do that repopulating
+	 * itself.  So for other pvscan commands (without --cache), we
+	 * need to check the lvmetad token and repopulate the cache
+	 * if it doesn't match.
+	 */
+	if (lvmetad_active() && !lvmetad_token_matches(cmd)) {
+		if (!lvmetad_pvscan_all_devs(cmd, NULL)) {
+			log_error("Failed to scan devices");
+			return ECMD_FAILED;
+		}
+	}
+
 	if (!lock_vol(cmd, VG_GLOBAL, LCK_VG_WRITE, NULL)) {
 		log_error("Unable to obtain global lock.");
 		return ECMD_FAILED;
diff --git a/tools/tools.h b/tools/tools.h
index 4ed893fc4..793a871ac 100644
--- a/tools/tools.h
+++ b/tools/tools.h
@@ -105,6 +105,8 @@ struct arg_value_group_list {
 #define LOCKD_VG_SH		0x00000020
 /* Command does not process any metadata. */
 #define NO_METADATA_PROCESSING	0x00000040
+/* Don't refresh lvmetad cache automatically, let command do it. */
+#define SKIP_AUTO_PVSCAN	0x00000080
  
 /* a register of the lvm commands */
 struct command {