settings: track profiles on disk that are shadowed by in-memory connections

Via Update2() D-Bus API there are three ways how a profile can be stored
(or migrated) to in-memory:

  - NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY
  - NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED
  - NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_ONLY

With the recent rework of settings I dropped NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY
and it had the same meaning as NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED.

However, the way NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED was implemented is
problematic. The problem is that it leaves the profile on disk but creates an
in-memory representation which shadows the persistent storage. Later,
when storing the profile to disk again, a new filename is chosen.
This allows via D-Bus API to toggle between NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED
and NM_SETTINGS_UPDATE2_FLAG_TO_DISK, and thereby pilling up profiles on disk.
Also, there is no D-Bus API to do anything sensible with these leaked, shadowed
profiles on disk.

Note that if we have a read-only profile in /usr/lib or in ifupdown
plugin, then the problem is not made any worse. That is, because via D-Bus
API such profiles can be made in-memory, and afterwards stored to /etc.
Thereby too the profile gets duplicate on disk, but this game only
works once. Afterwards, you cannot repeat it to create additional
profiles on disk. It means, you can only leak profiles once, and only
if they already exist in read-only storage to begin with.

This problem with NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED already existed
before the settings-delegate-storage rework, and is unrelated to whether in-memory
profiles now happen to be persisted to /run.

Note that NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_ONLY is simple and does not suffer
from this problem. When you move a profile to in-memory-only, it gets deleted from
persistent storage and no duplication happens.

The problem is that NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED used to
forget about the profile that it shadows, and that is wrong.

So, first re-add proper support for NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY. This
works by remembering the "shadowed-storage" path for in-memory profiles.
When later saving such a profile to disk again, the shadowed-storage
will be re-used. Likewise, when deleting such a profile, the shadowed
storage will be deleted.

Note that we keep NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED and it
also remembers the shadowed storage (but without "owning" it). That means,
when such a profile gets saved to disk again, the orginal storage is
reused too. As such, during future updates it behaves just like
NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY. The difference is when deleting
such a profile. In this case, the profile is left on storage and a
tombstone gets written. So, how is this better than before and why even
keep this complicated flag?
First, we keep this flag because we really want the ansible role to be
able to do in-memory changes only. That implies being able to delete a
profile from NetworkManager's view, but not from persistent storage. Without
this flag there is no way to do that. You can only modify an on-disk profile
by shadowing it, but you could not delete it form NetworkManager's view
while keeping it on disk.

The new form of NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED is safe and avoids
the duplication problem because also for tombstones it remembers the original
"shadowed-storage". That is, when the profile gets recreated later via
D-Bus API AddConnection, then the re-created profile will still reference
and reuse the shadowed storage that it had before deletion.

5 files changed, 468 insertions, 201 deletions

diff --git a/libnm-core/nm-dbus-interface.h b/libnm-core/nm-dbus-interface.h
index b2569af1ef..35496e82c2 100644
--- a/libnm-core/nm-dbus-interface.h
+++ b/libnm-core/nm-dbus-interface.h
@@ -1030,26 +1030,31 @@ typedef enum { /*< flags >*/
  * NMSettingsUpdate2Flags:
  * @NM_SETTINGS_UPDATE2_FLAG_NONE: an alias for numeric zero, no flags set.
  * @NM_SETTINGS_UPDATE2_FLAG_TO_DISK: to persist the connection to disk.
- * @NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY: to make the connection in-memory only.
- *   If the connection was previously persistent, the corresponding file on disk
- *   is not deleted but merely the connection is decoupled from the file
- *   on disk. If you later delete an in-memory connection, the connection
- *   on disk will be deleted as well.
- *   Note: with 1.20, this flag is no longer implemented because in-memory connections
- *   are also persisted under /run. For the moment, this behaves the same as
- *   @NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED.
- * @NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED: this is like @NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY,
- *   but if the connection has a corresponding file on disk, the association between
- *   the connection and the file is forgotten but the file is not modified.
- *   The difference to %NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY is if you later
- *   save the connection again to disk, a new file name will be chosen without
- *   overwriting the remaining file on disk. Also, if you delete the connection
- *   later, the file on disk will not be deleted.
+ * @NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY: makes the profile in-memory.
+ *   Note that such profiles are stored in keyfile format under /run.
+ *   If the file is already in-memory, the file in /run is updated in-place.
+ *   Otherwise, the previous storage for the profile is left unchanged
+ *   on disk, and the in-memory copy shadows it.
+ *   Note that the original filename of the previous persistent storage (if any)
+ *   is remembered. That means, when later persisting the profile again to disk,
+ *   the file on disk will be overwritten again.
+ *   Likewise, when finally deleting the profile, both the storage from /run
+ *   and persistent storage are deleted (or if the persistent storage does not
+ *   allow deletion, and nmmeta file is written to mark the UUID as deleted).
+ * @NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED: this is almost the same as
+ *   @NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY, with one difference: when later deleting
+ *   the profile, the original profile will not be deleted. Instead a nmmeta
+ *   file is written to /run to indicate that the profile is gone.
+ *   Note that if such a nmmeta tombstone file exists and hides a file in persistant
+ *   storage, then when re-adding the profile with the same UUID, then the original
+ *   storage is taken over again.
  * @NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_ONLY: this is like @NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY,
- *   but if the connection has a corresponding file on disk, the file on
- *   disk will be deleted.
+ *   but if the connection has a corresponding file on persistent storage, the file
+ *   will be deleted right away. If the profile is later again persisted to disk,
+ *   a new, unused filename will be chosen.
  * @NM_SETTINGS_UPDATE2_FLAG_VOLATILE: This can be specified with either
- *   %NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED or %NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_ONLY.
+ *   %NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY, %NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED
+ *   or %NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_ONLY.
  *   After making the connection in-memory only, the connection is marked
  *   as volatile. That means, if the connection is currently not active
  *   it will be deleted right away. Otherwise, it is marked to for deletion
diff --git a/src/devices/nm-device.c b/src/devices/nm-device.c
index 04a94db8a1..da34602427 100644
--- a/src/devices/nm-device.c
+++ b/src/devices/nm-device.c
@@ -12420,7 +12420,7 @@ nm_device_set_ip_config (NMDevice *self,
 
 			nm_settings_connection_update (settings_connection,
 			                               new_connection,
-			                               NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_DETACHED,
+			                               NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY,
 			                               NM_SETTINGS_CONNECTION_INT_FLAGS_NONE,
 			                               NM_SETTINGS_CONNECTION_INT_FLAGS_NONE,
 			                               NM_SETTINGS_CONNECTION_UPDATE_REASON_NONE,
diff --git a/src/settings/nm-settings-connection.c b/src/settings/nm-settings-connection.c
index 5da772ed96..2c1105258f 100644
--- a/src/settings/nm-settings-connection.c
+++ b/src/settings/nm-settings-connection.c
@@ -1514,8 +1514,9 @@ update_auth_cb (NMSettingsConnection *self,
 
 	if (NM_FLAGS_HAS (info->flags, NM_SETTINGS_UPDATE2_FLAG_TO_DISK))
 		persist_mode = NM_SETTINGS_CONNECTION_PERSIST_MODE_TO_DISK;
-	else if (NM_FLAGS_ANY (info->flags,   NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY
-	                                      | NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED))
+	else if (NM_FLAGS_ANY (info->flags, NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY))
+		persist_mode = NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY;
+	else if (NM_FLAGS_ANY (info->flags, NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED))
 		persist_mode = NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_DETACHED;
 	else if (NM_FLAGS_HAS (info->flags, NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_ONLY)) {
 		persist_mode = NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_ONLY;
@@ -1752,7 +1753,8 @@ impl_settings_connection_update2 (NMDBusObject *obj,
 	if (   (   NM_FLAGS_ANY (flags, _NM_SETTINGS_UPDATE2_FLAG_ALL_PERSIST_MODES)
 	        && !nm_utils_is_power_of_two (flags & _NM_SETTINGS_UPDATE2_FLAG_ALL_PERSIST_MODES))
 	    || (   NM_FLAGS_HAS (flags, NM_SETTINGS_UPDATE2_FLAG_VOLATILE)
-	        && !NM_FLAGS_ANY (flags,   NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED
+	        && !NM_FLAGS_ANY (flags,   NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY
+	                                 | NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED
 	                                 | NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_ONLY))) {
 		error = g_error_new_literal (NM_SETTINGS_ERROR,
 		                             NM_SETTINGS_ERROR_INVALID_ARGUMENTS,
diff --git a/src/settings/nm-settings-connection.h b/src/settings/nm-settings-connection.h
index ee58919abc..61d5c246c3 100644
--- a/src/settings/nm-settings-connection.h
+++ b/src/settings/nm-settings-connection.h
@@ -85,21 +85,46 @@ typedef enum {
 	 * if the profile is on-disk, update it on-disk, and keep it. */
 	NM_SETTINGS_CONNECTION_PERSIST_MODE_KEEP,
 
-	/* persist to disk. If the profile is currenly in-memory, remove
-	 * it from /run. */
+	/* persist to disk. If the profile is currently in-memory, remove
+	 * it from /run. Depending on the shadowed-storage, the pre-existing
+	 * file is reused when moving the storage.
+	 *
+	 * Corresponds to %NM_SETTINGS_UPDATE2_FLAG_TO_DISK. */
 	NM_SETTINGS_CONNECTION_PERSIST_MODE_TO_DISK,
 
-	/* persist to /run (in-memory). If the profile is currently on disk,
-	 * delete it from disk. */
-	NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_ONLY,
-
-	/* persist to /run (in-memory). If the profile is currently on disk,
-	 * forget about it, but don't delete it from disk. */
+	/* Update in-memory (i.e. persist to /run). If the profile is currently on disk,
+	 * then a reference to the profile is remembered as "shadowed-storage".
+	 * Later, when storing again to persistant storage, the shawowed-storage is
+	 * updated. When deleting the profile, the shadowed-storage is also deleted
+	 * from disk.
+	 *
+	 * Corresponds to %NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY. */
+	NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY,
+
+	/* Update in-memory (i.e. persist to /run). This is almost like
+	 * %NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY, except the in-memory profile
+	 * remembers not to own the shadowed-storage ("shadowed-owned").
+	 * The diffrence is that when deleting the in-memory profile, the original
+	 * profile is not deleted but instead the nmmeta tombstone remembers the
+	 * shadowed-storage and re-used it when re-adding the profile.
+	 *
+	 * Corresponds to %NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED. */
 	NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_DETACHED,
 
-	/* this only updates the connection in-memory. Note that "in-memory" above
-	 * means to write to keyfile in /run. This really means to not notify the
-	 * settings plugin about the change. */
+	/* Update in-memory (i.e. persist to /run). If the profile is currently on disk,
+	 * delete it from disk.
+	 *
+	 * If the profile is in-memory and has a shadowed-storage, the original profile
+	 * will be deleted from disk.
+	 *
+	 * Corresponds to %NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_ONLY. */
+	NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_ONLY,
+
+	/* This only updates the connection in-memory. Note that "in-memory" above
+	 * means to write to keyfile in /run. This mode really means to not notify the
+	 * settings plugin about the change. This should be only used for updating
+	 * secrets.
+	 */
 	NM_SETTINGS_CONNECTION_PERSIST_MODE_NO_PERSIST,
 
 } NMSettingsConnectionPersistMode;
diff --git a/src/settings/nm-settings.c b/src/settings/nm-settings.c
index 4075ca6040..85a3678a33 100644
--- a/src/settings/nm-settings.c
+++ b/src/settings/nm-settings.c
@@ -303,6 +303,35 @@ _sett_conn_entry_storage_find_conflicting_storage (SettConnEntry *sett_conn_entr
 	return NULL;
 }
 
+static NMSettingsStorage *
+_sett_conn_entry_find_shadowed_storage (SettConnEntry *sett_conn_entry,
+                                        const char *shadowed_storage_filename,
+                                        NMSettingsStorage *blacklisted_storage)
+{
+	StorageData *sd;
+
+	if (!shadowed_storage_filename)
+		return NULL;
+
+	c_list_for_each_entry (sd, &sett_conn_entry->sd_lst_head, sd_lst) {
+
+		nm_assert (NM_IS_SETTINGS_STORAGE (sd->storage));
+
+		if (!sd->connection)
+			continue;
+
+		if (blacklisted_storage == sd->storage)
+			continue;
+
+		if (!nm_streq0 (nm_settings_storage_get_filename_for_shadowed_storage (sd->storage), shadowed_storage_filename))
+			continue;
+
+		return sd->storage;
+	}
+
+	return NULL;
+}
+
 /*****************************************************************************/
 
 NM_GOBJECT_PROPERTIES_DEFINE (NMSettings,
@@ -1521,6 +1550,53 @@ _update_connection_to_plugin (NMSettings *self,
 	return success;
 }
 
+static void
+_set_nmmeta_tombstone (NMSettings *self,
+                       const char *uuid,
+                       gboolean tombstone_on_disk,
+                       gboolean tombstone_in_memory,
+                       const char *shadowed_storage)
+{
+	NMSettingsPrivate *priv = NM_SETTINGS_GET_PRIVATE (self);
+	gs_unref_object NMSettingsStorage *tombstone_1_storage = NULL;
+	gs_unref_object NMSettingsStorage *tombstone_2_storage = NULL;
+
+	if (tombstone_on_disk) {
+		if (!nms_keyfile_plugin_set_nmmeta_tombstone (priv->keyfile_plugin,
+		                                              FALSE,
+		                                              uuid,
+		                                              FALSE,
+		                                              TRUE,
+		                                              NULL,
+		                                              &tombstone_1_storage,
+		                                              NULL))
+			tombstone_in_memory = TRUE;
+		if (tombstone_1_storage)
+			_connection_changed_track (self, tombstone_1_storage, NULL, FALSE);
+	}
+
+	if (tombstone_in_memory) {
+		if (!nms_keyfile_plugin_set_nmmeta_tombstone (priv->keyfile_plugin,
+		                                              FALSE,
+		                                              uuid,
+		                                              TRUE,
+		                                              TRUE,
+		                                              shadowed_storage,
+		                                              &tombstone_2_storage,
+		                                              NULL)) {
+			nms_keyfile_plugin_set_nmmeta_tombstone (priv->keyfile_plugin,
+			                                         TRUE,
+			                                         uuid,
+			                                         TRUE,
+			                                         TRUE,
+			                                         shadowed_storage,
+			                                         &tombstone_2_storage,
+			                                         NULL);
+		}
+		_connection_changed_track (self, tombstone_2_storage, NULL, FALSE);
+	}
+}
+
 /**
  * nm_settings_add_connection:
  * @self: the #NMSettings object
@@ -1546,24 +1622,34 @@ nm_settings_add_connection (NMSettings *self,
                             NMSettingsConnection **out_sett_conn,
                             GError **error)
 {
+	NMSettingsPrivate *priv;
 	gs_unref_object NMConnection *connection_cloned_1 = NULL;
 	gs_unref_object NMConnection *new_connection = NULL;
 	gs_unref_object NMSettingsStorage *new_storage = NULL;
+	gs_unref_object NMSettingsStorage *shadowed_storage = NULL;
+	NMSettingsStorage *update_storage = NULL;
 	gs_free_error GError *local = NULL;
 	SettConnEntry *sett_conn_entry;
 	const char *uuid;
 	StorageData *sd;
+	gboolean new_in_memory;
+	gboolean success;
+	const char *shadowed_storage_filename = NULL;
+
+	priv = NM_SETTINGS_GET_PRIVATE (self);
 
 	nm_assert (NM_IN_SET (persist_mode, NM_SETTINGS_CONNECTION_PERSIST_MODE_TO_DISK,
 	                                    NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_ONLY));
 
-	nm_assert (!NM_FLAGS_ANY (sett_flags, ~_NM_SETTINGS_CONNECTION_INT_FLAGS_PERSISTENT_MASK));
+	new_in_memory = (persist_mode != NM_SETTINGS_CONNECTION_PERSIST_MODE_TO_DISK);
 
-	nm_assert (   !NM_FLAGS_HAS (sett_flags, NM_SETTINGS_CONNECTION_INT_FLAGS_NM_GENERATED)
-	           || persist_mode == NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_ONLY);
+	nm_assert (!NM_FLAGS_ANY (sett_flags, ~_NM_SETTINGS_CONNECTION_INT_FLAGS_PERSISTENT_MASK));
 
-	nm_assert (   !NM_FLAGS_HAS (sett_flags, NM_SETTINGS_CONNECTION_INT_FLAGS_VOLATILE)
-	           || persist_mode == NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_ONLY);
+	if (NM_FLAGS_ANY (sett_flags,   NM_SETTINGS_CONNECTION_INT_FLAGS_VOLATILE
+	                              | NM_SETTINGS_CONNECTION_INT_FLAGS_NM_GENERATED)) {
+		nm_assert (new_in_memory);
+		new_in_memory = TRUE;
+	}
 
 	nm_assert (!NM_FLAGS_ANY (add_reason, ~NM_SETTINGS_CONNECTION_ADD_REASON_BLOCK_AUTOCONNECT));
 
@@ -1596,23 +1682,106 @@ nm_settings_add_connection (NMSettings *self,
 	if (connection_cloned_1)
 		connection = connection_cloned_1;
 
-	if (!_add_connection_to_first_plugin (self,
-	                                      sett_conn_entry,
-	                                      connection,
-	                                      (   persist_mode != NM_SETTINGS_CONNECTION_PERSIST_MODE_TO_DISK
-	                                       || NM_FLAGS_ANY (sett_flags,   NM_SETTINGS_CONNECTION_INT_FLAGS_VOLATILE
-	                                                                    | NM_SETTINGS_CONNECTION_INT_FLAGS_NM_GENERATED)),
-	                                      sett_flags,
-	                                      NULL,
-	                                      FALSE,
-	                                      &new_storage,
-	                                      &new_connection,
-	                                      &local)) {
-		g_set_error (error,
-		             NM_SETTINGS_ERROR,
-		             NM_SETTINGS_ERROR_FAILED,
-		             "failure adding connection: %s",
-		             local->message);
+	if (sett_conn_entry) {
+		c_list_for_each_entry (sd, &sett_conn_entry->sd_lst_head, sd_lst) {
+			if (!nm_settings_storage_is_meta_data (sd->storage))
+				continue;
+			shadowed_storage = nm_g_object_ref (_sett_conn_entry_find_shadowed_storage (sett_conn_entry,
+			                                                                            nm_settings_storage_get_shadowed_storage (sd->storage, NULL),
+			                                                                            NULL));
+			if (shadowed_storage) {
+				/* We have a nmmeta tombstone that indicates that a storage is shadowed.
+				 *
+				 * This happens when deleting a in-memory profile that was decoupled from
+				 * the persitant storage with NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_DETACHED.
+				 * We need to take over this storage again... */
+				break;
+			}
+		}
+	}
+
+	if (   shadowed_storage
+	    && !new_in_memory) {
+		NMSettingsStorage *conflicting_storage;
+
+		conflicting_storage = _sett_conn_entry_storage_find_conflicting_storage (sett_conn_entry,
+		                                                                         nm_settings_storage_get_plugin (shadowed_storage),
+		                                                                         shadowed_storage,
+		                                                                         priv->plugins);
+		if (conflicting_storage) {
+			/* We cannot add the profile as @shadowed_storage, because there is another, existing storage
+			 * that would hide it. Just add it as new storage. In general, this leads to duplication of profiles,
+			 * but the circumstances where this happens are very exotic (you need at least one additional settings
+			 * plugin, then going through the paths of making shadowed_storage in-memory-detached and delete it,
+			 * and finally adding the conflicting storage outside of NM and restart/reload). */
+			_LOGT ("ignore shadowed storage "NM_SETTINGS_STORAGE_PRINT_FMT" due to conflicting storage "NM_SETTINGS_STORAGE_PRINT_FMT,
+			       NM_SETTINGS_STORAGE_PRINT_ARG (shadowed_storage),
+			       NM_SETTINGS_STORAGE_PRINT_ARG (conflicting_storage));
+		} else
+			update_storage = shadowed_storage;
+	}
+
+	shadowed_storage_filename =   (   shadowed_storage
+	                               && !update_storage)
+	                            ? nm_settings_storage_get_filename_for_shadowed_storage (shadowed_storage)
+	                            : NULL;
+
+again_add_connection:
+
+	if (!update_storage) {
+		success = _add_connection_to_first_plugin (self,
+		                                           sett_conn_entry,
+		                                           connection,
+		                                           new_in_memory,
+		                                           sett_flags,
+		                                           shadowed_storage_filename,
+		                                           FALSE,
+		                                           &new_storage,
+		                                           &new_connection,
+		                                           &local);
+	} else {
+		success = _update_connection_to_plugin (self,
+		                                        update_storage,
+		                                        connection,
+		                                        sett_flags,
+		                                        FALSE,
+		                                        shadowed_storage_filename,
+		                                        FALSE,
+		                                        &new_storage,
+		                                        &new_connection,
+		                                        &local);
+		if (!success) {
+			if (!NMS_IS_KEYFILE_STORAGE (update_storage)) {
+				/* hm, the intended storage is not keyfile (it's ifcfg-rh). This settings
+				 * plugin may not support the new connection. So step back and retry adding
+				 * the profile anew. */
+				_LOGT ("failure to add profile as existing storage \"%s\": %s",
+				       nm_settings_storage_get_filename (update_storage),
+				       local->message);
+				update_storage = NULL;
+				g_clear_object (&shadowed_storage);
+				shadowed_storage_filename = NULL;
+				g_clear_error (&local);
+				goto again_add_connection;
+			}
+		}
+	}
+
+	if (!success) {
+		if (!update_storage) {
+			g_set_error (error,
+			             NM_SETTINGS_ERROR,
+			             NM_SETTINGS_ERROR_FAILED,
+			             "failure adding connection: %s",
+			             local->message);
+		} else {
+			g_set_error (error,
+			             NM_SETTINGS_ERROR,
+			             NM_SETTINGS_ERROR_FAILED,
+			             "failure writing connection to existing storage \"%s\": %s",
+			             nm_settings_storage_get_filename (update_storage),
+			             local->message);
+		}
 		return FALSE;
 	}
 
@@ -1620,29 +1789,45 @@ nm_settings_add_connection (NMSettings *self,
 
 	c_list_for_each_entry (sd, &sett_conn_entry->sd_lst_head, sd_lst) {
 		const NMSettingsMetaData *meta_data;
+		gs_unref_object NMSettingsStorage *new_tombstone_storage = NULL;
+		gboolean in_memory;
+		gboolean simulate;
 
 		meta_data = nm_settings_storage_is_meta_data_alive (sd->storage);
-
 		if (   !meta_data
 		    || !meta_data->is_tombstone)
 			continue;
 
-		if (nm_settings_storage_is_keyfile_run (sd->storage)) {
-			/* We remove this file from /run. */
-		} else {
+		if (nm_settings_storage_is_keyfile_run (sd->storage))
+			in_memory = TRUE;
+		else {
 			if (nm_settings_storage_is_keyfile_run (new_storage)) {
 				/* Don't remove the file from /etc if we just wrote an in-memory connection */
 				continue;
 			}
+			in_memory = FALSE;
 		}
 
-		nm_settings_plugin_delete_connection (nm_settings_storage_get_plugin (sd->storage),
-		                                      sd->storage,
-		                                      NULL);
-
-		nm_assert (!_storage_data_is_alive (sd));
-
-		_connection_changed_track (self, sd->storage, NULL, FALSE);
+		simulate = FALSE;
+again_delete_tombstone:
+		if (!nms_keyfile_plugin_set_nmmeta_tombstone (priv->keyfile_plugin,
+		                                              simulate,
+		                                              uuid,
+		                                              in_memory,
+		                                              FALSE,
+		                                              NULL,
+		                                              &new_tombstone_storage,
+		                                              NULL)) {
+			/* Ups, something went wrong. We really need to get rid of the tombstone. At least
+			 * forget about it in-memory. Upong next restart/reload, this might be reverted
+			 * however :( .*/
+			if (!simulate) {
+				simulate = TRUE;
+				goto again_delete_tombstone;
+			}
+		}
+		if (new_tombstone_storage)
+			_connection_changed_track (self, new_tombstone_storage, NULL, FALSE);
 	}
 
 	_connection_changed_process_all_dirty (self,
@@ -1682,11 +1867,13 @@ nm_settings_update_connection (NMSettings *self,
 	NMConnection *new_connection_real;
 	gs_unref_object NMSettingsStorage *cur_storage = NULL;
 	gs_unref_object NMSettingsStorage *new_storage = NULL;
+	NMSettingsStorage *drop_storage = NULL;
+	SettConnEntry *sett_conn_entry;
 	gboolean cur_in_memory;
 	gboolean new_in_memory;
 	const char *uuid;
-	const char *shadowed_storage;
-	gboolean shadowed_owned;
+	gboolean tombstone_in_memory = FALSE;
+	gboolean tombstone_on_disk = FALSE;
 
 	g_return_val_if_fail (NM_IS_SETTINGS (self), FALSE);
 	g_return_val_if_fail (NM_IS_SETTINGS_CONNECTION (sett_conn), FALSE);
@@ -1697,6 +1884,7 @@ nm_settings_update_connection (NMSettings *self,
 	nm_assert (NM_IN_SET (persist_mode, NM_SETTINGS_CONNECTION_PERSIST_MODE_KEEP,
 	                                    NM_SETTINGS_CONNECTION_PERSIST_MODE_NO_PERSIST,
 	                                    NM_SETTINGS_CONNECTION_PERSIST_MODE_TO_DISK,
+	                                    NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY,
 	                                    NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_DETACHED,
 	                                    NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_ONLY));
 
@@ -1705,7 +1893,10 @@ nm_settings_update_connection (NMSettings *self,
 	uuid = nm_settings_storage_get_uuid (cur_storage);
 
 	nm_assert (NM_IS_SETTINGS_STORAGE (cur_storage));
-	nm_assert (_sett_conn_entry_get_conn (_sett_conn_entries_get (self, uuid)) == sett_conn);
+
+	sett_conn_entry = _sett_conn_entries_get (self, uuid);
+
+	nm_assert (_sett_conn_entry_get_conn (sett_conn_entry) == sett_conn);
 
 	if (connection) {
 		gs_free_error GError *local = NULL;
@@ -1736,7 +1927,7 @@ nm_settings_update_connection (NMSettings *self,
 
 	if (persist_mode == NM_SETTINGS_CONNECTION_PERSIST_MODE_KEEP) {
 		persist_mode =   cur_in_memory
-		               ? NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_DETACHED
+		               ? NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY
 		               : NM_SETTINGS_CONNECTION_PERSIST_MODE_TO_DISK;
 	}
 
@@ -1757,10 +1948,14 @@ nm_settings_update_connection (NMSettings *self,
 
 			default_wired_clear_tag (self, device, sett_conn, FALSE);
 
-			if (NM_IN_SET (persist_mode, NM_SETTINGS_CONNECTION_PERSIST_MODE_KEEP,
-			                             NM_SETTINGS_CONNECTION_PERSIST_MODE_NO_PERSIST)) {
-				/* making a default-wired-connection a regulard connection implies persisting
-				 * it to disk (unless specified differently). */
+			if (NM_IN_SET (persist_mode, NM_SETTINGS_CONNECTION_PERSIST_MODE_NO_PERSIST)) {
+				/* making a default-wired-connection a regular connection implies persisting
+				 * it to disk (unless specified differently).
+				 *
+				 * Actually, this line is probably unreached, because we should not use
+				 * NM_SETTINGS_CONNECTION_PERSIST_MODE_NO_PERSIST to toggle the nm-generated
+				 * flag. */
+				nm_assert_not_reached ();
 				persist_mode = NM_SETTINGS_CONNECTION_PERSIST_MODE_TO_DISK;
 			}
 		}
@@ -1776,12 +1971,13 @@ nm_settings_update_connection (NMSettings *self,
 		 * in-memory. The caller did not request to persist this to disk, however we need
 		 * to store the flags in run. */
 		nm_assert (cur_in_memory);
-		persist_mode = NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_DETACHED;
+		persist_mode = NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY;
 	}
 
 	if (persist_mode == NM_SETTINGS_CONNECTION_PERSIST_MODE_TO_DISK)
 		new_in_memory = FALSE;
-	else if (NM_IN_SET (persist_mode, NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_DETACHED,
+	else if (NM_IN_SET (persist_mode, NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY,
+	                                  NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_DETACHED,
 	                                  NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_ONLY))
 		new_in_memory = TRUE;
 	else {
@@ -1806,51 +2002,93 @@ nm_settings_update_connection (NMSettings *self,
 		                | NM_SETTINGS_CONNECTION_INT_FLAGS_VOLATILE);
 	}
 
-	/* TODO: set and handle shadowed-storages. */
-	shadowed_storage = NULL;
-	shadowed_owned = FALSE;
-
 	if (persist_mode == NM_SETTINGS_CONNECTION_PERSIST_MODE_NO_PERSIST) {
 		new_storage = g_object_ref (cur_storage);
-		new_connection = g_object_ref (connection);
+		new_connection_real = connection;
 		_LOGT ("update[%s]: %s: update profile \"%s\" (not persisted)",
 		       nm_settings_storage_get_uuid (cur_storage),
 		       log_context_name,
 		       nm_connection_get_id (connection));
 	} else {
-		gboolean success;
-		gboolean migrate_storage;
+		NMSettingsStorage *shadowed_storage;
+		const char *cur_shadowed_storage_filename;
+		const char *new_shadowed_storage_filename = NULL;
+		gboolean cur_shadowed_owned;
+		gboolean new_shadowed_owned = FALSE;
+		NMSettingsStorage *update_storage = NULL;
 		gs_free_error GError *local = NULL;
+		gboolean success;
+
+		cur_shadowed_storage_filename = nm_settings_storage_get_shadowed_storage (cur_storage, &cur_shadowed_owned);
+
+		shadowed_storage = _sett_conn_entry_find_shadowed_storage (sett_conn_entry, cur_shadowed_storage_filename, cur_storage);
+		if (!shadowed_storage) {
+			cur_shadowed_storage_filename = NULL;
+			cur_shadowed_owned = FALSE;
+		}
 
-		if (new_in_memory != cur_in_memory)
-			migrate_storage = TRUE;
-		else if  (   !new_in_memory
-		          && nm_settings_storage_is_keyfile_lib (cur_storage)) {
+		if (   new_in_memory
+		    && persist_mode == NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_ONLY) {
+			if (cur_in_memory) {
+				drop_storage = shadowed_storage;
+				update_storage = cur_storage;
+			} else
+				drop_storage = cur_storage;
+		} else if (   !new_in_memory
+		           && cur_in_memory
+		           && shadowed_storage) {
+			drop_storage = cur_storage;
+			update_storage = shadowed_storage;
+		} else if (new_in_memory != cur_in_memory) {
+			if (!new_in_memory)
+				drop_storage = cur_storage;
+			else if (persist_mode == NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_ONLY)
+				drop_storage = cur_storage;
+			else {
+				nm_assert (NM_IN_SET (persist_mode, NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY,
+				                                    NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_DETACHED));
+			}
+		} else if (nm_settings_storage_is_keyfile_lib (cur_storage)) {
 			/* the profile is a keyfile in /usr/lib. It cannot be overwritten, we must migrate it
 			 * from /usr/lib to /etc. */
-			migrate_storage = TRUE;
 		} else
-			migrate_storage = FALSE;
+			update_storage = cur_storage;
+
+		if (new_in_memory) {
+			if (persist_mode == NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_ONLY) {
+				/* pass */
+			} else if (!cur_in_memory) {
+				new_shadowed_storage_filename = nm_settings_storage_get_filename_for_shadowed_storage (cur_storage);
+				if (   new_shadowed_storage_filename
+				    && persist_mode != NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_DETACHED)
+					new_shadowed_owned = TRUE;
+			} else {
+				new_shadowed_storage_filename = cur_shadowed_storage_filename;
+				if (   new_shadowed_storage_filename
+				    && persist_mode == NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY)
+					new_shadowed_owned = TRUE;
+			}
+		}
 
-		if (migrate_storage) {
+		if (!update_storage) {
 			success = _add_connection_to_first_plugin (self,
-			                                           _sett_conn_entries_get (self, uuid),
+			                                           sett_conn_entry,
 			                                           connection,
 			                                           new_in_memory,
 			                                           sett_flags,
-			                                           shadowed_storage,
-			                                           shadowed_owned,
+			                                           new_shadowed_storage_filename,
+			                                           new_shadowed_owned,
 			                                           &new_storage,
 			                                           &new_connection,
 			                                           &local);
 		} else {
 			success = _update_connection_to_plugin (self,
-			                                        cur_storage,
+			                                        update_storage,
 			                                        connection,
 			                                        sett_flags,
 			                                        update_reason,
-			                                        shadowed_storage,
-			                                        shadowed_owned,
+			                                        new_shadowed_storage_filename,
+			                                        new_shadowed_owned,
 			                                        &new_storage,
 			                                        &new_connection,
 			                                        &local);
@@ -1864,7 +2102,7 @@ nm_settings_update_connection (NMSettings *self,
 			       nm_settings_storage_get_uuid (cur_storage),
 			       log_context_name,
 			       ignore_failure ? "ignore " : "",
-			       migrate_storage ? "write" : "update",
+			       update_storage ? "update" : "write",
 			       nm_connection_get_id (connection),
 			       local->message);
 			if (!ignore_failure) {
@@ -1872,76 +2110,77 @@ nm_settings_update_connection (NMSettings *self,
 				             NM_SETTINGS_ERROR,
 				             NM_SETTINGS_ERROR_INVALID_CONNECTION,
 				             "failed to %s connection: %s",
-				             migrate_storage ? "write" : "update",
+				             update_storage ? "update" : "write",
 				             local->message);
 				return FALSE;
 			}
+
 			new_storage = g_object_ref (cur_storage);
-			new_connection = g_object_ref (connection);
+			new_connection_real = connection;
 		} else {
+			gs_unref_variant GVariant *agent_owned_secrets = NULL;
+
 			_LOGT ("update[%s]: %s: %s profile \"%s\"",
 			       nm_settings_storage_get_uuid (cur_storage),
 			       log_context_name,
-			       migrate_storage ? "write" : "update",
+			       update_storage ? "update" : "write",
 			       nm_connection_get_id (connection));
-		}
-	}
 
-	nm_assert_valid_settings_storage (NULL, new_storage);
-	nm_assert (NM_IS_CONNECTION (new_connection));
-	nm_assert (nm_streq (uuid, nm_settings_storage_get_uuid (new_storage)));
-
-	if (persist_mode == NM_SETTINGS_CONNECTION_PERSIST_MODE_NO_PERSIST)
-		new_connection_real = new_connection;
-	else {
-		gs_unref_variant GVariant *agent_owned_secrets = NULL;
-
-		agent_owned_secrets = nm_connection_to_dbus (connection,
-		                                               NM_CONNECTION_SERIALIZE_ONLY_SECRETS
-		                                             | NM_CONNECTION_SERIALIZE_WITH_SECRETS_AGENT_OWNED);
-		new_connection_real = _connection_changed_normalize_connection (new_storage,
-		                                                                new_connection,
-		                                                                agent_owned_secrets,
-		                                                                &new_connection_cloned);
-		if (!new_connection_real) {
-			nm_assert_not_reached ();
-			new_connection_real = new_connection;
+			nm_assert_valid_settings_storage (NULL, new_storage);
+			nm_assert (NM_IS_CONNECTION (new_connection));
+			nm_assert (nm_streq (uuid, nm_settings_storage_get_uuid (new_storage)));
+
+
+			agent_owned_secrets = nm_connection_to_dbus (connection,
+			                                               NM_CONNECTION_SERIALIZE_ONLY_SECRETS
+			                                             | NM_CONNECTION_SERIALIZE_WITH_SECRETS_AGENT_OWNED);
+			new_connection_real = _connection_changed_normalize_connection (new_storage,
+			                                                                new_connection,
+			                                                                agent_owned_secrets,
+			                                                                &new_connection_cloned);
+			if (!new_connection_real) {
+				nm_assert_not_reached ();
+				new_connection_real = new_connection;
+			}
 		}
 	}
 
+	nm_assert (NM_IS_SETTINGS_STORAGE (new_storage));
 	nm_assert (NM_IS_CONNECTION (new_connection_real));
 
 	_connection_changed_track (self, new_storage, new_connection_real, TRUE);
 
-	if (new_storage != cur_storage) {
+	if (   drop_storage
+	    && drop_storage != new_storage) {
 		gs_free_error GError *local = NULL;
-		gboolean remove_from_disk;
 
-		if (persist_mode == NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_DETACHED)
-			remove_from_disk = FALSE;
-		else if (nm_settings_storage_is_keyfile_lib (cur_storage))
-			remove_from_disk = FALSE;
-		else
-			remove_from_disk = TRUE;
-
-		if (remove_from_disk) {
-			if (!nm_settings_plugin_delete_connection (nm_settings_storage_get_plugin (cur_storage),
-			                                           cur_storage,
-			                                           &local)) {
-				const char *filename;
-
-				filename = nm_settings_storage_get_filename (cur_storage);
-				_LOGW ("update[%s]: failed to delete moved storage "NM_SETTINGS_STORAGE_PRINT_FMT"%s%s%s: %s",
-				       nm_settings_storage_get_uuid (cur_storage),
-				       NM_SETTINGS_STORAGE_PRINT_ARG (cur_storage),
-				       local->message,
-				       NM_PRINT_FMT_QUOTED (filename, " (file \"", filename, "\")", ""));
-			}
+		if (!nm_settings_plugin_delete_connection (nm_settings_storage_get_plugin (drop_storage),
+		                                           drop_storage,
+		                                           &local)) {
+			const char *filename;
 
-			_connection_changed_track (self, cur_storage, NULL, FALSE);
-		}
+			filename = nm_settings_storage_get_filename (drop_storage);
+			_LOGT ("update[%s]: failed to delete moved storage "NM_SETTINGS_STORAGE_PRINT_FMT"%s%s%s: %s",
+			       nm_settings_storage_get_uuid (drop_storage),
+			       NM_SETTINGS_STORAGE_PRINT_ARG (drop_storage),
+			       NM_PRINT_FMT_QUOTED (filename, " (file \"", filename, "\")", ""),
+			       local->message);
+			/* there is no aborting back form this. We must get rid of the connection and
+			 * cannot do better than log a message. Proceed, but remember to write tombstones. */
+			if (nm_settings_storage_is_keyfile_run (cur_storage))
+				tombstone_in_memory = TRUE;
+			else
+				tombstone_on_disk = TRUE;
+		} else
+			_connection_changed_track (self, drop_storage, NULL, FALSE);
 	}
 
+	_set_nmmeta_tombstone (self,
+	                       uuid,
+	                       tombstone_on_disk,
+	                       tombstone_in_memory,
+	                       NULL);
+
 	_connection_changed_process_all_dirty (self,
 	                                       FALSE,
 	                                       sett_flags,
@@ -1957,23 +2196,24 @@ nm_settings_delete_connection (NMSettings *self,
                                NMSettingsConnection *sett_conn,
                                gboolean allow_add_to_no_auto_default)
 {
-	NMSettingsPrivate *priv;
 	NMSettingsStorage *cur_storage;
+	NMSettingsStorage *shadowed_storage;
+	NMSettingsStorage *shadowed_storage_unowned = NULL;
+	NMSettingsStorage *drop_storages[2] = { };
 	gs_free_error GError *local = NULL;
 	SettConnEntry *sett_conn_entry;
+	const char *cur_shadowed_storage_filename;
+	const char *new_shadowed_storage_filename = NULL;
+	gboolean cur_shadowed_owned;
 	const char *uuid;
-	gboolean delete;
 	gboolean tombstone_in_memory = FALSE;
 	gboolean tombstone_on_disk = FALSE;
-	gs_unref_object NMSettingsStorage *tombstone_1_storage = NULL;
-	gs_unref_object NMSettingsStorage *tombstone_2_storage = NULL;
+	int i;
 
 	g_return_if_fail (NM_IS_SETTINGS (self));
 	g_return_if_fail (NM_IS_SETTINGS_CONNECTION (sett_conn));
 	g_return_if_fail (nm_settings_has_connection (self, sett_conn));
 
-	priv = NM_SETTINGS_GET_PRIVATE (self);
-
 	cur_storage = nm_settings_connection_get_storage (sett_conn);
 
 	nm_assert (NM_IS_SETTINGS_STORAGE (cur_storage));
@@ -1992,39 +2232,63 @@ nm_settings_delete_connection (NMSettings *self,
 
 		if (NM_IN_SET (s->storage_type, NMS_KEYFILE_STORAGE_TYPE_RUN,
 		                                NMS_KEYFILE_STORAGE_TYPE_ETC))
-			delete = TRUE;
-		else {
+			drop_storages[0] = cur_storage;
+		else
 			tombstone_on_disk = TRUE;
-			delete = FALSE;
-		}
 	} else
-		delete = TRUE;
+		drop_storages[0] = cur_storage;
+
+	cur_shadowed_storage_filename = nm_settings_storage_get_shadowed_storage (cur_storage, &cur_shadowed_owned);
 
-	if (delete) {
+	shadowed_storage = _sett_conn_entry_find_shadowed_storage (sett_conn_entry, cur_shadowed_storage_filename, cur_storage);
+	if (shadowed_storage) {
+		if (!cur_shadowed_owned)
+			shadowed_storage_unowned = g_steal_pointer (&shadowed_storage);
+	}
+	drop_storages[1] = shadowed_storage;
+
+	for (i = 0; i < (int) G_N_ELEMENTS (drop_storages); i++) {
+		NMSettingsStorage *storage;
 		StorageData *sd;
 
-		if (!nm_settings_plugin_delete_connection (nm_settings_storage_get_plugin (cur_storage),
-		                                           cur_storage,
+		storage = drop_storages[i];
+		if (!storage)
+			continue;
+
+		if (!nm_settings_plugin_delete_connection (nm_settings_storage_get_plugin (storage),
+		                                           storage,
 		                                           &local)) {
-			_LOGW ("delete-connection: failed to delete storage "NM_SETTINGS_STORAGE_PRINT_FMT": %s",
-			       NM_SETTINGS_STORAGE_PRINT_ARG (cur_storage),
+			_LOGT ("delete-connection: failed to delete storage "NM_SETTINGS_STORAGE_PRINT_FMT": %s",
+			       NM_SETTINGS_STORAGE_PRINT_ARG (storage),
 			       local->message);
 			g_clear_error (&local);
 			/* there is no aborting back form this. We must get rid of the connection and
-			 * cannot do better than warn. Proceed... */
-			tombstone_in_memory = TRUE;
-		}
-
-		sett_conn_entry = _connection_changed_track (self, cur_storage, NULL, FALSE);
+			 * cannot do better than log a message. Proceed, but remember to write tombstones. */
+			if (nm_settings_storage_is_keyfile_run (cur_storage))
+				tombstone_in_memory = TRUE;
+			else
+				tombstone_on_disk = TRUE;
+			sett_conn_entry = _sett_conn_entries_get (self, uuid);
+		} else
+			sett_conn_entry = _connection_changed_track (self, storage, NULL, FALSE);
 
 		c_list_for_each_entry (sd, &sett_conn_entry->sd_lst_head, sd_lst) {
-			if (sd->storage == cur_storage)
+			if (NM_IN_SET (sd->storage, drop_storages[0],
+			                            drop_storages[1]))
 				continue;
 			if (!_storage_data_is_alive (sd))
 				continue;
 			if (nm_settings_storage_is_meta_data (sd->storage))
 				continue;
 
+			if (sd->storage == shadowed_storage_unowned) {
+				/* this only happens if we leak a profile on disk after NM_SETTINGS_CONNECTION_PERSIST_MODE_IN_MEMORY_DETACHED.
+				 * We need to write a tombstone and remember the shadowed-storage. */
+				tombstone_in_memory = TRUE;
+				new_shadowed_storage_filename = nm_settings_storage_get_filename (shadowed_storage_unowned);
+				continue;
+			}
+
 			/* we have still conflicting storages. We need to hide them with tombstones. */
 			if (nm_settings_storage_is_keyfile_run (sd->storage)) {
 				tombstone_in_memory = TRUE;
@@ -2034,40 +2298,11 @@ nm_settings_delete_connection (NMSettings *self,
 		}
 	}
 
-	if (tombstone_on_disk) {
-		if (!nms_keyfile_plugin_set_nmmeta_tombstone (priv->keyfile_plugin,
-		                                              FALSE,
-		                                              uuid,
-		                                              FALSE,
-		                                              TRUE,
-		                                              NULL,
-		                                              &tombstone_1_storage,
-		                                              NULL))
-			tombstone_in_memory = TRUE;
-		if (tombstone_1_storage)
-			_connection_changed_track (self, tombstone_1_storage, NULL, FALSE);
-	}
-
-	if (tombstone_in_memory) {
-		if (!nms_keyfile_plugin_set_nmmeta_tombstone (priv->keyfile_plugin,
-		                                              FALSE,
-		                                              uuid,
-		                                              TRUE,
-		                                              TRUE,
-		                                              NULL,
-		                                              &tombstone_2_storage,
-		                                              NULL)) {
-			nms_keyfile_plugin_set_nmmeta_tombstone (priv->keyfile_plugin,
-			                                         TRUE,
-			                                         uuid,
-			                                         TRUE,
-			                                         TRUE,
-			                                         NULL,
-			                                         &tombstone_2_storage,
-			                                         NULL);
-		}
-		_connection_changed_track (self, tombstone_2_storage, NULL, FALSE);
-	}
+	_set_nmmeta_tombstone (self,
+	                       uuid,
+	                       tombstone_on_disk,
+	                       tombstone_in_memory,
+	                       new_shadowed_storage_filename);
 
 	_connection_changed_process_all_dirty (self,
 	                                       allow_add_to_no_auto_default,