SERVER-65047 Strengthen guarantees for resharding aborting across FCVs

Previously, resharding operations were aborted after sending the request
to finalize the FCV version on the shards when upgrading, making it is
possible that a resharding recipient will update its FCV to its target
value prior to aborting. It was therefore not possible to differentiate
a resharding operation that has upgraded to the latest version since
starting from one that has been the latest version throughout. For this
reason, the resharding operations are now aborted first.

Furthermore, the resharding command currently ensures that the FCV
cannot change while setting up the coordinator. However, it did not
check to make sure that the current FCV is not currently in an upgrading
or downgrading state. After making the above change, this would allow
for the possibility for a new resharding operation to begin during an
FCV upgrade, after resharding operations are aborted, but before the
shards complete the FCV upgrade. This would have the consequence of the
operation running across FCVs without being aborted. As such, the
reshard command now fails if the current FCV is either upgrading or
downgrading.

These changes in combination should guarantee that during a version
change, a new resharding operation cannot begin and a previously running
resharding operation always aborts completely before reaching the target
version. Note that it is still possible for a resharding operation to
reach an upgrading or downgrading FCV before being aborted.

These changes were made in the interest of being able to assert that
newly added optional fields that should always be set were indeed set.
As such, this change also enables the assertion disabled by
SERVER-65039.

4 files changed, 20 insertions, 8 deletions

diff --git a/jstests/multiVersion/genericSetFCVUsage/setfcv_reshard_collection.js b/jstests/multiVersion/genericSetFCVUsage/setfcv_reshard_collection.js
index 1228ca42889..7fa818f7f99 100644
--- a/jstests/multiVersion/genericSetFCVUsage/setfcv_reshard_collection.js
+++ b/jstests/multiVersion/genericSetFCVUsage/setfcv_reshard_collection.js
@@ -111,6 +111,7 @@ function runTest(forcePooledConnectionsDropped) {
         },
         {
             expectedErrorCode: [
+                ErrorCodes.CommandNotSupported,
                 ErrorCodes.ReshardCollectionAborted,
                 ErrorCodes.Interrupted,
             ]
diff --git a/src/mongo/db/commands/set_feature_compatibility_version_command.cpp b/src/mongo/db/commands/set_feature_compatibility_version_command.cpp
index 01ce1c396b7..0212a664965 100644
--- a/src/mongo/db/commands/set_feature_compatibility_version_command.cpp
+++ b/src/mongo/db/commands/set_feature_compatibility_version_command.cpp
@@ -568,6 +568,11 @@ private:
                 !failUpgrading.shouldFail());
 
         if (serverGlobalParams.clusterRole == ClusterRole::ConfigServer) {
+            // Always abort the reshardCollection regardless of version to ensure that it will run
+            // on a consistent version from start to finish. This will ensure that it will be able
+            // to apply the oplog entries correctly.
+            abortAllReshardCollection(opCtx);
+
             // Tell the shards to enter phase-2 of setFCV (fully upgraded)
             auto requestPhase2 = request;
             requestPhase2.setFromConfigServer(true);
@@ -576,11 +581,6 @@ private:
             uassertStatusOK(
                 ShardingCatalogManager::get(opCtx)->setFeatureCompatibilityVersionOnShards(
                     opCtx, CommandHelpers::appendMajorityWriteConcern(requestPhase2.toBSON({}))));
-
-            // Always abort the reshardCollection regardless of version to ensure that it will run
-            // on a consistent version from start to finish. This will ensure that it will be able
-            // to apply the oplog entries correctly.
-            abortAllReshardCollection(opCtx);
         }
 
         // Create the pre-images collection if the feature flag is enabled on the requested version.
diff --git a/src/mongo/db/s/config/configsvr_reshard_collection_cmd.cpp b/src/mongo/db/s/config/configsvr_reshard_collection_cmd.cpp
index 4e5f53f72cc..f8a12c57f90 100644
--- a/src/mongo/db/s/config/configsvr_reshard_collection_cmd.cpp
+++ b/src/mongo/db/s/config/configsvr_reshard_collection_cmd.cpp
@@ -169,6 +169,13 @@ public:
                     return existingInstance;
                 }
 
+                // (Generic FCV reference): To run this command and ensure the consistency of the
+                // metadata we need to make sure we are on a stable state.
+                uassert(ErrorCodes::CommandNotSupported,
+                        "Resharding is not supported for this version, please update the FCV to "
+                        "latest.",
+                        !serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading());
+
                 const auto cm = uassertStatusOK(
                     Grid::get(opCtx)->catalogCache()->getShardedCollectionRoutingInfoWithRefresh(
                         opCtx, nss));
diff --git a/src/mongo/db/s/resharding/resharding_metrics_new.cpp b/src/mongo/db/s/resharding/resharding_metrics_new.cpp
index 6c80bd77e47..1d2f883d46c 100644
--- a/src/mongo/db/s/resharding/resharding_metrics_new.cpp
+++ b/src/mongo/db/s/resharding/resharding_metrics_new.cpp
@@ -61,11 +61,15 @@ BSONObj createOriginalCommand(const NamespaceString& nss, BSONObj shardKey) {
 }
 
 Date_t readStartTime(const CommonReshardingMetadata& metadata, ClockSource* fallbackSource) {
-    const auto& startTime = metadata.getStartTime();
-    if (startTime.has_value()) {
+    try {
+        const auto& startTime = metadata.getStartTime();
+        tassert(6503901,
+                "Metadata is missing start time despite feature flag being enabled",
+                startTime.has_value());
         return startTime.get();
+    } catch (const DBException&) {
+        return fallbackSource->now();
     }
-    return fallbackSource->now();
 }
 
 }  // namespace