sql: Integrate alembic

Switch to alembic for real by integrating it into the 'db sync' command
flow. From a user-facing perspective, things should remain pretty much
the same as before, with the key difference being that version
information (i.e. what's shown by 'keystone-manage db_sync --check' or
'keystone-manage db_version') will now take the form of a hash rather
than an integer. There are a few differences for contributors however.
The changes are described in the included release note and
documentation.

Note that there are a couple of important design decisions here that are
worth examining:

- We drop the idea of the 'data_migration' branch entirely and the
  'keystone-manage db_sync --migrate' command is now a no-op. Neutron
  doesn't do data migrations like we do and yet they manage just fine.
  Dropping this gets us closer to neutron's behavior, which is a good
  thing for users.

- We haven't re-added the ability to specify a version when doing
  'db_sync'. Neutron has this, but the logic needed to get this working
  is complex and of questionable value. We've managed without the
  ability to sync to a version since Newton and can continue to do so
  until someone asks for it (and does the work).

- sqlalchemy-migrate is not removed entirely. Instead, upon doing a
  'db_sync' we will apply all sqlalchemy-migrate migrations up to the
  final '079_expand_update_local_id_limit' migration and dummy apply the
  initial alembic migration, after which we will switch over to alembic.
  In a future release we can remove the sqlalchemy-migrate migrations
  and rely entirely on alembic. Until then, keeping this allows fast
  forward upgrades to continue as a thing.

- Related to the above, we always apply *all* sqlalchemy-migrate
  migrations when calling 'db_sync', even if this command is called with
  e.g. '--expand' (meaning only apply the expand branch). This is
  because there is at most one "real" migration to apply, the Xena-era
  '079_expand_update_local_id_limit' migration, which is an expand-only
  migration. There is no risk to applying the empty "data_migration" and
  "contract" parts of this migration, and applying everything in one go
  results in *much* simpler logic.

Future changes will update documentation and add developer tooling for
(auto-)generating new migrations, a la 'neutron-db-manage revision'.

Change-Id: Ia376cb87f5159a4e79e2cfbab8442b6bcead708f
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

1 files changed, 15 insertions, 6 deletions

diff --git a/keystone/common/sql/migrations/env.py b/keystone/common/sql/migrations/env.py
index 2d116f1bd..f5547a4e4 100644
--- a/keystone/common/sql/migrations/env.py
+++ b/keystone/common/sql/migrations/env.py
@@ -59,15 +59,24 @@ def run_migrations_online():
     In this scenario we need to create an Engine and associate a connection
     with the context.
     """
-    connectable = engine_from_config(
-        config.get_section(config.config_ini_section),
-        prefix="sqlalchemy.",
-        poolclass=pool.NullPool,
-    )
+    connectable = config.attributes.get('connection', None)
+
+    if connectable is None:
+        # only create Engine if we don't have a Connection from the outside
+        connectable = engine_from_config(
+            config.get_section(config.config_ini_section),
+            prefix="sqlalchemy.",
+            poolclass=pool.NullPool,
+        )
+
+    # when connectable is already a Connection object, calling connect() gives
+    # us a *branched connection*.
 
     with connectable.connect() as connection:
         context.configure(
-            connection=connection, target_metadata=target_metadata
+            connection=connection,
+            target_metadata=target_metadata,
+            render_as_batch=True,
         )
 
         with context.begin_transaction():