| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
| |
In change I1e71150ba6daeba464b6ed8d46163f1f34959db3 we removed the
legacy base test classes, first deprecated in 2015. We forgot to include
a release note, however. Address this now.
Change-Id: I4d66f0308b89a187143ef6c8495383fe60043c14
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Remaining issues encountered when running with SQLAlchemy 2.0 for real:
* Never call str() on a URL and expect it to be meaningful anymore.
The password is aggressively obfuscated now (users absolultely
wouldn't let us leave it as is)
* More utilities and fixtures that were calling begin() within a
block that would have already begun
* isnot is now called is_not; mocking "isnot" leads into too many
weird compat layers
* ORM InstrumentedAttribute and internals use __slots__ now, mock
seems to not be able to patch methods. Ideally these tests would use
a comparator subclass or something
* Connection.connection.connection is now called driver_connection,
SQLAlchemy keeps the old name available however oslo.db test suite
does not appear to tolerate the deprecation warning emitted,
so add a compat layer
* mapper() is fully removed from 2.0, not sure if there is another
not-yet-committed gerrit that removes mapper()
[1] https://docs.sqlalchemy.org/en/20/core/engines.html#sqlalchemy.create_engine.params.pool_pre_ping
[2] https://docs.sqlalchemy.org/en/20/changelog/changelog_20.html#change-2fe37eaf2295cebd3bb4ee8e5b8c575c
[3] https://github.com/sqlalchemy/sqlalchemy/issues/5648
Change-Id: Ifaca67c07f008d8bc0febeecd3e200cc7ee7a4b0
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
sqlalchemy-migrate does not support SQLAlchemy 2.x and we're not going
to invest the time in changing this. Remove integration of
sqlalchemy-migrate in oslo.db, allowing us to support SQLAlchemy 2.x
fully.
Note that we do not remove the 'migration_cli' module entirely yet.
While this is deprecated, it is possible to use this with alembic.
New users shouldn't be switching to it, but any existing users can
continue to use this module for some time yet.
Change-Id: Ic3d6bd318038d723b0d50d39e45f8e26289e9a57
Sem-Ver: api-break
|
| |
|
|
|
|
|
|
|
|
| |
The default value for wsrep_sync_wait option should be `None`.
However, since 0 is set incorrectly, an unintended process is executed.
This patch fixes default value for wsrep_sync_wait option to `None`
instead of `0`.
Change-Id: Ifb1dc7ddcb127a69ea01234922caa7ca5ab111ce
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When using Galera, the wsrep_sync_wait option [1]
can change the behavior of a variety of Galera DQL/DML statements
such that a particular operation will pause until outstanding
write-sets are fully persisted to the local node. The setting
supersedes the previous boolean parameter wsrep_causal_reads
which only affected SELECT, with an updated approach that allows
for fine-grained control of so-called "causality checks"
on individual statement types. The legacy-compatible setting
of '1' indicates that READ/SELECT/BEGIN operations should
proceed only after any pending writesets are fully available.
The use case for this setting is for an application that
is running operations on multiple Galera nodes simultaenously.
An application that commits data on one node, and then immediately
uses a different connection (on a potentially different node)
to SELECT that data, may fail to see those changes if
"causality checks" for SELECT are not enabled. While
a COMMIT operation in Galera will block locally until all other
nodes approve of the writeset, the operation does not block
for the subsequent period of time when other nodes are actually
persisting that writeset. Setting up "causal reads"
in this case indicates that a SELECT operation will wait until
any writesets in progress are available, thus maintaining
serialization between the COMMIT and subsequent SELECT.
As the name implies, wsrep_sync_wait adds...waiting! to the
operation, and thus directly impacts performance by adding
latency to SELECT operations or to the operations that have
been selected for causality checks, to the degree that
concurrent writesets are expected to be present.
Since it's not expected that most if any Openstack applications
actually need this setting in order to be effective with
Galera multi-master operation, and as the setting is available
within client session scope and also impacts performance,
making it available on a per-application basis means that
specific applications which may see issues under load can
choose to enable this setting, much in the way any other
"transaction isolation" settings might be made, without having
to add a cluster-wide performance penalty by setting it at the
Galera server level.
[1] https://mariadb.com/docs/ent/ref/mdb/system-variables/wsrep_sync_wait/
Change-Id: Iee7afcac8ba952a2d67a9ad9dd0e4eae3f42518e
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Traditionally, the MySQL support in oslo.db has assumed use of the
InnoDB storage engine. However, this isn't the only storage engine
available and a few years ago an effort was made to add support for
another storage engine, MySQL Cluster (NDB). The oslo.db aspects of this
effort were tracked via bug 1564110 [1] and from reading this bug and
looking at other patches related to this effort [2], it becomes obvious
that this was never seen through to the completion and the
OpenStack-wide effort never took off [3]. As a result, much of what is
here is in-effect dead code now.
Given no one is using this engine, there's no reason to keep it around.
Deprecate it with an eye on removing it sooner rather than later.
[1] https://bugs.launchpad.net/oslo.db/+bug/1564110
[2] https://review.opendev.org/q/owner:octave.orgeron%2540oracle.com
[3] https://review.opendev.org/c/openstack/openstack-specs/+/429940
Change-Id: Id5ddf1d6f47b8a572001f58ad8b9b8a7dbe4e8ac
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
These are not compatible with SQLAlchemy 2.0 due to their reliance on
nested transactions. We should deprecate them first but doing so would
push the boat out further wrt how long we have to wait before achieving
compatibility with this new version.
Change-Id: If3db4e8c1b681c0c62d3f04a57f92802639b3b9b
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This hasn't been supported by any project for many years. Quoting from
the nova patch that removed DB2 support [1]
This removes db2 support from tree completely. This is an oddball non
open database that made doing live data migrations difficult. It is
used by 0% of users in the OpenStack User Survey.
Supporting commercial software that doesn't have users at the cost of
delivering features and fixes to our community is the wrong tradeoff.
This corrects that.
There's no need to keep this around.
[1] Ifeb9929e4515e3483eb65d371126afd7672b92a4
Change-Id: I8c3f23083e09e5ac924e35d1b7b3248a0d074e1b
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The 'check_foreign_key' method was removed in oslo.db 1.4.0 [1] and
re-added but marked as deprecated in 1.4.1 [2] to allow for a
deprecation period. That was over 6 years ago so we can now remove it.
Meanwhile, the private '_walk_versions', '_migrate_down' and
'_migrate_up' methods have been marked as deprecated since oslo.db 0.5.0
(!!!) [3]. We can definitely remove these now. codesearch.o.o reveals no
users of any of these methods.
[1] I77a2bba2de08125b5a601de03c1d2d5c73fa33ee
[2] Id892567bd60d6b4b88765bbfe3cd5c5e75910b25
[3] Ic9358445e60a0dd43a5900e8bda7b12f2bebf679
Change-Id: Idb9596284df3e5d038c939d27fe14fe2cc2f4318
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |/
|
|
|
|
|
|
|
| |
This has been marked as experimented since before oslo.db was split out
of nova. We've no test coverage and no idea of whether it even works
nowadays. It's time to plan for its eventual removal.
Change-Id: Ib7a5a6e1124c44a742b003442f815022a866bcbb
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The option was replaced 4 years ago[1] by connection_recycle_time. The
option is not anymore present in sqlalchemy. It's the good time to
remove it.
[1] https://opendev.org/openstack/oslo.db/commit/6634218415906192fca891362b8fa0ac50b66284
Change-Id: I0c26c3e14ad9ad9a4f30af80757e369d6042441a
|
| |/
|
|
|
|
| |
Followup of change: Ic78059b625e73cece355541cb4d89e641abc1103
Change-Id: Idb0410a1946acbd7caf6a0bb1dad229bd5c2c346
|
| |
|
|
|
|
|
|
| |
The option was replaced 6 years ago[1] by max_pool_size.
[1] https://opendev.org/openstack/oslo.db/commit/1b7c295d522e26aa8bf09242dcc1355c3fb1ab3e
Change-Id: Ice1a90edc32603c85449b80fe78fd437aa7e8081
|
| |
|
|
|
|
|
|
|
|
|
| |
This API allowed users to abstract the differences between
sqlalchemy-migrate and alembic. However, no one is using it for that and
its continued existence necessitates continued support for
sqlalchemy-migrate. Deprecate it now and encourage the odd user to
switch to alembic directly.
Change-Id: Ia5979369f23b38a2c7307f4112651a8bedb45c01
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |
|
|
|
|
|
|
|
| |
This module hasn't been touched with years and 'sqlalchemy-migrate' is
effectively abandoned. It's time to drop support for this an encourage
the laggards to switch to alembic and use it directly.
Change-Id: I59335b4f318bae2e29ab139cdea089a4d6e14305
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |
|
|
| |
Change-Id: I7bbab4f96391828d0b0e095a152bdacca6585e3c
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
OpenStack is dropping the py2.7 support in Ussuri cycle.
Complete discussion & schedule can be found in
-
http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010142.html
- https://etherpad.openstack.org/p/drop-python2-support
Ussuri Communtiy-wide goal:
https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html
Change-Id: I02e2d11d38d2aa4182e8d466519b99f05943701f
|
| |
|
|
|
|
|
|
|
|
| |
The database option min_pool_size is not actually
used in oslo.db.
This is a folllow up to
Ib07b7b4200b5cf3a34b8eed441e6754da7d446cf
Change-Id: I89754e9e4b42c45259729b0b0bb0a4bf464a981c
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some module reloading scenarios such as that which occurs
within mod_wsgi mean that an existing module-level enginefacade
is already in the "started" state, however initialization
routines from the calling application may still attempt to
call the ``.configure`` method. Add a new flag is_started
to both _TransactionContextManager and _TransactionFactory so
that calling code can check for this state ahead of time;
additionally, promote the TypeError raised to a specific subclass
enginefacade.AlreadyStartedError to allow for better optimistic
schemes.
Change-Id: I2f5a9e35c2fae0c28b78beef3dcd2c4794362766
References: I704196711d30c1124e713ac31111a8ea6fa2f1ba
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The Nova project has made the decision to store the entire
contents of the "sql_connection" field in the database, with
alterations to the host value, such that various "cells" database
URLs can be located using Nova's database itself as a central
registry of database URLs.
This architecture has produced several problems. The first
is that it is impossible to apply parameters to the URL that
are local to the client machine; the leading example of this
is the MySQL "bind_host" variable, which must match the hostname
of the connecting host. Because cells puts these URLs into the
database and shares them with all controllers, we have to use a workaround
with the read_default_file parameter to specify a controller-local
file of options; this is not a standard thing
for other database drivers, and these parameters only apply to
the MySQL driver and not the SQLAlchemy engine.
The next issue is that it is inconvenient to add parameters
to the URL at all, once Nova has already been running, as one
must manually use Nova's command line tools
to alter all the URLs that have already been copied into the database
and alter the query parameters with each of those individually,
and then restart *all* services who will all receive the parameter
(no way to add params to just one controller).
Nova's "store the URL in the database" feature only needs to be able
to locate the host / database name of the alternate database,
and not change the URL tuning parameters. This patch adds a new
oslo.db parameter connection_parameters which allows the params to
be applied separately from the sql_connection parameter, so that
Nova can continue persisting sql_connection but the parameters remain
local to the nova.conf file. A URL parameter that truly had to remain
persisted in Nova's database (there aren't any) could still be applied
at the sql_connection level.
This feature is essential not just so that we can again place
simple parameters into controller-local files like "bind_host", but
also to allow for configuration of SQLAlchemy features such as
plugins that do connection pool monitoring.
Change-Id: Id4de4b09ec4719cbf8b372629fcf58cf368a33d4
|
| |/
|
|
|
|
|
|
|
|
|
| |
Repaired the "synchronous_reader" modifier of enginefacade so that it
refers to the "writer" engine when set to True, thereby allowing
"synchronous" behavior with the writer. When set to False, this is
"asynchronous", so this should be associated with the async engines.
The flag had the reverse behavior previously.
Change-Id: Id7fea7562ba90eb710176d497af103303f230531
Closes-bug: #1746116
|
| |
|
|
|
|
|
|
|
| |
In order to fade out MySQL-python from OpenStack we need to stop
testing it. Add release notes entry and adjust documentation
references accordingly.
Depends-On: Ie1f07062ed18350bcbb9e7b5e33c7ab2390be9ab
Change-Id: Ie17f4543fa3d72b507d88d2c4023a9a5a430419e
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
The "idle_timeout" name and documentation are misleading to users.
Rename the option to connection_recycle_time to more closely
match that of the underlying SQLAlchemy argument name
"pool_recycle", which refers to a time period for which
a connection present in the pool for longer than that number
of seconds will be closed and replaced with a new one *upon checkout*,
to prevent usage of stale connections that may have been closed by
server-side reapers.
Change-Id: I5e39d5ac81eaa8d85b1dfb18862fe27695ab8b96
|
| |
|
|
|
| |
Change-Id: I7e48a45e392c57901d27389f5797c8ce34492861
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
We deprecated config options sqlite_db in July, 2016 [1], and
it shows up in many projects' config files but these projects
don't use it either, so it's confusing to our users. It's time
to remove it, just use config option connection or slave_connection
to connect to the database.
[1] https://review.openstack.org/#/c/338047/
Closes-Bug: #1329086
Change-Id: Id269d921e40edf95eb977b011f1753f633b79d18
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This is provided out-of-box by SQLAlchemy versions >= 1.0.0. We've
had this class deprecated in docs for some time now, now do it
properly by the means of debtcollector and a release note.
Closes-Bug: #1439092
Change-Id: I562c35c265014c6b19a81961f236a7a0ca438765
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The original idea of enginefacade was that applications
would call upon the global _TransactionContextManager
given in oslo_db.sqlalchemy.enginefacade. However, as it
turns out, virtually no Openstack projects seem to be
using that technique, and instead, everyone is creating
their own ad-hoc _TransactionContextManager objects
and usually establishing it as a module-level global.
Nova has two of them.
Additionally, projects add configuration to these
enginefacades (which IS part of the original idea), and
this configuration in some cases is necessary to be present
for tests that run as well, a key example being the
sqlite_fks flag. The original DbFixture integration
provided no way of reusing this configuration.
Finally, projects very much tend to use custom fixtures
in order to define their database communication.
Test classes themselves don't really make use of
oslo_db's DbTestCase anymore.
This patch introduces a modernized fixture system
which, in conjunction with the recent provisioning
patch, addresses these use cases. Applications will typically
create their own subclasses of these fixtures up front
to suit the various testing cases they have, including
SQLite fixed, SQLite ad-hoc, and opportunistic.
In order to accommodate the fixture-based flow
along with the use of testresources for opportunistic
database provisioning, a mixin class OpportunisticDbTestMixin
is still needed when a test needs to use "opportunistic"
testing in order to provide the .resources attribute.
The calculation of .resources is moved into the fixture
system, but because this attribute is consulted before
setUp(), the "opportunistic" fixture must be created
early and stored.
Closes-Bug: #1548960
Change-Id: I0163e637ffef6d45d2573ebe29b5438911d01fce
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
Just in case mention the changes done in CR #318135, so that people
are aware of those (although, we've double checked everyone passes
the argument value explicitly now and that's the only reason the
decorator actually performs any retries in downstreams).
Change-Id: I263b2078535a3f06461e8905003b484a38d05f08
|
| | |
| |
| |
| |
| |
| |
| | |
There is a set allowable minimum and maximum to the connection_debug
option of 0 and 100. The config option should enforce this.
Change-Id: Ica610ad8fc0875a8be1042f7daba5d0a0c1aad03
|
| |/
|
|
|
|
|
|
|
|
| |
We should use config option connection or slave_connection to connect
to the database. We can remove it when all consuming projects don't use
it anymore.
Related-Bug: #1329086
Change-Id: I1960e47b27b5f98f19a07f9899079269bac025cc
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The decorator form can now be used with bound methods after a fix
for the related bug was merged. Mention this in docs and release
notes, so that people are aware of it.
A new test is added to make sure this also works with class methods
(as well as to check that the proposed decorator applying order
is actually correct).
Related-Bug: #1520195
Change-Id: Ifea08114d6d89de9d67fcae397eb94c0afc4d339
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turned out that distribution of WSGI requests between forks is not
even when eventlet is used, which can cause pool timeout issues under
load, while there are idle workers with open DB connections, which
should have served these HTTP requests.
The default max_overflow value should be increased to allow DB oriented
services (mostly APIs) handle spikes in number of concurrent requests
trying to use a DB.
At the same time, the default number of greenlets is decreased:
I65b40b9906b75146a0085bbe168f1e6bcae82f21
which effectively causes a particular worker to stop accepting new
requests it has no resources (DB connections) to handle and allow
other forks accept()'ing on the same FD to proceed.
Testing was performed using this script:
https://gist.github.com/zzzeek/c69138fd0d0b3e553a1f
With 100 greenthreads in the pool, 50 DB connections allowed overflow
no pool timeout issues were seen with up to 500 concurrent requests
done by ab, while current default values (1000/10) could not handle
even 100 concurrent requests.
See this ML thread for details:
http://lists.openstack.org/pipermail/openstack-dev/2015-December/082717.html
DocImpact
This change potentially increases the number of connections open to the
RDBMS server. If you start seeing "too many connections" errors, please
check these settings and adjust them appropriately:
https://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_max_connections
http://www.postgresql.org/docs/current/static/runtime-config-connection.html#GUC-MAX-CONNECTIONS
Closes-Bug: #1535375
Change-Id: I2e9c2a71d8231e0dfbefc6293ad319e1e459beec
|
|
|
Change-Id: I6574d2bb21fa8c64290b06ab493f8f7c8b1b7c16
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
|
