summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/testing/engines.py
Commit message (Collapse)AuthorAgeFilesLines
* happy new year 2023Mike Bayer2023-01-031-1/+1
| | | | Change-Id: I14db8e9c69a832b0f5dae8036db3c0a70bb49edd
* test sqlite w/ savepoint workaround in session fixture testMike Bayer2022-03-071-0/+14
| | | | | | Fixes: #7795 Change-Id: Ib790581555656c088f86c00080c70d19ca295a03 (cherry picked from commit fbacb1991585202a5bf22acb0d36b5c979bcfad8)
* update zimportsMike Bayer2022-02-091-1/+3
| | | | | | | | | includes new fix for formatting like black does. also runs black on a few outliers. Change-Id: I67446660a6bc10b73eb710389ae6d3f122af9302 (cherry picked from commit 2579ed2b8f295c7e0ad3d875bf57535623f8df0d)
* Remove dispose warning on async engines when running testsFederico Caselli2022-01-211-1/+6
| | | | | | Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Change-Id: Ia3357959ed286dc7d2ce264b5ddcadf309351ff7 (cherry picked from commit 9b2cd1ede5951fff7180d64bb39aa3a601ec1900)
* happy new year 2022Mike Bayer2022-01-061-1/+1
| | | | Change-Id: Ic38dbc640aa0fe8a784a5b5e57c45a41eb0ea01b
* Surface driver connection object when using a proxied dialectFederico Caselli2021-09-171-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Improve the interface used by adapted drivers, like the asyncio ones, to access the actual connection object returned by the driver. The :class:`_engine._ConnectionRecord` and :class:`_engine._ConnectionFairy` now have two new attributes: * ``dbapi_connection`` always represents a DBAPI compatible object. For pep-249 drivers, this is the DBAPI connection as it always has been, previously accessed under the ``.connection`` attribute. For asyncio drivers that SQLAlchemy adapts into a pep-249 interface, the returned object will normally be a SQLAlchemy adaption object called :class:`_engine.AdaptedConnection`. * ``driver_connection`` always represents the actual connection object maintained by the third party pep-249 DBAPI or async driver in use. For standard pep-249 DBAPIs, this will always be the same object as that of the ``dbapi_connection``. For an asyncio driver, it will be the underlying asyncio-only connection object. The ``.connection`` attribute remains available and is now a legacy alias of ``.dbapi_connection``. Fixes: #6832 Change-Id: Ib72f97deefca96dce4e61e7c38ba430068d6a82e
* Replace all http:// links to https://Federico Caselli2021-07-041-1/+1
| | | | | | Also replace http://pypi.python.org/pypi with https://pypi.org/project Change-Id: I84b5005c39969a82140706472989f2a30b0c7685
* Add new "sync once" mode for pool.connectMike Bayer2021-04-211-2/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed critical regression caused by the change in :ticket`5497` where the connection pool "init" phase no longer occurred within mutexed isolation, allowing other threads to proceed with the dialect uninitialized, which could then impact the compilation of SQL statements. This issue is essentially the same regression which was fixed many years ago in :ticket:`2964` in dd32540dabbee0678530fb1b0868d1eb41572dca, which was missed this time as the test suite fo that issue only tested the pool in isolation, and assumed the "first_connect" event would be used by the Engine. However :ticket:`5497` stopped using "first_connect" and no test detected the lack of mutexing, that has been resolved here through the addition of more tests. This fix also identifies what is probably a bug in earlier versions of SQLAlchemy where the "first_connect" handler would be cancelled if the initializer failed; this is evidenced by test_explode_in_initializer which was doing a reconnect due to c.rollback() yet wasn't hanging. We now solve this issue by preventing the manufactured Connection from ever reconnecting inside the first_connect handler. Also remove the "_sqla_unwrap" test attribute; this is almost not used anymore however we can use a more targeted wrapper supplied by the testing.engines.proxying_engine function. See if we can also open up Oracle for "ad hoc engines" tests now that we have better connection management logic. Fixes: #6337 Change-Id: I4a3476625c4606f1a304dbc940d500325e8adc1a
* Add support for aiosqliteFederico Caselli2021-03-241-1/+13
| | | | | | | | Added support for the aiosqlite database driver for use with the SQLAlchemy asyncio extension. Fixes: #5920 Change-Id: Id11a320516a44e886a6f518d2866a0f992413e55
* Create explicit GC ordering between ConnectionFairy/ConnectionRecordMike Bayer2021-01-151-0/+14
| | | | | | | | | | | | | | Fixed issue where connection pool would not return connections to the pool or otherwise be finalized upon garbage collection under pypy if the checked out connection fell out of scope without being closed. This is a long standing issue due to pypy's difference in GC behavior that does not call weakref finalizers if they are relative to another object that is also being garbage collected. A strong reference to the related record is now maintained so that the weakref has a strong-referenced "base" to trigger off of. Fixes: #5842 Change-Id: Id5448fdacb6cceaac1ea40b2fbc851f052ed8e86
* reinvent xdist hooks in terms of pytest fixturesMike Bayer2021-01-131-79/+100
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To allow the "connection" pytest fixture and others work correctly in conjunction with setup/teardown that expects to be external to the transaction, remove and prevent any usage of "xdist" style names that are hardcoded by pytest to run inside of fixtures, even function level ones. Instead use pytest autouse fixtures to implement our own r"setup|teardown_test(?:_class)?" methods so that we can ensure function-scoped fixtures are run within them. A new more explicit flow is set up within plugin_base and pytestplugin such that the order of setup/teardown steps, which there are now many, is fully documented and controllable. New granularity has been added to the test teardown phase to distinguish between "end of the test" when lock-holding structures on connections should be released to allow for table drops, vs. "end of the test plus its teardown steps" when we can perform final cleanup on connections and run assertions that everything is closed out. From there we can remove most of the defensive "tear down everything" logic inside of engines which for many years would frequently dispose of pools over and over again, creating for a broken and expensive connection flow. A quick test shows that running test/sql/ against a single Postgresql engine with the new approach uses 75% fewer new connections, creating 42 new connections total, vs. 164 new connections total with the previous system. As part of this, the new fixtures metadata/connection/future_connection have been integrated such that they can be combined together effectively. The fixture_session(), provide_metadata() fixtures have been improved, including that fixture_session() now strongly references sessions which are explicitly torn down before table drops occur afer a test. Major changes have been made to the ConnectionKiller such that it now features different "scopes" for testing engines and will limit its cleanup to those testing engines corresponding to end of test, end of test class, or end of test session. The system by which it tracks DBAPI connections has been reworked, is ultimately somewhat similar to how it worked before but is organized more clearly along with the proxy-tracking logic. A "testing_engine" fixture is also added that works as a pytest fixture rather than a standalone function. The connection cleanup logic should now be very robust, as we now can use the same global connection pools for the whole suite without ever disposing them, while also running a query for PostgreSQL locks remaining after every test and assert there are no open transactions leaking between tests at all. Additional steps are added that also accommodate for asyncio connections not explicitly closed, as is the case for legacy sync-style tests as well as the async tests themselves. As always, hundreds of tests are further refined to use the new fixtures where problems with loose connections were identified, largely as a result of the new PostgreSQL assertions, many more tests have moved from legacy patterns into the newest. An unfortunate discovery during the creation of this system is that autouse fixtures (as well as if they are set up by @pytest.mark.usefixtures) are not usable at our current scale with pytest 4.6.11 running under Python 2. It's unclear if this is due to the older version of pytest or how it implements itself for Python 2, as well as if the issue is CPU slowness or just large memory use, but collecting the full span of tests takes over a minute for a single process when any autouse fixtures are in place and on CI the jobs just time out after ten minutes. So at the moment this patch also reinvents a small version of "autouse" fixtures when py2k is running, which skips generating the real fixture and instead uses two global pytest fixtures (which don't seem to impact performance) to invoke the "autouse" fixtures ourselves outside of pytest. This will limit our ability to do more with fixtures until we can remove py2k support. py.test is still observed to be much slower in collection in the 4.6.11 version compared to modern 6.2 versions, so add support for new TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that will run the suite for fewer backends under Python 2. For Python 3 pin pytest to modern 6.2 versions where performance for collection has been improved greatly. Includes the following improvements: Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would be raised rather than :class:`.exc.TimeoutError`. Also repaired the :paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using the async engine, which previously would ignore the timeout and block rather than timing out immediately as is the behavior with regular :class:`.QueuePool`. For asyncio the connection pool will now also not interact at all with an asyncio connection whose ConnectionFairy is being garbage collected; a warning that the connection was not properly closed is emitted and the connection is discarded. Within the test suite the ConnectionKiller is now maintaining strong references to all DBAPI connections and ensuring they are released when tests end, including those whose ConnectionFairy proxies are GCed. Identified cx_Oracle.stmtcachesize as a major factor in Oracle test scalability issues, this can be reset on a per-test basis rather than setting it to zero across the board. the addition of this flag has resolved the long-standing oracle "two task" error problem. For SQL Server, changed the temp table style used by the "suite" tests to be the double-pound-sign, i.e. global, variety, which is much easier to test generically. There are already reflection tests that are more finely tuned to both styles of temp table within the mssql test suite. Additionally, added an extra step to the "dropfirst" mechanism for SQL Server that will remove all foreign key constraints first as some issues were observed when using this flag when multiple schemas had not been torn down. Identified and fixed two subtle failure modes in the engine, when commit/rollback fails in a begin() context manager, the connection is explicitly closed, and when "initialize()" fails on the first new connection of a dialect, the transactional state on that connection is still rolled back. Fixes: #5826 Fixes: #5827 Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
* happy new yearMike Bayer2021-01-041-1/+1
| | | | Change-Id: Ic5bb19ca8be3cb47c95a0d3315d84cb484bac47c
* remove metadata.bind use from test suiteMike Bayer2021-01-031-1/+1
| | | | | | | | | | | | | | importantly this means we can remove bound metadata from the fixtures that are used by Alembic's test suite. hopefully this is the last one that has to happen to allow Alembic to be fully 1.4/2.0. Start moving from @testing.provide_metadata to a pytest metadata fixture. This does not seem to have any negative effects even though TablesTest uses a "self.metadata" attribute. Change-Id: Iae6ab95938a7e92b6d42086aec534af27b5577d3
* Repair async test refactorMike Bayer2021-01-021-4/+12
| | | | | | | | | | | | | | | in I4940d184a4dc790782fcddfb9873af3cca844398 we reworked how async tests run but apparently the async tests in test/ext/asyncio are reporting success without being run. This patch pushes pytestplugin further so that it won't instrument any test or function overall that declares itself async. This removes the need for the __async_wrap__ flag and also allows us to use a more strict "run_async_test" function that always runs the asyncio event loop from the top. Also start working asyncio into main testing suite. Change-Id: If7144e951a9db67eb7ea73b377f81c4440d39819
* Support testing of async drivers without fallback modeFederico Caselli2020-12-301-4/+2
| | | | Change-Id: I4940d184a4dc790782fcddfb9873af3cca844398
* Don't link on_connect to first_connect event handlerMike Bayer2020-08-071-0/+3
| | | | | | | | | | | | | Adjusted the dialect initialization process such that the :meth:`_engine.Dialect.on_connect` is not called a second time on the first connection. The hook is called first, then the :meth:`_engine.Dialect.initialize` is called if that connection is the first for that dialect, then no more events are called. This eliminates the two calls to the "on_connect" function which can produce very difficult debugging situations. Fixes: #5497 Change-Id: Icefc2e884e30ee7b4ac84b99dc54bf992a6085e3
* Create initial 2.0 engine implementationMike Bayer2020-04-161-2/+5
| | | | | | | | | | | | | | | | | | | Implemented the SQLAlchemy 2 :func:`.future.create_engine` function which is used for forwards compatibility with SQLAlchemy 2. This engine features always-transactional behavior with autobegin. Allow execution options per statement execution. This includes that the before_execute() and after_execute() events now accept an additional dictionary with these options, empty if not passed; a legacy event decorator is added for backwards compatibility which now also emits a deprecation warning. Add some basic tests for execution, transactions, and the new result object. Build out on a new testing fixture that swaps in the future engine completely to start with. Change-Id: I70e7338bb3f0ce22d2f702537d94bb249bd9fb0a Fixes: #4644
* Fix databases -> dialectsMike Bayer2020-04-011-1/+1
| | | | | | | | | The change in I5343a2d270ed5a8c654b9fe13dff40cdf54649ed left the name "databases" present in one spot, which is usually not reached unless tests are being run on only partial files where not every dialect has been imported. Change-Id: I5a8776b412eaf2476f8a93eebc31448b6f9218ee
* Deprecate the databases packagesFederico Caselli2020-02-281-2/+2
| | | | | | | The databases package is an old alias to the dialect package, and its usage is now deprecated Change-Id: I5343a2d270ed5a8c654b9fe13dff40cdf54649ed
* Do away with pool._refsMike Bayer2020-02-011-6/+6
| | | | | | | | | | | | | This collection was added only for the benefit of unit tests and is unnecessary for the pool to function. As SQLAlchemy 2.0 will be removing the automatic handling of connections that are garbage collection, remove this collection so that we ultimately don't need a weakref handler to do anything within the pool. The handler will do nothing other than emit a warning that a connection was dereferenced without being explicitly returned to the pool, invalidated, or detached. Change-Id: I4ca196270d5714efbac44dbf6f034e8c7f0af58a
* Remove jython code, remove all jython / pypy symbolsMike Bayer2020-01-171-1/+1
| | | | | | | | | | | | | | | | | | | | | | Removed all dialect code related to support for Jython and zxJDBC. Jython has not been supported by SQLAlchemy for many years and it is not expected that the current zxJDBC code is at all functional; for the moment it just takes up space and adds confusion by showing up in documentation. At the moment, it appears that Jython has achieved Python 2.7 support in its releases but not Python 3. If Jython were to be supported again, the form it should take is against the Python 3 version of Jython, and the various zxJDBC stubs for various backends should be implemented as a third party dialect. Additionally modernized logic that distinguishes between "cpython" and "pypy" to instead look at platform.python_distribution() which reliably tells us if we are cPython or not; all booleans which previously checked for pypy and sometimes jython are now converted to be "not cpython", this impacts the test suite for tests that are cPython centric. Fixes: #5094 Change-Id: I226cb55827f997daf6b4f4a755c18e7f4eb8d9ad
* happy new yearMike Bayer2020-01-011-1/+1
| | | | Change-Id: I08440dc25e40ea1ccea1778f6ee9e28a00808235
* Source base cleanupsMike Bayer2020-01-011-3/+1
| | | | | | | | | | | | | | | in trying to apply 2020 copyright to files, the pre-commit hooks complain about random file issues. - remove old corrections.py utility, this had something to do with repairing refs in the sphinx docs - run pre commit hooks on all files - formatting adjustments to work around code formatting collisions (long import lines that zimports can't rewrite correctly) Change-Id: I260744866f69e902eb93665c7c728ee94d3371a2
* Remove threadlocal engine strategy, engine strategies pool threadlocalMike Bayer2019-07-151-3/+3
| | | | | | | | | | | | The "threadlocal" execution strategy, deprecated in 1.3, has been removed for 1.4, as well as the concept of "engine strategies" and the ``Engine.contextual_connect`` method. The "strategy='mock'" keyword argument is still accepted for now with a deprecation warning; use :func:`.create_mock_engine` instead for this use case. Fixes: #4632 Change-Id: I8a351f9fa1f7dfa2a56eec1cd2d1a4b9d65765a2 (cherry picked from commit b368c49b44c5716d93c7428ab22b6761c6ca7cf5)
* Add deprecation warnings to all deprecated APIsMike Bayer2019-01-231-0/+3
| | | | | | | | | | | | | | | A large change throughout the library has ensured that all objects, parameters, and behaviors which have been noted as deprecated or legacy now emit ``DeprecationWarning`` warnings when invoked. As the Python 3 interpreter now defaults to displaying deprecation warnings, as well as that modern test suites based on tools like tox and pytest tend to display deprecation warnings, this change should make it easier to note what API features are obsolete. See the notes added to the changelog and migration notes for further details. Fixes: #4393 Change-Id: If0ea11a1fc24f9a8029352eeadfc49a7a54c0a1b
* happy new yearMike Bayer2019-01-111-1/+1
| | | | Change-Id: I6a71f4924d046cf306961c58dffccf21e9c03911
* Post black reformattingMike Bayer2019-01-061-3/+5
| | | | | | | | | | | | | Applied on top of a pure run of black -l 79 in I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes resolves all remaining flake8 conditions for those codes we have enabled in setup.cfg. Included are resolutions for all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
* Run black -l 79 against all source filesMike Bayer2019-01-061-30/+30
| | | | | | | | | | | | | | This is a straight reformat run using black as is, with no edits applied at all. The black run will format code consistently, however in some cases that are prevalent in SQLAlchemy code it produces too-long lines. The too-long lines will be resolved in the following commit that will resolve all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
* Don't call rollback on DBAPI connection that's "closed"Mike Bayer2018-10-141-0/+6
| | | | | | | | | | Use the existence of ConnectionRecord.connection to estimate that this connection is likely closed, and if so, don't try to call "rollback" on it. This rollback is normally harmless but is causing segfaults in mysqlclient due to https://github.com/PyMySQL/mysqlclient-python/issues/270. Change-Id: I1d7c5f5a520527d8268b6334795c2051f7ceeea6
* happy new yearMike Bayer2018-01-121-1/+1
| | | | Change-Id: I3ef36bfd0cb0ba62b3123c8cf92370a43156cf8f
* Use config.db global opts as defaults for all testing_engine()Mike Bayer2017-03-221-0/+3
| | | | | | | | | | Some options need to be passed to engines in all cases, such as currently the oracle 12516 workaround. make sure calls to testing_engine also set up the dictionary with defaults even if options is passed. not clear if this affects other backends yet. Change-Id: I5a1f7634e4ce5af6fe55dc21a24db6afacd19bb7
* Integrate "pre-ping" into connection pool.Mike Bayer2017-03-201-3/+19
| | | | | | | | | | | | | | | Added native "pessimistic disconnection" handling to the :class:`.Pool` object. The new parameter :paramref:`.Pool.pre_ping`, available from the engine as :paramref:`.create_engine.pool_pre_ping`, applies an efficient form of the "pre-ping" recipe featured in the pooling documentation, which upon each connection check out, emits a simple statement, typically "SELECT 1", to test the connection for liveness. If the existing connection is no longer able to respond to commands, the connection is transparently recycled, and all other connections made prior to the current timestamp are invalidated. Change-Id: I89700d0075e60abd2250e54b9bd14daf03c71c00 Fixes: #3919
* update for 2017 copyrightMike Bayer2017-01-041-1/+1
| | | | Change-Id: I4e8c2aa8fe817bb2af8707410fa0201f938781de
* - happy new yearMike Bayer2016-01-291-1/+1
|
* - repair that we use db_opts when the url here is different,Mike Bayer2015-06-191-3/+8
| | | | as we are using db_opts for the first time with legacy_schema_args
* - repair a regression caused by #3282, where we no longer wereMike Bayer2015-04-221-1/+8
| | | | | | | | | | applying any topological sort to tables on SQLite. See the changelog for details, but we now continue to sort tables for SQLite on DROP, prohibit the sort from considering alter, and only warn if we encounter an unresolvable cycle, in which case, then we forego the ordering. use_alter as always is used to break such a cycle. fixes #3378
* - copyright 2015Mike Bayer2015-03-101-1/+1
|
* - force the _has_events flag to True on engines, so thatMike Bayer2014-12-081-0/+3
| | | | | profiling is more predictable - restore the profiling from before this change
* - rework the assert_sql system so that we have a context manager to work with,Mike Bayer2014-12-071-3/+0
| | | | use events that are local to the engine and to the run and are removed afterwards.
* Merge branch 'master' into fdbsql-testsScott Dugas2014-11-031-4/+0
|\ | | | | | | | | Conflicts: lib/sqlalchemy/testing/exclusions.py
| * cleanup exception handling - use new exception hierarchy (since python 2.5)pr/140ndparker2014-10-021-4/+0
| |
* | Removed accidental print statementsScott Dugas2014-10-301-2/+0
| |
* | Support additional args/kwargs on cursor methodScott Dugas2014-10-221-4/+6
|/ | | | | fdbsql has an optional nested kwarg, which is supported in the actual code, but not in the testing proxy
* - rework profiling, zoomark tests into single tests so thatMike Bayer2014-08-161-112/+0
| | | | they can be used under xdist
* - apply pep8 formatting to sqlalchemy/sql, sqlalchemy/util, sqlalchemy/dialects,Brian Jarrett2014-07-201-18/+19
| | | | sqlalchemy/orm, sqlalchemy/event, sqlalchemy/testing
* - break up the <authors> copyright comment as part of a passMike Bayer2014-07-091-1/+2
| | | | to get all flake8 passing
* Fix many typos throughout the codebasepr/85Alex Gaynor2014-04-261-1/+1
| | | | Found using: https://github.com/intgr/topy
* - Support has been added for pytest to run tests. This runnerMike Bayer2014-03-031-18/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | is currently being supported in addition to nose, and will likely be preferred to nose going forward. The nose plugin system used by SQLAlchemy has been split out so that it works under pytest as well. There are no plans to drop support for nose at the moment and we hope that the test suite itself can continue to remain as agnostic of testing platform as possible. See the file README.unittests.rst for updated information on running tests with pytest. The test plugin system has also been enhanced to support running tests against mutiple database URLs at once, by specifying the ``--db`` and/or ``--dburi`` flags multiple times. This does not run the entire test suite for each database, but instead allows test cases that are specific to certain backends make use of that backend as the test is run. When using pytest as the test runner, the system will also run specific test suites multiple times, once for each database, particularly those tests within the "dialect suite". The plan is that the enhanced system will also be used by Alembic, and allow Alembic to run migration operation tests against multiple backends in one run, including third-party backends not included within Alembic itself. Third party dialects and extensions are also encouraged to standardize on SQLAlchemy's test suite as a basis; see the file README.dialects.rst for background on building out from SQLAlchemy's test platform.
* - _cursor_execute() will close the cursor on error; oracle doesn't allow ↵Mike Bayer2014-01-131-2/+2
| | | | | | double close - ensure no iterator changed size issues in testing.engines
* - continue with [ticket:2907] and further clean up how we set upMike Bayer2014-01-131-1/+5
| | | | | | | | | | | | | | _reset_agent, so that it's local to the various begin_impl(), rollback_impl(), etc. this allows setting/resetting of the flag to be symmetric. - don't set _reset_agent if it's not None, don't unset it if it isn't our own transaction. - make sure we clean it out in close(). - basically, we're dealing here with pools using "threadlocal" that have a counter, other various mismatches that the tests bring up - test for recover() now has to invalidate() the previous connection, because closing it actually rolls it back (e.g. this test was relying on the broken behavior).
View Lorry Controller Status