summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/testing/assertions.py
Commit message (Collapse)AuthorAgeFilesLines
* fix test suite warningsMike Bayer2023-05-091-10/+32
| | | | | | | | | | | | | | | | fix a handful of warnings that were emitting but not raising, usually because they were inside an "expect_warnings" block. modify "expect_warnings" to always use "raise_on_any_unexpected" behavior; remove this parameter. Fixed issue in semi-private ``await_only()`` and ``await_fallback()`` concurrency functions where the given awaitable would remain un-awaited if the function threw a ``GreenletError``, which could cause "was not awaited" warnings later on if the program continued. In this case, the given awaitable is now cancelled before the exception is thrown. Change-Id: I33668c5e8c670454a3d879e559096fb873b57244
* use clone, not constructor, in BindParameter.render_literal_execute()Mike Bayer2023-03-221-0/+4
| | | | | | | | | | | | | Fixed issue where the :meth:`_sql.BindParameter.render_literal_execute` method would fail when called on a parameter that also had ORM annotations associated with it. In practice, this would be observed as a failure of SQL compilation when using some combinations of a dialect that uses "FETCH FIRST" such as Oracle along with a :class:`_sql.Select` construct that uses :meth:`_sql.Select.limit`, within some ORM contexts, including if the statement were embedded within a relationship primaryjoin expression. Fixes: #9526 Change-Id: I2f512b6760a90293d274e60b06a891f10b276ecc
* add set_shard_id() loader option for horizontal shardMike Bayer2023-01-251-11/+12
| | | | | | | | | | | | | | Added new option to horizontal sharding API :class:`_horizontal.set_shard_id` which sets the effective shard identifier to query against, for both the primary query as well as for all secondary loaders including relationship eager loaders as well as relationship and column lazy loaders. Modernize sharding examples with new-style mappings, add new asyncio example. Fixes: #7226 Fixes: #7028 Change-Id: Ie69248060c305e8de04f75a529949777944ad511
* fix stringify for CreateSchemaMike Bayer2023-01-241-0/+2
| | | | | | | | Fixed stringify for a the :class:`.CreateSchema` DDL construct, which would fail with an ``AttributeError`` when stringified without a dialect. Fixes: #7664 Change-Id: Ifc1769604bc5219c060f5112f7bdea0f780f1a1c
* happy new year 2023Mike Bayer2023-01-031-1/+1
| | | | Change-Id: I625af65b3fb1815b1af17dc2ef47dd697fdc3fb1
* fix construct_params() for render_postcompile; add new APIMike Bayer2022-12-081-3/+23
| | | | | | | | | | | | | | | | | | | | | The :meth:`.SQLCompiler.construct_params` method, as well as the :attr:`.SQLCompiler.params` accessor, will now return the exact parameters that correspond to a compiled statement that used the ``render_postcompile`` parameter to compile. Previously, the method returned a parameter structure that by itself didn't correspond to either the original parameters or the expanded ones. Passing a new dictionary of parameters to :meth:`.SQLCompiler.construct_params` for a :class:`.SQLCompiler` that was constructed with ``render_postcompile`` is now disallowed; instead, to make a new SQL string and parameter set for an alternate set of parameters, a new method :meth:`.SQLCompiler.construct_expanded_state` is added which will produce a new expanded form for the given parameter set, using the :class:`.ExpandedState` container which includes a new SQL statement and new parameter dictionary, as well as a positional parameter tuple. Fixes: #6114 Change-Id: I9874905bb90f86799b82b244d57369558b18fd93
* Fix positional compiling bugsFederico Caselli2022-12-011-3/+61
| | | | | | | | | | | Fixed a series of issues regarding positionally rendered bound parameters, such as those used for SQLite, asyncpg, MySQL and others. Some compiled forms would not maintain the order of parameters correctly, such as the PostgreSQL ``regexp_replace()`` function as well as within the "nesting" feature of the :class:`.CTE` construct first introduced in :ticket:`4123`. Fixes: #8827 Change-Id: I9813ed7c358cc5c1e26725c48df546b209a442cb
* ensure anon_map is passed for most annotated traversalsMike Bayer2022-11-111-0/+11
| | | | | | | | | | | | | | | | | | | | | | | | We can cache the annotated cache key for Table, but for selectables it's not safe, as it fails to pass the anon_map along and creates many redudant structures in observed test scenario. It is likely safe for a Column that's mapped to a Table also, however this is not implemented here. Will have to see if that part needs adjusting. Fixed critical memory issue identified in cache key generation, where for very large and complex ORM statements that make use of lots of ORM aliases with subqueries, cache key generation could produce excessively large keys that were orders of magnitude bigger than the statement itself. Much thanks to Rollo Konig Brock for their very patient, long term help in finally identifying this issue. Also within TypeEngine objects, when we generate elements for instance variables, skip the None elements at least. this also saves on tuple complexity. Fixes: #8790 Change-Id: I448ddbfb45ae0a648815be8dad4faad7d1977427
* Reflect expression-based indexes on PostgreSQLFederico Caselli2022-07-281-0/+31
| | | | | | | | | | | The PostgreSQL dialect now supports reflection of expression based indexes. The reflection is supported both when using :meth:`_engine.Inspector.get_indexes` and when reflecting a :class:`_schema.Table` using :paramref:`_schema.Table.autoload_with`. Thanks to immerrr and Aidan Kane for the help on this ticket. Fixes: #7442 Change-Id: I3e36d557235286c0f7f6d8276272ff9225058d48
* Merge "create new approach for deeply nested post loader options" into mainmike bayer2022-06-181-1/+1
|\
| * create new approach for deeply nested post loader optionsMike Bayer2022-06-181-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added very experimental feature to the :func:`_orm.selectinload` and :func:`_orm.immediateload` loader options called :paramref:`_orm.selectinload.recursion_depth` / :paramref:`_orm.immediateload.recursion_depth` , which allows a single loader option to automatically recurse into self-referential relationships. Is set to an integer indicating depth, and may also be set to -1 to indicate to continue loading until no more levels deep are found. Major internal changes to :func:`_orm.selectinload` and :func:`_orm.immediateload` allow this feature to work while continuing to make correct use of the compilation cache, as well as not using arbitrary recursion, so any level of depth is supported (though would emit that many queries). This may be useful for self-referential structures that must be loaded fully eagerly, such as when using asyncio. A warning is also emitted when loader options are connected together with arbitrary lengths (that is, without using the new ``recursion_depth`` option) when excessive recursion depth is detected in related object loading. This operation continues to use huge amounts of memory and performs extremely poorly; the cache is disabled when this condition is detected to protect the cache from being flooded with arbitrary statements. Fixes: #8126 Change-Id: I9f162e0a09c1ed327dd19498aac193f649333a01
* | Merge "rearchitect reflection for batched performance" into mainFederico Caselli2022-06-181-10/+20
|\ \ | |/ |/|
| * rearchitect reflection for batched performanceFederico Caselli2022-06-181-10/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Rearchitected the schema reflection API to allow some dialects to make use of high performing batch queries to reflect the schemas of many tables at once using much fewer queries. The new performance features are targeted first at the PostgreSQL and Oracle backends, and may be applied to any dialect that makes use of SELECT queries against system catalog tables to reflect tables (currently this omits the MySQL and SQLite dialects which instead make use of parsing the "CREATE TABLE" statement, however these dialects do not have a pre-existing performance issue with reflection. MS SQL Server is still a TODO). The new API is backwards compatible with the previous system, and should require no changes to third party dialects to retain compatibility; third party dialects can also opt into the new system by implementing batched queries for schema reflection. Along with this change is an updated reflection API that is fully :pep:`484` typed, features many new methods and some changes. Fixes: #4379 Change-Id: I897ec09843543aa7012bcdce758792ed3d415d08
* | Revert "add auto_recurse option to selectinload, immediateload"Mike Bayer2022-06-161-1/+1
|/ | | | | | | | | | this option works very badly with caching and the API is likely not what we want either. Work continues for #8126 including that the additional work in I9f162e0a09c1ed327dd19498aac193f649333a01 tries to add new recursive features. This reverts commit b3a1162553879d1369154e920f3f4129020bb88e.
* add auto_recurse option to selectinload, immediateloadMike Bayer2022-06-111-1/+1
| | | | | | | | | | | | | | Added very experimental feature to the :func:`_orm.selectinload` and :func:`_orm.immediateload` loader options called :paramref:`_orm.selectinload.auto_recurse` / :paramref:`_orm.immediateload.auto_recurse` , which when set to True will cause a self-referential relationship load to continue loading with arbitrary depth until no further objects are found. This may be useful for self-referential structures that must be loaded fully eagerly, such as when using asyncio. Fixes: #8126 Change-Id: I5bbd00bd0ca43f4649b44680fea1e84680f0a5db
* inline mypy config; files ignoring type errors for the momentMike Bayer2022-04-281-0/+2
| | | | | | | | | | | | | | | | | | | to simplify pyproject.toml change the remaining files that aren't going to be typed on this first pass (unless of course someone wants to type some of these) to include # mypy: ignore-errors. for the moment, only a handful of ORM modules are to have more type checking implemented. It's important that ignore-errors is used and not "# type: ignore", as in the latter case, mypy doesn't even read the existing types in the file, which makes it impossible to type any files that refer to those modules at all. to simplify ongoing typing work use inline mypy config for remaining files that are "done" for now, indicating the level of type checking they currently have. Change-Id: I98669c1a305c2f0adba85d10b5425541f3fe9533
* pep-484 for sqlalchemy.event; use future annotationsMike Bayer2022-02-151-0/+2
| | | | | | | | | | __future__.annotations mode allows us to use non-string annotations for argument and return types in most cases, but more importantly it removes a large amount of runtime overhead that would be spent in evaluating the annotations. Change-Id: I2f5b6126fe0019713fc50001be3627b664019ede References: #6810
* dont test squelched warnings against the filterMike Bayer2022-01-231-0/+2
| | | | | | | | | | | I spent days on Ibcf09af25228d39ee5a943fda82d8a9302433726 reading it over and over again and noticed this slight inaccuracy 10 seconds after I merged it. the assert_warns_message() and assert_warns() functions should not consider a mismatched warning class as valid for a match. Change-Id: Ib8944dd95bcec1a7e4963917a5f4829e2ba27732
* dont use exception catches for warnings; modernize xdist detectionMike Bayer2022-01-221-5/+54
| | | | | | | | | | | | | | | | | Improvements to the test suite's integration with pytest such that the "warnings" plugin, if manually enabled, will not interfere with the test suite, such that third parties can enable the warnings plugin or make use of the ``-W`` parameter and SQLAlchemy's test suite will continue to pass. Additionally, modernized the detection of the "pytest-xdist" plugin so that plugins can be globally disabled using PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 without breaking the test suite if xdist were still installed. Warning filters that promote deprecation warnings to errors are now localized to SQLAlchemy-specific warnings, or within SQLAlchemy-specific sources for general Python deprecation warnings, so that non-SQLAlchemy deprecation warnings emitted from pytest plugins should also not impact the test suite. Fixes: #7599 Change-Id: Ibcf09af25228d39ee5a943fda82d8a9302433726
* happy new year 2022Mike Bayer2022-01-061-1/+1
| | | | Change-Id: I49abf2607e0eb0623650efdf0091b1fb3db737ea
* Remove all remaining removed_in_20 warnings slated for removalMike Bayer2022-01-051-7/+3
| | | | | | | | | | | | | | | | | | | | | | | | | Finalize all remaining removed-in-2.0 changes so that we can begin doing pep-484 typing without old things getting in the way (we will also have to do public_factory). note there are a few "moved_in_20()" and "became_legacy_in_20()" warnings still in place. The SQLALCHEMY_WARN_20 variable is now removed. Also removed here are the legacy "in place mutators" for Select statements, and some keyword-only argument signatures in Core have been added. Also in the big change department, the ORM mapper() function is removed entirely; the Mapper class is otherwise unchanged, just the public-facing API function. Mappers are now always given a registry in which to participate, however the argument signature of Mapper is not changed. ideally "registry" would be the first positional argument. Fixes: #7257 Change-Id: Ic70c57b9f1cf7eb996338af5183b11bdeb3e1623
* replace Variant with direct feature inside of TypeEngineMike Bayer2021-12-291-0/+4
| | | | | | | | | | | | | The :meth:`_sqltypes.TypeEngine.with_variant` method now returns a copy of the original :class:`_sqltypes.TypeEngine` object, rather than wrapping it inside the ``Variant`` class, which is effectively removed (the import symbol remains for backwards compatibility with code that may be testing for this symbol). While the previous approach maintained in-Python behaviors, maintaining the original type allows for clearer type checking and debugging. Fixes: #6980 Change-Id: I158c7e56306b886b5b82b040205c428a5c4a242c
* Warn when caching is disabled / documentMike Bayer2021-12-061-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds new warnings for all elements that don't indicate their caching behavior, including user-defined ClauseElement subclasses and third party dialects. it additionally adds new documentation to discuss an apparent performance degradation in 1.4 when caching is disabled as a result in the significant expense incurred by ORM lazy loaders, which in 1.3 used BakedQuery so were actually cached. As a result of adding the warnings, a fair degree of lesser used SQL expression objects identified that they did not define caching behavior so would have been producing ``[no key]``, including PostgreSQL constructs ``hstore`` and ``array``. These have been amended to use inherit cache where appropriate. "on conflict" constructs in PostgreSQL, MySQL, SQLite still explicitly don't generate a cache key at this time. The change also adds a test for all constructs via assert_compile() to assert they will not generate cache warnings. Fixes: #7394 Change-Id: I85958affbb99bfad0f5efa21bc8f2a95e7e46981
* Clean up most py3k compatFederico Caselli2021-11-241-29/+10
| | | | Change-Id: I8172fdcc3103ff92aa049827728484c8779af6b7
* Remove object in class definitionFederico Caselli2021-11-221-6/+6
| | | | | References: #4600 Change-Id: I2a62ddfe00bc562720f0eae700a497495d7a987a
* fully support isolation_level parameter in base dialectMike Bayer2021-11-181-3/+7
| | | | | | | | | | | | | | | | | | | | Generalized the :paramref:`_sa.create_engine.isolation_level` parameter to the base dialect so that it is no longer dependent on individual dialects to be present. This parameter sets up the "isolation level" setting to occur for all new database connections as soon as they are created by the connection pool, where the value then stays set without being reset on every checkin. The :paramref:`_sa.create_engine.isolation_level` parameter is essentially equivalent in functionality to using the :paramref:`_engine.Engine.execution_options.isolation_level` parameter via :meth:`_engine.Engine.execution_options` for an engine-wide setting. The difference is in that the former setting assigns the isolation level just once when a connection is created, the latter sets and resets the given level on each connection checkout. Fixes: #6342 Change-Id: Id81d6b1c1a94371d901ada728a610696e09e9741
* 2.0 removals: LegacyRow, connectionless execution, close_with_resultMike Bayer2021-10-311-5/+1
| | | | | | | | | | | | | | | | | | | in order to remove LegacyRow / LegacyResult, we have to also lose close_with_result, which connectionless execution relies upon. also includes a new profiles.txt file that's all against py310, as that's what CI is on now. some result counts changed by one function call which was enough to fail the low-count result tests. Replaces Connectable as the common interface between Connection and Engine with EngineEventsTarget. Engine is no longer Connectable. Connection and MockConnection still are. References: #7257 Change-Id: Iad5eba0313836d347e65490349a22b061356896a
* deprecation warnings: with_parent, aliased, from_joinpointMike Bayer2021-10-271-1/+1
| | | | | | | | | most of the work for aliased / from_joinpoint has been done already as I added all new tests for these and moved most aliased/from_joinpoint to test/orm/test_deprecations.py already Change-Id: Ia23e332dec183de17b2fb9d89d946af8d5e89ae7
* warn or deprecate for auto-aliasing in joinsMike Bayer2021-09-281-41/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | An extra layer of warning messages has been added to the functionality of :meth:`_orm.Query.join` and the ORM version of :meth:`_sql.Select.join`, where a few places where "automatic aliasing" continues to occur will now be called out as a pattern to avoid, mostly specific to the area of joined table inheritance where classes that share common base tables are being joined together without using explicit aliases. One case emits a legacy warning for a pattern that's not recommended, the other case is fully deprecated. The automatic aliasing within ORM join() which occurs for overlapping mapped tables does not work consistently with all APIs such as ``contains_eager()``, and rather than continue to try to make these use cases work everywhere, replacing with a more user-explicit pattern is clearer, less prone to bugs and simplifies SQLAlchemy's internals further. The warnings include links to the errors.rst page where each pattern is demonstrated along with the recommended pattern to fix. * Improved the exception message generated when configuring a mapping with joined table inheritance where the two tables either have no foreign key relationships set up, or where they have multiple foreign key relationships set up. The message is now ORM specific and includes context that the :paramref:`_orm.Mapper.inherit_condition` parameter may be needed particularly for the ambiguous foreign keys case. * Add explicit support in the _expect_warnings() assertion for nested _expect_warnings calls * generalize the NoCache fixture, which we also need to catch warnings during compilation consistently * generalize the __str__() method for the HasCode mixin so all warnings and errors include the code link in their string Fixes: #6974 Change-Id: I84ed79ba2112c39eaab7973b6d6f46de7fa80842
* Replace all http:// links to https://Federico Caselli2021-07-041-1/+1
| | | | | | Also replace http://pypi.python.org/pypi with https://pypi.org/project Change-Id: I84b5005c39969a82140706472989f2a30b0c7685
* init extra_criteria_entities in fromstatement w/ DMLMike Bayer2021-06-071-4/+4
| | | | | | | | | | | | Fixed issue in experimental "select ORM objects from INSERT/UPDATE" use case where an error was raised if the statement were against a single-table-inheritance subclass. Additionally makes some adjustments in the SQL assertion fixture to test a FromStatement w/ DML. Fixes: #6591 Change-Id: I53a627ab18a01dc6d9b5037e28312a1177891327
* delete exception traceback in _expect_raisesMike Bayer2021-06-021-0/+7
| | | | | | | | | | a new segfault is observed in python 3.10.0b2 in conjunction with asyncio and possibly greenlet. Ensuring the traceback object is deleted from scope here, which is a good idea anyway, apparently seems to resolve the issue. Change-Id: Ia83bafb088ef19853044f1d436db259cbfd1e5f4 References: https://github.com/python-greenlet/greenlet/issues/242
* Support DEFAULT VALUES and VALUES(DEFAULT) individuallyMike Bayer2021-04-141-0/+3
| | | | | | | | | | | | | Fixed regression where the introduction of the INSERT syntax "INSERT... VALUES (DEFAULT)" was not supported on some backends that do however support "INSERT..DEFAULT VALUES", including SQLite. The two syntaxes are now each individually supported or non-supported for each dialect, for example MySQL supports "VALUES (DEFAULT)" but not "DEFAULT VALUES". Support for Oracle is still not enabled as there are unresolved issues in using RETURNING at the same time. Fixes: #6254 Change-Id: I47959bc826e3d9d2396ccfa290eb084841b02e77
* Remove internal use of string attr in loader optionMike Bayer2021-03-231-3/+14
| | | | | | | | | | | | | | | | | | Fixed issue where a "removed in 2.0" warning were generated internally by the relationship loader mechanics. This changeset started the effort of converting all string usage in the test suite, however this is a much longer job as the use of strings in loader options is widespread. In particular I'm not totally comfortable with strings not being accepted in obvious spots like Load(User).load_only("x", "y", "z"), which points to a new string expecting functionality that's not what's there now. However at the moment it seems like we need to continue removing all support for strings and then figure out "immediate strings from an explicit class" later. Fixes: #6115 Change-Id: I6b314d135d2bc049fd66500914b772c1fe60b5b3
* Ensure all Query statements compile w/ orm, fix test harnessMike Bayer2021-03-051-5/+3
| | | | | | | | | | | | | | | | | Fixed regression where :meth:`_orm.Query.join` would produce no effect if the query itself as well as the join target were against a :class:`_schema.Table` object, rather than a mapped class. This was part of a more systemic issue where the legacy ORM query compiler would not be correctly used from a :class:`_orm.Query` if the statement produced had not ORM entities present within it. Also repair the assert_compile() method, which was using Query._compile_state() which was bypassing the bug. A handful of ORM tests with Query objects and Core-only objects were actually failing if the default compilation path were used. Fixes: #6003 Change-Id: I97478b2d06bf6c01fe1f09ee118e1f2ac4c849bc
* Revert AppenderQuery modifications from ORMMike Bayer2021-02-251-3/+0
| | | | | | | | | | | | | | We are unfortunately stuck with this class completely until we get rid of "dynamic" altogether. The usage contract includes the "query_class" mixin feature where users add their own methods, and this use case very well in line with 2.0's contract. As Query is not going away in any case this has to stay in "legacy" style, there's no point trying to change it as the new version was still fully dependent on Query. Fixes: #5981 Change-Id: I1bc623b17d976b4bb417ab623248d4ac227db74d
* Track a second from_linter for lateral subqueriesMike Bayer2021-02-051-0/+4
| | | | | | | | | | | | | | | Fixed bug where the "cartesian product" assertion was not correctly accommodating for joins between tables that relied upon the use of LATERAL to connect from a subquery to another subquery in the enclosing context. Additionally, enabled from_linting for the base assert_compile(), however it remains off by default; to enable by default we would have to make sure it isn't set for DDL compiles and there's also a lot of tests that would also need to turn it off, so leaving this off for expediency. Fixes: #5924 Change-Id: I22604baf572f8c4d96befcc610b3dcb79c13fc4a
* Render NULL for bindparam w/ None value/literal_binds, warnMike Bayer2021-01-281-0/+3
| | | | | | | | | | | | | | | | | Adjusted the "literal_binds" feature of :class:`_sql.Compiler` to render NULL for a bound parameter that has ``None`` as the value, either explicitly passed or omitted. The previous error message "bind parameter without a renderable value" is removed, and a missing or ``None`` value will now render NULL in all cases. Previously, rendering of NULL was starting to happen for DML statements due to internal refactorings, but was not explicitly part of test coverage, which it now is. While no error is raised, when the context is within that of a column comparison, and the operator is not "IS"/"IS NOT", a warning is emitted that this is not generally useful from a SQL perspective. Fixes: #5888 Change-Id: Id5939d8dbfb1156a9f8a7f7e76cf18327155331a
* fix double is_none()Mike Bayer2021-01-141-8/+4
| | | | | | | | I added an extra is_none() by mistake the other day and for some reason it sneaked past flake8. it's breaking all the builds so get it back in Change-Id: I17b311341169571efa856e062c6be7e8f362618f
* Merge "reinvent xdist hooks in terms of pytest fixtures"mike bayer2021-01-141-0/+8
|\
| * reinvent xdist hooks in terms of pytest fixturesMike Bayer2021-01-131-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To allow the "connection" pytest fixture and others work correctly in conjunction with setup/teardown that expects to be external to the transaction, remove and prevent any usage of "xdist" style names that are hardcoded by pytest to run inside of fixtures, even function level ones. Instead use pytest autouse fixtures to implement our own r"setup|teardown_test(?:_class)?" methods so that we can ensure function-scoped fixtures are run within them. A new more explicit flow is set up within plugin_base and pytestplugin such that the order of setup/teardown steps, which there are now many, is fully documented and controllable. New granularity has been added to the test teardown phase to distinguish between "end of the test" when lock-holding structures on connections should be released to allow for table drops, vs. "end of the test plus its teardown steps" when we can perform final cleanup on connections and run assertions that everything is closed out. From there we can remove most of the defensive "tear down everything" logic inside of engines which for many years would frequently dispose of pools over and over again, creating for a broken and expensive connection flow. A quick test shows that running test/sql/ against a single Postgresql engine with the new approach uses 75% fewer new connections, creating 42 new connections total, vs. 164 new connections total with the previous system. As part of this, the new fixtures metadata/connection/future_connection have been integrated such that they can be combined together effectively. The fixture_session(), provide_metadata() fixtures have been improved, including that fixture_session() now strongly references sessions which are explicitly torn down before table drops occur afer a test. Major changes have been made to the ConnectionKiller such that it now features different "scopes" for testing engines and will limit its cleanup to those testing engines corresponding to end of test, end of test class, or end of test session. The system by which it tracks DBAPI connections has been reworked, is ultimately somewhat similar to how it worked before but is organized more clearly along with the proxy-tracking logic. A "testing_engine" fixture is also added that works as a pytest fixture rather than a standalone function. The connection cleanup logic should now be very robust, as we now can use the same global connection pools for the whole suite without ever disposing them, while also running a query for PostgreSQL locks remaining after every test and assert there are no open transactions leaking between tests at all. Additional steps are added that also accommodate for asyncio connections not explicitly closed, as is the case for legacy sync-style tests as well as the async tests themselves. As always, hundreds of tests are further refined to use the new fixtures where problems with loose connections were identified, largely as a result of the new PostgreSQL assertions, many more tests have moved from legacy patterns into the newest. An unfortunate discovery during the creation of this system is that autouse fixtures (as well as if they are set up by @pytest.mark.usefixtures) are not usable at our current scale with pytest 4.6.11 running under Python 2. It's unclear if this is due to the older version of pytest or how it implements itself for Python 2, as well as if the issue is CPU slowness or just large memory use, but collecting the full span of tests takes over a minute for a single process when any autouse fixtures are in place and on CI the jobs just time out after ten minutes. So at the moment this patch also reinvents a small version of "autouse" fixtures when py2k is running, which skips generating the real fixture and instead uses two global pytest fixtures (which don't seem to impact performance) to invoke the "autouse" fixtures ourselves outside of pytest. This will limit our ability to do more with fixtures until we can remove py2k support. py.test is still observed to be much slower in collection in the 4.6.11 version compared to modern 6.2 versions, so add support for new TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that will run the suite for fewer backends under Python 2. For Python 3 pin pytest to modern 6.2 versions where performance for collection has been improved greatly. Includes the following improvements: Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would be raised rather than :class:`.exc.TimeoutError`. Also repaired the :paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using the async engine, which previously would ignore the timeout and block rather than timing out immediately as is the behavior with regular :class:`.QueuePool`. For asyncio the connection pool will now also not interact at all with an asyncio connection whose ConnectionFairy is being garbage collected; a warning that the connection was not properly closed is emitted and the connection is discarded. Within the test suite the ConnectionKiller is now maintaining strong references to all DBAPI connections and ensuring they are released when tests end, including those whose ConnectionFairy proxies are GCed. Identified cx_Oracle.stmtcachesize as a major factor in Oracle test scalability issues, this can be reset on a per-test basis rather than setting it to zero across the board. the addition of this flag has resolved the long-standing oracle "two task" error problem. For SQL Server, changed the temp table style used by the "suite" tests to be the double-pound-sign, i.e. global, variety, which is much easier to test generically. There are already reflection tests that are more finely tuned to both styles of temp table within the mssql test suite. Additionally, added an extra step to the "dropfirst" mechanism for SQL Server that will remove all foreign key constraints first as some issues were observed when using this flag when multiple schemas had not been torn down. Identified and fixed two subtle failure modes in the engine, when commit/rollback fails in a begin() context manager, the connection is explicitly closed, and when "initialize()" fails on the first new connection of a dialect, the transactional state on that connection is still rolled back. Fixes: #5826 Fixes: #5827 Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
* | Implement connection binding for AsyncSessionMike Bayer2021-01-071-0/+4
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | Implemented "connection-binding" for :class:`.AsyncSession`, the ability to pass an :class:`.AsyncConnection` to create an :class:`.AsyncSession`. Previously, this use case was not implemented and would use the associated engine when the connection were passed. This fixes the issue where the "join a session to an external transaction" use case would not work correctly for the :class:`.AsyncSession`. Additionally, added methods :meth:`.AsyncConnection.in_transaction`, :meth:`.AsyncConnection.in_nested_transaction`, :meth:`.AsyncConnection.get_transaction`. The :class:`.AsyncEngine`, :class:`.AsyncConnection` and :class:`.AsyncTransaction` objects may be compared using Python ``==`` or ``!=``, which will compare the two given objects based on the "sync" object they are proxying towards. This is useful as there are cases particularly for :class:`.AsyncTransaction` where multiple instances of :class:`.AsyncTransaction` can be proxying towards the same sync :class:`_engine.Transaction`, and are actually equivalent. The :meth:`.AsyncConnection.get_transaction` method will currently return a new proxying :class:`.AsyncTransaction` each time as the :class:`.AsyncTransaction` is not otherwise statefully associated with its originating :class:`.AsyncConnection`. Fixes: #5811 Change-Id: I5a3a6b2f088541eee7b0e0f393510e61bc9f986b
* happy new yearMike Bayer2021-01-041-1/+1
| | | | Change-Id: Ic5bb19ca8be3cb47c95a0d3315d84cb484bac47c
* remove metadata.bind use from test suiteMike Bayer2021-01-031-2/+2
| | | | | | | | | | | | | | importantly this means we can remove bound metadata from the fixtures that are used by Alembic's test suite. hopefully this is the last one that has to happen to allow Alembic to be fully 1.4/2.0. Start moving from @testing.provide_metadata to a pytest metadata fixture. This does not seem to have any negative effects even though TablesTest uses a "self.metadata" attribute. Change-Id: Iae6ab95938a7e92b6d42086aec534af27b5577d3
* Ensure no compiler visit method tries to access .statementMike Bayer2020-10-191-1/+56
| | | | | | | | | | | | Fixed structural compiler issue where some constructs such as MySQL / PostgreSQL "on conflict / on duplicate key" would rely upon the state of the :class:`_sql.Compiler` object being fixed against their statement as the top level statement, which would fail in cases where those statements are branched from a different context, such as a DDL construct linked to a SQL statement. Fixes: #5656 Change-Id: I568bf40adc7edcf72ea6c7fd6eb9d07790de189e
* upgrade to black 20.8b1Mike Bayer2020-09-281-3/+6
| | | | | | | It's better, the majority of these changes look more readable to me. also found some docstrings that had formatting / quoting issues. Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
* Add deprecation warning for .join().alias()Gord Thompson2020-09-281-3/+3
| | | | | | | | | The :meth:`_sql.Join.alias` method is deprecated and will be removed in SQLAlchemy 2.0. An explicit select + subquery, or aliasing of the inner tables, should be used instead. Fixes: #5010 Change-Id: Ic913afc31f0d70b0605f9a7af2742a0de1f9ad19
* Merge "internal test framework files for standardization of is_not/not_in;"mike bayer2020-08-301-2/+10
|\
| * internal test framework files for standardization of is_not/not_in;jonathan vanasco2020-08-291-2/+10
| | | | | | | | | | | | this is safe for 1.3.x Change-Id: Icba38fdc20f5d8ac407383a4278ccb346e09af38
* | Emit v2.0 deprecation warning for "implicit autocommit"Gord Thompson2020-08-281-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | "Implicit autocommit", which is the COMMIT that occurs when a DML or DDL statement is emitted on a connection, is deprecated and won't be part of SQLAlchemy 2.0. A 2.0-style warning is emitted when autocommit takes effect, so that the calling code may be adjusted to use an explicit transaction. As part of this change, DDL methods such as :meth:`_schema.MetaData.create_all` when used against a :class:`_engine.Engine` or :class:`_engine.Connection` will run the operation in a BEGIN block if one is not started already. The MySQL and MariaDB dialects now query from the information_schema.tables system view in order to determine if a particular table exists or not. Previously, the "DESCRIBE" command was used with an exception catch to detect non-existent, which would have the undesirable effect of emitting a ROLLBACK on the connection. There appeared to be legacy encoding issues which prevented the use of "SHOW TABLES", for this, but as MySQL support is now at 5.0.2 or above due to :ticket:`4189`, the information_schema tables are now available in all cases. Fixes: #4846 Change-Id: I733a7e0e17477a63607fb9931c87c393bbd7ac57
View Lorry Controller Status