summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/engine/create.py
Commit message (Collapse)AuthorAgeFilesLines
* Add pool creation functionsFederico Caselli2023-04-121-11/+59
| | | | | | | | | | Added :func:`_sa.create_pool_from_url` and :func:`_asyncio.create_async_pool_from_url` to create a :class:`_pool.Pool` instance from an input url passed as string or :class:`_sa.URL`. Fixes: #9613 Change-Id: Icd8aa3f2849e6fd1bc5341114f3ef8d216a2c543
* remove "listeners" docstringMike Bayer2023-03-151-5/+0
| | | | | | | this should have been removed with #4638. Fixes: #9492 Change-Id: If82dba7e63382e921aceb0c01d88f0977b7f5e8d
* sqlite provisioning is ridiculousMike Bayer2023-03-101-0/+1
| | | | | | try to get file naming to be more sane for pysqlite file databases Change-Id: I68ad8c2f6c6c25930fbffdd79b8d429cd7a7dd9a
* happy new year 2023Mike Bayer2023-01-031-1/+1
| | | | Change-Id: I625af65b3fb1815b1af17dc2ef47dd697fdc3fb1
* Try running pyupgrade on the codeFederico Caselli2022-11-161-4/+4
| | | | | | | | command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>" pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not exists in sqlalchemy fixtures Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
* Improve typings of execution optionsFederico Caselli2022-11-021-2/+2
| | | | | Fixes: #8605 Change-Id: I4aec83b9f321462427c3f4ac941c3b272255c088
* ensure pool.reset event always called for resetMike Bayer2022-10-301-1/+1
| | | | | | | | | | | | | | | | | | | | | | Added new parameter :paramref:`.PoolEvents.reset.reset_state` parameter to the :meth:`.PoolEvents.reset` event, with deprecation logic in place that will continue to accept event hooks using the previous set of arguments. This indicates various state information about how the reset is taking place and is used to allow custom reset schemes to take place with full context given. Within this change a fix that's also backported to 1.4 is included which re-enables the :meth:`.PoolEvents.reset` event to continue to take place under all circumstances, including when :class:`.Connection` has already "reset" the connection. The two changes together allow custom reset schemes to be implemented using the :meth:`.PoolEvents.reset` event, instead of the :meth:`.PoolEvents.checkin` event (which continues to function as it always has). Change-Id: Ie17c4f55d02beb6f570b9de6b3044baffa7d6df6 Fixes: #8717
* implement batched INSERT..VALUES () () for executemanyMike Bayer2022-09-241-0/+28
| | | | | | | | | | | | | | | | | | | | the feature is enabled for all built in backends when RETURNING is used, except for Oracle that doesn't need it, and on psycopg2 and mssql+pyodbc it is used for all INSERT statements, not just those that use RETURNING. third party dialects would need to opt in to the new feature by setting use_insertmanyvalues to True. Also adds dialect-level guards against using returning with executemany where we dont have an implementation to suit it. execute single w/ returning still defers to the server without us checking. Fixes: #6047 Fixes: #7907 Change-Id: I3936d3c00003f02e322f2e43fb949d0e6e568304
* Generalize RETURNING and suppor for MariaDB / SQLiteDaniel Black2022-06-021-13/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As almost every dialect supports RETURNING now, RETURNING is also made more of a default assumption. * the default compiler generates a RETURNING clause now when specified; CompileError is no longer raised. * The dialect-level implicit_returning parameter now has no effect. It's not fully clear if there are real world cases relying on the dialect-level parameter, so we will see once 2.0 is released. ORM-level RETURNING can be disabled at the table level, and perhaps "implicit returning" should become an ORM-level option at some point as that's where it applies. * Altered ORM update() / delete() to respect table-level implicit returning for fetch. * Since MariaDB doesnt support UPDATE returning, "full_returning" is now split into insert_returning, update_returning, delete_returning * Crazy new thing. Dialects that have *both* cursor.lastrowid *and* returning. so now we can pick between them for SQLite and mariadb. so, we are trying to keep it on .lastrowid for simple inserts with an autoincrement column, this helps with some edge case test scenarios and i bet .lastrowid is faster anyway. any return_defaults() / multiparams etc then we use returning * SQLite decided they dont want to return rows that match in ON CONFLICT. this is flat out wrong, but for now we need to work with it. Fixes: #6195 Fixes: #7011 Closes: #7047 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7047 Pull-request-sha: d25d5ea3abe094f282c53c7dd87f5f53a9e85248 Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Change-Id: I9908ce0ff7bdc50bd5b27722081767c31c19a950
* adapt create_engine from sqlalchemy2-stubsMike Bayer2022-03-071-0/+59
| | | | | | | | this is much simplified, will try to see if _IsolationLevel can work out, technically some driver can have custom values here but in practice this might not be a thing Change-Id: I6085ccb559c377fab03c8ce79f0eecb240c56f7a
* pep-484 for engineMike Bayer2022-03-011-13/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | All modules in sqlalchemy.engine are strictly typed with the exception of cursor, default, and reflection. cursor and default pass with non-strict typing, reflection is waiting on the multi-reflection refactor. Behavioral changes: * create_connect_args() methods return a tuple of list, dict, rather than a list of list, dict * removed allow_chars parameter from pyodbc connector ._get_server_version_info() method * the parameter list passed to do_executemany is now a list in all cases. previously, this was being run through dialect.execute_sequence_format, which defaults to tuple and was only intended for individual tuple params. * broke up dialect.dbapi into dialect.import_dbapi class method and dialect.dbapi module object. added a deprecation path for legacy dialects. it's not really feasible to type a single attr as a classmethod vs. module type. The "type_compiler" attribute also has this problem with greater ability to work around, left that one for now. * lots of constants changing to be Enum, so that we can type them. for fixed tuple-position constants in cursor.py / compiler.py (which are used to avoid the speed overhead of namedtuple), using Literal[value] which seems to work well * some tightening up in Row regarding __getitem__, which we can do since we are on full 2.0 style result use * altered the set_connection_execution_options and set_engine_execution_options event flows so that the dictionary of options may be mutated within the event hook, where it will then take effect as the actual options used. Previously, changing the dict would be silently ignored which seems counter-intuitive and not very useful. * A lot of DefaultDialect/DefaultExecutionContext methods and attributes, including underscored ones, move to interfaces. This is not fully ideal as it means the Dialect/ExecutionContext interfaces aren't publicly subclassable directly, but their current purpose is more of documentation for dialect authors who should (and certainly are) still be subclassing the DefaultXYZ versions in all cases Overall, Result was the most extremely difficult class hierarchy to type here as this hierarchy passes through largely amorphous "row" datatypes throughout, which can in fact by all kinds of different things, like raw DBAPI rows, or Row objects, or "scalar"/Any, but at the same time these types have meaning so I tried still maintaining some level of semantic markings for these, it highlights how complex Result is now, as it's trying to be extremely efficient and inlined while also being very open-ended and extensible. Change-Id: I98b75c0c09eab5355fc7a33ba41dd9874274f12a
* pep-484 for poolMike Bayer2022-02-171-3/+8
| | | | | | | | | | | | | | | | | also extends into some areas of utils, events and others as needed. Formalizes a public hierarchy for pool API, with ManagesConnection -> PoolProxiedConnection / ConnectionPoolEntry for connectionfairy / connectionrecord, which are now what's exposed in the event API and other APIs. all public API docs moved to the new objects. Corrects the mypy plugin's check for sqlalchemy-stubs not being insatlled, which has to be imported using the dash in the name to be effective. Change-Id: I16c2cb43b2e840d28e70a015f370a768e70f3581
* pep-484 for sqlalchemy.event; use future annotationsMike Bayer2022-02-151-0/+2
| | | | | | | | | | __future__.annotations mode allows us to use non-string annotations for argument and return types in most cases, but more importantly it removes a large amount of runtime overhead that would be spent in evaluating the annotations. Change-Id: I2f5b6126fe0019713fc50001be3627b664019ede References: #6810
* establish mypy / typing approach for v2.0Mike Bayer2022-02-131-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | large patch to get ORM / typing efforts started. this is to support adding new test cases to mypy, support dropping sqlalchemy2-stubs entirely from the test suite, validate major ORM typing reorganization to eliminate the need for the mypy plugin. * New declarative approach which uses annotation introspection, fixes: #7535 * Mapped[] is now at the base of all ORM constructs that find themselves in classes, to support direct typing without plugins * Mypy plugin updated for new typing structures * Mypy test suite broken out into "plugin" tests vs. "plain" tests, and enhanced to better support test structures where we assert that various objects are introspected by the type checker as we expect. as we go forward with typing, we will add new use cases to "plain" where we can assert that types are introspected as we expect. * For typing support, users will be much more exposed to the class names of things. Add these all to "sqlalchemy" import space. * Column(ForeignKey()) no longer needs to be `@declared_attr` if the FK refers to a remote table * composite() attributes mapped to a dataclass no longer need to implement a `__composite_values__()` method * with_variant() accepts multiple dialect names Change-Id: I22797c0be73a8fbbd2d6f5e0c0b7258b17fe145d Fixes: #7535 Fixes: #7551 References: #6810
* mypy: sqlalchemy.utilMike Bayer2022-01-241-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Starting to set up practices and conventions to get the library typed. Key goals for typing are: 1. whole library can pass mypy without any strict turned on. 2. we can incrementally turn on some strict flags on a per-package/ module basis, as here we turn on more strictness for sqlalchemy.util, exc, and log 3. mypy ORM plugin tests work fully without sqlalchemy2-stubs installed 4. public facing methods all have return types, major parameter signatures filled in also 5. Foundational elements like util etc. are typed enough so that we can use them in fully typed internals higher up the stack. Conventions set up here: 1. we can use lots of config in setup.cfg to limit where mypy is throwing errors and how detailed it should be in different packages / modules. We can use this to push up gerrits that will pass tests fully without everything being typed. 2. a new tox target pep484 is added. this links to a new jenkins pep484 job that works across all projects (alembic, dogpile, etc.) We've worked around some mypy bugs that will likely be around for awhile, and also set up some core practices for how to deal with certain things such as public_factory modules (mypy won't accept a module from a callable at all, so need to use simple type checking conditionals). References: #6810 Change-Id: I80be58029896a29fd9f491aa3215422a8b705e12
* remove dbapi_proxy moduleMike Bayer2022-01-141-3/+0
| | | | | | | | | | | This module was not documented nor part of any test suite, and it's unlikely it was working correctly. It's not likely that this module was ever used after the first year or so of SQLAlchemy, and it's stayed around because it is so obscure that I never remembered to remove it. Change-Id: I0ed9030438982e935add87c51abbfff50e7382be References: #7257
* happy new year 2022Mike Bayer2022-01-061-1/+1
| | | | Change-Id: I49abf2607e0eb0623650efdf0091b1fb3db737ea
* provide connectionfairy on initializeMike Bayer2021-11-291-2/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is so that dialect methods that are called within init can assume the same argument structure as when they are called in other places; we can nail down the type of object as well. This change seems to mostly impact the isolation level routines in the dialects, as these are called during initialize() as well as on established connections. these methods can now assume a non-proxied DBAPI connection object in all cases, as it is commonly required that attributes like ".autocommit" are set on the object which don't work well in a proxied situation. Other changes: * adds an interface for the "connectionfairy" concept called PoolProxiedConnection. * Removes ``Connectable`` superclass of Connection. ``Connectable`` was originally meant to provide for the "method which accepts connection or engine" theme. As this pattern is greatly reduced in 2.0 and Engine no longer extends from it, the ``Connectable`` superclass doesnt serve any real purpose. Leading from that, to set this in I also applied pep 484 annotations to the Dialect base, and then in the interests of seeing some of the typing information show up in my IDE did a little bit for Engine, Connection and others. I hope that it's feasible that we can add annotations to specific classes and attributes ahead of when we actually try to mass-populate the whole library. This was the original spirit of pep-484 that we can apply annotations gradually. I do of course want to try to do a mass-populate although i think even in that case we will end up doing a lot of manual work anyway (in particular for the changes here which are distinct from what the stubs have). Fixes: #7122 Change-Id: I5dd7fbff8a7ae520a81c165091af12a6a68826db
* Added support for ``psycopg`` dialect.Federico Caselli2021-11-261-1/+5
| | | | | | | Both sync and async versions are supported. Fixes: #6842 Change-Id: I57751c5028acebfc6f9c43572562405453a2f2a4
* Merge "fully support isolation_level parameter in base dialect" into mainmike bayer2021-11-181-24/+26
|\
| * fully support isolation_level parameter in base dialectMike Bayer2021-11-181-24/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Generalized the :paramref:`_sa.create_engine.isolation_level` parameter to the base dialect so that it is no longer dependent on individual dialects to be present. This parameter sets up the "isolation level" setting to occur for all new database connections as soon as they are created by the connection pool, where the value then stays set without being reset on every checkin. The :paramref:`_sa.create_engine.isolation_level` parameter is essentially equivalent in functionality to using the :paramref:`_engine.Engine.execution_options.isolation_level` parameter via :meth:`_engine.Engine.execution_options` for an engine-wide setting. The difference is in that the former setting assigns the isolation level just once when a connection is created, the latter sets and resets the given level on each connection checkout. Fixes: #6342 Change-Id: Id81d6b1c1a94371d901ada728a610696e09e9741
* | Merge "Deprecate create_engine.implicit_returning" into mainmike bayer2021-11-181-4/+5
|\ \ | |/ |/|
| * Deprecate create_engine.implicit_returningjonathan vanasco2021-11-091-4/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The :paramref:`_sa.create_engine.implicit_returning` parameter is deprecated on the :func:`_sa.create_engine` function only; the parameter remains available on the :class:`_schema.Table` object. This parameter was originally intended to enable the "implicit returning" feature of SQLAlchemy when it was first developed and was not enabled by default. Under modern use, there's no reason this parameter should be disabled, and it has been observed to cause confusion as it degrades performance and makes it more difficult for the ORM to retrieve recently inserted server defaults. The parameter remains available on :class:`_schema.Table` to specifically suit database-level edge cases which make RETURNING infeasible, the sole example currently being SQL Server's limitation that INSERT RETURNING may not be used on a table that has INSERT triggers on it. Also removed from the Oracle dialect some logic that would upgrade an Oracle 8/8i server version to use implicit returning if the parameter were explictly passed; these versions of Oracle still support RETURNING so the feature is now enabled for all Oracle versions. Fixes: #6962 Change-Id: Ib338e300cd7c8026c3083043f645084a8211aed8
* | removals: all unicode encoding / decodingMike Bayer2021-11-101-72/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Removed here includes: * convert_unicode parameters * encoding create_engine() parameter * description encoding support * "non-unicode fallback" modes under Python 2 * String symbols regarding Python 2 non-unicode fallbacks * any concept of DBAPIs that don't accept unicode statements, unicode bound parameters, or that return bytes for strings anywhere except an explicit Binary / BLOB type * unicode processors in Python / C Risk factors: * Whether all DBAPIs do in fact return Unicode objects for all entries in cursor.description now * There was logic for mysql-connector trying to determine description encoding. A quick test shows Unicode coming back but it's not clear if there are still edge cases where they return bytes. if so, these are bugs in that driver, and at most we would only work around it in the mysql-connector DBAPI itself (but we won't do that either). * It seems like Oracle 8 was not expecting unicode bound parameters. I'm assuming this was all Python 2 stuff and does not apply for modern cx_Oracle under Python 3. * third party dialects relying upon built in unicode encoding/decoding but it's hard to imagine any non-SQLAlchemy database driver not dealing exclusively in Python unicode strings in Python 3 Change-Id: I97d762ef6d4dd836487b714d57d8136d0310f28a References: #7257
* | De-emphasize notion of "default driver" (DBAPI)Gord Thompson2021-11-091-2/+2
|/ | | | | | | | | | | Fixes: #6960 Even though a default driver still exists for each dialect, remove most usages of `dialect://` to encourage users to explicitly specify `dialect+driver://` Change-Id: I0ad42167582df509138fca64996bbb53e379b1af
* fully implement future engine and remove legacyMike Bayer2021-11-071-12/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | The major action here is to lift and move future.Connection and future.Engine fully into sqlalchemy.engine.base. This removes lots of engine concepts, including: * autocommit * Connection running without a transaction, autobegin is now present in all cases * most "autorollback" is obsolete * Core-level subtransactions (i.e. MarkerTransaction) * "branched" connections, copies of connections * execution_options() returns self, not a new connection * old argument formats, distill_params(), simplifies calling scheme between engine methods * before/after_execute() events (oriented towards compiled constructs) don't emit for exec_driver_sql(). before/after_cursor_execute() is still included for this * old helper methods superseded by context managers, connection.transaction(), engine.transaction() engine.run_callable() * ancient engine-level reflection methods has_table(), table_names() * sqlalchemy.testing.engines.proxying_engine References: #7257 Change-Id: Ib20ed816642d873b84221378a9ec34480e01e82c
* remove case_sensitive create_engine parameterMike Bayer2021-11-011-11/+0
| | | | | | | | | | | | Removed the previously deprecated ``case_sensitive`` parameter from :func:`_sa.create_engine`, which would impact only the lookup of string column names in Core-only result set rows; it had no effect on the behavior of the ORM. The effective behavior of what ``case_sensitive`` refers towards remains at its default value of ``True``, meaning that string names looked up in ``row._mapping`` will match case-sensitively, just like any other Python mapping. Change-Id: I0dc4be3fac37d30202b1603db26fa10a110b618d
* changelog, doc editsMike Bayer2021-09-221-8/+12
| | | | Change-Id: Iafb50de7e28626d9cee755db9c05ac7189b4d963
* clarify "encoding" parameter furtherMike Bayer2021-09-201-8/+16
| | | | | | | | | Prevent any reading of this parameter that would omit that it is not used under Python 3 and in Python 2 is not used very much either. Fixes: #7050 Change-Id: Iaf619f1ee164fc58afe710d11627ed6368d74343
* Replace all http:// links to https://Federico Caselli2021-07-041-2/+2
| | | | | | Also replace http://pypi.python.org/pypi with https://pypi.org/project Change-Id: I84b5005c39969a82140706472989f2a30b0c7685
* Pass URL object, not the string, to on_connect_urlMike Bayer2021-06-061-2/+1
| | | | | | | | | | | | The fix for pysqlcipher released in version 1.4.3 :ticket:`5848` was unfortunately non-working, in that the new ``on_connect_url`` hook was erroneously not receiving a ``URL`` object under normal usage of :func:`_sa.create_engine` and instead received a string that was unhandled; the test suite failed to fully set up the actual conditions under which this hook is called. This has been fixed. Fixes: #6586 Change-Id: I3bf738daec35877a10fdad740f08dca9e7420829
* Add new "sync once" mode for pool.connectMike Bayer2021-04-211-9/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed critical regression caused by the change in :ticket`5497` where the connection pool "init" phase no longer occurred within mutexed isolation, allowing other threads to proceed with the dialect uninitialized, which could then impact the compilation of SQL statements. This issue is essentially the same regression which was fixed many years ago in :ticket:`2964` in dd32540dabbee0678530fb1b0868d1eb41572dca, which was missed this time as the test suite fo that issue only tested the pool in isolation, and assumed the "first_connect" event would be used by the Engine. However :ticket:`5497` stopped using "first_connect" and no test detected the lack of mutexing, that has been resolved here through the addition of more tests. This fix also identifies what is probably a bug in earlier versions of SQLAlchemy where the "first_connect" handler would be cancelled if the initializer failed; this is evidenced by test_explode_in_initializer which was doing a reconnect due to c.rollback() yet wasn't hanging. We now solve this issue by preventing the manufactured Connection from ever reconnecting inside the first_connect handler. Also remove the "_sqla_unwrap" test attribute; this is almost not used anymore however we can use a more targeted wrapper supplied by the testing.engines.proxying_engine function. See if we can also open up Oracle for "ad hoc engines" tests now that we have better connection management logic. Fixes: #6337 Change-Id: I4a3476625c4606f1a304dbc940d500325e8adc1a
* Repair pysqlcipher and use sqlcipher3Mike Bayer2021-03-241-1/+1
| | | | | | | | | | | | | | | | | | | | | | | The ``pysqlcipher`` dialect now imports the ``sqlcipher3`` module for Python 3 by default. Regressions have been repaired such that the connection routine was not working. To better support the post-connection steps of the pysqlcipher dialect, a new hook Dialect.on_connect_url() is added, which supersedes Dialect.on_connect() and is passed the URL object. The dialect now pulls the passphrase and other cipher args from the URL directly without including them in the "connect" args. This will allow any user-defined extensibility to connecting to work as it would for other dialects. The commit also builds upon the extended routines in sqlite/provisioning.py to better support running tests against multiple simultaneous SQLite database files. Additionally enables backend for test_sqlite which was skipping everything for aiosqlite too, fortunately everything there is passing. Fixes: #5848 Change-Id: I43f53ebc62298a84a4abe149e1eb699a027b7915
* work around formatting issue with the :param: tagMike Bayer2021-03-171-1/+3
| | | | | Fixes: #6075 Change-Id: Ia3f6109e3a038ddcf513d3e887b4cad0f776f0a6
* Add emphasis to URL documentation re: URL encodingMike Bayer2021-02-141-2/+7
| | | | | Fixes: #5715 Change-Id: I2ac16541d34f49b25070e00c43596bcd71aff72d
* Implement per-connection logging tokenMike Bayer2021-02-031-0/+2
| | | | | | | | | | | | | | | Added new execution option :paramref:`_engine.Connection.execution_options.logging_token`. This option will add an additional per-message token to log messages generated by the :class:`_engine.Connection` as it executes statements. This token is not part of the logger name itself (that part can be affected using the existing :paramref:`_sa.create_engine.logging_name` parameter), so is appropriate for ad-hoc connection use without the side effect of creating many new loggers. The option can be set at the level of :class:`_engine.Connection` or :class:`_engine.Engine`. Fixes: #5911 Change-Id: Iec9c39b868b3578fcedc1c094dace5b6f64bacea
* reinvent xdist hooks in terms of pytest fixturesMike Bayer2021-01-131-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To allow the "connection" pytest fixture and others work correctly in conjunction with setup/teardown that expects to be external to the transaction, remove and prevent any usage of "xdist" style names that are hardcoded by pytest to run inside of fixtures, even function level ones. Instead use pytest autouse fixtures to implement our own r"setup|teardown_test(?:_class)?" methods so that we can ensure function-scoped fixtures are run within them. A new more explicit flow is set up within plugin_base and pytestplugin such that the order of setup/teardown steps, which there are now many, is fully documented and controllable. New granularity has been added to the test teardown phase to distinguish between "end of the test" when lock-holding structures on connections should be released to allow for table drops, vs. "end of the test plus its teardown steps" when we can perform final cleanup on connections and run assertions that everything is closed out. From there we can remove most of the defensive "tear down everything" logic inside of engines which for many years would frequently dispose of pools over and over again, creating for a broken and expensive connection flow. A quick test shows that running test/sql/ against a single Postgresql engine with the new approach uses 75% fewer new connections, creating 42 new connections total, vs. 164 new connections total with the previous system. As part of this, the new fixtures metadata/connection/future_connection have been integrated such that they can be combined together effectively. The fixture_session(), provide_metadata() fixtures have been improved, including that fixture_session() now strongly references sessions which are explicitly torn down before table drops occur afer a test. Major changes have been made to the ConnectionKiller such that it now features different "scopes" for testing engines and will limit its cleanup to those testing engines corresponding to end of test, end of test class, or end of test session. The system by which it tracks DBAPI connections has been reworked, is ultimately somewhat similar to how it worked before but is organized more clearly along with the proxy-tracking logic. A "testing_engine" fixture is also added that works as a pytest fixture rather than a standalone function. The connection cleanup logic should now be very robust, as we now can use the same global connection pools for the whole suite without ever disposing them, while also running a query for PostgreSQL locks remaining after every test and assert there are no open transactions leaking between tests at all. Additional steps are added that also accommodate for asyncio connections not explicitly closed, as is the case for legacy sync-style tests as well as the async tests themselves. As always, hundreds of tests are further refined to use the new fixtures where problems with loose connections were identified, largely as a result of the new PostgreSQL assertions, many more tests have moved from legacy patterns into the newest. An unfortunate discovery during the creation of this system is that autouse fixtures (as well as if they are set up by @pytest.mark.usefixtures) are not usable at our current scale with pytest 4.6.11 running under Python 2. It's unclear if this is due to the older version of pytest or how it implements itself for Python 2, as well as if the issue is CPU slowness or just large memory use, but collecting the full span of tests takes over a minute for a single process when any autouse fixtures are in place and on CI the jobs just time out after ten minutes. So at the moment this patch also reinvents a small version of "autouse" fixtures when py2k is running, which skips generating the real fixture and instead uses two global pytest fixtures (which don't seem to impact performance) to invoke the "autouse" fixtures ourselves outside of pytest. This will limit our ability to do more with fixtures until we can remove py2k support. py.test is still observed to be much slower in collection in the 4.6.11 version compared to modern 6.2 versions, so add support for new TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that will run the suite for fewer backends under Python 2. For Python 3 pin pytest to modern 6.2 versions where performance for collection has been improved greatly. Includes the following improvements: Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would be raised rather than :class:`.exc.TimeoutError`. Also repaired the :paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using the async engine, which previously would ignore the timeout and block rather than timing out immediately as is the behavior with regular :class:`.QueuePool`. For asyncio the connection pool will now also not interact at all with an asyncio connection whose ConnectionFairy is being garbage collected; a warning that the connection was not properly closed is emitted and the connection is discarded. Within the test suite the ConnectionKiller is now maintaining strong references to all DBAPI connections and ensuring they are released when tests end, including those whose ConnectionFairy proxies are GCed. Identified cx_Oracle.stmtcachesize as a major factor in Oracle test scalability issues, this can be reset on a per-test basis rather than setting it to zero across the board. the addition of this flag has resolved the long-standing oracle "two task" error problem. For SQL Server, changed the temp table style used by the "suite" tests to be the double-pound-sign, i.e. global, variety, which is much easier to test generically. There are already reflection tests that are more finely tuned to both styles of temp table within the mssql test suite. Additionally, added an extra step to the "dropfirst" mechanism for SQL Server that will remove all foreign key constraints first as some issues were observed when using this flag when multiple schemas had not been torn down. Identified and fixed two subtle failure modes in the engine, when commit/rollback fails in a begin() context manager, the connection is explicitly closed, and when "initialize()" fails on the first new connection of a dialect, the transactional state on that connection is still rolled back. Fixes: #5826 Fixes: #5827 Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
* happy new yearMike Bayer2021-01-041-1/+1
| | | | Change-Id: Ic5bb19ca8be3cb47c95a0d3315d84cb484bac47c
* QueuePool: support subsecond timeoutJordan Pittier2020-11-211-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes: #5719 <!-- Provide a general summary of your proposed changes in the Title field above --> ### Description Make it explicit in the documentation and in the default value for the 'timeout' parameter that `timeout` can be a float. Because Python timing is not very accurate, warn about the precision. ### Checklist <!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once) --> This pull request is: - [ ] A documentation / typographical error fix - Good to go, no issue or tests are needed - [x] A short code fix - please include the issue number, and create an issue if none exists, which must include a complete example of the issue. one line code fixes without an issue and demonstration will not be accepted. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. one line code fixes without tests will not be accepted. - [ ] A new feature implementation - please include the issue number, and create an issue if none exists, which must include a complete example of how the feature would look. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. **Have a nice day!** Closes: #5710 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5710 Pull-request-sha: 5f4eef8b4aba756d32e14ea41f71ef2919c26b84 Change-Id: I462524b1624ca5cc76d083a1d58e5dc89501c1a9
* Support pool.connect() event firing before all elseMike Bayer2020-11-191-11/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed regression where a connection pool event specified with a keyword, most notably ``insert=True``, would be lost when the event were set up. This would prevent startup events that need to fire before dialect-level events from working correctly. The internal mechanics of the engine connection routine has been altered such that it's now guaranteed that a user-defined event handler for the :meth:`_pool.PoolEvents.connect` handler, when established using ``insert=True``, will allow an event handler to run that is definitely invoked **before** any dialect-specific initialization starts up, most notably when it does things like detect default schema name. Previously, this would occur in most cases but not unconditionally. A new example is added to the schema documentation illustrating how to establish the "default schema name" within an on-connect event (upcoming as part of I882edd5bbe06ee5b4d0a9c148854a57b2bcd4741) Addiional changes to support setting default schema name: The Oracle dialect now uses ``select sys_context( 'userenv', 'current_schema' ) from dual`` to get the default schema name, rather than ``SELECT USER FROM DUAL``, to accommodate for changes to the session-local schema name under Oracle. Added a read/write ``.autocommit`` attribute to the DBAPI-adaptation layer for the asyncpg dialect. This so that when working with DBAPI-specific schemes that need to use "autocommit" directly with the DBAPI connection, the same ``.autocommit`` attribute which works with both psycopg2 as well as pg8000 is available. Fixes: #5716 Fixes: #5708 Change-Id: I7dce56b4345ffc720e25e2aaccb7e42bb29e5671
* Support "sqlalchemy.future" for engine_from_configMike Bayer2020-11-061-1/+1
| | | | | | | | | Added the "future" keyword to the list of words that are known by the :func:`_sa.engine_from_config` function, so that the values "true" and "false" may be configured as "boolean" values when using a key such as ``sqlalchemy.future = true`` or ``sqlalchemy.future = false``. Change-Id: Ib4bba748497cc68e4c913dde54c23a4bb08b4deb
* Reduce import time overheadMike Bayer2020-11-031-1/+1
| | | | | | | | | | | | | | | | | | * Fix subclass traversals to not run classes multiple times * switch compiler visitor to use an attrgetter, to avoid an eval() at startup time * don't pre-generate traversal functions, there's lots of these which are expensive to generate at once and most applications won't use them all; have it generate them on first use instead * Some ideas about removing asyncio imports, they don't seem to be too signficant, apply some more simplicity to the overall "greenlet fallback" situation Fixes: #5681 Change-Id: Ib564ddaddb374787ce3e11ff48026e99ed570933
* improve engine logging docsMike Bayer2020-10-241-0/+20
| | | | | | | the text here was a little confusing and didn't refer to major configurational elements such as hide_parameters. Change-Id: I4e2179e5a64c326d30b65a8871b924725c41b453
* Fix typo in :paramref:`_engine.create_engine.isolation_level` namesFederico Caselli2020-09-081-2/+2
| | | | | Fixes: #5563 Change-Id: I29204fdf679d750c66ed17daf70bc8d7cb1b7f65
* make URL immutableMike Bayer2020-08-251-4/+1
| | | | | | | | | | | | | | it's not really correct that URL is mutable and doesn't do any argument checking. propose replacing it with an immutable named tuple with rich copy-and-mutate methods. At the moment this makes a hard change to the CreateEnginePlugin docs that previously recommended url.query.pop(). I can't find any plugins on github other than my own that are using this feature, so see if we can just make a hard change on this one. Fixes: #5526 Change-Id: I28a0a471d80792fa8c28f4fa573d6352966a4a79
* Implement rudimentary asyncio support w/ asyncpgMike Bayer2020-08-131-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Using the approach introduced at https://gist.github.com/zzzeek/6287e28054d3baddc07fa21a7227904e We can now create asyncio endpoints that are then handled in "implicit IO" form within the majority of the Core internals. Then coroutines are re-exposed at the point at which we call into asyncpg methods. Patch includes: * asyncpg dialect * asyncio package * engine, result, ORM session classes * new test fixtures, tests * some work with pep-484 and a short plugin for the pyannotate package, which seems to have so-so results Change-Id: Idbcc0eff72c4cad572914acdd6f40ddb1aef1a7d Fixes: #3414
* Don't link on_connect to first_connect event handlerMike Bayer2020-08-071-4/+12
| | | | | | | | | | | | | Adjusted the dialect initialization process such that the :meth:`_engine.Dialect.on_connect` is not called a second time on the first connection. The hook is called first, then the :meth:`_engine.Dialect.initialize` is called if that connection is the first for that dialect, then no more events are called. This eliminates the two calls to the "on_connect" function which can produce very difficult debugging situations. Fixes: #5497 Change-Id: Icefc2e884e30ee7b4ac84b99dc54bf992a6085e3
* Robustness for lambdas, lambda statementsMike Bayer2020-08-051-1/+1
| | | | | | | | | | | | | | | | | | | | in order to accommodate relationship loaders with lambda caching, a lot more is needed. This is a full refactor of the lambda system such that it now has two levels of caching; the first level caches what can be known from the __code__ element, then the next level of caching is against the lambda itself and the contents of __closure__. This allows for the elements inside the lambdas, like columns and entities, to change and then be part of the cache key. Lazy/selectinloads' use of baked queries had to add distinct cache key elements, which was attempted here but overall things needed to be more robust than that. This commit is broken out from the very long and sprawling commit at Id6b5c03b1ce9ddb7b280f66792212a0ef0a1c541 . Change-Id: I29a513c98917b1d503abfdd61e6b6e8800851aa8
* Add future=True to create_engine/Session; unify select()Mike Bayer2020-07-081-1/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several weeks of using the future_select() construct has led to the proposal there be just one select() construct again which features the new join() method, and otherwise accepts both the 1.x and 2.x argument styles. This would make migration simpler and reduce confusion. However, confusion may be increased by the fact that select().join() is different Current thinking is we may be better off with a few hard behavioral changes to old and relatively unknown APIs rather than trying to play both sides within two extremely similar but subtly different APIs. At the moment, the .join() thing seems to be the only behavioral change that occurs without the user taking any explicit steps. Session.execute() will still behave the old way as we are adding a future flag. This change also adds the "future" flag to Session() and session.execute(), so that interpretation of the incoming statement, as well as that the new style result is returned, does not occur for existing applications unless they add the use of this flag. The change in general is moving the "removed in 2.0" system further along where we want the test suite to fully pass even if the SQLALCHEMY_WARN_20 flag is set. Get many tests to pass when SQLALCHEMY_WARN_20 is set; this should be ongoing after this patch merges. Improve the RemovedIn20 warning; these are all deprecated "since" 1.4, so ensure that's what the messages read. Make sure the inforamtion link is on all warnings. Add deprecation warnings for parameters present and add warnings to all FromClause.select() types of methods. Fixes: #5379 Fixes: #5284 Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51 References: #5159
* introduce deferred lambdasMike Bayer2020-07-031-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The coercions system allows us to add in lambdas as arguments to Core and ORM elements without changing them at all. By allowing the lambda to produce a deterministic cache key where we can also cheat and yank out literal parameters means we can move towards having 90% of "baked" functionality in a clearer way right in Core / ORM. As a second step, we can have whole statements inside the lambda, and can then add generation with __add__(), so then we have 100% of "baked" functionality with full support of ad-hoc literal values. Adds some more short_selects tests for the moment for comparison. Other tweaks inside cache key generation as we're trying to approach a certain level of performance such that we can remove the use of "baked" from the loader strategies. As we have not yet closed #4639, however the caching feature has been fully integrated as of b0cfa7379cf8513a821a3dbe3028c4965d9f85bd, we will also add complete caching documentation here and close that issue as well. Closes: #4639 Fixes: #5380 Change-Id: If91f61527236fd4d7ae3cad1f24c38be921c90ba
View Lorry Controller Status