summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/engine
Commit message (Collapse)AuthorAgeFilesLines
* more docs for autocommit isolation levelMike Bayer2020-07-121-10/+8
| | | | | | | | this concept is not clear that we offer real DBAPI autocommit everywhere. backport 1.3 with edits as well Change-Id: I2e8328b7fb6e1cdc5453ab29c94276f60c7ca149
* Convert remaining ORM APIs to support 2.0 styleMike Bayer2020-07-112-1/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is kind of a mixed bag of all kinds to help get us to 1.4 betas. The documentation stuff is a work in progress. Lots of other relatively small changes to APIs and things. More commits will follow to continue improving the documentation and transitioning to the 1.4/2.0 hybrid documentation. In particular some refinements to Session usage models so that it can match Engine's scoping / transactional patterns, and a decision to start moving away from "subtransactions" completely. * add select().from_statement() to produce FromStatement in an ORM context * begin referring to select() that has "plugins" for the few edge cases where select() will have ORM-only behaviors * convert dynamic.AppenderQuery to its own object that can use select(), though at the moment it uses Query to support legacy join calling forms. * custom query classes for AppenderQuery are replaced by do_orm_execute() hooks for custom actions, a separate gerrit will document this * add Session.get() to replace query.get() * Deprecate session.begin->subtransaction. propose within the test suite a hypothetical recipe for apps that rely on this pattern * introduce Session construction level context manager, sessionmaker context manager, rewrite the whole top of the session_transaction.rst documentation. Establish context manager patterns for Session that are identical to engine * ensure same begin_nested() / commit() behavior as engine * devise all new "join into an external transaction" recipe, add test support for it, add rules into Session so it just works, write new docs. need to ensure this doesn't break anything * vastly reduce the verbosity of lots of session docs as I dont think people read this stuff and it's difficult to keep current in any case * constructs like case(), with_only_columns() really need to move to *columns, add a coercion rule to just change these. * docs need changes everywhere I look. in_() is not in the Core tutorial? how do people even know about it? Remove tons of cruft from Select docs, etc. * build a system for common ORM options like populate_existing and autoflush to populate from execution options. * others? Change-Id: Ia4bea0f804250e54d90b3884cf8aab8b66b82ecf
* Add future=True to create_engine/Session; unify select()Mike Bayer2020-07-083-6/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several weeks of using the future_select() construct has led to the proposal there be just one select() construct again which features the new join() method, and otherwise accepts both the 1.x and 2.x argument styles. This would make migration simpler and reduce confusion. However, confusion may be increased by the fact that select().join() is different Current thinking is we may be better off with a few hard behavioral changes to old and relatively unknown APIs rather than trying to play both sides within two extremely similar but subtly different APIs. At the moment, the .join() thing seems to be the only behavioral change that occurs without the user taking any explicit steps. Session.execute() will still behave the old way as we are adding a future flag. This change also adds the "future" flag to Session() and session.execute(), so that interpretation of the incoming statement, as well as that the new style result is returned, does not occur for existing applications unless they add the use of this flag. The change in general is moving the "removed in 2.0" system further along where we want the test suite to fully pass even if the SQLALCHEMY_WARN_20 flag is set. Get many tests to pass when SQLALCHEMY_WARN_20 is set; this should be ongoing after this patch merges. Improve the RemovedIn20 warning; these are all deprecated "since" 1.4, so ensure that's what the messages read. Make sure the inforamtion link is on all warnings. Add deprecation warnings for parameters present and add warnings to all FromClause.select() types of methods. Fixes: #5379 Fixes: #5284 Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51 References: #5159
* introduce deferred lambdasMike Bayer2020-07-034-5/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The coercions system allows us to add in lambdas as arguments to Core and ORM elements without changing them at all. By allowing the lambda to produce a deterministic cache key where we can also cheat and yank out literal parameters means we can move towards having 90% of "baked" functionality in a clearer way right in Core / ORM. As a second step, we can have whole statements inside the lambda, and can then add generation with __add__(), so then we have 100% of "baked" functionality with full support of ad-hoc literal values. Adds some more short_selects tests for the moment for comparison. Other tweaks inside cache key generation as we're trying to approach a certain level of performance such that we can remove the use of "baked" from the loader strategies. As we have not yet closed #4639, however the caching feature has been fully integrated as of b0cfa7379cf8513a821a3dbe3028c4965d9f85bd, we will also add complete caching documentation here and close that issue as well. Closes: #4639 Fixes: #5380 Change-Id: If91f61527236fd4d7ae3cad1f24c38be921c90ba
* Merge "ORM executemany returning"mike bayer2020-06-281-3/+4
|\
| * ORM executemany returningMike Bayer2020-06-271-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Build on #5401 to allow the ORM to take advanage of executemany INSERT + RETURNING. Implemented the feature updated tests to support INSERT DEFAULT VALUES, needed to come up with a new syntax for compiler INSERT INTO table (anycol) VALUES (DEFAULT) which can then be iterated out for executemany. Added graceful degrade to plain executemany for PostgreSQL <= 8.2 Renamed EXECUTEMANY_DEFAULT to EXECUTEMANY_PLAIN Fix issue where unicode identifiers or parameter names wouldn't work with execute_values() under Py2K, because we have to encode the statement and therefore have to encode the insert_single_values_expr too. Correct issue from #5401 to support executemany + return_defaults for a PK that is explicitly pre-generated, meaning we aren't actually getting RETURNING but need to return it from compiled_parameters. Fixes: #5263 Change-Id: Id68e5c158c4f9ebc33b61c06a448907921c2a657
* | Merge "Fix a wide variety of typos and broken links"mike bayer2020-06-2610-52/+52
|\ \ | |/ |/|
| * Fix a wide variety of typos and broken linksaplatkouski2020-06-2510-52/+52
| | | | | | | | | | | | | | | | | | | | | | | | Note the PR has a few remaining doc linking issues listed in the comment that must be addressed separately. Signed-off-by: aplatkouski <5857672+aplatkouski@users.noreply.github.com> Closes: #5371 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5371 Pull-request-sha: 7e7d233cf3a0c66980c27db0fcdb3c7d93bc2510 Change-Id: I9c36e8d8804483950db4b42c38ee456e384c59e3
* | Merge "Default psycopg2 executemany mode to "values_only""mike bayer2020-06-263-73/+98
|\ \ | |/ |/|
| * Default psycopg2 executemany mode to "values_only"Mike Bayer2020-06-253-73/+98
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The psycopg2 dialect now defaults to using the very performant ``execute_values()`` psycopg2 extension for compiled INSERT statements, and also impements RETURNING support when this extension is used. This allows INSERT statements that even include an autoincremented SERIAL or IDENTITY value to run very fast while still being able to return the newly generated primary key values. The ORM will then integrate this new feature in a separate change. Implements RETURNING for insert with executemany Adds support to return_defaults() mode and inserted_primary_key to support mutiple INSERTed rows, via return_defauls_rows and inserted_primary_key_rows accessors. within default execution context, new cached compiler getters are used to fetch primary keys from rows inserted_primary_key now returns a plain tuple. this is not yet a row-like object however this can be added. Adds distinct "values_only" and "batch" modes, as "values" has a lot of benefits but "batch" breaks cursor.rowcount psycopg2 minimum version 2.7 so we can remove the large number of checks for very old versions of psycopg2 simplify tests to no longer distinguish between native and non-native json Fixes: #5401 Change-Id: Ic08fd3423d4c5d16ca50994460c0c234868bd61c
* | Use time.perf_counter() for cache time measurementMike Bayer2020-06-241-2/+1
|/ | | | | | See https://twitter.com/raymondh/status/1275937373080023040 Change-Id: Iaa0abb0c433ccedfbd88d00e3970120242ba379b
* Propose using RETURNING for bulk updates, deletesMike Bayer2020-06-232-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch makes several improvements in the area of bulk updates and deletes as well as the new session mechanics. RETURNING is now used for an UPDATE or DELETE statement emitted for a diaelct that supports "full returning" in order to satisfy the "fetch" strategy; this currently includes PostgreSQL and SQL Server. The Oracle dialect does not support RETURNING for more than one row, so a new dialect capability "full_returning" is added in addition to the existing "implicit_returning", indicating this dialect supports RETURNING for zero or more rows, not just a single identity row. The "fetch" strategy will gracefully degrade to the previous SELECT mechanics for dialects that do not support RETURNING. Additionally, the "fetch" strategy will attempt to use evaluation for the VALUES that were UPDATEd, rather than just expiring the updated attributes. Values should be evalutable in all cases where the value is not a SQL expression. The new approach also incurs some changes in the session.execute mechanics, where do_orm_execute() event handlers can now be chained to each return results; this is in turn used by the handler to detect on a per-bind basis if the fetch strategy needs to do a SELECT or if it can do RETURNING. A test suite is added to test_horizontal_shard that breaks up a single UPDATE or DELETE operation among multiple backends where some are SQLite and don't support RETURNING and others are PostgreSQL and do. The session event mechanics are corrected in terms of the "orm pre execute" hook, which now receives a flag "is_reentrant" so that the two ORM implementations for this can skip on their work if they are being called inside of ORMExecuteState.invoke(), where previously bulk update/delete were calling its SELECT a second time. In order for "fetch" to get the correct identity when called as pre-execute, it also requests the identity_token for each mapped instance which is now added as an optional capability of a SELECT for ORM columns. the identity_token that's placed by horizontal_sharding is now made available within each result row, so that even when fetching a merged result of plain rows we can tell which row belongs to which identity token. The evaluator that takes place within the ORM bulk update and delete for synchronize_session="evaluate" now supports the IN and NOT IN operators. Tuple IN is also supported. Fixes: #1653 Change-Id: I2292b56ae004b997cef0ba4d3fc350ae1dd5efc1
* Merge "Added reflection method :meth:`.Inspector.get_sequence_names`"mike bayer2020-06-192-7/+41
|\
| * Added reflection method :meth:`.Inspector.get_sequence_names`Federico Caselli2020-06-032-7/+41
| | | | | | | | | | | | | | | | | | | | Added new reflection method :meth:`.Inspector.get_sequence_names` which returns all the sequences defined. Support for this method has been added to the backend that support :class:`.Sequence`: PostgreSql, Oracle, MSSQL and MariaDB >= 10.3. Fixes: #2056 Change-Id: I0949696a39aa28c849edf2504779241f7443778a
* | perf tweaksMike Bayer2020-06-191-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | - avoid abc checks in distill_20 - ColumnEntity subclasses are unique to their compile state and have no querycontext specific state. They can do a simple memoize of their fetch_column without using attributes, and they can memoize their _getter() too so that it goes into the cache, just like instance_processor() does. - unify ORMColumnEntity and RawColumnEntity for the row processor part, add some test coverage for the case where it is used in a from_statement - do a faster generate if there are no memoized entries - query._params is always immutabledict Change-Id: I1e2dfe607a1749b5b434fc11f9348ee631501dfa
* | Warn when transaction context manager ends on inactive transactionMike Bayer2020-06-121-3/+6
| | | | | | | | | | | | | | | | | | | | if .rollback() or .commit() is called inside the transaction context manager, the transaction object is deactivated. the context manager continues but will not be able to correctly fulfill it's closing state. Ensure a warning is emitted when this happens. Change-Id: I8fc3a73f7c21575dda5bcbd6fb74ddb679771630
* | Turn on caching everywhere, add loggingMike Bayer2020-06-104-100/+122
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A variety of caching issues found by running all tests with statement caching turned on. The cache system now has a more conservative approach where any subclass of a SQL element will by default invalidate the cache key unless it adds the flag inherit_cache=True at the class level, or if it implements its own caching. Add working caching to a few elements that were omitted previously; fix some caching implementations to suit lesser used edge cases such as json casts and array slices. Refine the way BaseCursorResult and CursorMetaData interact with caching; to suit cases like Alembic modifying table structures, don't cache the cursor metadata if it were created against a cursor.description using non-positional matching, e.g. "select *". if a table re-ordered its columns or added/removed, now that data is obsolete. Additionally we have to adapt the cursor metadata _keymap regardless of if we just processed cursor.description, because if we ran against a cached SQLCompiler we won't have the right columns in _keymap. Other refinements to how and when we do this adaption as some weird cases were exposed in the Postgresql dialect, a text() construct that names just one column that is not actually in the statement. Fixed that also as it looks like a cut-and-paste artifact that doesn't actually affect anything. Various issues with re-use of compiled result maps and cursor metadata in conjunction with tables being changed, such as change in order of columns. mappers can be cleared but the class remains, meaning a mapper has to use itself as the cache key not the class. lots of bound parameter / literal issues, due to Alembic creating a straight subclass of bindparam that renders inline directly. While we can update Alembic to not do this, we have to assume other people might be doing this, so bindparam() implements the inherit_cache=True logic as well that was a bit involved. turn on cache stats in logging. Includes a fix to subqueryloader which moves all setup to the create_row_processor() phase and elminates any storage within the compiled context. This includes some changes to create_row_processor() signature and a revising of the technique used to determine if the loader can participate in polymorphic queries, which is also applied to selectinloading. DML update.values() and ordered_values() now coerces the keys as we have tests that pass an arbitrary class here which only includes __clause_element__(), so the key can't be cached unless it is coerced. this in turn changed how composite attributes support bulk update to use the standard approach of ClauseElement with annotations that are parsed in the ORM context. memory profiling successfully caught that the Session from Query was getting passed into _statement_20() so that was a big win for that test suite. Apparently Compiler had .execute() and .scalar() methods stuck on it, these date back to version 0.4 and there was a single test in the PostgreSQL dialect tests that exercised it for no apparent reason. Removed these methods as well as the concept of a Compiler holding onto a "bind". Fixes: #5386 Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
* | Convert bulk update/delete to new execution modelMike Bayer2020-06-062-16/+78
|/ | | | | | | | | | | | | | | This reorganizes the BulkUD model in sqlalchemy.orm.persistence to be based on the CompileState concept and to allow plain update() / delete() to be passed to session.execute() where the ORM synchronize session logic will take place. Also gets "synchronize_session='fetch'" working with horizontal sharding. Adding a few more result.scalar_one() types of methods as scalar_one() seems like what is normally desired. Fixes: #5160 Change-Id: I8001ebdad089da34119eb459709731ba6c0ba975
* Merge "Improve rendering of core statements w/ ORM elements"mike bayer2020-06-011-2/+0
|\
| * Improve rendering of core statements w/ ORM elementsMike Bayer2020-05-311-2/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch contains a variety of ORM and expression layer tweaks to support ORM constructs in select() statements, without the 1.3.x requiremnt in Query that a full _compile_context() + new select() is needed in order to get a working statement object. Includes such tweaks as the ability to implement aliased class of an aliased class, as we are looking to fully support ACs against subqueries, as well as the ability to access anonymously-labeled ColumnProperty expressions within subqueries by naming the ".key" of the label after the property key. Some tuning to query.join() as well as ORMJoin internals to allow things to work more smoothly. Change-Id: Id810f485c5f7ed971529489b84694e02a3356d6d
* | Add support for "real" sequences in mssqlGord Thompson2020-05-291-0/+3
|/ | | | | | | | | | | | | | | | | Added support for "CREATE SEQUENCE" and full :class:`.Sequence` support for Microsoft SQL Server. This removes the deprecated feature of using :class:`.Sequence` objects to manipulate IDENTITY characteristics which should now be performed using ``mssql_identity_start`` and ``mssql_identity_increment`` as documented at :ref:`mssql_identity`. The change includes a new parameter :paramref:`.Sequence.data_type` to accommodate SQL Server's choice of datatype, which for that backend includes INTEGER and BIGINT. The default starting value for SQL Server's version of :class:`.Sequence` has been set at 1; this default is now emitted within the CREATE SEQUENCE DDL for all backends. Fixes: #4235 Fixes: #4633 Change-Id: I6aa55c441e8146c2f002e2e201a7f645e667b916
* callcount reductions and refinement for cached queriesMike Bayer2020-05-286-415/+421
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit includes that we've removed the "_orm_query" attribute from compile state as well as query context. The attribute created reference cycles and also added method call overhead. As part of this change, the interface for ORMExecuteState changes a bit, as well as the interface for the horizontal sharding extension which now deprecates the "query_chooser" callable in favor of "execute_chooser", which receives the contextual object. This will also work more nicely when we implement the new execution path for bulk updates and deletes. Pre-merge execution options for statement, connection, arguments all up front in Connection. that way they can be passed to the before_execute / after_execute events, and the ExecutionContext doesn't have to merge as second time. Core execute is pretty close to 1.3 now. baked wasn't using the new one()/first()/one_or_none() methods, fixed that. Convert non-buffered cursor strategy to be a stateless singleton. inline all the paths by which the strategy gets chosen, oracle and SQL Server dialects make use of the already-invoked post_exec() hook to establish the alternate strategies, and this is actually much nicer than it was before. Add caching to mapper instance processor for getters. Identified a reference cycle per query that was showing up as a lot of gc cleanup, fixed that. After all that, performance not budging much. Even test_baked_query now runs with significantly fewer function calls than 1.3, still 40% slower. Basically something about the new patterns just makes this slower and while I've walked a whole bunch of them back, it hardly makes a dent. that said, the performance issues are relatively small, in the 20-40% time increase range, and the new caching feature does provide for regular ORM and Core queries that are cached, and they are faster than non-cached. Change-Id: I7b0b0d8ca550c05f79e82f75cd8eff0bbfade053
* Convert execution to move through SessionMike Bayer2020-05-255-59/+172
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the ORM execution flow with a single pathway through Session.execute() for all queries, including Core and ORM. Currently included is full support for ORM Query, Query.from_statement(), select(), as well as the baked query and horizontal shard systems. Initial changes have also been made to the dogpile caching example, which like baked query makes use of a new ORM-specific execution hook that replaces the use of both QueryEvents.before_compile() as well as Query._execute_and_instances() as the central ORM interception hooks. select() and Query() constructs alike can be passed to Session.execute() where they will return ORM results in a Results object. This API is currently used internally by Query. Full support for Session.execute()->results to behave in a fully 2.0 fashion will be in later changesets. bulk update/delete with ORM support will also be delivered via the update() and delete() constructs, however these have not yet been adapted to the new system and may follow in a subsequent update. Performance is also beginning to lag as of this commit and some previous ones. It is hoped that a few central functions such as the coercions functions can be rewritten in C to re-gain performance. Additionally, query caching is now available and some subsequent patches will attempt to cache more of the per-execution work from the ORM layer, e.g. column getters and adapters. This patch also contains initial "turn on" of the caching system enginewide via the query_cache_size parameter to create_engine(). Still defaulting at zero for "no caching". The caching system still needs adjustments in order to gain adequate performance. Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
* Unify Query and select() , move all processing to compile phaseMike Bayer2020-05-241-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | Convert Query to do virtually all compile state computation in the _compile_context() phase, and organize it all such that a plain select() construct may also be used as the source of information in order to generate ORM query state. This makes it such that Query is not needed except for its additional methods like from_self() which are all to be deprecated. The construction of ORM state will occur beyond the caching boundary when the new execution model is integrated. future select() gains a working join() and filter_by() method. as we continue to rebase and merge each commit in the steps, callcounts continue to bump around. will have to look at the final result when it's all in. References: #5159 References: #4705 References: #4639 References: #4871 References: #5010 Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
* inline one_or_noneMike Bayer2020-05-242-45/+76
| | | | | | | Remove a bunch of unnecessary functions for this case. add test coverage to ensure uniqueness logic works. Change-Id: I2e6232c5667a3277b0ec8d7e47085a267f23d75f
* Avoid proxy functions in row functionsFederico Caselli2020-05-232-26/+37
| | | | | | | | | | | This streamlines a bit for non-C implementations, however also adds and tests behavioral contracts that mappings should not allow integer or slice access and should behave like a Python mapping in that it raises KeyError for an integer and TypeError for a slice. Py3/Py2/C/noC :) References: #5340 Change-Id: Id3cef452dc8a526b8371c90c5ca2bbb240b25c26
* Merge "Add with_for_update mysql new functionalities"mike bayer2020-05-231-0/+3
|\
| * Add with_for_update mysql new functionalitiesRobotScribe2020-05-151-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes: #4860 # Description Add nowait, skip_lock, of arguments to for_update_clause for mysql ### Checklist This pull request is: - [ ] A documentation / typographical error fix - Good to go, no issue or tests are needed - [ ] A short code fix - please include the issue number, and create an issue if none exists, which must include a complete example of the issue. one line code fixes without an issue and demonstration will not be accepted. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. one line code fixes without tests will not be accepted. - [x] A new feature implementation - please include the issue number, and create an issue if none exists, which must include a complete example of how the feature would look. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. **Have a nice day!** Closes: #5290 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5290 Pull-request-sha: 490e822e73e92ffe63cf45df9c49f3b31af1954d Change-Id: Ibd2acc47b538c601c69c8fb954776035ecab4c6c
* | Merge "Fix query string escaping in engine URLs"mike bayer2020-05-221-1/+1
|\ \
| * | Fix query string escaping in engine URLsMiguel Grinberg2020-05-211-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed issue in :class:`.URL` object where stringifying the object would not URL encode special characters, preventing the URL from being re-consumable as a real URL. Pull request courtesy Miguel Grinberg. Fixes: #5341 Closes: #5342 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5342 Pull-request-sha: 362ca3398336a3a892e8020530f0c68d4f2d1d01 Change-Id: Ief6218122d1ec0c70479eb1a90e1c16433801924
* | | Cross reference do_connect() event w/ creatorMike Bayer2020-05-222-0/+16
| | | | | | | | | | | | | | | | | | | | | | | | two questions today involving creator / do_connect, do_connect is not well known enough, ensure docs are present and prominent. Change-Id: I85d518b9fc7b9b069a18010969abefa360134fe9
* | | Structural / performance refinementsMike Bayer2020-05-222-39/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * state connection schema_translate_map entirely in terms of execution options, support for per-execution options as well * use slots for role impls, remove superclass of the roles themselves as this is not needed * tighten loop in resolve, might become a C function Change-Id: Ib98ac9b65022fbf976e49c6060e4c37573528c5f
* | | Performance fixes for new result setMike Bayer2020-05-215-166/+261
|/ / | | | | | | | | | | | | | | | | | | | | A few small mistakes led to huge callcounts. Additionally, the warn-on-get behavior which is attempting to warn for deprecated access in SQLAlchemy 2.0 is very expensive; it's not clear if its feasible to have this warning or to somehow alter how it works. Fixes: #5340 Change-Id: I73bdd2d7b6f1b25cc0222accabd585cf761a5af4
* | Disable "check unicode returns" under Python 3Mike Bayer2020-05-191-5/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Disabled the "unicode returns" check that runs on dialect startup when running under Python 3, which for many years has occurred in order to test the current DBAPI's behavior for whether or not it returns Python Unicode or Py2K strings for the VARCHAR and NVARCHAR datatypes. The check still occurs by default under Python 2, however the mechanism to test the behavior will be removed in SQLAlchemy 2.0 when Python 2 support is also removed. This logic was very effective when it was needed, however now that Python 3 is standard, all DBAPIs are expected to return Python 3 strings for character datatypes. In the unlikely case that a third party DBAPI does not support this, the conversion logic within :class:`.String` is still available and the third party dialect may specify this in its upfront dialect flags by setting the dialect level flag ``returns_unicode_strings`` to one of :attr:`.String.RETURNS_CONDITIONAL` or :attr:`.String.RETURNS_BYTES`, both of which will enable Unicode conversion even under Python 3. As part of this change, disabling testing of the doctest tutorials under Python 2. Fixes: #5315 Change-Id: I1260e894611409d3b7fe1a92bd90c52043bbcf19
* | Update transaction / connection handlingMike Bayer2020-05-173-264/+458
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | step one, do away with __connection attribute and using awkward AttributeError logic step two, move all management of "connection._transaction" into the transaction objects themselves where it's easier to follow. build MarkerTransaction that takes the role of "do-nothing block" new connection datamodel is: connection._transaction, always a root, connection._nested_transaction, always a nested. nested transactions still chain to each other as this is still sort of necessary but they consider the root transaction separately, and the marker transactions not at all. introduce new InvalidRequestError subclass PendingRollbackError. Apply to connection and session for all cases where a transaction needs to be rolled back before continuing. Within Connection, both PendingRollbackError as well as ResourceClosedError are now raised directly without being handled by handle_dbapi_error(); this removes these two exception cases from the handle_error event handler as well as from StatementError wrapping, as these two exceptions are not statement oriented and are instead programmatic issues, that the application is failing to handle database errors properly. Revise savepoints so that when a release fails, they set themselves as inactive so that their rollback() method does not throw another exception. Give savepoints another go on MySQL, can't get release working however get support for basic round trip going Fixes: #5327 Change-Id: Ia3cbbf56d4882fcc7980f90519412f1711fae74d
* | Actively unset reset agent in discard transactionMike Bayer2020-05-141-15/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The assumptions in _discard_transaction from 916e1fea25afcd07fa1d1d2f72043b372cd02223 were too narrow, assuming that if the given transaction were not our "current" one, that this would not be the reset agent. however as the legacy behvaior is that even a "nested" transaction gets set as "self._transaction", this did not accommodate for the nested transaction being thrown away. We will attempt to refine all of this logic in #5327 for 1.4 /master assuming this is feasible for the full suite of current use cases. Fixes: #5326 Change-Id: I6787e82c9e50c23317f87d0d094122c6a6f066da
* | Assert reset agent always set correctly and is activeMike Bayer2020-05-131-3/+21
|/ | | | | | | | | | | | | | | | | | | | Fixed fairly critical issue where the DBAPI connection could be returned to the connection pool while still in an un-rolled-back state. The reset agent responsible for rolling back the connection could be corrupted in the case that the transaction was "closed" without being rolled back or committed, which can occur in some scenarios when using ORM sessions and emitting .close() in a certain pattern involving savepoints. The fix ensures that the reset agent is always active. note that the reset agent will go away in 2.0 and the only real purpose of it is for logging of ROLLBACK. Apparently with the SQLite singleton engine in the test suite, there are some strucutral mismatches in the test fixtures where the reset agent is getting set differently than the transaction likely due to the same connection being shared in multiple context, though it's unclear. Fixes: #5326 Change-Id: If056870ea70a2d9a1749768988d5e023f3061b31
* Clarify create_engine encoding; update cx_OracleMike Bayer2020-05-061-24/+28
| | | | | | | | | | | | | | The create_engine()->encoding parameter is mostly irrelevant under Python 3. make it clear this parameter is not generally useful anymore and refer readers to the dialect documenation. Update cx_Oracle documentation to feature many examples of the encoding / nencoding parameters, remove extra detail that is not generally useful and reorganize information into more specific sections, de-emphasizing legacy / Python 2 specific sections. Change-Id: I42dafb85de0a585c9e05e1e3d787d8d6bfa632c0
* Documentation updates for ResultProxy -> ResultMike Bayer2020-05-015-16/+80
| | | | | | | | | This is based off of I8091919d45421e3f53029b8660427f844fee0228 and includes all documentation-only changes as a separate merge, once the parent is merged. Change-Id: I711adea23df0f9f0b1fe7c76210bd2de6d31842d
* Propose Result as immediate replacement for ResultProxyMike Bayer2020-05-015-1540/+2609
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As progress is made on the _future.Result, including breaking it out such that DBAPI behaviors are local to specific implementations, it becomes apparent that the Result object is a functional superset of ResultProxy and that basic operations like fetchone(), fetchall(), and fetchmany() behave pretty much exactly the same way on the new object. Reorganize things so that ResultProxy is now referred to as LegacyCursorResult, which subclasses CursorResult that represents the DBAPI-cursor version of Result, making use of a multiple inheritance pattern so that the functionality of Result is also available in non-DBAPI contexts, as will be necessary for some ORM patterns. Additionally propose the composition system for Result that will form the basis for ORM-alternative result systems such as horizontal sharding and dogpile cache. As ORM results will soon be coming directly from instances of Result, these extensions will instead build their own ResultFetchStrategies that perform the special steps to create composed or cached result sets. Also considering at the moment not emitting deprecation warnings for fetchXYZ() methods; the immediate issue is Keystone tests are calling upon it, but as the implementations here are proving to be not in any kind of conflict with how Result works, there's not too much issue leaving them around and deprecating at some later point. References: #5087 References: #4395 Fixes: #4959 Change-Id: I8091919d45421e3f53029b8660427f844fee0228
* Create initial 2.0 engine implementationMike Bayer2020-04-166-118/+432
| | | | | | | | | | | | | | | | | | | Implemented the SQLAlchemy 2 :func:`.future.create_engine` function which is used for forwards compatibility with SQLAlchemy 2. This engine features always-transactional behavior with autobegin. Allow execution options per statement execution. This includes that the before_execute() and after_execute() events now accept an additional dictionary with these options, empty if not passed; a legacy event decorator is added for backwards compatibility which now also emits a deprecation warning. Add some basic tests for execution, transactions, and the new result object. Build out on a new testing fixture that swaps in the future engine completely to start with. Change-Id: I70e7338bb3f0ce22d2f702537d94bb249bd9fb0a Fixes: #4644
* Set up absolute references for create_engine and relatedMike Bayer2020-04-148-95/+121
| | | | | | | includes more replacements for create_engine(), Connection, disambiguation of Result from future/baked Change-Id: Icb60a79ee7a6c45ea9056c211ffd1be110da3b5e
* Run search and replace of symbolic module namesMike Bayer2020-04-148-363/+472
| | | | | | | | Replaces a wide array of Sphinx-relative doc references with an abbreviated absolute form now supported by zzzeeksphinx. Change-Id: I94bffcc3f37885ffdde6238767224296339698a2
* Enable zzzeeksphinx module prefixesMike Bayer2020-04-142-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | zzzeeksphinx 1.1.2 in git can now convert short prefix names in a configured lookup to fully qualified module names, so that we can have succinct and portable pyrefs that still resolve absolutely. It also includes a formatter that will format all pyrefs in a fully consistent way regardless of the package path, by unconditionally removing all package tokens but always leaving class names in place including for methods, which means we no longer have to deal with tildes in pyrefs. The most immediate goal of the absolute prefixes is that we have lots of "ambiguous" names that appear in muliple places, like select(), ARRAY, ENUM etc. With the incoming future packages there is going to be lots of name overlap so it is necessary that all names eventually use absolute package paths when Sphinx receives them. In multiple stages, pyrefs will be converted using the zzzeeksphinx tools/fix_xrefs.py tool so that doclinks can be made absolute using symbolic prefixes. For this review, the actual search and replace of symbols is not performed, instead some general cleanup to prepare the docs as well as a lookup file used by the tool to do the conversion. this relatively small patch will be backported with appropriate changes to 1.3, 1.2, 1.1 and the tool can then be run on each branch individually. We are shooting for almost no warnings at all for master (still a handful I can't figure out which don't seem to have any impact) , very few for 1.3, and for 1.2 / 1.1 we hope for a significant reduction in warnings. Overall for all versions pyrefs should always point to the correct target, if they are in fact hyperlinked. it's better for a ref to go nowhere and be plain text than go to the wrong thing. Right now, hundreds of API links are pointing to the wrong thing as they are ambiguous names such as refresh(), insert(), update(), select(), join(), JSON etc. and Sphinx sends these all to essesntially random destinations among as many as five or six possible choices per symbol. A shorthand system that allows us to use absolute refs without having to type out a full blown absoulte module is the only way this is going to work, and we should ultimately seek to abandon any use of prefix dot for lookups. Everything should be on an underscore token so at the very least the module spaces can be reorganized without having to search and replace the entire documentation every time. Change-Id: I484a7329034af275fcdb322b62b6255dfeea9151
* Fix almost all read-level sphinx warningsMike Bayer2020-04-111-0/+1
| | | | | | | | There are some related to changelog that I can't figure out and are likely due to something in the changelog extension. also one thing with a "collection" I can't figure out. Change-Id: I0a9e6f4291c3589aa19a4abcb9245cd22a266fe0
* The :meth:`.Inspector.reflecttable` was renamed to ↵Federico Caselli2020-04-101-2/+15
| | | | | | | :meth:`.Inspector.reflect_table`. Fixes: #5244 Change-Id: I2b12fd69ed24ce1ede8f6ed5cb14cc7761308ee3
* Remove code deprecated before version 1.1Federico Caselli2020-04-096-151/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Remove deprecated method ``get_primary_keys` in the :class:`.Dialect` and :class:`.Inspector` classes. - Remove deprecated event ``dbapi_error`` and the method ``ConnectionEvents.dbapi_error`. - Remove support for deprecated engine URLs of the form ``postgres://``. - Remove deprecated dialect ``mysql+gaerdbms``. - Remove deprecated parameter ``quoting`` from :class:`.mysql.ENUM` and :class:`.mysql.SET` in the ``mysql`` dialect. - Remove deprecated function ``comparable_property``. and function ``comparable_using`` in the declarative extension. - Remove deprecated function ``compile_mappers``. - Remove deprecated method ``collection.linker``. - Remove deprecated method ``Session.prune`` and parameter ``Session.weak_identity_map``. This change also removes the class ``StrongInstanceDict``. - Remove deprecated parameter ``mapper.order_by``. - Remove deprecated parameter ``Session._enable_transaction_accounting`. - Remove deprecated parameter ``Session.is_modified.passive``. - Remove deprecated class ``Binary``. Please use :class:`.LargeBinary`. - Remove deprecated methods ``Compiled.compile``, ``ClauseElement.__and__`` and ``ClauseElement.__or__`` and attribute ``Over.func``. - Remove deprecated ``FromClause.count`` method. - Remove deprecated parameter ``Table.useexisting``. - Remove deprecated parameters ``text.bindparams`` and ``text.typemap``. - Remove boolean support for the ``passive`` parameter in ``get_history``. - Remove deprecated ``adapt_operator`` in ``UserDefinedType.Comparator``. Fixes: #4643 Change-Id: Idcd390c77bf7b0e9957907716993bdaa3f1a1763
* Broaden is[not]_distinct_from supportGord Thompson2020-04-021-0/+2
| | | | | | | Added support for .is[not]_distinct_from to SQL Server, MySQL, and Oracle. Fixes: #5137 Change-Id: I3b4d3b199821a55687f83c9a5b63a95d07a64cd5
* Try to measure new style caching in the ORM, take twoMike Bayer2020-04-013-10/+77
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf Trying to see if by making the cache key memoized, we still can have the older "identity" form of caching which is the cheapest of all, at the same time as the newer "cache key each time" version that is not nearly as cheap; but still much cheaper than no caching at all. Also needed is a per-execution update of _keymap when we invoke from a cached select, so that Column objects that are anonymous or otherwise adapted will match up. this is analogous to the adaption of bound parameters from the cache key. Adds test coverage for the keymap / construct_params() changes related to caching. Also hones performance to a large extent for statement construction and cache key generation. Also includes a new memoized attribute approach that vastly simplifies the previous approach of "group_expirable_memoized_property" and finally integrates cleanly with _clone(), _generate(), etc. no more hardcoding of attributes is needed, as well as that most _reset_memoization() calls are no longer needed as the reset is inherent in a _generate() call; this also has dramatic performance improvements. Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
* String compiler can now literal compile datetime objectsFederico Caselli2020-03-291-0/+26
| | | | | | | | | | Add ability to literal compile a :class:`DateTime`, :class:`Date` or :class:"Time" when using the string dialect for debugging purposes. This change does not impact real dialect implementation that retain their current behavior. Fixes: #5052 Change-Id: Ia3fad2be905c6d35b0106b9a2388c7508f067e90
View Lorry Controller Status