summaryrefslogtreecommitdiff
path: root/lib
Commit message (Collapse)AuthorAgeFilesLines
...
* | | Small callcount reductions and refinement for cached queriesMike Bayer2020-05-252-74/+87
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | baked wasn't using the new one()/first()/one_or_none() methods, fixed that. loading._instance_processor() can skip setting up the quick populators every time because it can cache the getters. Callcounts have gone below what 1.3 does for the test_baked_query performance suite, however runtime for continued inexplicable reasons has not :(. still suspecting the result tuples but this seems so hard to believe. Change-Id: Ifbca04834d27350e0fa82cb8512e66112abc8729
* | | Convert execution to move through SessionMike Bayer2020-05-2540-1031/+2007
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the ORM execution flow with a single pathway through Session.execute() for all queries, including Core and ORM. Currently included is full support for ORM Query, Query.from_statement(), select(), as well as the baked query and horizontal shard systems. Initial changes have also been made to the dogpile caching example, which like baked query makes use of a new ORM-specific execution hook that replaces the use of both QueryEvents.before_compile() as well as Query._execute_and_instances() as the central ORM interception hooks. select() and Query() constructs alike can be passed to Session.execute() where they will return ORM results in a Results object. This API is currently used internally by Query. Full support for Session.execute()->results to behave in a fully 2.0 fashion will be in later changesets. bulk update/delete with ORM support will also be delivered via the update() and delete() constructs, however these have not yet been adapted to the new system and may follow in a subsequent update. Performance is also beginning to lag as of this commit and some previous ones. It is hoped that a few central functions such as the coercions functions can be rewritten in C to re-gain performance. Additionally, query caching is now available and some subsequent patches will attempt to cache more of the per-execution work from the ORM layer, e.g. column getters and adapters. This patch also contains initial "turn on" of the caching system enginewide via the query_cache_size parameter to create_engine(). Still defaulting at zero for "no caching". The caching system still needs adjustments in order to gain adequate performance. Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
* | | Unify Query and select() , move all processing to compile phaseMike Bayer2020-05-2437-2730/+4781
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Convert Query to do virtually all compile state computation in the _compile_context() phase, and organize it all such that a plain select() construct may also be used as the source of information in order to generate ORM query state. This makes it such that Query is not needed except for its additional methods like from_self() which are all to be deprecated. The construction of ORM state will occur beyond the caching boundary when the new execution model is integrated. future select() gains a working join() and filter_by() method. as we continue to rebase and merge each commit in the steps, callcounts continue to bump around. will have to look at the final result when it's all in. References: #5159 References: #4705 References: #4639 References: #4871 References: #5010 Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
* | inline one_or_noneMike Bayer2020-05-242-45/+76
|/ | | | | | | Remove a bunch of unnecessary functions for this case. add test coverage to ensure uniqueness logic works. Change-Id: I2e6232c5667a3277b0ec8d7e47085a267f23d75f
* Avoid proxy functions in row functionsFederico Caselli2020-05-234-32/+84
| | | | | | | | | | | This streamlines a bit for non-C implementations, however also adds and tests behavioral contracts that mappings should not allow integer or slice access and should behave like a Python mapping in that it raises KeyError for an integer and TypeError for a slice. Py3/Py2/C/noC :) References: #5340 Change-Id: Id3cef452dc8a526b8371c90c5ca2bbb240b25c26
* Merge "Add python 3.8 profiles; remove zoomark tests"mike bayer2020-05-231-209/+0
|\
| * Add python 3.8 profiles; remove zoomark testsFederico Caselli2020-05-231-209/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The zoomark tests have served us well for many years. At this point, they have been using a very antiquated calling style for many years and are no longer where we catch performance issues. The performance suite now has a large number of individual tests that catch issues very specifically and additionally record just one performance count per test. This also allows us to remove the "replay" fixtures that were not used for anything else. Fixes: #5347 Change-Id: I0a8d078e7b7240602f4f3f7068f231e98a40f17e
* | Merge "Correctly apply self_group in type_coerce element."mike bayer2020-05-231-6/+10
|\ \
| * | Correctly apply self_group in type_coerce element.Federico Caselli2020-05-221-6/+10
| |/ | | | | | | | | | | | | | | The type coerce element did not correctly apply grouping rules when using in an expression Fixes: #5344 Change-Id: Id67b0e60ac54f8992f931aaed62731672f60c96c
* | Merge "Add with_for_update mysql new functionalities"mike bayer2020-05-232-2/+32
|\ \
| * | Add with_for_update mysql new functionalitiesRobotScribe2020-05-152-2/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes: #4860 # Description Add nowait, skip_lock, of arguments to for_update_clause for mysql ### Checklist This pull request is: - [ ] A documentation / typographical error fix - Good to go, no issue or tests are needed - [ ] A short code fix - please include the issue number, and create an issue if none exists, which must include a complete example of the issue. one line code fixes without an issue and demonstration will not be accepted. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. one line code fixes without tests will not be accepted. - [x] A new feature implementation - please include the issue number, and create an issue if none exists, which must include a complete example of how the feature would look. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. **Have a nice day!** Closes: #5290 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5290 Pull-request-sha: 490e822e73e92ffe63cf45df9c49f3b31af1954d Change-Id: Ibd2acc47b538c601c69c8fb954776035ecab4c6c
* | | Merge "Improve error message when using :meth:`_query.Query.filter_by` in a ↵mike bayer2020-05-231-1/+10
|\ \ \ | | | | | | | | | | | | query where the first entity is not a mapped class."
| * | | Improve error message when using :meth:`_query.Query.filter_by` inFederico Caselli2020-05-151-1/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | a query where the first entity is not a mapped class. Fixes: #5260 Change-Id: I18cfccc0f3e587cb3df3e1f5f23b6885d5f2f466
* | | | Merge "Add note re: pymssql not in CI"mike bayer2020-05-231-0/+5
|\ \ \ \
| * | | | Add note re: pymssql not in CIGord Thompson2020-05-161-0/+5
| | | | | | | | | | | | | | | | | | | | Change-Id: I45f78131ffe1881a3965e8aa41bbc46da7d43a5b
* | | | | Merge "fix typo in reference documentation"mike bayer2020-05-232-2/+2
|\ \ \ \ \
| * | | | | fix typo in reference documentationFederico Caselli2020-05-232-2/+2
| | |_|_|/ | |/| | | | | | | | | | | | | Change-Id: I5720635863f0d8a8ebfa4ddc51008bb93c1067b4
* | | | | Stop converting mssql datetime.time parameters to datetime.datetimeGord Thompson2020-05-231-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | Fixes: #5339 Change-Id: Ida75422d8c3fdfc7adae68e547d88df49368a693
* | | | | Merge "Add immutabledict C code"mike bayer2020-05-238-24/+553
|\ \ \ \ \ | |/ / / / |/| | | |
| * | | | Add immutabledict C codeMike Bayer2020-05-239-25/+553
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Start trying to convert fundamental objects to C as we now rely on a fairly small core of things, and 1.4 is having problems with complexity added being slower than the performance gains we are trying to build in. immutabledict here does seem to bench as twice as fast as the Python one, see below. However, it does not appear to be used prominently enough to make any dent in the performance tests. at the very least it may provide us some more lift-and-copy code for more C extensions. import timeit from sqlalchemy.util._collections import not_immutabledict, immutabledict def run(dict_cls): for i in range(1000000): d1 = dict_cls({"x": 5, "y": 4}) d2 = d1.union({"x": 17, "new key": "some other value"}, None) assert list(d2) == ["x", "y", "new key"] print( timeit.timeit( "run(d)", "from __main__ import run, not_immutabledict as d", number=1 ) ) print( timeit.timeit( "run(d)", "from __main__ import run, immutabledict as d", number=1 ) ) output: python: 1.8799766399897635 C code: 0.8880784640205093 Change-Id: I29e7104dc21dcc7cdf895bf274003af2e219bf6d
* | | | | Merge "Fix query string escaping in engine URLs"mike bayer2020-05-221-1/+1
|\ \ \ \ \
| * | | | | Fix query string escaping in engine URLsMiguel Grinberg2020-05-211-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed issue in :class:`.URL` object where stringifying the object would not URL encode special characters, preventing the URL from being re-consumable as a real URL. Pull request courtesy Miguel Grinberg. Fixes: #5341 Closes: #5342 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5342 Pull-request-sha: 362ca3398336a3a892e8020530f0c68d4f2d1d01 Change-Id: Ief6218122d1ec0c70479eb1a90e1c16433801924
* | | | | | Merge "Don't emit pyodbc "no driver" warning for empty URL"mike bayer2020-05-221-1/+2
|\ \ \ \ \ \
| * | | | | | Don't emit pyodbc "no driver" warning for empty URLMike Bayer2020-05-221-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed an issue in the pyodbc connector such that a warning about pyodbc "drivername" would be emitted when using a totally empty URL. Empty URLs are normal when producing a non-connected dialect object or when using the "creator" argument to create_engine(). The warning now only emits if the driver name is missing but other parameters are still present. Fixes: #5346 Change-Id: I0ee6f5fd5af7faca63bf0d7034410942f40834a8
* | | | | | | Cross reference do_connect() event w/ creatorMike Bayer2020-05-222-0/+16
|/ / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | two questions today involving creator / do_connect, do_connect is not well known enough, ensure docs are present and prominent. Change-Id: I85d518b9fc7b9b069a18010969abefa360134fe9
* | | | | | Merge "Structural / performance refinements"mike bayer2020-05-223-115/+129
|\ \ \ \ \ \
| * | | | | | Structural / performance refinementsMike Bayer2020-05-223-115/+129
| | |/ / / / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * state connection schema_translate_map entirely in terms of execution options, support for per-execution options as well * use slots for role impls, remove superclass of the roles themselves as this is not needed * tighten loop in resolve, might become a C function Change-Id: Ib98ac9b65022fbf976e49c6060e4c37573528c5f
* | | | | | Don't incref on new reference key_styleMike Bayer2020-05-221-1/+1
|/ / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | in 4550983e0ce2f35b3585e53894c941c23693e71d we added a new attribute key_style. remove an erroneous Py_INCREF when we acquire it from PyLong_FromLong as we already own the reference. since this is a new reference we actualy need to Py_DECREF it because we aren't returning it. Change-Id: I61470513a173c76863ec6f7f5ff9b2ec13582f08
* | | | | Performance fixes for new result setMike Bayer2020-05-2113-230/+415
|/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A few small mistakes led to huge callcounts. Additionally, the warn-on-get behavior which is attempting to warn for deprecated access in SQLAlchemy 2.0 is very expensive; it's not clear if its feasible to have this warning or to somehow alter how it works. Fixes: #5340 Change-Id: I73bdd2d7b6f1b25cc0222accabd585cf761a5af4
* | | | fix table.schema version to 1.3.18Mike Bayer2020-05-191-1/+1
| | | | | | | | | | | | | | | | | | | | Fixes: #5338 Change-Id: I12ce8021f5627c306c768b74da991ef4af0a5b55
* | | | Merge "SQLite 3.31 added support for computed column."mike bayer2020-05-191-14/+67
|\ \ \ \
| * | | | SQLite 3.31 added support for computed column.Federico Caselli2020-05-191-14/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change enables their support in SQLAlchemy when targeting SQLite. Fixes: #5297 Change-Id: Ia9f21a49e58fc977e3c669b8176036c95d93b9c8
* | | | | Merge "Introduce :class:`.IdentityOptions` to store common parameters for ↵mike bayer2020-05-193-17/+75
|\ \ \ \ \ | | | | | | | | | | | | | | | | | | sequences and identity columns."
| * | | | | Introduce :class:`.IdentityOptions` to store common parameters forFederico Caselli2020-05-153-17/+75
| | |_|/ / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | sequences and identity columns. References: #5324 Change-Id: I72f7fc1a003456206b004d3d26306940f9c36414
* | | | | Disable "check unicode returns" under Python 3Mike Bayer2020-05-192-6/+79
| |/ / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Disabled the "unicode returns" check that runs on dialect startup when running under Python 3, which for many years has occurred in order to test the current DBAPI's behavior for whether or not it returns Python Unicode or Py2K strings for the VARCHAR and NVARCHAR datatypes. The check still occurs by default under Python 2, however the mechanism to test the behavior will be removed in SQLAlchemy 2.0 when Python 2 support is also removed. This logic was very effective when it was needed, however now that Python 3 is standard, all DBAPIs are expected to return Python 3 strings for character datatypes. In the unlikely case that a third party DBAPI does not support this, the conversion logic within :class:`.String` is still available and the third party dialect may specify this in its upfront dialect flags by setting the dialect level flag ``returns_unicode_strings`` to one of :attr:`.String.RETURNS_CONDITIONAL` or :attr:`.String.RETURNS_BYTES`, both of which will enable Unicode conversion even under Python 3. As part of this change, disabling testing of the doctest tutorials under Python 2. Fixes: #5315 Change-Id: I1260e894611409d3b7fe1a92bd90c52043bbcf19
* | | | Streamline visitors.iterateMike Bayer2020-05-187-133/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This method might be used more significantly in the ORM refactor, so further refine it. * all get_children() methods now work entirely based on iterators. Basically only select() was sensitive to this anymore and it now chains the iterators together * remove all kinds of flags like column_collections, schema_visitor that apparently aren't used anymore. * remove the "depthfirst" visitors as these don't seem to be used either. * make sure select() yields its columns first as these will be used to determine the current mapper. Change-Id: I05273a2d5306a57c2d1b0979050748cf3ac964bf
* | | | Merge "Update transaction / connection handling"mike bayer2020-05-178-290/+485
|\ \ \ \
| * | | | Update transaction / connection handlingMike Bayer2020-05-178-290/+485
| |/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | step one, do away with __connection attribute and using awkward AttributeError logic step two, move all management of "connection._transaction" into the transaction objects themselves where it's easier to follow. build MarkerTransaction that takes the role of "do-nothing block" new connection datamodel is: connection._transaction, always a root, connection._nested_transaction, always a nested. nested transactions still chain to each other as this is still sort of necessary but they consider the root transaction separately, and the marker transactions not at all. introduce new InvalidRequestError subclass PendingRollbackError. Apply to connection and session for all cases where a transaction needs to be rolled back before continuing. Within Connection, both PendingRollbackError as well as ResourceClosedError are now raised directly without being handled by handle_dbapi_error(); this removes these two exception cases from the handle_error event handler as well as from StatementError wrapping, as these two exceptions are not statement oriented and are instead programmatic issues, that the application is failing to handle database errors properly. Revise savepoints so that when a release fails, they set themselves as inactive so that their rollback() method does not throw another exception. Give savepoints another go on MySQL, can't get release working however get support for basic round trip going Fixes: #5327 Change-Id: Ia3cbbf56d4882fcc7980f90519412f1711fae74d
* | | | Reword delete-orphan on many error message and documentMike Bayer2020-05-162-5/+25
| |/ / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For many years we have encountered users making use of the "single_parent" flag in response to the error message for "delete-orphan" expressing this as a means to cancel the current error. However, the actual issue here is usually a misuse of the delete-orphan cascade setting. Reword the error message to be much more descriptive about what this means and add new error link sections describing the situation in as much detail as possible. Fixes: #5329 Change-Id: I7ba710378b2935479ab22ff9a0a79c692dbf69a6
* | | Merge "Add 'schema' parameter to table"mike bayer2020-05-151-3/+14
|\ \ \
| * | | Add 'schema' parameter to tableDylan Modesitt2020-05-101-3/+14
| | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added a "schema" parameter to the :func:`_expression.table` construct, allowing ad-hoc table expressions to also include a schema name. Pull request courtesy Dylan Modesitt. Fixes: #5309 Closes: #5310 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5310 Pull-request-sha: ce85681050500186678131f948b6ea277a65dc17 Change-Id: I32015d593e1ee1121c7426fbffdcc565d025fad1
* | | Merge "Adjust documentation wording for firebird and sybase"mike bayer2020-05-152-13/+16
|\ \ \
| * | | Adjust documentation wording for firebird and sybaseGord Thompson2020-05-152-13/+16
| | |/ | |/| | | | | | | Change-Id: Ied2cdab1c2dbdee681a5500285a5bcdd4d589afa
* | | Merge "Move supports_sane_rowcount_returning = False to dialect level"mike bayer2020-05-152-1/+5
|\ \ \ | |/ / |/| |
| * | Move supports_sane_rowcount_returning = False to dialect levelGord Thompson2020-05-112-1/+5
| | | | | | | | | | | | | | | Fixes: #5321 Change-Id: Id83e98e9013818424c133297a850746302633158
* | | Actively unset reset agent in discard transactionMike Bayer2020-05-141-15/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The assumptions in _discard_transaction from 916e1fea25afcd07fa1d1d2f72043b372cd02223 were too narrow, assuming that if the given transaction were not our "current" one, that this would not be the reset agent. however as the legacy behvaior is that even a "nested" transaction gets set as "self._transaction", this did not accommodate for the nested transaction being thrown away. We will attempt to refine all of this logic in #5327 for 1.4 /master assuming this is feasible for the full suite of current use cases. Fixes: #5326 Change-Id: I6787e82c9e50c23317f87d0d094122c6a6f066da
* | | Merge "Assert reset agent always set correctly and is active"mike bayer2020-05-132-5/+39
|\ \ \
| * | | Assert reset agent always set correctly and is activeMike Bayer2020-05-132-5/+39
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed fairly critical issue where the DBAPI connection could be returned to the connection pool while still in an un-rolled-back state. The reset agent responsible for rolling back the connection could be corrupted in the case that the transaction was "closed" without being rolled back or committed, which can occur in some scenarios when using ORM sessions and emitting .close() in a certain pattern involving savepoints. The fix ensures that the reset agent is always active. note that the reset agent will go away in 2.0 and the only real purpose of it is for logging of ROLLBACK. Apparently with the SQLite singleton engine in the test suite, there are some strucutral mismatches in the test fixtures where the reset agent is getting set differently than the transaction likely due to the same connection being shared in multiple context, though it's unclear. Fixes: #5326 Change-Id: If056870ea70a2d9a1749768988d5e023f3061b31
* | | Load external firebird or sybase dialect if availableGord Thompson2020-05-111-3/+9
|/ / | | | | | | | | | | | | | | Fixes: #5318 Extension of I1660abb11c02656fbf388f2f9c4257075111be58 Change-Id: I32b678430497327f9b08f821bd345a2557e34b1f
* | Merge "Use cx_oracle.LONG_STRING /LONG_BINARY for CLOB/BLOB"mike bayer2020-05-092-8/+27
|\ \
View Lorry Controller Status