summaryrefslogtreecommitdiff
path: root/test/aaa_profiling/test_orm.py
Commit message (Collapse)AuthorAgeFilesLines
* Try running pyupgrade on the codeFederico Caselli2022-11-161-3/+3
| | | | | | | | command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>" pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not exists in sqlalchemy fixtures Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
* update flake8 noqa skips with proper syntaxFederico Caselli2022-04-111-9/+9
| | | | Change-Id: I42ed77f559e3ee5b8c600d98457ee37803ef0ea6
* pep484 - sql.selectableMike Bayer2022-04-041-54/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | the pep484 task becomes more intense as there is mounting pressure to come up with a consistency in how data moves from end-user to instance variable. current thinking is coming into: 1. there are _typing._XYZArgument objects that represent "what the user sent" 2. there's the roles, which represent a kind of "filter" for different kinds of objects. These are mostly important as the argument we pass to coerce(). 3. there's the thing that coerce() returns, which should be what the construct uses as its internal representation of the thing. This is _typing._XYZElement. but there's some controversy over whether or not we should pass actual ClauseElements around by their role or not. I think we shouldn't at the moment, but this makes the "role-ness" of something a little less portable. Like, we have to set DMLTableRole for TableClause, Join, and Alias, but then also we have to repeat those three types in order to set up _DMLTableElement. Other change introduced here, there was a deannotate=True for the left/right of a sql.join(). All tests pass without that. I'd rather not have that there as if we have a join(A, B) where A, B are mapped classes, we want them inside of the _annotations. The rationale seems to be performance, but this performance can be illustrated to be on the compile side which we hope is cached in the normal case. CTEs now accommodate for text selects including recursive. Get typing to accommodate "util.preloaded" cleanly; add "preloaded" as a real module. This seemed like we would have needed pep562 `__getattr__()` but we don't, just set names in globals() as we import them. References: #6810 Change-Id: I34d17f617de2fe2c086fc556bd55748dc782faf0
* correct for non-deterministic gc artifactsMike Bayer2022-02-151-9/+27
| | | | | | | | | | | | Observed the tests here have different profiling counts when run individually vs. as a group, and this seems to be due to whether or not results of each query are garbage collected or not. for all but one test, ensuring results stay between query runs seems to meet the current profiling counts. Change-Id: I5aca5db08936757ad2a6055c5fc077cc58979bdd
* Remove all remaining removed_in_20 warnings slated for removalMike Bayer2022-01-051-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | Finalize all remaining removed-in-2.0 changes so that we can begin doing pep-484 typing without old things getting in the way (we will also have to do public_factory). note there are a few "moved_in_20()" and "became_legacy_in_20()" warnings still in place. The SQLALCHEMY_WARN_20 variable is now removed. Also removed here are the legacy "in place mutators" for Select statements, and some keyword-only argument signatures in Core have been added. Also in the big change department, the ORM mapper() function is removed entirely; the Mapper class is otherwise unchanged, just the public-facing API function. Mappers are now always given a registry in which to participate, however the argument signature of Mapper is not changed. ideally "registry" would be the first positional argument. Fixes: #7257 Change-Id: Ic70c57b9f1cf7eb996338af5183b11bdeb3e1623
* Update Black's target-version to py37Hugo van Kemenade2022-01-051-15/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | <!-- Provide a general summary of your proposed changes in the Title field above --> ### Description <!-- Describe your changes in detail --> Black's `target-version` was still set to `['py27', 'py36']`. Set it to `[py37]` instead. Also update Black and other pre-commit hooks and re-format with Black. ### Checklist <!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once) --> This pull request is: - [ ] A documentation / typographical error fix - Good to go, no issue or tests are needed - [ ] A short code fix - please include the issue number, and create an issue if none exists, which must include a complete example of the issue. one line code fixes without an issue and demonstration will not be accepted. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. one line code fixes without tests will not be accepted. - [ ] A new feature implementation - please include the issue number, and create an issue if none exists, which must include a complete example of how the feature would look. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. **Have a nice day!** Closes: #7536 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7536 Pull-request-sha: b3aedf5570d7e0ba6c354e5989835260d0591b08 Change-Id: I8be85636fd2c9449b07a8626050c8bd35bd119d5
* Removals: strings for join(), loader_options().Mike Bayer2021-12-081-1/+4
| | | | | | | | | | | | | | | | * The :meth:`_orm.Query.join` method no longer accepts strings for relationship names; the long-documented approach of using ``Class.attrname`` for join targets is now standard. * Loader options no longer accept strings for attribute names. The long-documented approach of using ``Class.attrname`` for loader option targets is now standard. It is hoped that a subsequent commit can refactor loader options to no longer need "UnboundLoad" for most cases. Change-Id: If4629882c40523dccbf4459256bf540fb468b618 References: #6986
* fully implement future engine and remove legacyMike Bayer2021-11-071-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | The major action here is to lift and move future.Connection and future.Engine fully into sqlalchemy.engine.base. This removes lots of engine concepts, including: * autocommit * Connection running without a transaction, autobegin is now present in all cases * most "autorollback" is obsolete * Core-level subtransactions (i.e. MarkerTransaction) * "branched" connections, copies of connections * execution_options() returns self, not a new connection * old argument formats, distill_params(), simplifies calling scheme between engine methods * before/after_execute() events (oriented towards compiled constructs) don't emit for exec_driver_sql(). before/after_cursor_execute() is still included for this * old helper methods superseded by context managers, connection.transaction(), engine.transaction() engine.run_callable() * ancient engine-level reflection methods has_table(), table_names() * sqlalchemy.testing.engines.proxying_engine References: #7257 Change-Id: Ib20ed816642d873b84221378a9ec34480e01e82c
* Modernize tests - session_query_getGord Thompson2021-10-281-2/+2
| | | | | Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Change-Id: I92013aad471baf32df1b51b756e86d95449b5cfd
* Modernize tests - calling_mapper_directlyGord Thompson2021-09-301-33/+48
| | | | | | | | | | | | | a few changes for py2k: * map_imperatively() includes the check that a class is being sent, this was only working for mapper() before * the test suite didn't place the py2k "autouse" workaround in the correct order, seemingly, tried to adjust the per-test ordering setup in pytestplugin.py Change-Id: I4cc39630724e810953cfda7b2afdadc8b948e3c2
* warn or deprecate for auto-aliasing in joinsMike Bayer2021-09-281-14/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | An extra layer of warning messages has been added to the functionality of :meth:`_orm.Query.join` and the ORM version of :meth:`_sql.Select.join`, where a few places where "automatic aliasing" continues to occur will now be called out as a pattern to avoid, mostly specific to the area of joined table inheritance where classes that share common base tables are being joined together without using explicit aliases. One case emits a legacy warning for a pattern that's not recommended, the other case is fully deprecated. The automatic aliasing within ORM join() which occurs for overlapping mapped tables does not work consistently with all APIs such as ``contains_eager()``, and rather than continue to try to make these use cases work everywhere, replacing with a more user-explicit pattern is clearer, less prone to bugs and simplifies SQLAlchemy's internals further. The warnings include links to the errors.rst page where each pattern is demonstrated along with the recommended pattern to fix. * Improved the exception message generated when configuring a mapping with joined table inheritance where the two tables either have no foreign key relationships set up, or where they have multiple foreign key relationships set up. The message is now ORM specific and includes context that the :paramref:`_orm.Mapper.inherit_condition` parameter may be needed particularly for the ambiguous foreign keys case. * Add explicit support in the _expect_warnings() assertion for nested _expect_warnings calls * generalize the NoCache fixture, which we also need to catch warnings during compilation consistently * generalize the __str__() method for the HasCode mixin so all warnings and errors include the code link in their string Fixes: #6974 Change-Id: I84ed79ba2112c39eaab7973b6d6f46de7fa80842
* simplify relationship caching optionsMike Bayer2021-06-091-1/+1
| | | | | | | | | | | | | | | | | | | | | | Clarified the current purpose of the :paramref:`_orm.relationship.bake_queries` flag, which in 1.4 is to enable or disable "lambda caching" of statements within the "lazyload" and "selectinload" loader strategies; this is separate from the more foundational SQL query cache that is used for most statements. Additionally, the lazy loader no longer uses its own cache for many-to-one SQL queries, which was an implementation quirk that doesn't exist for any other loader scenario. Finally, the "lru cache" warning that the lazyloader and selectinloader strategies could emit when handling a wide array of class/relationship combinations has been removed; based on analysis of some end-user cases, this warning doesn't suggest any significant issue. While setting ``bake_queries=False`` for such a relationship will remove this cache from being used, there's no particular performance gain in this case as using no caching vs. using a cache that needs to refresh often likely still wins out on the caching being used side. Fixes: #6072 Fixes: #6487 Change-Id: Ida61f09b837d3acdafa07344d7d747d7f3ab226a
* Fix adaption in AnnotatedLabel; repair needless expense in coercionMike Bayer2021-05-281-0/+16
| | | | | | | | | | | | | | | Fixed regression involving clause adaption of labeled ORM compound elements, such as single-table inheritance discriminator expressions with conditionals or CASE expressions, which could cause aliased expressions such as those used in ORM join / joinedload operations to not be adapted correctly, such as referring to the wrong table in the ON clause in a join. This change also improves a performance bump that was located within the process of invoking :meth:`_sql.Select.join` given an ORM attribute as a target. Fixes: #6550 Change-Id: I98906476f0cce6f41ea00b77c789baa818e9d167
* reinvent xdist hooks in terms of pytest fixturesMike Bayer2021-01-131-4/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To allow the "connection" pytest fixture and others work correctly in conjunction with setup/teardown that expects to be external to the transaction, remove and prevent any usage of "xdist" style names that are hardcoded by pytest to run inside of fixtures, even function level ones. Instead use pytest autouse fixtures to implement our own r"setup|teardown_test(?:_class)?" methods so that we can ensure function-scoped fixtures are run within them. A new more explicit flow is set up within plugin_base and pytestplugin such that the order of setup/teardown steps, which there are now many, is fully documented and controllable. New granularity has been added to the test teardown phase to distinguish between "end of the test" when lock-holding structures on connections should be released to allow for table drops, vs. "end of the test plus its teardown steps" when we can perform final cleanup on connections and run assertions that everything is closed out. From there we can remove most of the defensive "tear down everything" logic inside of engines which for many years would frequently dispose of pools over and over again, creating for a broken and expensive connection flow. A quick test shows that running test/sql/ against a single Postgresql engine with the new approach uses 75% fewer new connections, creating 42 new connections total, vs. 164 new connections total with the previous system. As part of this, the new fixtures metadata/connection/future_connection have been integrated such that they can be combined together effectively. The fixture_session(), provide_metadata() fixtures have been improved, including that fixture_session() now strongly references sessions which are explicitly torn down before table drops occur afer a test. Major changes have been made to the ConnectionKiller such that it now features different "scopes" for testing engines and will limit its cleanup to those testing engines corresponding to end of test, end of test class, or end of test session. The system by which it tracks DBAPI connections has been reworked, is ultimately somewhat similar to how it worked before but is organized more clearly along with the proxy-tracking logic. A "testing_engine" fixture is also added that works as a pytest fixture rather than a standalone function. The connection cleanup logic should now be very robust, as we now can use the same global connection pools for the whole suite without ever disposing them, while also running a query for PostgreSQL locks remaining after every test and assert there are no open transactions leaking between tests at all. Additional steps are added that also accommodate for asyncio connections not explicitly closed, as is the case for legacy sync-style tests as well as the async tests themselves. As always, hundreds of tests are further refined to use the new fixtures where problems with loose connections were identified, largely as a result of the new PostgreSQL assertions, many more tests have moved from legacy patterns into the newest. An unfortunate discovery during the creation of this system is that autouse fixtures (as well as if they are set up by @pytest.mark.usefixtures) are not usable at our current scale with pytest 4.6.11 running under Python 2. It's unclear if this is due to the older version of pytest or how it implements itself for Python 2, as well as if the issue is CPU slowness or just large memory use, but collecting the full span of tests takes over a minute for a single process when any autouse fixtures are in place and on CI the jobs just time out after ten minutes. So at the moment this patch also reinvents a small version of "autouse" fixtures when py2k is running, which skips generating the real fixture and instead uses two global pytest fixtures (which don't seem to impact performance) to invoke the "autouse" fixtures ourselves outside of pytest. This will limit our ability to do more with fixtures until we can remove py2k support. py.test is still observed to be much slower in collection in the 4.6.11 version compared to modern 6.2 versions, so add support for new TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that will run the suite for fewer backends under Python 2. For Python 3 pin pytest to modern 6.2 versions where performance for collection has been improved greatly. Includes the following improvements: Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would be raised rather than :class:`.exc.TimeoutError`. Also repaired the :paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using the async engine, which previously would ignore the timeout and block rather than timing out immediately as is the behavior with regular :class:`.QueuePool`. For asyncio the connection pool will now also not interact at all with an asyncio connection whose ConnectionFairy is being garbage collected; a warning that the connection was not properly closed is emitted and the connection is discarded. Within the test suite the ConnectionKiller is now maintaining strong references to all DBAPI connections and ensuring they are released when tests end, including those whose ConnectionFairy proxies are GCed. Identified cx_Oracle.stmtcachesize as a major factor in Oracle test scalability issues, this can be reset on a per-test basis rather than setting it to zero across the board. the addition of this flag has resolved the long-standing oracle "two task" error problem. For SQL Server, changed the temp table style used by the "suite" tests to be the double-pound-sign, i.e. global, variety, which is much easier to test generically. There are already reflection tests that are more finely tuned to both styles of temp table within the mssql test suite. Additionally, added an extra step to the "dropfirst" mechanism for SQL Server that will remove all foreign key constraints first as some issues were observed when using this flag when multiple schemas had not been torn down. Identified and fixed two subtle failure modes in the engine, when commit/rollback fails in a begin() context manager, the connection is explicitly closed, and when "initialize()" fails on the first new connection of a dialect, the transactional state on that connection is still rolled back. Fixes: #5826 Fixes: #5827 Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
* remove more bound metadataMike Bayer2021-01-051-25/+26
| | | | | | | | | | | | | | in Iae6ab95938a7e92b6d42086aec534af27b5577d3 I missed that the "bind" was being stuck onto the MetaData in TablesTest, which led thousands of ORM tests to still use bound metadata. Keep looking for bound metadata. standardize all ORM tests on a single means of getting a Session when the Session API isn't the thing we are directly testing, using a new function fixture_session() that replaces create_session() and uses modern defaults. Change-Id: Iaf71206e9ee568151496d8bc213a069504bf65ef
* Major revisals to lambdasMike Bayer2020-12-161-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Improve coercions._deep_is_literal to check sequences for clause elements, thus allowing a phrase like lambda: col.in_([literal("x"), literal("y")]) to be handled 2. revise closure variable caching completely. All variables entering must be part of a closure cache key or rejected. only objects that can be resolved to HasCacheKey or FunctionType are accepted; all other types are rejected. This adds a high degree of strictness to lambdas and will make them a little more awkward to use in some cases, however prevents several classes of critical issues: a. previously, a lambda that had an expression derived from some kind of state, like "self.x", or "execution_context.session.foo" would produce a closure cache key from "self" or "execution_context", objects that can very well be per-execution and would therefore cause a AnalyzedFunction objects to overflow. (memory won't leak as it looks like an LRUCache is already used for these) b. a lambda, such as one used within DeferredLamdaElement, that produces different SQL expressions based on the arguments (which is in fact what it's supposed to do), however it would through the use of conditionals produce different bound parameter combinations, leading to literal parameters not tracked properly. These are now rejected as uncacheable whereas previously they would again be part of the closure cache key, causing an overflow of AnalyizedFunction objects. 3. Ensure non-mapped mixins are handled correctly by with_loader_criteria(). 4. Fixed bug in lambda SQL system where we are not supposed to allow a Python function to be embedded in the lambda, since we can't predict a bound value from it. While there was an error condition added for this, it was not tested and wasn't working; an informative error is now raised. 5. new docs for lambdas 6. consolidated changelog for all of these Fixes: #5760 Fixes: #5765 Fixes: #5766 Fixes: #5768 Fixes: #5770 Change-Id: Iedaa636c3225fad496df23b612c516c8ab247ab7
* Don't emit warnings on descriptor accessMike Bayer2020-11-201-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit is revising 5162f2bc5fc0ac239f26a76fc9f0c2, which when I did it felt a little rushed but I couldn't find anything wrong. Well here we are :). Fixed issue where a :class:`.RemovedIn20Warning` would erroneously emit when the ``.bind`` attribute were accessed internally on objects, particularly when stringifying a SQL construct. Alter the deprecated() decorator so that we can use it just to add docstring warnings but not actually warn when the function is accessed, adding new argument enable_warnings that can be set to False. Added a safety feature to deprecated_20() that will disallow an ":attr:" from proceeding if enable_warnings=False isn't present, unless there's an extra flag warn_on_attribute_access, since we want Session.transaction to emit a deprecation warning. This is a little hacky but it's essentially modifying the decorator to require a positive assertion that a deprecation decorator on a descriptor should actually warn on access. Remove the warning filter for session.transaction and get tests to pass to ensure this is not also being called internally. Added tests to ensure that common places .bind can be passed as a parameter definitely warn as I was not able to find this otherwise. Fixes: #5717 Change-Id: Ia586b4f9ee6b212f3a71104b1caf40b5edd399e2
* Reduce import time overheadMike Bayer2020-11-031-1/+1
| | | | | | | | | | | | | | | | | | * Fix subclass traversals to not run classes multiple times * switch compiler visitor to use an attrgetter, to avoid an eval() at startup time * don't pre-generate traversal functions, there's lots of these which are expensive to generate at once and most applications won't use them all; have it generate them on first use instead * Some ideas about removing asyncio imports, they don't seem to be too signficant, apply some more simplicity to the overall "greenlet fallback" situation Fixes: #5681 Change-Id: Ib564ddaddb374787ce3e11ff48026e99ed570933
* upgrade to black 20.8b1Mike Bayer2020-09-281-1/+6
| | | | | | | It's better, the majority of these changes look more readable to me. also found some docstrings that had formatting / quoting issues. Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
* Raise if unique() not applied to 2.0 joined eager load resultsMike Bayer2020-09-211-1/+1
| | | | | | | | | | | | | | | | | | | | | | The automatic uniquing of rows on the client side is turned off for the new :term:`2.0 style` of ORM querying. This improves both clarity and performance. However, uniquing of rows on the client side is generally necessary when using joined eager loading for collections, as there will be duplicates of the primary entity for each element in the collection because a join was used. This uniquing must now be manually enabled and can be achieved using the new :meth:`_engine.Result.unique` modifier. To avoid silent failure, the ORM explicitly requires the method be called when the result of an ORM query in 2.0 style makes use of joined load collections. The newer :func:`_orm.selectinload` strategy is likely preferable for eager loading of collections in any case. This changeset also fixes an issue where ORM-style "single entity" results would not apply unique() correctly if results were returned as tuples. Fixes: #4395 Change-Id: Ie62e0cb68ef2a6d2120e968b79575a70d057212e
* Convert lazy loader, selectinload, load_on_ident to lambda statementsMike Bayer2020-08-051-4/+7
| | | | | | | | | Building on newly robust lambdas in I29a513c98917b1d503abfdd61e6b6e8800851aa8, convert key loading off of the "baked" system so that baked is no longer used by the ORM. Change-Id: I3abfb45dd6e50f84f29d39434caa0b550ce27864
* Remove _generate_path_cache_key()Mike Bayer2020-06-291-41/+0
| | | | | | | | | | loader options can now make a deterministic cache key based on the structure they are given, and this accommodates for aliased classes as well so that these cache keys are now "safe". Have baked query call upon the regular cache key method. Change-Id: Iaa2ef4064cfb16146f415ca73080f32003dd830d
* Propose using RETURNING for bulk updates, deletesMike Bayer2020-06-231-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch makes several improvements in the area of bulk updates and deletes as well as the new session mechanics. RETURNING is now used for an UPDATE or DELETE statement emitted for a diaelct that supports "full returning" in order to satisfy the "fetch" strategy; this currently includes PostgreSQL and SQL Server. The Oracle dialect does not support RETURNING for more than one row, so a new dialect capability "full_returning" is added in addition to the existing "implicit_returning", indicating this dialect supports RETURNING for zero or more rows, not just a single identity row. The "fetch" strategy will gracefully degrade to the previous SELECT mechanics for dialects that do not support RETURNING. Additionally, the "fetch" strategy will attempt to use evaluation for the VALUES that were UPDATEd, rather than just expiring the updated attributes. Values should be evalutable in all cases where the value is not a SQL expression. The new approach also incurs some changes in the session.execute mechanics, where do_orm_execute() event handlers can now be chained to each return results; this is in turn used by the handler to detect on a per-bind basis if the fetch strategy needs to do a SELECT or if it can do RETURNING. A test suite is added to test_horizontal_shard that breaks up a single UPDATE or DELETE operation among multiple backends where some are SQLite and don't support RETURNING and others are PostgreSQL and do. The session event mechanics are corrected in terms of the "orm pre execute" hook, which now receives a flag "is_reentrant" so that the two ORM implementations for this can skip on their work if they are being called inside of ORMExecuteState.invoke(), where previously bulk update/delete were calling its SELECT a second time. In order for "fetch" to get the correct identity when called as pre-execute, it also requests the identity_token for each mapped instance which is now added as an optional capability of a SELECT for ORM columns. the identity_token that's placed by horizontal_sharding is now made available within each result row, so that even when fetching a merged result of plain rows we can tell which row belongs to which identity token. The evaluator that takes place within the ORM bulk update and delete for synchronize_session="evaluate" now supports the IN and NOT IN operators. Tuple IN is also supported. Fixes: #1653 Change-Id: I2292b56ae004b997cef0ba4d3fc350ae1dd5efc1
* Turn on caching everywhere, add loggingMike Bayer2020-06-101-12/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A variety of caching issues found by running all tests with statement caching turned on. The cache system now has a more conservative approach where any subclass of a SQL element will by default invalidate the cache key unless it adds the flag inherit_cache=True at the class level, or if it implements its own caching. Add working caching to a few elements that were omitted previously; fix some caching implementations to suit lesser used edge cases such as json casts and array slices. Refine the way BaseCursorResult and CursorMetaData interact with caching; to suit cases like Alembic modifying table structures, don't cache the cursor metadata if it were created against a cursor.description using non-positional matching, e.g. "select *". if a table re-ordered its columns or added/removed, now that data is obsolete. Additionally we have to adapt the cursor metadata _keymap regardless of if we just processed cursor.description, because if we ran against a cached SQLCompiler we won't have the right columns in _keymap. Other refinements to how and when we do this adaption as some weird cases were exposed in the Postgresql dialect, a text() construct that names just one column that is not actually in the statement. Fixed that also as it looks like a cut-and-paste artifact that doesn't actually affect anything. Various issues with re-use of compiled result maps and cursor metadata in conjunction with tables being changed, such as change in order of columns. mappers can be cleared but the class remains, meaning a mapper has to use itself as the cache key not the class. lots of bound parameter / literal issues, due to Alembic creating a straight subclass of bindparam that renders inline directly. While we can update Alembic to not do this, we have to assume other people might be doing this, so bindparam() implements the inherit_cache=True logic as well that was a bit involved. turn on cache stats in logging. Includes a fix to subqueryloader which moves all setup to the create_row_processor() phase and elminates any storage within the compiled context. This includes some changes to create_row_processor() signature and a revising of the technique used to determine if the loader can participate in polymorphic queries, which is also applied to selectinloading. DML update.values() and ordered_values() now coerces the keys as we have tests that pass an arbitrary class here which only includes __clause_element__(), so the key can't be cached unless it is coerced. this in turn changed how composite attributes support bulk update to use the standard approach of ClauseElement with annotations that are parsed in the ORM context. memory profiling successfully caught that the Session from Query was getting passed into _statement_20() so that was a big win for that test suite. Apparently Compiler had .execute() and .scalar() methods stuck on it, these date back to version 0.4 and there was a single test in the PostgreSQL dialect tests that exercised it for no apparent reason. Removed these methods as well as the concept of a Compiler holding onto a "bind". Fixes: #5386 Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
* Convert bulk update/delete to new execution modelMike Bayer2020-06-061-0/+1
| | | | | | | | | | | | | | | This reorganizes the BulkUD model in sqlalchemy.orm.persistence to be based on the CompileState concept and to allow plain update() / delete() to be passed to session.execute() where the ORM synchronize session logic will take place. Also gets "synchronize_session='fetch'" working with horizontal sharding. Adding a few more result.scalar_one() types of methods as scalar_one() seems like what is normally desired. Fixes: #5160 Change-Id: I8001ebdad089da34119eb459709731ba6c0ba975
* Inline a few ORM arguments, othersMike Bayer2020-06-031-1/+3
| | | | | | small changes Change-Id: Id89a0651196c431d0aaf6935f5a4e7b12dd70c6c
* Improve rendering of core statements w/ ORM elementsMike Bayer2020-05-311-1/+4
| | | | | | | | | | | | | | | | | | | | This patch contains a variety of ORM and expression layer tweaks to support ORM constructs in select() statements, without the 1.3.x requiremnt in Query that a full _compile_context() + new select() is needed in order to get a working statement object. Includes such tweaks as the ability to implement aliased class of an aliased class, as we are looking to fully support ACs against subqueries, as well as the ability to access anonymously-labeled ColumnProperty expressions within subqueries by naming the ".key" of the label after the property key. Some tuning to query.join() as well as ORMJoin internals to allow things to work more smoothly. Change-Id: Id810f485c5f7ed971529489b84694e02a3356d6d
* Remove loader option cycleMike Bayer2020-05-291-0/+6
| | | | | | | | removed a reference cycle set up by loader options due to the attribute dictionary containing Load objects that reference that dictionary. Change-Id: Ie3159a084f819ae44ca4992b0dbe094fb69b2fa7
* Convert execution to move through SessionMike Bayer2020-05-251-7/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the ORM execution flow with a single pathway through Session.execute() for all queries, including Core and ORM. Currently included is full support for ORM Query, Query.from_statement(), select(), as well as the baked query and horizontal shard systems. Initial changes have also been made to the dogpile caching example, which like baked query makes use of a new ORM-specific execution hook that replaces the use of both QueryEvents.before_compile() as well as Query._execute_and_instances() as the central ORM interception hooks. select() and Query() constructs alike can be passed to Session.execute() where they will return ORM results in a Results object. This API is currently used internally by Query. Full support for Session.execute()->results to behave in a fully 2.0 fashion will be in later changesets. bulk update/delete with ORM support will also be delivered via the update() and delete() constructs, however these have not yet been adapted to the new system and may follow in a subsequent update. Performance is also beginning to lag as of this commit and some previous ones. It is hoped that a few central functions such as the coercions functions can be rewritten in C to re-gain performance. Additionally, query caching is now available and some subsequent patches will attempt to cache more of the per-execution work from the ORM layer, e.g. column getters and adapters. This patch also contains initial "turn on" of the caching system enginewide via the query_cache_size parameter to create_engine(). Still defaulting at zero for "no caching". The caching system still needs adjustments in order to gain adequate performance. Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
* Unify Query and select() , move all processing to compile phaseMike Bayer2020-05-241-4/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | Convert Query to do virtually all compile state computation in the _compile_context() phase, and organize it all such that a plain select() construct may also be used as the source of information in order to generate ORM query state. This makes it such that Query is not needed except for its additional methods like from_self() which are all to be deprecated. The construction of ORM state will occur beyond the caching boundary when the new execution model is integrated. future select() gains a working join() and filter_by() method. as we continue to rebase and merge each commit in the steps, callcounts continue to bump around. will have to look at the final result when it's all in. References: #5159 References: #4705 References: #4639 References: #4871 References: #5010 Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
* repair test_fetch_resultsMike Bayer2020-04-301-1/+2
| | | | | | | this profiling test was not actually loading the related objects. Change-Id: I9d18a44f50f72f6653f736708829365eb561160e
* Pass connection to TablesTest.insert_data()Mike Bayer2020-04-151-18/+22
| | | | | | | | | | towards the goal of reducing verbosity and repetition in test fixtures as well as that we are moving to connection only for execution, move the insert_data() classmethod to accept a connection and adjust all fixtures to use it. Change-Id: I3bf534acca0d5f4cda1d4da8ae91f1155b829b09
* Ensure context.attrbutes dictionary is reset each timeMike Bayer2020-04-041-0/+2
| | | | | | | A memory growth issue was identified in this test which caused the profiling results to be inaccurate. Change-Id: I248dcce5833feada947bc91bdf09a8f925d31d65
* Fix tests failing for SQLite file databases; repair provisioningGord Thompson2020-03-131-1/+1
| | | | | | | | | | | | | | | | | | | | | | 1. ensure provision.py loads dialect implementations when running reap_dbs.py. Reapers haven't been working since 598f2f7e557073f29563d4d567f43931fc03013f . 2. add some exclusion rules to allow the sqlite_file target to work; add to tox. 3. add reap dbs target for SQLite, repair SQLite drop_db routine which also wasn't doing the right thing for memory databases etc. 4. Fix logging in provision files, as the main provision logger is the one that's enabled by reap_dbs and maybe others, have all the provision files use the provision logger. Fixes: #5180 Fixes: #5168 Change-Id: Ibc1b0106394d20f5bcf847f37b09d185f26ac9b5
* Limit non-backend critical profiling tests to SQLiteMike Bayer2020-02-171-18/+42
| | | | | | | | | | | | | | | | | | | | issues with backend-specific profiling should be limited to tests that are explcitly against resultset, compiler, etc. MySQL in particular has an often varying callcount that isn't worth running these tests against nor is it worth profiling them for other backends like Oracle and SQL Server. Also add the REQUIRE_SQLALCHEMY_CEXT flag to the regen_callcounts.tox.ini script, which is part of some review somewhere but is needed here to generate callcounts correctly. Add a "warmup" phase for some of the ORM tests for join conditions that have varying profile counts based on whether mappings have been used already or not; profiling should always be against the "warmed up" version of a function. Change-Id: If483820235fa4cc4360cbd067a9b68d83512d587
* Implement explicit autobegin step for SessionMike Bayer2020-01-031-0/+6
| | | | | | | | | | | | | | | | | | | | The :class:`.Session` object no longer initates a :class:`.SessionTransaction` object immediately upon construction or after the previous transaction is closed; instead, "autobegin" logic now initiates the new :class:`.SessionTransaction` on demand when it is next needed. Rationale includes to remove reference cycles from a :class:`.Session` that has been closed out, as well as to remove the overhead incurred by the creation of :class:`.SessionTransaction` objects that are often discarded immediately. This change affects the behavior of the :meth:`.SessionEvents.after_transaction_create` hook in that the event will be emitted when the :class:`.Session` first requires a :class:`.SessionTransaction` be present, rather than whenever the :class:`.Session` were created or the previous :class:`.SessionTransaction` were closed. Interactions with the :class:`.Engine` and the database itself remain unaffected. Fixes: #5074 Change-Id: I00b656eb5ee03d87104257a214214617aacae16c
* Don't apply aliasing + adaption for simple relationship joinsMike Bayer2019-12-191-0/+91
| | | | | | | | | | | Identified a performance issue in the system by which a join is constructed based on a mapped relationship. The clause adaption system would be used for the majority of join expressions including in the common case where no adaptation is needed. The conditions under which this adaptation occur have been refined so that average non-aliased joins along a simple relationship without a "secondary" table use about 70% less function calls. Change-Id: Ifbe04214576e5a9fac86ca80c1dc7145c27cd50a
* Add anonymizing context to cache keys, comparison; convert traversalMike Bayer2019-11-041-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Created new visitor system called "internal traversal" that applies a data driven approach to the concept of a class that defines its own traversal steps, in contrast to the existing style of traversal now known as "external traversal" where the visitor class defines the traversal, i.e. the SQLCompiler. The internal traversal system now implements get_children(), _copy_internals(), compare() and _cache_key() for most Core elements. Core elements with special needs like Select still implement some of these methods directly however most of these methods are no longer explicitly implemented. The data-driven system is also applied to ORM elements that take part in SQL expressions so that these objects, like mappers, aliasedclass, query options, etc. can all participate in the cache key process. Still not considered is that this approach to defining traversibility will be used to create some kind of generic introspection system that works across Core / ORM. It's also not clear if real statement caching using the _cache_key() method is feasible, if it is shown that running _cache_key() is nearly as expensive as compiling in any case. Because it is data driven, it is more straightforward to optimize using inlined code, as is the case now, as well as potentially using C code to speed it up. In addition, the caching sytem now accommodates for anonymous name labels, which is essential so that constructs which have anonymous labels can be cacheable, that is, their position within a statement in relation to other anonymous names causes them to generate an integer counter relative to that construct which will be the same every time. Gathering of bound parameters from any cache key generation is also now required as there is no use case for a cache key that does not extract bound parameter values. Applies-to: #4639 Change-Id: I0660584def8627cad566719ee98d3be045db4b8d
* Simplify _ColumnEntity, relatedMike Bayer2019-09-301-0/+4
| | | | | | | | | | | | | | | In the interests of making Query much more lightweight up front, rework the calculations done at the top when the entities are constructed to be much less inolved. Use the new coercion system for _ColumnEntity and stop accepting plain strings, this will need to emit a deprecation warning in 1.3.x. Use annotations and other techniques to reduce the decisionmaking and complexity of Query. For the use case of subquery(), .statement, etc. we would like to do minimal work in order to get the columns clause. Change-Id: I7e459bbd3bb10ec71235f75ef4f3b0a969bec590
* bump variance on test_merge_loadMike Bayer2019-08-231-1/+1
| | | | | | | This test is very sensitive and fluctuates a lot, failing builds, bump the variance to try and resolve. Change-Id: Ia19bb8871b432059cb3917ca0050a68f75c0a0f2
* Add autoincrement to test_orm AnnotatedOverheadTest for OracleMike Bayer2019-05-111-1/+3
| | | | | Fixes: #4675 Change-Id: I593c3a891462818e7095a30bf6cd7795ca143ad1
* Post black reformattingMike Bayer2019-01-061-17/+18
| | | | | | | | | | | | | Applied on top of a pure run of black -l 79 in I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes resolves all remaining flake8 conditions for those codes we have enabled in setup.cfg. Included are resolutions for all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
* Run black -l 79 against all source filesMike Bayer2019-01-061-331/+380
| | | | | | | | | | | | | | This is a straight reformat run using black as is, with no edits applied at all. The black run will format code consistently, however in some cases that are prevalent in SQLAlchemy code it produces too-long lines. The too-long lines will be resolved in the following commit that will resolve all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
* Deannotate ORM columns in ColumnEntityMike Bayer2018-11-021-1/+148
| | | | | | | | | | | Fixed a minor performance issue which could in some cases add unnecessary overhead to result fetching, involving the use of ORM columns and entities that include those same columns at the same time within a query. The issue has to do with hash / eq overhead when referring to the column in different ways. Fixes: #4347 Change-Id: I191d4d1b1623898060a9accdfd186de16f89a6b7
* Iterate options per path for baked cache keyMike Bayer2018-06-061-2/+191
| | | | | | | | | | | | | | | | Fixed an issue that was both a performance regression in 1.2 as well as an incorrect result regarding the "baked" lazy loader, involving the generation of cache keys from the original :class:`.Query` object's loader options. If the loader options were built up in a "branched" style using common base elements for multiple options, the same options would be rendered into the cache key repeatedly, causing both a performance issue as well as generating the wrong cache key. This is fixed, along with a performance improvement when such "branched" options are applied via :meth:`.Query.options` to prevent the same option objects from being applied repeatedly. Change-Id: I955fe2f50186abd8e753ad490fd3eb8f017e26f9 Fixes: #4270
* Add selectin loadingMike Bayer2017-04-261-1/+91
| | | | | | | | | | Adding a new kind of relationship loader that is a cross between the "immediateload" and the "subquery" eager loader, using an IN criteria to load related items in bulk immediately after the lead query result is loaded. Change-Id: If13713fba9b465865aef8fd50b5b6b977fe3ef7d Fixes: #3944
* Performance within instances()Mike Bayer2017-02-231-0/+20
| | | | | | | | | | | | | | | | Continuing from Ie43beecf37945b2bb7fff0aaa597a597293daa18, also observed is the overhead of PathRegsitry memoized token functions, as these paths are not cached in the case of long joinedloader chains. The memoizations here were made with short paths in mind, and have been replaced with an inlined straight create of these paths up front, producing callcounts very similar to 0.8. Combined with the previous optimizations, 1.1 now runs the "joined eager load of one row" worst case test in about 40% fewer calls than 0.8 and 60% fewer than 1.1.5. Change-Id: Ib5e1c1345a1dd8edfbdb3fed06eb717d4e164d31 Fixes: #3915
* Memoize AliasedClass, ClauseAdapter objects in joined eager loadMike Bayer2017-02-161-1/+142
| | | | | | | | | | | | | | | Addressed some long unattended performance concerns within the joined eager loader query construction system. The use of ad-hoc :class:`.AliasedClass` objects per query, which produces lots of column lookup overhead each time, has been replaced with a cached approach that makes use of a small pool of :class:`.AliasedClass` objects that are reused between invocations of joined eager loading. Callcount reduction of SQL query generation for worst-case joined loader scenarios (lots of joins, lots of columns) is reduced by approximately 270%. Change-Id: Ie43beecf37945b2bb7fff0aaa597a597293daa18 Fixes: #3915
* PEP8 cleanup in /test/aaa_profilingEric Streeper2015-03-181-0/+1
|
* - start locking down querying for cols after observing yesterdaysMike Bayer2015-03-091-0/+52
| | | | | callcount bump due to the slots thing - rewrite profiles using new technique
View Lorry Controller Status