summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/sql/visitors.py
Commit message (Collapse)AuthorAgeFilesLines
* Remove `typing.Self` workaroundYurii Karabas2023-02-081-9/+2
| | | | | | | | | | | | Remove ``typing.Self`` workaround, now using :pep:`673` for most methods that return ``Self``. Pull request courtesy Yurii Karabas. Fixes: #9254 Closes: #9255 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/9255 Pull-request-sha: 2947df8ada79f5c3afe9c838e65993302199c2f7 Change-Id: Ic32015ad52e95a61f3913d43ea436aa9402804df
* happy new year 2023Mike Bayer2023-01-031-1/+1
| | | | Change-Id: I625af65b3fb1815b1af17dc2ef47dd697fdc3fb1
* disable polymorphic adaption in most casesMike Bayer2022-12-071-3/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Improved a fix first made in version 1.4 for :ticket:`8456` which scaled back the usage of internal "polymorphic adapters", that are used to render ORM queries when the :paramref:`_orm.Mapper.with_polymorphic` parameter is used. These adapters, which are very complex and error prone, are now used only in those cases where an explicit user-supplied subquery is used for :paramref:`_orm.Mapper.with_polymorphic`, which includes only the use case of concrete inheritance mappings that use the :func:`_orm.polymorphic_union` helper, as well as the legacy use case of using an aliased subquery for joined inheritance mappings, which is not needed in modern use. For the most common case of joined inheritance mappings that use the built-in polymorphic loading scheme, which includes those which make use of the :paramref:`_orm.Mapper.polymorphic_load` parameter set to ``inline``, polymorphic adapters are now no longer used. This has both a positive performance impact on the construction of queries as well as a substantial simplification of the internal query rendering process. The specific issue targeted was to allow a :func:`_orm.column_property` to refer to joined-inheritance classes within a scalar subquery, which now works as intuitively as is feasible. ORM context, mapper, strategies now use ORMAdapter in all cases instead of straight ColumnAdapter; added some more parameters to ORMAdapter to make this possible. ORMAdapter now includes a "trace" enumeration that identifies the use path for the adapter and can aid in debugging. implement __slots__ for the ExternalTraversal hierarchy up to ORMAdapter. Within this change, we have to change the ClauseAdapter.wrap() method, which is only used in one polymorphic codepath, to use copy.copy() instead of `__dict__` access (apparently `__reduce_ex__` is implemented for objects with `__slots__`), and we also remove pickling ability, which should not be needed for adapters (this might have been needed for 1.3 and earlier in order for Query to be picklable, but none of that state is present within Query / select() / etc. anymore). Fixes: #8168 Change-Id: I3f6593eb02ab5e5964807c53a9fa4894c826d017
* try to support mypy 0.990Mike Bayer2022-11-091-1/+1
| | | | | | | | | | | mypy introduces a crash we need to work around, also some new rules. It also has either a behavioral change regarding how output is rendered in relationship to files being within sys.path or not, so work around that for test_mypy_plugin_py3k.py References: https://github.com/python/mypy/issues/14027 Change-Id: I689c7fe27dc52abee932de9e0fb23b2a2eba76fa
* revenge of pep 484Mike Bayer2022-05-151-12/+34
| | | | | | trying to get remaining must-haves for ORM Change-Id: I66a3ecbbb8e5ba37c818c8a92737b576ecf012f7
* update for flake8-future-imports 0.0.5Mike Bayer2022-05-141-2/+6
| | | | | | | | a whole bunch of errors were apparently blocked by 0.0.4 being installed. Fixes: #8020 Change-Id: I22a0faeaabe03de501897893391946d677c2df7e
* pep484 ORM / SQL result supportMike Bayer2022-04-271-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | after some experimentation it seems mypy is more amenable to the generic types being fully integrated rather than having separate spin-off types. so key structures like Result, Row, Select become generic. For DML Insert, Update, Delete, these are spun into type-specific subclasses ReturningInsert, ReturningUpdate, ReturningDelete, which is fine since the "row-ness" of these constructs doesn't happen until returning() is called in any case. a Tuple based model is then integrated so that these objects can carry along information about their return types. Overloads at the .execute() level carry through the Tuple from the invoked object to the result. To suit the issue of AliasedClass generating attributes that are dynamic, experimented with a custom subclass AsAliased, but then just settled on having aliased() lie to the type checker and return `Type[_O]`, essentially. will need some type-related accessors for with_polymorphic() also. Additionally, identified an issue in Update when used "mysql style" against a join(), it basically doesn't work if asked to UPDATE two tables on the same column name. added an error message to the specific condition where it happens with a very non-specific error message that we hit a thing we can't do right now, suggest multi-table update as a possible cause. Change-Id: I5eff7eefe1d6166ee74160b2785c5e6a81fa8b95
* pep-484: ORM public API, constructorsMike Bayer2022-04-201-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | for the moment, abandoning using @overload with relationship() and mapped_column(). The overloads are very difficult to get working at all, and the overloads that were there all wouldn't pass on mypy. various techniques of getting them to "work", meaning having right hand side dictate what's legal on the left, have mixed success and wont give consistent results; additionally, it's legal to have Optional / non-optional independent of nullable in any case for columns. relationship cases are less ambiguous but mypy was not going along with things. we have a comprehensive system of allowing left side annotations to drive the right side, in the absense of explicit settings on the right. so type-centric SQLAlchemy will be left-side driven just like dataclasses, and the various flags and switches on the right side will just not be needed very much. in other matters, one surprise, forgot to remove string support from orm.join(A, B, "somename") or do deprecations for it in 1.4. This is a really not-directly-used structure barely mentioned in the docs for many years, the example shows a relationship being used, not a string, so we will just change it to raise the usual error here. Change-Id: Iefbbb8d34548b538023890ab8b7c9a5d9496ec6e
* Don't pass vistor to immutables in cloned traverseMike Bayer2022-04-121-4/+30
| | | | | | | | | | Saw someone using cloned_traverse to move columns around (changing their .table) and not surprisingly having poor results. As cloned traversal is to provide a hook for in-place mutation of elements, it should not be given Immutable objects as these should not be changed once they are structurally composed. Change-Id: I43b22f52f243ef481a75d2cf5ecc73d50f110a81
* update flake8 noqa skips with proper syntaxFederico Caselli2022-04-111-2/+2
| | | | Change-Id: I42ed77f559e3ee5b8c600d98457ee37803ef0ea6
* pep484 - sql.selectableMike Bayer2022-04-041-10/+133
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | the pep484 task becomes more intense as there is mounting pressure to come up with a consistency in how data moves from end-user to instance variable. current thinking is coming into: 1. there are _typing._XYZArgument objects that represent "what the user sent" 2. there's the roles, which represent a kind of "filter" for different kinds of objects. These are mostly important as the argument we pass to coerce(). 3. there's the thing that coerce() returns, which should be what the construct uses as its internal representation of the thing. This is _typing._XYZElement. but there's some controversy over whether or not we should pass actual ClauseElements around by their role or not. I think we shouldn't at the moment, but this makes the "role-ness" of something a little less portable. Like, we have to set DMLTableRole for TableClause, Join, and Alias, but then also we have to repeat those three types in order to set up _DMLTableElement. Other change introduced here, there was a deannotate=True for the left/right of a sql.join(). All tests pass without that. I'd rather not have that there as if we have a join(A, B) where A, B are mapped classes, we want them inside of the _annotations. The rationale seems to be performance, but this performance can be illustrated to be on the compile side which we hope is cached in the normal case. CTEs now accommodate for text selects including recursive. Get typing to accommodate "util.preloaded" cleanly; add "preloaded" as a real module. This seemed like we would have needed pep562 `__getattr__()` but we don't, just set names in globals() as we import them. References: #6810 Change-Id: I34d17f617de2fe2c086fc556bd55748dc782faf0
* pep-484 - SQL column operationsMike Bayer2022-03-151-1/+1
| | | | | | | | | note we are taking out the ColumnOperartors[SQLCoreOperations] thing; not really clear why that was needed and at the moment it seems I was likely confused. Change-Id: I834b75f9b44f91b97e29f2e1a7b1029bd910e0a1
* pep-484: sqlalchemy.sql pass oneMike Bayer2022-03-131-244/+354
| | | | | | | | | | | | | | | | | | sqlalchemy.sql will require many passes to get all modules even gradually typed. Will have to pick and choose what modules can be strictly typed vs. which can be gradual. in this patch, emphasis is on visitors.py, cache_key.py, annotations.py for strict typing, compiler.py is on gradual typing but has much more structure, in particular where it connects with the outside world. The work within compiler.py also reached back out to engine/cursor.py , default.py quite a bit. References: #6810 Change-Id: I6e8a29f6013fd216e43d45091bc193f8be0368fd
* pep-484 for engineMike Bayer2022-03-011-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | All modules in sqlalchemy.engine are strictly typed with the exception of cursor, default, and reflection. cursor and default pass with non-strict typing, reflection is waiting on the multi-reflection refactor. Behavioral changes: * create_connect_args() methods return a tuple of list, dict, rather than a list of list, dict * removed allow_chars parameter from pyodbc connector ._get_server_version_info() method * the parameter list passed to do_executemany is now a list in all cases. previously, this was being run through dialect.execute_sequence_format, which defaults to tuple and was only intended for individual tuple params. * broke up dialect.dbapi into dialect.import_dbapi class method and dialect.dbapi module object. added a deprecation path for legacy dialects. it's not really feasible to type a single attr as a classmethod vs. module type. The "type_compiler" attribute also has this problem with greater ability to work around, left that one for now. * lots of constants changing to be Enum, so that we can type them. for fixed tuple-position constants in cursor.py / compiler.py (which are used to avoid the speed overhead of namedtuple), using Literal[value] which seems to work well * some tightening up in Row regarding __getitem__, which we can do since we are on full 2.0 style result use * altered the set_connection_execution_options and set_engine_execution_options event flows so that the dictionary of options may be mutated within the event hook, where it will then take effect as the actual options used. Previously, changing the dict would be silently ignored which seems counter-intuitive and not very useful. * A lot of DefaultDialect/DefaultExecutionContext methods and attributes, including underscored ones, move to interfaces. This is not fully ideal as it means the Dialect/ExecutionContext interfaces aren't publicly subclassable directly, but their current purpose is more of documentation for dialect authors who should (and certainly are) still be subclassing the DefaultXYZ versions in all cases Overall, Result was the most extremely difficult class hierarchy to type here as this hierarchy passes through largely amorphous "row" datatypes throughout, which can in fact by all kinds of different things, like raw DBAPI rows, or Row objects, or "scalar"/Any, but at the same time these types have meaning so I tried still maintaining some level of semantic markings for these, it highlights how complex Result is now, as it's trying to be extremely efficient and inlined while also being very open-ended and extensible. Change-Id: I98b75c0c09eab5355fc7a33ba41dd9874274f12a
* pep-484 for sqlalchemy.event; use future annotationsMike Bayer2022-02-151-0/+2
| | | | | | | | | | __future__.annotations mode allows us to use non-string annotations for argument and return types in most cases, but more importantly it removes a large amount of runtime overhead that would be spent in evaluating the annotations. Change-Id: I2f5b6126fe0019713fc50001be3627b664019ede References: #6810
* Fix overlapping slots, base classes without slotsArie Bovenberg2022-02-011-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some `__slots__` were not in order. Fixes #7527 ### Description I'm fixing two types of slots mistakes: - [x] remove overlapping slots (i.e. slots already defined on a base class) - [x] fix broken inheritance (i.e. slots class inheriting from a non-slots class) - [x] slots added to base class `TransactionalContext`. It seemed to use two attributes, which I've added as slots. - [x] empty slots removed from `ORMOption`. Its base class explicitly makes use of `__dict__` so empty slots don't add anything. - [x] empty slots added to `PostLoader`. It doesn't appear to use any slots not already defined on its base classes. - [x] empty slots added to `IterateMappersMixin`. It doesn't appear to use any slots not already defined on its subclasses. - [x] empty slots added to `ImmutableContainer`. It doesn't use any fields. - [x] empty slots added to `OperatorType`. It's a protocol. - [x] empty slots added to `InternalTraversal`, `_HasTraversalDispatch`. They don't seem to use attributes on their own. ### Checklist This pull request is: - [x] A short code fix - please include the issue number, and create an issue if none exists, which must include a complete example of the issue. one line code fixes without an issue and demonstration will not be accepted. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. one line code fixes without tests will not be accepted. **Have a nice day!** Closes: #7589 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7589 Pull-request-sha: 70a9c4d46916b7c6907eb1d3ad4f7033ec964191 Change-Id: I6c6e3e69c3c34d0f3bdda7f0684849834fdd1863
* Add new infrastructure to support greater use of __slots__Mike Bayer2022-01-211-40/+65
| | | | | | | | | | | | | | | | | | | * Changed AliasedInsp to use __slots__ * Migrated all of strategy_options to use __slots__ for objects. Adds new infrastructure to traversals to support shallow copy, to dict and from dict based on internal traversal attributes. Load / _LoadElement then leverage this to provide clone / generative / getstate without the need for __dict__ or explicit attribute lists. Doing this change revealed that there are lots of things that trigger off of whether or not a class has a __visit_name__ attribute. so to suit that we've gone back to having Visitable, which is a better name than Traversible at this point (I think Traversible is mis-spelled too). Change-Id: I13d04e494339fac9dbda0b8e78153418abebaf72 References: #7527
* remove internal use of metaclassesMike Bayer2022-01-111-134/+136
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | All but one metaclass used internally can now be replaced using __init_subclass__(). Within this patch we remove: * events._EventMeta * sql.visitors.TraversibleType * sql.visitors.InternalTraversibleType * testing.fixtures.FindFixture * testing.fixtures.FindFixtureDeclarative * langhelpers.EnsureKWArgType * sql.functions._GenericMeta * sql.type_api.VisitableCheckKWArg (was a mixture of TraversibleType and EnsureKWArgType) The remaining internal class is MetaOptions used by the sql.Options object which is in turn currently mostly for ORM internal use, as this type implements class level overrides for the ``+`` operator. For declarative, removing DeclarativeMeta in place of an `__init_subclass__()` class would not be fully feasible as it would break backwards compatibility with applications that refer to this class explicitly, but also DeclarativeMeta intercepts class-level attribute set and delete operations which is a widely used pattern. An option for declarative base to use `__init_subclass__()` should be provided but this is out of scope for this particular change. Change-Id: I8aa898c7ab59d887739037d34b1cbab36521ab78 References: #6810
* happy new year 2022Mike Bayer2022-01-061-1/+1
| | | | Change-Id: I49abf2607e0eb0623650efdf0091b1fb3db737ea
* accommodate for "clone" of ColumnClauseMike Bayer2021-12-211-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | for use with the ClauseElement.params() method, altered ColumnClause._clone() so that while the element stays immutable, if the column is associated with a subquery, it returns a new version of itself as corresponding to a clone of the subquery. this allows processing functions to access the parameters in the subquery and produce a copy of it. The use case here is the expanded use of .params() within loader strategies that use HasCacheKey._apply_params_to_element(). Fixed issue in new "loader criteria" method :meth:`_orm.PropComparator.and_` where usage with a loader strategy like :func:`_orm.selectinload` against a column that was a member of the ``.c.`` collection of a subquery object, where the subquery would be dynamically added to the FROM clause of the statement, would be subject to stale parameter values within the subquery in the SQL statement cache, as the process used by the loader strategy to replace the parameters at execution time would fail to accommodate the subquery when received in this form. Fixes: #7489 Change-Id: Ibb3b6af140b8a62a2c8d05b2ac92e86ca3013c46
* Clean up most py3k compatFederico Caselli2021-11-241-3/+3
| | | | Change-Id: I8172fdcc3103ff92aa049827728484c8779af6b7
* Support lightweight compiler column elements w/ slotsMike Bayer2021-11-221-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | the _CompileLabel class included ``__slots__`` but these weren't used as the superclasses included slots. Create a ``__slots__`` superclass for ``ClauseElement``, creating a new class of compilable SQL elements that don't include heavier features like caching, annotations and cloning, which are meant to be used only in an ad-hoc compiler fashion. Create new ``CompilerColumnElement`` from that which serves in column-oriented contexts, but similarly does not include any expression operator support as it is intended to be used only to generate a string. Apply this to both ``_CompileLabel`` as well as PostgreSQL ``_ColonCast``, which does not actually subclass ``ColumnElement`` as this class has memoized attributes that aren't worth changing, and does not include SQL operator capabilities as these are not needed for these compiler-only objects. this allows us to more inexpensively add new ad-hoc labels / casts etc. at compile time, as we will be seeking to expand out the typecasts that are needed for PostgreSQL dialects in a subsequent patch. Change-Id: I52973ae3295cb6e2eb0d7adc816c678a626643ed
* Remove object in class definitionFederico Caselli2021-11-221-1/+1
| | | | | References: #4600 Change-Id: I2a62ddfe00bc562720f0eae700a497495d7a987a
* Support generics in code to allow Column[int] etcFederico Caselli2021-07-271-0/+4
| | | | | | Fixes: #6804 Fixes: #6759 Change-Id: Ie7f32c38a22dbfa059b5709b883ff464b16031ae
* Replace all http:// links to https://Federico Caselli2021-07-041-2/+2
| | | | | | Also replace http://pypi.python.org/pypi with https://pypi.org/project Change-Id: I84b5005c39969a82140706472989f2a30b0c7685
* memoize current options and joins w with_entities/with_only_colsMike Bayer2021-06-171-0/+35
| | | | | | | | | | | | | | | | Fixed further regressions in the same area as that of :ticket:`6052` where loader options as well as invocations of methods like :meth:`_orm.Query.join` would fail if the left side of the statement for which the option/join depends upon were replaced by using the :meth:`_orm.Query.with_entities` method, or when using 2.0 style queries when using the :meth:`_sql.Select.with_only_columns` method. A new set of state has been added to the objects which tracks the "left" entities that the options / join were made against which is memoized when the lead entities are changed. Fixes: #6503 Fixes: #6253 Change-Id: I211b2af98b0b20d1263fb15dc513884dcc5de6a4
* Fix minor glitches in the visitors.py docstrings (#6522)Lele Gaifax2021-05-251-5/+5
| | | | | | | | | | | * Fix docstring typo * Capitalize first sentence word * Properly markup class name This is just for consistency with nearby references: the HasCacheKey class has no docstring so it does not appear in the documentation, moreover it's in the traversals module, so Sphinx cannot render it as a link anyway.
* Correct cache key for proxy_owner, with_context_optionsMike Bayer2021-05-101-0/+2
| | | | | | | | | | | | | | | | | | Fixed issue in subquery loader strategy which prevented caching from working correctly. This would have been seen in the logs as a "generated" message instead of "cached" for all subqueryload SQL emitted, which by saturating the cache with new keys would degrade overall performance; it also would produce "LRU size alert" warnings. In this issue we also observe that the local LRU cache for lazyloader and selectinloader will get used for all subsequent loads as well, which makes it more likely to hit the limit of 30. However rather than trying to work this out, it would be better if we removed the loader-local LRU caches altogether once we are confident these are working well. Fixes: #6459 Change-Id: Id953e8f75536bb87f7e3315929cebcd8f84a5a50
* Adapt loader_criteria params for current queryMike Bayer2021-03-261-3/+5
| | | | | | | | | | | | | | | | | | Fixed critical issue in the new :meth:`_orm.PropComparator.and_` feature where loader strategies that emit secondary SELECT statements such as :func:`_orm.selectinload` and :func:`_orm.lazyload` would fail to accommodate for bound parameters in the user-defined criteria in terms of the current statement being executed, as opposed to the cached statement, causing stale bound values to be used. This also adds a warning for the case where an object that uses :func:`_orm.lazyload` in conjunction with :meth:`_orm.PropComparator.and_` is attempted to be serialized; the loader criteria cannot reliably be serialized and deserialized and eager loading should be used for this case. Fixes: #6139 Change-Id: I5a638bbecb7b583db2d3c0b76469f5a25c13dd3b
* Implement support for functions as FROM with columns clause supportMike Bayer2021-02-031-4/+2
| | | | | | | | | | | | | | | | Implemented support for "table valued functions" along with additional syntaxes supported by PostgreSQL, one of the most commonly requested features. Table valued functions are SQL functions that return lists of values or rows, and are prevalent in PostgreSQL in the area of JSON functions, where the "table value" is commonly referred towards as the "record" datatype. Table valued functions are also supported by Oracle and SQL Server. Moved from I5b093b72533ef695293e737eb75850b9713e5e03 due to accidental push Fixes: #3566 Change-Id: Iea36d04c80a5ed3509dcdd9ebf0701687143fef5
* happy new yearMike Bayer2021-01-041-1/+1
| | | | Change-Id: Ic5bb19ca8be3cb47c95a0d3315d84cb484bac47c
* Reduce import time overheadMike Bayer2020-11-031-18/+17
| | | | | | | | | | | | | | | | | | * Fix subclass traversals to not run classes multiple times * switch compiler visitor to use an attrgetter, to avoid an eval() at startup time * don't pre-generate traversal functions, there's lots of these which are expensive to generate at once and most applications won't use them all; have it generate them on first use instead * Some ideas about removing asyncio imports, they don't seem to be too signficant, apply some more simplicity to the overall "greenlet fallback" situation Fixes: #5681 Change-Id: Ib564ddaddb374787ce3e11ff48026e99ed570933
* Update select usage to use the new 1.4 formatFederico Caselli2020-09-081-1/+1
| | | | | | | | | | | | | | | | This change includes mainly that the bracketed use within select() is moved to positional, and keyword arguments are removed from calls to the select() function. it does not yet fully address other issues such as keyword arguments passed to the table.select(). Additionally, allows False / None to both be considered as "disable" for all of select.correlate(), select.correlate_except(), query.correlate(), which establishes consistency with passing of ``False`` for the legact select(correlate=False) argument. Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
* Robustness for lambdas, lambda statementsMike Bayer2020-08-051-8/+26
| | | | | | | | | | | | | | | | | | | | in order to accommodate relationship loaders with lambda caching, a lot more is needed. This is a full refactor of the lambda system such that it now has two levels of caching; the first level caches what can be known from the __code__ element, then the next level of caching is against the lambda itself and the contents of __closure__. This allows for the elements inside the lambdas, like columns and entities, to change and then be part of the cache key. Lazy/selectinloads' use of baked queries had to add distinct cache key elements, which was attempted here but overall things needed to be more robust than that. This commit is broken out from the very long and sprawling commit at Id6b5c03b1ce9ddb7b280f66792212a0ef0a1c541 . Change-Id: I29a513c98917b1d503abfdd61e6b6e8800851aa8
* Fix a wide variety of typos and broken linksaplatkouski2020-06-251-22/+22
| | | | | | | | | | | | Note the PR has a few remaining doc linking issues listed in the comment that must be addressed separately. Signed-off-by: aplatkouski <5857672+aplatkouski@users.noreply.github.com> Closes: #5371 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5371 Pull-request-sha: 7e7d233cf3a0c66980c27db0fcdb3c7d93bc2510 Change-Id: I9c36e8d8804483950db4b42c38ee456e384c59e3
* Turn on caching everywhere, add loggingMike Bayer2020-06-101-7/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A variety of caching issues found by running all tests with statement caching turned on. The cache system now has a more conservative approach where any subclass of a SQL element will by default invalidate the cache key unless it adds the flag inherit_cache=True at the class level, or if it implements its own caching. Add working caching to a few elements that were omitted previously; fix some caching implementations to suit lesser used edge cases such as json casts and array slices. Refine the way BaseCursorResult and CursorMetaData interact with caching; to suit cases like Alembic modifying table structures, don't cache the cursor metadata if it were created against a cursor.description using non-positional matching, e.g. "select *". if a table re-ordered its columns or added/removed, now that data is obsolete. Additionally we have to adapt the cursor metadata _keymap regardless of if we just processed cursor.description, because if we ran against a cached SQLCompiler we won't have the right columns in _keymap. Other refinements to how and when we do this adaption as some weird cases were exposed in the Postgresql dialect, a text() construct that names just one column that is not actually in the statement. Fixed that also as it looks like a cut-and-paste artifact that doesn't actually affect anything. Various issues with re-use of compiled result maps and cursor metadata in conjunction with tables being changed, such as change in order of columns. mappers can be cleared but the class remains, meaning a mapper has to use itself as the cache key not the class. lots of bound parameter / literal issues, due to Alembic creating a straight subclass of bindparam that renders inline directly. While we can update Alembic to not do this, we have to assume other people might be doing this, so bindparam() implements the inherit_cache=True logic as well that was a bit involved. turn on cache stats in logging. Includes a fix to subqueryloader which moves all setup to the create_row_processor() phase and elminates any storage within the compiled context. This includes some changes to create_row_processor() signature and a revising of the technique used to determine if the loader can participate in polymorphic queries, which is also applied to selectinloading. DML update.values() and ordered_values() now coerces the keys as we have tests that pass an arbitrary class here which only includes __clause_element__(), so the key can't be cached unless it is coerced. this in turn changed how composite attributes support bulk update to use the standard approach of ClauseElement with annotations that are parsed in the ORM context. memory profiling successfully caught that the Session from Query was getting passed into _statement_20() so that was a big win for that test suite. Apparently Compiler had .execute() and .scalar() methods stuck on it, these date back to version 0.4 and there was a single test in the PostgreSQL dialect tests that exercised it for no apparent reason. Removed these methods as well as the concept of a Compiler holding onto a "bind". Fixes: #5386 Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
* Improve rendering of core statements w/ ORM elementsMike Bayer2020-05-311-1/+10
| | | | | | | | | | | | | | | | | | | | This patch contains a variety of ORM and expression layer tweaks to support ORM constructs in select() statements, without the 1.3.x requiremnt in Query that a full _compile_context() + new select() is needed in order to get a working statement object. Includes such tweaks as the ability to implement aliased class of an aliased class, as we are looking to fully support ACs against subqueries, as well as the ability to access anonymously-labeled ColumnProperty expressions within subqueries by naming the ".key" of the label after the property key. Some tuning to query.join() as well as ORMJoin internals to allow things to work more smoothly. Change-Id: Id810f485c5f7ed971529489b84694e02a3356d6d
* Convert execution to move through SessionMike Bayer2020-05-251-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the ORM execution flow with a single pathway through Session.execute() for all queries, including Core and ORM. Currently included is full support for ORM Query, Query.from_statement(), select(), as well as the baked query and horizontal shard systems. Initial changes have also been made to the dogpile caching example, which like baked query makes use of a new ORM-specific execution hook that replaces the use of both QueryEvents.before_compile() as well as Query._execute_and_instances() as the central ORM interception hooks. select() and Query() constructs alike can be passed to Session.execute() where they will return ORM results in a Results object. This API is currently used internally by Query. Full support for Session.execute()->results to behave in a fully 2.0 fashion will be in later changesets. bulk update/delete with ORM support will also be delivered via the update() and delete() constructs, however these have not yet been adapted to the new system and may follow in a subsequent update. Performance is also beginning to lag as of this commit and some previous ones. It is hoped that a few central functions such as the coercions functions can be rewritten in C to re-gain performance. Additionally, query caching is now available and some subsequent patches will attempt to cache more of the per-execution work from the ORM layer, e.g. column getters and adapters. This patch also contains initial "turn on" of the caching system enginewide via the query_cache_size parameter to create_engine(). Still defaulting at zero for "no caching". The caching system still needs adjustments in order to gain adequate performance. Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
* Unify Query and select() , move all processing to compile phaseMike Bayer2020-05-241-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | Convert Query to do virtually all compile state computation in the _compile_context() phase, and organize it all such that a plain select() construct may also be used as the source of information in order to generate ORM query state. This makes it such that Query is not needed except for its additional methods like from_self() which are all to be deprecated. The construction of ORM state will occur beyond the caching boundary when the new execution model is integrated. future select() gains a working join() and filter_by() method. as we continue to rebase and merge each commit in the steps, callcounts continue to bump around. will have to look at the final result when it's all in. References: #5159 References: #4705 References: #4639 References: #4871 References: #5010 Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
* Streamline visitors.iterateMike Bayer2020-05-181-73/+19
| | | | | | | | | | | | | | | | | | | | This method might be used more significantly in the ORM refactor, so further refine it. * all get_children() methods now work entirely based on iterators. Basically only select() was sensitive to this anymore and it now chains the iterators together * remove all kinds of flags like column_collections, schema_visitor that apparently aren't used anymore. * remove the "depthfirst" visitors as these don't seem to be used either. * make sure select() yields its columns first as these will be used to determine the current mapper. Change-Id: I05273a2d5306a57c2d1b0979050748cf3ac964bf
* Run search and replace of symbolic module namesMike Bayer2020-04-141-28/+42
| | | | | | | | Replaces a wide array of Sphinx-relative doc references with an abbreviated absolute form now supported by zzzeeksphinx. Change-Id: I94bffcc3f37885ffdde6238767224296339698a2
* Fix almost all read-level sphinx warningsMike Bayer2020-04-111-1/+1
| | | | | | | | There are some related to changelog that I can't figure out and are likely due to something in the changelog extension. also one thing with a "collection" I can't figure out. Change-Id: I0a9e6f4291c3589aa19a4abcb9245cd22a266fe0
* Try to measure new style caching in the ORM, take twoMike Bayer2020-04-011-0/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf Trying to see if by making the cache key memoized, we still can have the older "identity" form of caching which is the cheapest of all, at the same time as the newer "cache key each time" version that is not nearly as cheap; but still much cheaper than no caching at all. Also needed is a per-execution update of _keymap when we invoke from a cached select, so that Column objects that are anonymous or otherwise adapted will match up. this is analogous to the adaption of bound parameters from the cache key. Adds test coverage for the keymap / construct_params() changes related to caching. Also hones performance to a large extent for statement construction and cache key generation. Also includes a new memoized attribute approach that vastly simplifies the previous approach of "group_expirable_memoized_property" and finally integrates cleanly with _clone(), _generate(), etc. no more hardcoding of attributes is needed, as well as that most _reset_memoization() calls are no longer needed as the reset is inherent in a _generate() call; this also has dramatic performance improvements. Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
* Correct ambiguous func / class linksMike Bayer2020-03-251-1/+1
| | | | | | | | | :func:`.sql.expression.select`, :func:`.sql.expression.insert` and :class:`.sql.expression.Insert` were hitting many ambiguous symbol errors, due to future.select, as well as the PG/MySQL variants of Insert. Change-Id: Iac862bfc172a7f7f0cbba5353a83dc203bed376c
* Decouple compiler state from DML objects; make cacheableMike Bayer2020-03-061-0/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Targeting select / insert / update / delete, the goal is to minimize overhead of construction and generative methods so that only the raw arguments passed are handled. An interim stage that converts the raw state into more compiler-ready state is added, which is analogous to the ORM QueryContext which will also be rolled in to be a similar concept, as is currently being prototyped in I19e05b3424b07114cce6c439b05198ac47f7ac10. the ORM update/delete BulkUD concept is also going to be rolled onto this idea. So while the compiler-ready state object, here called DMLState, looks a little thin, it's the base of a bigger pattern that will allow for ORM functionality to embed itself directly into the compiler, execution context, and result set objects. This change targets the DML objects, primarily focused on the values() method which is the most complex process. The work done by values() is minimized as much as possible while still being able to create a cache key. Additional computation is then offloaded to a new object ValuesState that is handled by the compiler. Architecturally, a big change here is that insert.values() and update.values() will generate BindParameter objects for the values now, which are then carefully received by crud.py so that they generate the expected names. This is so that the values() portion of these constructs is cacheable. for the "multi-values" version of Insert, this is all skipped and the plan right now is that a multi-values insert is not worth caching (can always be revisited). Using the coercions system in values() also gets us nicer validation for free, we can remove the NotAClauseElement thing from schema, and we also now require scalar_subquery() is called for an insert/update that uses a SELECT as a column value, 1.x deprecation path is added. The traversal system is then applied to the DML objects including tests so that they have traversal, cloning, and cache key support. cloning is not a use case for DML however having it present allows better validation of the structure within the tests. Special per-dialect DML is explicitly not cacheable at the moment, more as a proof of concept that third party DML constructs can exist as gracefully not-cacheable rather than producing an incomplete cache key. A few selected performance improvements have been added as well, simplifying the immutabledict.union() method and adding a new SQLCompiler function that can generate delimeter-separated clauses like WHERE and ORDER BY without having to build a ClauseList object at all. The use of ClauseList will be removed from Select in an upcoming commit. Overall, ClaustList is unnecessary for internal use and only adds overhead to statement construction and will likely be removed as much as possible except for explcit use of conjunctions like and_() and or_(). Change-Id: I408e0b8be91fddd77cf279da97f55020871f75a9
* Ensure all nested exception throws have a causeMike Bayer2020-03-021-3/+4
| | | | | | | | | | | | | | | Applied an explicit "cause" to most if not all internally raised exceptions that are raised from within an internal exception catch, to avoid misleading stacktraces that suggest an error within the handling of an exception. While it would be preferable to suppress the internally caught exception in the way that the ``__suppress_context__`` attribute would, there does not as yet seem to be a way to do this without suppressing an enclosing user constructed context, so for now it exposes the internally caught exception as the cause so that full information about the context of the error is maintained. Fixes: #4849 Change-Id: I55a86b29023675d9e5e49bc7edc5a2dc0bcd4751
* Discontinue dynamic __visit_name__Mike Bayer2020-03-011-27/+20
| | | | | | | | | Removed very antiquated logic that checks if __visit_name__ is a property. There's no need for this as the compiler can handle switching between implementations. Convert _compile_dispatch() to be fully inlined. Change-Id: Ic0c7247c2d7dfed93a27f09250a8ed6352370764
* fix documentation typosFederico Caselli2020-02-081-1/+1
| | | | Change-Id: I4376910ae1cf6bf27226f049d4cf2d0e6ba2a83b
* happy new yearMike Bayer2020-01-011-1/+1
| | | | Change-Id: I08440dc25e40ea1ccea1778f6ee9e28a00808235
* Test for short term reference cycles and resolve as many as possibleMike Bayer2019-12-301-1/+3
| | | | | | | | Added test support and repaired a wide variety of unnecessary reference cycles created for short-lived objects, mostly in the area of ORM queries. Fixes: #5056 Change-Id: Ifd93856eba550483f95f9ae63d49f36ab068b85a
View Lorry Controller Status