summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/sql/elements.py
Commit message (Collapse)AuthorAgeFilesLines
* Remove code deprecated before version 1.1Federico Caselli2020-04-091-81/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Remove deprecated method ``get_primary_keys` in the :class:`.Dialect` and :class:`.Inspector` classes. - Remove deprecated event ``dbapi_error`` and the method ``ConnectionEvents.dbapi_error`. - Remove support for deprecated engine URLs of the form ``postgres://``. - Remove deprecated dialect ``mysql+gaerdbms``. - Remove deprecated parameter ``quoting`` from :class:`.mysql.ENUM` and :class:`.mysql.SET` in the ``mysql`` dialect. - Remove deprecated function ``comparable_property``. and function ``comparable_using`` in the declarative extension. - Remove deprecated function ``compile_mappers``. - Remove deprecated method ``collection.linker``. - Remove deprecated method ``Session.prune`` and parameter ``Session.weak_identity_map``. This change also removes the class ``StrongInstanceDict``. - Remove deprecated parameter ``mapper.order_by``. - Remove deprecated parameter ``Session._enable_transaction_accounting`. - Remove deprecated parameter ``Session.is_modified.passive``. - Remove deprecated class ``Binary``. Please use :class:`.LargeBinary`. - Remove deprecated methods ``Compiled.compile``, ``ClauseElement.__and__`` and ``ClauseElement.__or__`` and attribute ``Over.func``. - Remove deprecated ``FromClause.count`` method. - Remove deprecated parameter ``Table.useexisting``. - Remove deprecated parameters ``text.bindparams`` and ``text.typemap``. - Remove boolean support for the ``passive`` parameter in ``get_history``. - Remove deprecated ``adapt_operator`` in ``UserDefinedType.Comparator``. Fixes: #4643 Change-Id: Idcd390c77bf7b0e9957907716993bdaa3f1a1763
* Try to measure new style caching in the ORM, take twoMike Bayer2020-04-011-39/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf Trying to see if by making the cache key memoized, we still can have the older "identity" form of caching which is the cheapest of all, at the same time as the newer "cache key each time" version that is not nearly as cheap; but still much cheaper than no caching at all. Also needed is a per-execution update of _keymap when we invoke from a cached select, so that Column objects that are anonymous or otherwise adapted will match up. this is analogous to the adaption of bound parameters from the cache key. Adds test coverage for the keymap / construct_params() changes related to caching. Also hones performance to a large extent for statement construction and cache key generation. Also includes a new memoized attribute approach that vastly simplifies the previous approach of "group_expirable_memoized_property" and finally integrates cleanly with _clone(), _generate(), etc. no more hardcoding of attributes is needed, as well as that most _reset_memoization() calls are no longer needed as the reset is inherent in a _generate() call; this also has dramatic performance improvements. Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
* Add a third labeling mode for SELECT statementsMike Bayer2020-03-291-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Enhanced the disambiguating labels feature of the :func:`~.sql.expression.select` construct such that when a select statement is used in a subquery, repeated column names from different tables are now automatically labeled with a unique label name, without the need to use the full "apply_labels()" feature that conbines tablename plus column name. The disambigated labels are available as plain string keys in the .c collection of the subquery, and most importantly the feature allows an ORM :func:`.orm.aliased` construct against the combination of an entity and an arbitrary subquery to work correctly, targeting the correct columns despite same-named columns in the source tables, without the need for an "apply labels" warning. The existing labeling style is now called LABEL_STYLE_TABLENAME_PLUS_COL. This labeling style will remain used throughout the ORM as has been the case for over a decade, however, the new disambiguation scheme could theoretically replace this scheme entirely. The new scheme would dramatically alter how SQL looks when rendered from the ORM to be more succinct but arguably harder to read. The tablename_columnname scheme used by Join.c is unaffected here, as that's still hardcoded to that scheme. Fixes: #5221 Change-Id: Ib47d9e0f35046b3afc77bef6e65709b93d0c3026
* Correct ambiguous func / class linksMike Bayer2020-03-251-19/+19
| | | | | | | | | :func:`.sql.expression.select`, :func:`.sql.expression.insert` and :class:`.sql.expression.Insert` were hitting many ambiguous symbol errors, due to future.select, as well as the PG/MySQL variants of Insert. Change-Id: Iac862bfc172a7f7f0cbba5353a83dc203bed376c
* Remove deprecated elements from selectable.py; remove lockmodeMike Bayer2020-03-231-3/+1
| | | | | | | | | | | | | Removed autocommit and legacy "for update" / "lockmode" elements from selectable.py / query.py. lockmode was removed from selectable in 693938dd6fb2f3ee3e031aed4c62355ac97f3ceb however was not removed from the ORM. Also removes the ignore_nonexistent_tables option on join(). Change-Id: I0cfcf9e6a8d4ef6432c9e25ef75173b3b3f5fd87 Partially-fixes: #4643
* Merge "Rework select(), CompoundSelect() in terms of CompileState"mike bayer2020-03-111-57/+81
|\
| * Rework select(), CompoundSelect() in terms of CompileStateMike Bayer2020-03-101-57/+81
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Continuation of I408e0b8be91fddd77cf279da97f55020871f75a9 - add an options() method to the base Generative construct. this will be where ORM options can go - Change Null, False_, True_ to be singletons, so that we aren't instantiating them and having to use isinstance. The previous issue with this was that they would produce dupe labels in SELECT statements. Apply the duplicate column logic, newly added in 1.4, to these objects as well as to non-apply-labels SELECT statements in general as a means of improving this. - create a revised system for generating ClauseList compilation constructs that simplfies up front creation to not actually use ClauseList; a simple tuple is rendered by the compiler using the same constrcution rules as what are used for ClauseList but without creating the actual object. Apply to Select, CompoundSelect, revise Update, Delete - Select, CompoundSelect get an initial CompileState implementation. All methods used only within compilation are moved here - refine update/insert/delete compile state to not require an outside boolean - refine and simplify Select._copy_internals - rework bind(), which is going away, to not use some of the internal traversal stuff - remove "autocommit", "for_update" parameters from Select, references #4643 - remove "autocommit" parameter from TextClause , references #4643 - add deprecation warnings for statement.execute(), engine.execute(), statement.scalar(), engine.scalar(). Fixes: #5193 Change-Id: I04ca0152b046fd42c5054ba10f37e43fc6e5a57b
* | Simplified module pre-loading strategy and made it linter friendlyFederico Caselli2020-03-071-4/+6
|/ | | | | | | | | | | | | | | | | Introduced a modules registry to register modules that should be lazily loaded in the package init. This ensures that they are in the system module cache, avoiding potential thread safety issues as when importing them directly in the function that uses them. The module registry is used to obtain these modules directly, ensuring that the all the lazily loaded modules are resolved at the proper time This replaces dependency_for decorator and the dependencies decorator logic, removing the need to pass the resolved modules as arguments of the decodated functions and removes possible errors caused by linters. Fixes: #4689 Fixes: #4656 Change-Id: I2e291eba4297867fc0ddb5d875b9f7af34751d01
* Decouple compiler state from DML objects; make cacheableMike Bayer2020-03-061-9/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Targeting select / insert / update / delete, the goal is to minimize overhead of construction and generative methods so that only the raw arguments passed are handled. An interim stage that converts the raw state into more compiler-ready state is added, which is analogous to the ORM QueryContext which will also be rolled in to be a similar concept, as is currently being prototyped in I19e05b3424b07114cce6c439b05198ac47f7ac10. the ORM update/delete BulkUD concept is also going to be rolled onto this idea. So while the compiler-ready state object, here called DMLState, looks a little thin, it's the base of a bigger pattern that will allow for ORM functionality to embed itself directly into the compiler, execution context, and result set objects. This change targets the DML objects, primarily focused on the values() method which is the most complex process. The work done by values() is minimized as much as possible while still being able to create a cache key. Additional computation is then offloaded to a new object ValuesState that is handled by the compiler. Architecturally, a big change here is that insert.values() and update.values() will generate BindParameter objects for the values now, which are then carefully received by crud.py so that they generate the expected names. This is so that the values() portion of these constructs is cacheable. for the "multi-values" version of Insert, this is all skipped and the plan right now is that a multi-values insert is not worth caching (can always be revisited). Using the coercions system in values() also gets us nicer validation for free, we can remove the NotAClauseElement thing from schema, and we also now require scalar_subquery() is called for an insert/update that uses a SELECT as a column value, 1.x deprecation path is added. The traversal system is then applied to the DML objects including tests so that they have traversal, cloning, and cache key support. cloning is not a use case for DML however having it present allows better validation of the structure within the tests. Special per-dialect DML is explicitly not cacheable at the moment, more as a proof of concept that third party DML constructs can exist as gracefully not-cacheable rather than producing an incomplete cache key. A few selected performance improvements have been added as well, simplifying the immutabledict.union() method and adding a new SQLCompiler function that can generate delimeter-separated clauses like WHERE and ORDER BY without having to build a ClauseList object at all. The use of ClauseList will be removed from Select in an upcoming commit. Overall, ClaustList is unnecessary for internal use and only adds overhead to statement construction and will likely be removed as much as possible except for explcit use of conjunctions like and_() and or_(). Change-Id: I408e0b8be91fddd77cf279da97f55020871f75a9
* Ensure all nested exception throws have a causeMike Bayer2020-03-021-22/+44
| | | | | | | | | | | | | | | Applied an explicit "cause" to most if not all internally raised exceptions that are raised from within an internal exception catch, to avoid misleading stacktraces that suggest an error within the handling of an exception. While it would be preferable to suppress the internally caught exception in the way that the ``__suppress_context__`` attribute would, there does not as yet seem to be a way to do this without suppressing an enclosing user constructed context, so for now it exposes the internally caught exception as the cause so that full information about the context of the error is maintained. Fixes: #4849 Change-Id: I55a86b29023675d9e5e49bc7edc5a2dc0bcd4751
* Remove print statement in favor of print() function in docs and examplesAlbert Tugushev2020-02-261-13/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | <!-- Provide a general summary of your proposed changes in the Title field above --> ### Description <!-- Describe your changes in detail --> Remove print statements ### Checklist <!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once) --> This pull request is: - [X] A documentation / typographical error fix - Good to go, no issue or tests are needed - [ ] A short code fix - please include the issue number, and create an issue if none exists, which must include a complete example of the issue. one line code fixes without an issue and demonstration will not be accepted. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. one line code fixes without tests will not be accepted. - [ ] A new feature implementation - please include the issue number, and create an issue if none exists, which must include a complete example of how the feature would look. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. **Have a nice day!** Closes: #5166 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5166 Pull-request-sha: 04a7394f71298322188f0861b4dfe93e5485839d Change-Id: Ib90a59fac929661a18748c6e44966fb87e3978c6
* Deprecate empty or_() and and_()Federico Caselli2020-01-251-18/+88
| | | | | | | | | | | | | | | | | | | | | | | Creating an :func:`.and_` or :func:`.or_` construct with no arguments or empty ``*args`` will now emit a deprecation warning, as the SQL produced is a no-op (i.e. it renders as a blank string). This behavior is considered to be non-intuitive, so for empty or possibly empty :func:`.and_` or :func:`.or_` constructs, an appropriate default boolean should be included, such as ``and_(True, *args)`` or ``or_(False, *args)``. As has been the case for many major versions of SQLAlchemy, these particular boolean values will not render if the ``*args`` portion is non-empty. As there are some internal cases where an empty and_() construct is used in order to build an optional WHERE expression, a private utility function is added to suit this use case. Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Fixes: #5054 Closes: #5062 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5062 Pull-request-sha: 5ca2f27281977d74e390148c0fb8deaa0e0e4ad9 Change-Id: I599b9c8befa64d9a59a35ad7dd84ff400e3aa647
* Alter unique bound parameter key on deserializeMike Bayer2020-01-101-0/+7
| | | | | | | | | | | | | | Fixed bug in sqlalchemy.ext.serializer where a unique :class:`.BindParameter` object could conflict with itself if it were present in the mapping itself, as well as the filter condition of the query, as one side would be used against the non-deserialized version and the other side would use the deserialized version. Logic is added to :class:`.BindParameter` similar to its "clone" method which will uniquify the parameter name upon deserialize so that it doesn't conflict with its original. Fixes: #5086 Change-Id: Ie1edce137e92ac496c822831d038999be5d1fc2d
* Enable F821Mike Bayer2020-01-041-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | In Ia63a510f9c1d08b055eef62cf047f1f427f0450c we introduced "lambda combinations" which use a bit of function closure inspection in order to allow for testing combinations that make use of symbols that come from test fixtures, or from the test itself. Two problems. One is that we can't use F821 flake8 rule without either adding lots of noqas, skipping the file, or adding arguments to the lambdas themselves that are then populated, which makes for a very verbose system. The other is that the system is already verbose with all those lambdas and the magic in use is a non-explicit kind, hence F821 reminds us that if we can improve upon this, we should. So let's improve upon it by making it so that the "lambda" is just once and up front for the whole thing, and let it accept the arguments directly. This still requires magic, because these test cases need to resolve at test collection time, not test runtime. But we will instead substitute a namespace up front that can be coerced into its desired form within the tests. Additionally, there's a little bit of py2k compatible type annotations present; f821 is checking these, so we have to add those imports also using the TYPE_CHECKING boolean so they don't take place in py2k. Change-Id: Idb7e7a0c8af86d9ab133f548511306ef68cdba14
* happy new yearMike Bayer2020-01-011-1/+1
| | | | Change-Id: I08440dc25e40ea1ccea1778f6ee9e28a00808235
* Test for short term reference cycles and resolve as many as possibleMike Bayer2019-12-301-0/+6
| | | | | | | | Added test support and repaired a wide variety of unnecessary reference cycles created for short-lived objects, mostly in the area of ORM queries. Fixes: #5056 Change-Id: Ifd93856eba550483f95f9ae63d49f36ab068b85a
* Use expanding IN for all literal value IN expressionsMike Bayer2019-12-221-17/+27
| | | | | | | | | | | | | | | | | | | | | | | | The "expanding IN" feature, which generates IN expressions at query execution time which are based on the particular parameters associated with the statement execution, is now used for all IN expressions made against lists of literal values. This allows IN expressions to be fully cacheable independently of the list of values being passed, and also includes support for empty lists. For any scenario where the IN expression contains non-literal SQL expressions, the old behavior of pre-rendering for each position in the IN is maintained. The change also completes support for expanding IN with tuples, where previously type-specific bind processors weren't taking effect. As part of this change, a more explicit separation between "literal execute" and "post compile" bound parameters is being made; as the "ansi bind rules" feature is rendering bound parameters inline, as we now support "postcompile" generically, these should be used here, however we have to render literal values at execution time even for "expanding" parameters. new test fixtures etc. are added to assert everything goes to the right place. Fixes: #4645 Change-Id: Iaa2b7bfbfaaf5b80799ee17c9b8507293cba6ed1
* Ensure cache keys are hashable in the testMike Bayer2019-12-141-1/+9
| | | | Change-Id: I962ff15194e2416844086f03dddadb49f48a6c8d
* Traversal and clause generation performance improvementsMike Bayer2019-12-141-10/+37
| | | | | | | Added one traversal test, callcounts have been brought from 29754 to 5173 so far. Change-Id: I164e9831600709ee214c1379bb215fdad73b39aa
* Remove ORM elements from annotations at the schema level.Mike Bayer2019-11-261-3/+8
| | | | | | | | | | | | | | | | | Fixed issue where when constructing constraints from ORM-bound columns, primarily :class:`.ForeignKey` objects but also :class:`.UniqueConstraint`, :class:`.CheckConstraint` and others, the ORM-level :class:`.InstrumentedAttribute` is discarded entirely, and all ORM-level annotations from the columns are removed; this is so that the constraints are still fully pickleable without the ORM-level entities being pulled in. These annotations are not necessary to be present at the schema/metadata level. Fully implemented coercions for constraint columns within schema.py, including for FK referenced columns. Fixes: #5001 Change-Id: I895400dd979310be034085d207f096707c635909
* Add anonymizing context to cache keys, comparison; convert traversalMike Bayer2019-11-041-329/+186
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Created new visitor system called "internal traversal" that applies a data driven approach to the concept of a class that defines its own traversal steps, in contrast to the existing style of traversal now known as "external traversal" where the visitor class defines the traversal, i.e. the SQLCompiler. The internal traversal system now implements get_children(), _copy_internals(), compare() and _cache_key() for most Core elements. Core elements with special needs like Select still implement some of these methods directly however most of these methods are no longer explicitly implemented. The data-driven system is also applied to ORM elements that take part in SQL expressions so that these objects, like mappers, aliasedclass, query options, etc. can all participate in the cache key process. Still not considered is that this approach to defining traversibility will be used to create some kind of generic introspection system that works across Core / ORM. It's also not clear if real statement caching using the _cache_key() method is feasible, if it is shown that running _cache_key() is nearly as expensive as compiling in any case. Because it is data driven, it is more straightforward to optimize using inlined code, as is the case now, as well as potentially using C code to speed it up. In addition, the caching sytem now accommodates for anonymous name labels, which is essential so that constructs which have anonymous labels can be cacheable, that is, their position within a statement in relation to other anonymous names causes them to generate an integer counter relative to that construct which will be the same every time. Gathering of bound parameters from any cache key generation is also now required as there is no use case for a cache key that does not extract bound parameter values. Applies-to: #4639 Change-Id: I0660584def8627cad566719ee98d3be045db4b8d
* Merge "Use default repr() for quoted_name under python 3"mike bayer2019-10-241-4/+7
|\
| * Use default repr() for quoted_name under python 3Mike Bayer2019-10-241-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Changed the ``repr()`` of the :class:`.quoted_name` construct to use regular string repr() under Python 3, rather than running it through "backslashreplace" escaping, which can be misleading. Modified the approach of "name normalization" for the Oracle and Firebird dialects, which converts from the UPPERCASE-as-case-insensitive convention of these dialects into lowercase-as-case-insensitive for SQLAlchemy, to not automatically apply the :class:`.quoted_name` construct to a name that matches itself under upper or lower case conversion, as is the case for many non-european characters. All names used within metadata structures are converted to :class:`.quoted_name` objects in any case; the change here would only affect the output of some inspection functions. Moved name normalize to be under default dialect, added test coverage in test/sql/test_quote.py Fixes: #4931 Change-Id: Ic121b20e07249824710a54423e321d94a425362f
* | Document unique bound parameters for text()Mike Bayer2019-10-231-0/+29
|/ | | | | | | | Will backport the feature from part of 36e8fe4 / #4808 to 1.3. Fixes: #4933 Change-Id: Ide4069ff5cccd5ed83a5f314e5f21e51dfe08b7d
* Use separate label generator for column_label naming conventionMike Bayer2019-10-141-16/+21
| | | | | | | | | | | | | | | | | | | | Fixed bug where a table that would have a column label overlap with a plain column name, such as "foo.id AS foo_id" vs. "foo.foo_id", would prematurely generate the ``._label`` attribute for a column before this overlap could be detected due to the use of the ``index=True`` or ``unique=True`` flag on the column in conjunction with the default naming convention of ``"column_0_label"``. This would then lead to failures when ``._label`` were used later to generate a bound parameter name, in particular those used by the ORM when generating the WHERE clause for an UPDATE statement. The issue has been fixed by using an alternate ``._label`` accessor for DDL generation that does not affect the state of the :class:`.Column`. The accessor also bypasses the key-deduplication step as it is not necessary for DDL, the naming is now consistently ``"<tablename>_<columnname>"`` without any subsequent numeric symbols when used in DDL. Fixes: #4911 Change-Id: Iabf5fd3250738d800d6e41a2a3a27a7ce2405e7d
* create second level deduping when use_labels is turned onMike Bayer2019-10-071-1/+11
| | | | | | | | | | | | | | | | | | | | | As of #4753 we allow duplicate columns. This creates some new problems that there can be duplicate columns in a subquery which are then not addressible on the outside because they are ambiguous (Postgresql has this behavior at least). Additionally it creates situations where we are making an anon label of an anon label which is leaking into the query. New logic for generating anon labels handles this situation and also alters the .c collection of a subquery such that we are only getting the first column from the derived selectable that has that name, the subsequent ones have a new deduping label with two underscores and are not exposed in .c. The dedupe logic when rendering the columns will handle duplicate label names for different columns, vs. the same column repeated, as separate cases. Fixes: #4892 Change-Id: I929fbd8da14bcc239e0481c24bbd9b5ce826e8fa
* Deprecate textual column matching in RowMike Bayer2019-10-041-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Deprecate query.instances() without a context Deprecate string alias with contains_eager() Deprecated the behavior by which a :class:`.Column` can be used as the key in a result set row lookup, when that :class:`.Column` is not part of the SQL selectable that is being selected; that is, it is only matched on name. A deprecation warning is now emitted for this case. Various ORM use cases, such as those involving :func:`.text` constructs, have been improved so that this fallback logic is avoided in most cases. Calling the :meth:`.Query.instances` method without passing a :class:`.QueryContext` is deprecated. The original use case for this was that a :class:`.Query` could yield ORM objects when given only the entities to be selected as well as a DBAPI cursor object. However, for this to work correctly there is essential metadata that is passed from a SQLAlchemy :class:`.ResultProxy` that is derived from the mapped column expressions, which comes originally from the :class:`.QueryContext`. To retrieve ORM results from arbitrary SELECT statements, the :meth:`.Query.from_statement` method should be used. Note there is a small bump in test_zoomark because the column._label is being calculated for each of those columns within baseline_3_properties, as it is now part of the result map. This label can't be calculated when the column is attached to the table because it needs to have all the columns present to do this correctly. Another approach here would be to pre-load the _label before the test runs however the zoomark tests don't have an easy place for this to happen and it's not really worth it. Fixes: #4877 Fixes: #4719 Change-Id: I9bd29e72e6dce7c855651d69ba68d7383469acbc
* Add note on the use of tuple_() for the IN operator in the docsStepland2019-09-161-1/+2
| | | | | | | | Closes: #4861 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4861 Pull-request-sha: c7379d390752d0c10d6488872b163b06ee30d952 Change-Id: I223008f720fe64951e2a0bf95aab955ece22516b
* Strip special chars in anonymized bind namesMike Bayer2019-09-051-1/+7
| | | | | | | | | | | | | | | | | | | | | | Characters that interfere with "pyformat" or "named" formats in bound parameters, namely ``%, (, )`` and the space character, as well as a few other typically undesirable characters, are stripped early for a :func:`.bindparam` that is using an anonymized name, which is typically generated automatically from a named column which itself includes these characters in its name and does not use a ``.key``, so that they do not interfere either with the SQLAlchemy compiler's use of string formatting or with the driver-level parsing of the parameter, both of which could be demonstrated before the fix. The change only applies to anonymized parameter names that are generated and consumed internally, not end-user defined names, so the change should have no impact on any existing code. Applies in particular to the psycopg2 driver which does not otherwise quote special parameter names, but also strips leading underscores to suit Oracle (but not yet leading numbers, as some anon parameters are currently entirely numeric/underscore based); Oracle in any case continues to quote parameter names that include special characters. Fixes: #4837 Change-Id: I21cb654c3e4ef786114160b8b4295242720bf3f9
* Merge "Annotate session-bind-lookup entity in Query-produced selectables"mike bayer2019-08-301-32/+8
|\
| * Annotate session-bind-lookup entity in Query-produced selectablesMike Bayer2019-08-301-32/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | Added new entity-targeting capabilities to the :class:`.Query` object to help with the case where the :class:`.Session` is using a bind dictionary against mapped classes, rather than a single bind, and the :class:`.Query` is against a Core statement that was ultimately generated from a method such as :meth:`.Query.subquery`; a deep search is performed to locate any ORM entity related to the query in order to locate a mapper if one is not otherwise present. Fixes: #4829 Change-Id: I95cf325a5aba21baec4b313246c6f4d692284820
* | Render LIMIT/OFFSET conditions after compile on select dialectsMike Bayer2019-08-301-3/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added new "post compile parameters" feature. This feature allows a :func:`.bindparam` construct to have its value rendered into the SQL string before being passed to the DBAPI driver, but after the compilation step, using the "literal render" feature of the compiler. The immediate rationale for this feature is to support LIMIT/OFFSET schemes that don't work or perform well as bound parameters handled by the database driver, while still allowing for SQLAlchemy SQL constructs to be cacheable in their compiled form. The immediate targets for the new feature are the "TOP N" clause used by SQL Server (and Sybase) which does not support a bound parameter, as well as the "ROWNUM" and optional "FIRST_ROWS()" schemes used by the Oracle dialect, the former of which has been known to perform better without bound parameters and the latter of which does not support a bound parameter. The feature builds upon the mechanisms first developed to support "expanding" parameters for IN expressions. As part of this feature, the Oracle ``use_binds_for_limits`` feature is turned on unconditionally and this flag is now deprecated. - adds limited support for "unique" bound parameters within a text() construct. - adds an additional int() check within the literal render function of the Integer datatype and tests that non-int values raise ValueError. Fixes: #4808 Change-Id: Iace97d544d1a7351ee07db970c6bc06a19c712c6
* | Label simple column transformations as the column nameMike Bayer2019-08-281-3/+51
|/ | | | | | | | | | | | | | | | | | | | Additional logic has been added such that certain SQL expressions which typically wrap a single database column will use the name of that column as their "anonymous label" name within a SELECT statement, potentially making key-based lookups in result tuples more intutive. The primary example of this is that of a CAST expression, e.g. ``CAST(table.colname AS INTEGER)``, which will export its default name as "colname", rather than the usual "anon_1" label, that is, ``CAST(table.colname AS INTEGER) AS colname``. If the inner expression doesn't have a name, then the previous "anonymous label" logic is used. When using SELECT statements that make use of :meth:`.Select.apply_labels`, such as those emitted by the ORM, the labeling logic will produce ``<tablename>_<inner column name>`` in the same was as if the column were named alone. The logic applies right now to the :func:`.cast` and :func:`.type_coerce` constructs as well as some single-element boolean expressions. Fixes: #4449 Change-Id: Ie3b73470e3bea53f2386cd86514cdc556491564e
* Support tuple IN VALUES for SQLite, othersMike Bayer2019-07-191-6/+10
| | | | | | | | | | | | Added support for composite (tuple) IN operators with SQLite, by rendering the VALUES keyword for this backend. As other backends such as DB2 are known to use the same syntax, the syntax is enabled in the base compiler using a dialect-level flag ``tuple_in_values``. The change also includes support for "empty IN tuple" expressions for SQLite when using "in_()" between a tuple value and an empty set. Fixes: #4766 Change-Id: I416e1af29b31d78f9ae06ec3c3a48ef6d6e813f5
* self_group() for FunctionFilterMike Bayer2019-07-121-0/+6
| | | | | | | | | Fixed issue where the :class:`.array_agg` construct in combination with :meth:`.FunctionElement.filter` would not produce the correct operator precedence between the FILTER keyword and the array index operator. Fixes: #4760 Change-Id: Ic662cd3da3330554ec673bafd80495b3f1506098
* Allow duplicate columns in from clauses and selectablesMike Bayer2019-07-111-13/+15
| | | | | | | | | | | | | | | | | | The :func:`.select` construct and related constructs now allow for duplication of column labels and columns themselves in the columns clause, mirroring exactly how column expressions were passed in. This allows the tuples returned by an executed result to match what was SELECTed for in the first place, which is how the ORM :class:`.Query` works, so this establishes better cross-compatibility between the two constructs. Additionally, it allows column-positioning-sensitive structures such as UNIONs (i.e. :class:`.CompoundSelect`) to be more intuitively constructed in those cases where a particular column might appear in more than one place. To support this change, the :class:`.ColumnCollection` has been revised to support duplicate columns as well as to allow integer index access. Fixes: #4753 Change-Id: Ie09a8116f05c367995c1e43623c51e07971d3bf0
* SelectBase no longer a FromClauseMike Bayer2019-07-061-56/+53
| | | | | | | | | | | | | | | | | | | | As part of the SQLAlchemy 2.0 migration project, a conceptual change has been made to the role of the :class:`.SelectBase` class hierarchy, which is the root of all "SELECT" statement constructs, in that they no longer serve directly as FROM clauses, that is, they no longer subclass :class:`.FromClause`. For end users, the change mostly means that any placement of a :func:`.select` construct in the FROM clause of another :func:`.select` requires first that it be wrapped in a subquery first, which historically is through the use of the :meth:`.SelectBase.alias` method, and is now also available through the use of :meth:`.SelectBase.subquery`. This was usually a requirement in any case since several databases don't accept unnamed SELECT subqueries in their FROM clause in any case. See the documentation in this change for lots more detail. Fixes: #4617 Change-Id: I0f6174ee24b9a1a4529168e52e855e12abd60667
* Add tutorial section for cast(), type_coerce()Mike Bayer2019-07-041-2/+24
| | | | Change-Id: I49f635f0ad4d07abe8ef2681c9660ec7fcf5f99b
* Rework proxy_cache fix to restore performanceMike Bayer2019-07-021-2/+12
| | | | | | | Adjustment to the fix made in I7fb134cac3604f8fe62e220fb24a0945d0a1c56f. Fixes: #4747 Change-Id: I2f1010b0abc1faa892f5e346e58f9c4a3867622f
* Clear proxy_set cache when creating an annotated columnMike Bayer2019-07-021-6/+6
| | | | | | | | | | | | Fixed an unlikely issue where the "corresponding column" routine for unions and other :class:`.CompoundSelect` objects could return the wrong column in some overlapping column situtations, thus potentially impacting some ORM operations when set operations are in use, if the underlying :func:`.select` constructs were used previously in other similar kinds of routines, due to a cached value not being cleared. Fixes: #4747 Change-Id: I7fb134cac3604f8fe62e220fb24a0945d0a1c56f
* Turn off the is_literal flag when proxying literal_column() to LabelMike Bayer2019-06-151-5/+20
| | | | | | | | | | | | | | | | | | | Fixed a series of quoting issues which all stemmed from the concept of the :func:`.literal_column` construct, which when being "proxied" through a subquery to be referred towards by a label that matches its text, the label would not have quoting rules applied to it, even if the string in the :class:`.Label` were set up as a :class:`.quoted_name` construct. Not applying quoting to the text of the :class:`.Label` is a bug because this text is strictly a SQL identifier name and not a SQL expression, and the string should not have quotes embedded into it already unlike the :func:`.literal_column` which it may be applied towards. The existing behavior of a non-labeled :func:`.literal_column` being propagated as is on the outside of a subquery is maintained in order to help with manual quoting schemes, although it's not clear if valid SQL can be generated for such a construct in any case. Fixes: #4730 Change-Id: I300941f27872fc4298c74a1d1ed65aef1a5cdd82
* Implement new ClauseElement role and coercion systemMike Bayer2019-05-181-434/+244
| | | | | | | | | | | | | | | | | | | | A major refactoring of all the functions handle all detection of Core argument types as well as perform coercions into a new class hierarchy based on "roles", each of which identify a syntactical location within a SQL statement. In contrast to the ClauseElement hierarchy that identifies "what" each object is syntactically, the SQLRole hierarchy identifies the "where does it go" of each object syntactically. From this we define a consistent type checking and coercion system that establishes well defined behviors. This is a breakout of the patch that is reorganizing select() constructs to no longer be in the FromClause hierarchy. Also includes a rename of as_scalar() into scalar_subquery(); deprecates automatic coercion to scalar_subquery(). Partially-fixes: #4617 Change-Id: I26f1e78898693c6b99ef7ea2f4e7dfd0e8e1a1bd
* Add _cache_key implementation.Mike Bayer2019-04-291-0/+191
| | | | | | | This leverages the work started in #4336 to allow ClauseElement structures to be cachable based on structure, not just identity. Change-Id: Ia99ddeb5353496dd7d61243245685f02b98d8100
* Reimplement .compare() in terms of a visitorMike Bayer2019-04-291-118/+5
| | | | | | | | | | | | Reworked the :meth:`.ClauseElement.compare` methods in terms of a new visitor-based approach, and additionally added test coverage ensuring that all :class:`.ClauseElement` subclasses can be accurately compared against each other in terms of structure. Structural comparison capability is used to a small degree within the ORM currently, however it also may form the basis for new caching features. Fixes: #4336 Change-Id: I581b667d8e1642a6c27165cc9f4aded1c66effc6
* negate True/False separately from other elementsMike Bayer2019-04-181-7/+4
| | | | | | | | | Fixed issue where double negation of a boolean column wouldn't reset the "NOT" operator. Fixes: #4618 Change-Id: Ica280a0d6b5b0870aa2d05c4d059a1e559e6b12a (cherry picked from commit 18f25f50353d9736e6638266585b2cb3ef7b0ea4)
* Remove all remaining text() coercions and ensure identifiers are safeMike Bayer2019-02-061-28/+59
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fully removed the behavior of strings passed directly as components of a :func:`.select` or :class:`.Query` object being coerced to :func:`.text` constructs automatically; the warning that has been emitted is now an ArgumentError or in the case of order_by() / group_by() a CompileError. This has emitted a warning since version 1.0 however its presence continues to create concerns for the potential of mis-use of this behavior. Note that public CVEs have been posted for order_by() / group_by() which are resolved by this commit: CVE-2019-7164 CVE-2019-7548 Added "SQL phrase validation" to key DDL phrases that are accepted as plain strings, including :paramref:`.ForeignKeyConstraint.on_delete`, :paramref:`.ForeignKeyConstraint.on_update`, :paramref:`.ExcludeConstraint.using`, :paramref:`.ForeignKeyConstraint.initially`, for areas where a series of SQL keywords only are expected.Any non-space characters that suggest the phrase would need to be quoted will raise a :class:`.CompileError`. This change is related to the series of changes committed as part of :ticket:`4481`. Fixed issue where using an uppercase name for an index type (e.g. GIST, BTREE, etc. ) or an EXCLUDE constraint would treat it as an identifier to be quoted, rather than rendering it as is. The new behavior converts these types to lowercase and ensures they contain only valid SQL characters. Quoting is applied to :class:`.Function` names, those which are usually but not necessarily generated from the :attr:`.sql.func` construct, at compile time if they contain illegal characters, such as spaces or punctuation. The names are as before treated as case insensitive however, meaning if the names contain uppercase or mixed case characters, that alone does not trigger quoting. The case insensitivity is currently maintained for backwards compatibility. Fixes: #4481 Fixes: #4473 Fixes: #4467 Change-Id: Ib22a27d62930e24702e2f0f7c74a0473385a08eb
* Fix many spell glitchesLele Gaifax2019-01-251-1/+1
| | | | | | | | | | | | This affects mostly docstrings, except in orm/events.py::dispose_collection() where one parameter gets renamed: given that the method is empty, it seemed reasonable to me to fix that too. Closes: #4440 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4440 Pull-request-sha: 779ed75acb6142e1f1daac467b5b14134529bb4b Change-Id: Ic0553fe97853054b09c2453af76d96363de6eb0e
* Add deprecation warnings to all deprecated APIsMike Bayer2019-01-231-41/+53
| | | | | | | | | | | | | | | A large change throughout the library has ensured that all objects, parameters, and behaviors which have been noted as deprecated or legacy now emit ``DeprecationWarning`` warnings when invoked. As the Python 3 interpreter now defaults to displaying deprecation warnings, as well as that modern test suites based on tools like tox and pytest tend to display deprecation warnings, this change should make it easier to note what API features are obsolete. See the notes added to the changelog and migration notes for further details. Fixes: #4393 Change-Id: If0ea11a1fc24f9a8029352eeadfc49a7a54c0a1b
* Remove version directives for 0.6, 0.7, 0.8Mike Bayer2019-01-151-5/+0
| | | | | | | | | - fix a few "seealso"s - ComparableProprerty's "superseded in 0.7" becomes deprecated in 0.7 Backport to currently maintained doc versions 1.2, 1.1 Change-Id: Ib1fcb2df8673dbe5c4ffc47f3896a60d1dfcb4b2
* Merge "use ..deprecated directive w/ version in all cases"mike bayer2019-01-121-38/+26
|\
View Lorry Controller Status