summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/sql/compiler.py
Commit message (Collapse)AuthorAgeFilesLines
* Apply underscore naming to several more operatorsjonathan vanasco2020-10-301-16/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | The operator changes are: * `isfalse` is now `is_false` * `isnot_distinct_from` is now `is_not_distinct_from` * `istrue` is now `is_true` * `notbetween` is now `not_between` * `notcontains` is now `not_contains` * `notendswith` is now `not_endswith` * `notilike` is now `not_ilike` * `notlike` is now `not_like` * `notmatch` is now `not_match` * `notstartswith` is now `not_startswith` * `nullsfirst` is now `nulls_first` * `nullslast` is now `nulls_last` Because these are core operators, the internal migration strategy for this change is to support legacy terms for an extended period of time -- if not indefinitely -- but update all documentation, tutorials, and internal usage to the new terms. The new terms are used to define the functions, and the legacy terms have been deprecated into aliases of the new terms. Fixes: #5435 Change-Id: Ifbd7cb1cdda5981990243c4fc4b4ff467dc132ac
* Ensure no compiler visit method tries to access .statementMike Bayer2020-10-191-1/+42
| | | | | | | | | | | | Fixed structural compiler issue where some constructs such as MySQL / PostgreSQL "on conflict / on duplicate key" would rely upon the state of the :class:`_sql.Compiler` object being fixed against their statement as the top level statement, which would fail in cases where those statements are branched from a different context, such as a DDL construct linked to a SQL statement. Fixes: #5656 Change-Id: I568bf40adc7edcf72ea6c7fd6eb9d07790de189e
* Ensure escaping of percent signs in columns, parametersMike Bayer2020-10-171-6/+32
| | | | | | | | | | | | | | | | | | | | | | | | Improved support for column names that contain percent signs in the string, including repaired issues involving anoymous labels that also embedded a column name with a percent sign in it, as well as re-established support for bound parameter names with percent signs embedded on the psycopg2 dialect, using a late-escaping process similar to that used by the cx_Oracle dialect. * Added new constructor for _anonymous_label() that ensures incoming string tokens based on column or table names will have percent signs escaped; abstracts away the format of the label. * generalized cx_Oracle's quoted_bind_names facility into the compiler itself, and leveraged this for the psycopg2 dialect's issue with percent signs in names as well. the parameter substitution is now integrated with compiler.construct_parameters() as well as the recently reworked set_input_sizes(), reducing verbosity in the cx_Oracle dialect. Fixes: #5653 Change-Id: Ia2ad13ea68b4b0558d410026e5a33f5cb3fbab2c
* Genericize setinputsizes and support pyodbcMike Bayer2020-10-161-4/+4
| | | | | | | | | | | | Reworked the "setinputsizes()" set of dialect hooks to be correctly extensible for any arbirary DBAPI, by allowing dialects individual hooks that may invoke cursor.setinputsizes() in the appropriate style for that DBAPI. In particular this is intended to support pyodbc's style of usage which is fundamentally different from that of cx_Oracle. Added support for pyodbc. Fixes: #5649 Change-Id: I9f1794f8368bf3663a286932cfe3992dae244a10
* Fetch first supportFederico Caselli2020-10-021-10/+26
| | | | | | | | | Add support to ``FETCH {FIRST | NEXT} [ count ] {ROW | ROWS} {ONLY | WITH TIES}`` in the select for the supported backends, currently PostgreSQL, Oracle and MSSQL. Fixes: #5576 Change-Id: Ibb5871a457c0555f82b37e354e7787d15575f1f7
* Rename Core expression isnot, not_in_jonathan vanasco2020-09-141-2/+2
| | | | | | | | | | | | | | | | | | | Several operators are renamed to achieve more consistent naming across SQLAlchemy. The operator changes are: * `isnot` is now `is_not` * `not_in_` is now `not_in` Because these are core operators, the internal migration strategy for this change is to support legacy terms for an extended period of time -- if not indefinitely -- but update all documentation, tutorials, and internal usage to the new terms. The new terms are used to define the functions, and the legacy terms have been deprecated into aliases of the new terms. Fixes: #5429 Change-Id: Ia1e66e7a50ac35d3f6260d8bf6ba3ce8087cbad2
* Add support for regular expression on supported backend.Federico Caselli2020-08-271-0/+32
| | | | | | | | | | | | Two operations have been defined: * :meth:`~.ColumnOperators.regexp_match` implementing a regular expression match like function. * :meth:`~.ColumnOperators.regexp_replace` implementing a regular expression string replace function. Fixes: #1390 Change-Id: I44556846e4668ccf329023613bd26861d5c674e6
* Updates for MariaDB sequencesFederico Caselli2020-08-221-0/+4
| | | | | | | | | | | | MariaDB should not run a Sequence if it has optional=True. Additionally, rework the rules in crud.py to accommodate the new combination MariaDB brings us, which is a dialect that supports both cursor.lastrowid, explicit sequences, *and* no support for returning. Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Fixes: #5528 Change-Id: I9a8ea69a34983affa95dfd22186e2908fdf0d58c
* Add support for identity columnsFederico Caselli2020-08-191-17/+37
| | | | | | | | | | | | | | Added the :class:`_schema.Identity` construct that can be used to configure identity columns rendered with GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY. Currently the supported backends are PostgreSQL >= 10, Oracle >= 12 and MSSQL (with different syntax and a subset of functionalities). Fixes: #5362 Fixes: #5324 Fixes: #5360 Change-Id: Iecea6f3ceb36821e8b96f0b61049b580507a1875
* Create a real type for Tuple() and handle appropriately in compilerMike Bayer2020-08-171-19/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | Improved the :func:`_sql.tuple_` construct such that it behaves predictably when used in a columns-clause context. The SQL tuple is not supported as a "SELECT" columns clause element on most backends; on those that do (PostgreSQL, not surprisingly), the Python DBAPI does not have a "nested type" concept so there are still challenges in fetching rows for such an object. Use of :func:`_sql.tuple_` in a :func:`_sql.select` or :class:`_orm.Query` will now raise a :class:`_exc.CompileError` at the point at which the :func:`_sql.tuple_` object is seen as presenting itself for fetching rows (i.e., if the tuple is in the columns clause of a subquery, no error is raised). For ORM use,the :class:`_orm.Bundle` object is an explicit directive that a series of columns should be returned as a sub-tuple per row and is suggested by the error message. Additionally ,the tuple will now render with parenthesis in all contexts. Previously, the parenthesization would not render in a columns context leading to non-defined behavior. As part of this change, Tuple receives a dedicated datatype which appears to allow us the very desirable change of removing the bindparam._expanding_in_types attribute as well as ClauseList._tuple_values (which might already have not been needed due to #4645). Fixes: #5127 Change-Id: Iecafa0e0aac2f1f37ec8d0e1631d562611c90200
* render INSERT/UPDATE column expressions up front; pass stateMike Bayer2020-08-081-12/+7
| | | | | | | | | | | | | | | | | | | | | | Fixes related to rendering of complex UPDATE DML which was not correctly preserving positional parameter order in conjunction with DML features that are only known to work on the PostgreSQL database. Both pg8000 and asyncpg use positional parameters which is why these issues are suddenly apparent. crud.py now takes on the task of rendering the column expressions for SET or VALUES so that for the very unusual case that the column expression is a compound expression that includes a bound parameter (namely an array index), the bound parameter order is preserved. Additionally, crud.py passes through the positional_names keyword argument into bindparam_string() which is necessary when CTEs are being rendered, as PG supports complex CTE / INSERT / UPDATE scenarios. Change-Id: I7f03920500e19b721636b84594de78a5bfdcbc82
* Pass schema_translate_map from DDLCompiler to SQLCompilerMike Bayer2020-08-071-1/+3
| | | | | | | | | | | | Fixed issue where the :paramref:`_engine.Connection.execution_options.schema_translate_map` feature would not take effect when the :meth:`_schema.Sequence.next_value` function function for a :class:`_schema.Sequence` were used in the :paramref:`_schema.Column.server_default` parameter and the create table DDL were emitted. Fixes: #5500 Change-Id: I74a9fa13d22749d06c8202669f9ea220d9d984d9
* Implement relationship AND criteria; global loader criteriaMike Bayer2020-08-051-0/+4
| | | | | | | | | | | | | | | | | | | Added the ability to add arbitrary criteria to the ON clause generated by a relationship attribute in a query, which applies to methods such as :meth:`_query.Query.join` as well as loader options like :func:`_orm.joinedload`. Additionally, a "global" version of the option allows limiting criteria to be applied to particular entities in a query globally. Documentation is minimal at this point, new examples will be coming in a subsequent commit. Some adjustments to execution options in how they are represented in the ORMExecuteState as well as well as a few ORM tests that forgot to get merged in a preceding commit. Fixes: #4472 Change-Id: I2b8fc57092dedf35ebd16f6343ad0f0d7d332beb
* Convert lazy loader, selectinload, load_on_ident to lambda statementsMike Bayer2020-08-051-2/+10
| | | | | | | | | Building on newly robust lambdas in I29a513c98917b1d503abfdd61e6b6e8800851aa8, convert key loading off of the "baked" system so that baked is no longer used by the ORM. Change-Id: I3abfb45dd6e50f84f29d39434caa0b550ce27864
* Genericize str() for typesMike Bayer2020-08-011-1/+23
| | | | | | | | | | | | | | Remove lookup logic that attempts to locate a dialect for a type, just use StrSQLTypeCompiler. Cleaned up the internal ``str()`` for datatypes so that all types produce a string representation without any dialect present, including that it works for third-party dialect types without that dialect being present. The string representation defaults to being the UPPERCASE name of that type with nothing else. Fixes: #4262 Change-Id: I02149e8a1ba1e7336149e962939b07ae0df83c6b
* Revise setinputsizes approachMike Bayer2020-07-191-0/+63
| | | | | | | | | | | | | | | | | | | | | | | in order to support asyncpg as well as pg8000, we need to revise setinputsizes to work for more cases as well as adjust NativeForEmulated a bit to work more completely with the INTERVAL datatype. - put most of the setinputsizes work into the compiler where the computation can be cached. - support per-element setinputsizes for a tuple - adjust TypeDecorator so that _unwrapped_dialect_impl will honor a type that the dialect links to directly in it's adaption mapping. Decouble _unwrapped_dialect_impl from TypeDecorator._gen_dialect_impl() which has a different purpose. This allows setinputsizes to do the right thing with the INTERVAL datatype. - test cases for Oracle with Variant continue to work Change-Id: I9e1ea33aeca3b92b365daa4a356d778191070c03
* introduce deferred lambdasMike Bayer2020-07-031-9/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The coercions system allows us to add in lambdas as arguments to Core and ORM elements without changing them at all. By allowing the lambda to produce a deterministic cache key where we can also cheat and yank out literal parameters means we can move towards having 90% of "baked" functionality in a clearer way right in Core / ORM. As a second step, we can have whole statements inside the lambda, and can then add generation with __add__(), so then we have 100% of "baked" functionality with full support of ad-hoc literal values. Adds some more short_selects tests for the moment for comparison. Other tweaks inside cache key generation as we're trying to approach a certain level of performance such that we can remove the use of "baked" from the loader strategies. As we have not yet closed #4639, however the caching feature has been fully integrated as of b0cfa7379cf8513a821a3dbe3028c4965d9f85bd, we will also add complete caching documentation here and close that issue as well. Closes: #4639 Fixes: #5380 Change-Id: If91f61527236fd4d7ae3cad1f24c38be921c90ba
* Merge "Fix a wide variety of typos and broken links"mike bayer2020-06-261-2/+2
|\
| * Fix a wide variety of typos and broken linksaplatkouski2020-06-251-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | Note the PR has a few remaining doc linking issues listed in the comment that must be addressed separately. Signed-off-by: aplatkouski <5857672+aplatkouski@users.noreply.github.com> Closes: #5371 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5371 Pull-request-sha: 7e7d233cf3a0c66980c27db0fcdb3c7d93bc2510 Change-Id: I9c36e8d8804483950db4b42c38ee456e384c59e3
* | Merge "Default psycopg2 executemany mode to "values_only""mike bayer2020-06-261-10/+84
|\ \ | |/ |/|
| * Default psycopg2 executemany mode to "values_only"Mike Bayer2020-06-251-10/+84
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The psycopg2 dialect now defaults to using the very performant ``execute_values()`` psycopg2 extension for compiled INSERT statements, and also impements RETURNING support when this extension is used. This allows INSERT statements that even include an autoincremented SERIAL or IDENTITY value to run very fast while still being able to return the newly generated primary key values. The ORM will then integrate this new feature in a separate change. Implements RETURNING for insert with executemany Adds support to return_defaults() mode and inserted_primary_key to support mutiple INSERTed rows, via return_defauls_rows and inserted_primary_key_rows accessors. within default execution context, new cached compiler getters are used to fetch primary keys from rows inserted_primary_key now returns a plain tuple. this is not yet a row-like object however this can be added. Adds distinct "values_only" and "batch" modes, as "values" has a lot of benefits but "batch" breaks cursor.rowcount psycopg2 minimum version 2.7 so we can remove the large number of checks for very old versions of psycopg2 simplify tests to no longer distinguish between native and non-native json Fixes: #5401 Change-Id: Ic08fd3423d4c5d16ca50994460c0c234868bd61c
* | Use time.perf_counter() for cache time measurementMike Bayer2020-06-241-2/+1
|/ | | | | | See https://twitter.com/raymondh/status/1275937373080023040 Change-Id: Iaa0abb0c433ccedfbd88d00e3970120242ba379b
* Turn on caching everywhere, add loggingMike Bayer2020-06-101-25/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A variety of caching issues found by running all tests with statement caching turned on. The cache system now has a more conservative approach where any subclass of a SQL element will by default invalidate the cache key unless it adds the flag inherit_cache=True at the class level, or if it implements its own caching. Add working caching to a few elements that were omitted previously; fix some caching implementations to suit lesser used edge cases such as json casts and array slices. Refine the way BaseCursorResult and CursorMetaData interact with caching; to suit cases like Alembic modifying table structures, don't cache the cursor metadata if it were created against a cursor.description using non-positional matching, e.g. "select *". if a table re-ordered its columns or added/removed, now that data is obsolete. Additionally we have to adapt the cursor metadata _keymap regardless of if we just processed cursor.description, because if we ran against a cached SQLCompiler we won't have the right columns in _keymap. Other refinements to how and when we do this adaption as some weird cases were exposed in the Postgresql dialect, a text() construct that names just one column that is not actually in the statement. Fixed that also as it looks like a cut-and-paste artifact that doesn't actually affect anything. Various issues with re-use of compiled result maps and cursor metadata in conjunction with tables being changed, such as change in order of columns. mappers can be cleared but the class remains, meaning a mapper has to use itself as the cache key not the class. lots of bound parameter / literal issues, due to Alembic creating a straight subclass of bindparam that renders inline directly. While we can update Alembic to not do this, we have to assume other people might be doing this, so bindparam() implements the inherit_cache=True logic as well that was a bit involved. turn on cache stats in logging. Includes a fix to subqueryloader which moves all setup to the create_row_processor() phase and elminates any storage within the compiled context. This includes some changes to create_row_processor() signature and a revising of the technique used to determine if the loader can participate in polymorphic queries, which is also applied to selectinloading. DML update.values() and ordered_values() now coerces the keys as we have tests that pass an arbitrary class here which only includes __clause_element__(), so the key can't be cached unless it is coerced. this in turn changed how composite attributes support bulk update to use the standard approach of ClauseElement with annotations that are parsed in the ORM context. memory profiling successfully caught that the Session from Query was getting passed into _statement_20() so that was a big win for that test suite. Apparently Compiler had .execute() and .scalar() methods stuck on it, these date back to version 0.4 and there was a single test in the PostgreSQL dialect tests that exercised it for no apparent reason. Removed these methods as well as the concept of a Compiler holding onto a "bind". Fixes: #5386 Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
* Convert bulk update/delete to new execution modelMike Bayer2020-06-061-0/+4
| | | | | | | | | | | | | | | This reorganizes the BulkUD model in sqlalchemy.orm.persistence to be based on the CompileState concept and to allow plain update() / delete() to be passed to session.execute() where the ORM synchronize session logic will take place. Also gets "synchronize_session='fetch'" working with horizontal sharding. Adding a few more result.scalar_one() types of methods as scalar_one() seems like what is normally desired. Fixes: #5160 Change-Id: I8001ebdad089da34119eb459709731ba6c0ba975
* Inline a few ORM arguments, othersMike Bayer2020-06-031-4/+3
| | | | | | small changes Change-Id: Id89a0651196c431d0aaf6935f5a4e7b12dd70c6c
* Add support for "real" sequences in mssqlGord Thompson2020-05-291-3/+6
| | | | | | | | | | | | | | | | | Added support for "CREATE SEQUENCE" and full :class:`.Sequence` support for Microsoft SQL Server. This removes the deprecated feature of using :class:`.Sequence` objects to manipulate IDENTITY characteristics which should now be performed using ``mssql_identity_start`` and ``mssql_identity_increment`` as documented at :ref:`mssql_identity`. The change includes a new parameter :paramref:`.Sequence.data_type` to accommodate SQL Server's choice of datatype, which for that backend includes INTEGER and BIGINT. The default starting value for SQL Server's version of :class:`.Sequence` has been set at 1; this default is now emitted within the CREATE SEQUENCE DDL for all backends. Fixes: #4235 Fixes: #4633 Change-Id: I6aa55c441e8146c2f002e2e201a7f645e667b916
* Render table hints in generic SQLMike Bayer2020-05-271-0/+3
| | | | | | | | | | | | Added :meth:`.Select.with_hint` output to the generic SQL string that is produced when calling ``str()`` on a statement. Previously, this clause would be omitted under the assumption that it was dialect specific. The hint text is presented within brackets to indicate the rendering of such hints varies among backends. Fixes: #5353 References: #4667 Change-Id: I01d97d6baa993e495519036ec7ecd5ae62856c16
* Convert execution to move through SessionMike Bayer2020-05-251-34/+92
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the ORM execution flow with a single pathway through Session.execute() for all queries, including Core and ORM. Currently included is full support for ORM Query, Query.from_statement(), select(), as well as the baked query and horizontal shard systems. Initial changes have also been made to the dogpile caching example, which like baked query makes use of a new ORM-specific execution hook that replaces the use of both QueryEvents.before_compile() as well as Query._execute_and_instances() as the central ORM interception hooks. select() and Query() constructs alike can be passed to Session.execute() where they will return ORM results in a Results object. This API is currently used internally by Query. Full support for Session.execute()->results to behave in a fully 2.0 fashion will be in later changesets. bulk update/delete with ORM support will also be delivered via the update() and delete() constructs, however these have not yet been adapted to the new system and may follow in a subsequent update. Performance is also beginning to lag as of this commit and some previous ones. It is hoped that a few central functions such as the coercions functions can be rewritten in C to re-gain performance. Additionally, query caching is now available and some subsequent patches will attempt to cache more of the per-execution work from the ORM layer, e.g. column getters and adapters. This patch also contains initial "turn on" of the caching system enginewide via the query_cache_size parameter to create_engine(). Still defaulting at zero for "no caching". The caching system still needs adjustments in order to gain adequate performance. Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
* Unify Query and select() , move all processing to compile phaseMike Bayer2020-05-241-0/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | Convert Query to do virtually all compile state computation in the _compile_context() phase, and organize it all such that a plain select() construct may also be used as the source of information in order to generate ORM query state. This makes it such that Query is not needed except for its additional methods like from_self() which are all to be deprecated. The construction of ORM state will occur beyond the caching boundary when the new execution model is integrated. future select() gains a working join() and filter_by() method. as we continue to rebase and merge each commit in the steps, callcounts continue to bump around. will have to look at the final result when it's all in. References: #5159 References: #4705 References: #4639 References: #4871 References: #5010 Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
* Performance fixes for new result setMike Bayer2020-05-211-10/+9
| | | | | | | | | | | A few small mistakes led to huge callcounts. Additionally, the warn-on-get behavior which is attempting to warn for deprecated access in SQLAlchemy 2.0 is very expensive; it's not clear if its feasible to have this warning or to somehow alter how it works. Fixes: #5340 Change-Id: I73bdd2d7b6f1b25cc0222accabd585cf761a5af4
* Propose Result as immediate replacement for ResultProxyMike Bayer2020-05-011-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As progress is made on the _future.Result, including breaking it out such that DBAPI behaviors are local to specific implementations, it becomes apparent that the Result object is a functional superset of ResultProxy and that basic operations like fetchone(), fetchall(), and fetchmany() behave pretty much exactly the same way on the new object. Reorganize things so that ResultProxy is now referred to as LegacyCursorResult, which subclasses CursorResult that represents the DBAPI-cursor version of Result, making use of a multiple inheritance pattern so that the functionality of Result is also available in non-DBAPI contexts, as will be necessary for some ORM patterns. Additionally propose the composition system for Result that will form the basis for ORM-alternative result systems such as horizontal sharding and dogpile cache. As ORM results will soon be coming directly from instances of Result, these extensions will instead build their own ResultFetchStrategies that perform the special steps to create composed or cached result sets. Also considering at the moment not emitting deprecation warnings for fetchXYZ() methods; the immediate issue is Keystone tests are calling upon it, but as the implementations here are proving to be not in any kind of conflict with how Result works, there's not too much issue leaving them around and deprecating at some later point. References: #5087 References: #4395 Fixes: #4959 Change-Id: I8091919d45421e3f53029b8660427f844fee0228
* Deprecate ``DISTINCT ON`` when not targeting PostgreSQLFederico Caselli2020-04-201-1/+10
| | | | | | | | | Deprecate usage of ``DISTINCT ON`` in dialect other than PostgreSQL. Previously this was silently ignored. Deprecate old usage of string distinct in MySQL dialect Fixes: #4002 Change-Id: I38fc64aef75e77748083c11d388ec831f161c9c9
* Create initial 2.0 engine implementationMike Bayer2020-04-161-2/+6
| | | | | | | | | | | | | | | | | | | Implemented the SQLAlchemy 2 :func:`.future.create_engine` function which is used for forwards compatibility with SQLAlchemy 2. This engine features always-transactional behavior with autobegin. Allow execution options per statement execution. This includes that the before_execute() and after_execute() events now accept an additional dictionary with these options, empty if not passed; a legacy event decorator is added for backwards compatibility which now also emits a deprecation warning. Add some basic tests for execution, transactions, and the new result object. Build out on a new testing fixture that swaps in the future engine completely to start with. Change-Id: I70e7338bb3f0ce22d2f702537d94bb249bd9fb0a Fixes: #4644
* Run search and replace of symbolic module namesMike Bayer2020-04-141-7/+10
| | | | | | | | Replaces a wide array of Sphinx-relative doc references with an abbreviated absolute form now supported by zzzeeksphinx. Change-Id: I94bffcc3f37885ffdde6238767224296339698a2
* Remove code deprecated before version 1.1Federico Caselli2020-04-091-14/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Remove deprecated method ``get_primary_keys` in the :class:`.Dialect` and :class:`.Inspector` classes. - Remove deprecated event ``dbapi_error`` and the method ``ConnectionEvents.dbapi_error`. - Remove support for deprecated engine URLs of the form ``postgres://``. - Remove deprecated dialect ``mysql+gaerdbms``. - Remove deprecated parameter ``quoting`` from :class:`.mysql.ENUM` and :class:`.mysql.SET` in the ``mysql`` dialect. - Remove deprecated function ``comparable_property``. and function ``comparable_using`` in the declarative extension. - Remove deprecated function ``compile_mappers``. - Remove deprecated method ``collection.linker``. - Remove deprecated method ``Session.prune`` and parameter ``Session.weak_identity_map``. This change also removes the class ``StrongInstanceDict``. - Remove deprecated parameter ``mapper.order_by``. - Remove deprecated parameter ``Session._enable_transaction_accounting`. - Remove deprecated parameter ``Session.is_modified.passive``. - Remove deprecated class ``Binary``. Please use :class:`.LargeBinary`. - Remove deprecated methods ``Compiled.compile``, ``ClauseElement.__and__`` and ``ClauseElement.__or__`` and attribute ``Over.func``. - Remove deprecated ``FromClause.count`` method. - Remove deprecated parameter ``Table.useexisting``. - Remove deprecated parameters ``text.bindparams`` and ``text.typemap``. - Remove boolean support for the ``passive`` parameter in ``get_history``. - Remove deprecated ``adapt_operator`` in ``UserDefinedType.Comparator``. Fixes: #4643 Change-Id: Idcd390c77bf7b0e9957907716993bdaa3f1a1763
* Repair caching / traversals for valuesMike Bayer2020-04-011-2/+7
| | | | | | | | | | | | | | The test suite wasn't running the copy_internals most fixtures, enable that and try to get all cases working. Set up selectable.values to do tuple conversion within compilation step. at the same time, disable caching for selectable.values for the moment and make it equivalent to dml_multi_values. fix cache / compare / copy cases for dml_values and dml_multi_values which weren't fully tested or covered. Change-Id: I484ca6e9cb2b66c2e6a321698f2abc0838db1460
* Try to measure new style caching in the ORM, take twoMike Bayer2020-04-011-10/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf Trying to see if by making the cache key memoized, we still can have the older "identity" form of caching which is the cheapest of all, at the same time as the newer "cache key each time" version that is not nearly as cheap; but still much cheaper than no caching at all. Also needed is a per-execution update of _keymap when we invoke from a cached select, so that Column objects that are anonymous or otherwise adapted will match up. this is analogous to the adaption of bound parameters from the cache key. Adds test coverage for the keymap / construct_params() changes related to caching. Also hones performance to a large extent for statement construction and cache key generation. Also includes a new memoized attribute approach that vastly simplifies the previous approach of "group_expirable_memoized_property" and finally integrates cleanly with _clone(), _generate(), etc. no more hardcoding of attributes is needed, as well as that most _reset_memoization() calls are no longer needed as the reset is inherent in a _generate() call; this also has dramatic performance improvements. Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
* Add a third labeling mode for SELECT statementsMike Bayer2020-03-291-21/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Enhanced the disambiguating labels feature of the :func:`~.sql.expression.select` construct such that when a select statement is used in a subquery, repeated column names from different tables are now automatically labeled with a unique label name, without the need to use the full "apply_labels()" feature that conbines tablename plus column name. The disambigated labels are available as plain string keys in the .c collection of the subquery, and most importantly the feature allows an ORM :func:`.orm.aliased` construct against the combination of an entity and an arbitrary subquery to work correctly, targeting the correct columns despite same-named columns in the source tables, without the need for an "apply labels" warning. The existing labeling style is now called LABEL_STYLE_TABLENAME_PLUS_COL. This labeling style will remain used throughout the ORM as has been the case for over a decade, however, the new disambiguation scheme could theoretically replace this scheme entirely. The new scheme would dramatically alter how SQL looks when rendered from the ORM to be more succinct but arguably harder to read. The tablename_columnname scheme used by Join.c is unaffected here, as that's still hardcoded to that scheme. Fixes: #5221 Change-Id: Ib47d9e0f35046b3afc77bef6e65709b93d0c3026
* Correct ambiguous func / class linksMike Bayer2020-03-251-5/+6
| | | | | | | | | :func:`.sql.expression.select`, :func:`.sql.expression.insert` and :class:`.sql.expression.Insert` were hitting many ambiguous symbol errors, due to future.select, as well as the PG/MySQL variants of Insert. Change-Id: Iac862bfc172a7f7f0cbba5353a83dc203bed376c
* Merge "Implement SQL VALUES in core."mike bayer2020-03-241-0/+40
|\
| * Implement SQL VALUES in core.Gord Thompson2020-03-241-0/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | Added a core :class:`Values` object that enables a VALUES construct to be used in the FROM clause of an SQL statement for databases that support it (mainly PostgreSQL and SQL Server). Fixes: #4868 Closes: #5030 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5030 Pull-request-sha: 84684038a8efa93b460318e0db53f6c644554588 Change-Id: Ib8109b63bc1a9dc04ab987c5322ca3375f7e824d
* | Convert schema_translate to a post compileMike Bayer2020-03-241-25/+75
|/ | | | | | | | | | | | | | | | Revised the :paramref:`.Connection.execution_options.schema_translate_map` feature such that the processing of the SQL statement to receive a specific schema name occurs within the execution phase of the statement, rather than at the compile phase. This is to support the statement being efficiently cached. Previously, the current schema being rendered into the statement for a particular run would be considered as part of the cache key itself, meaning that for a run against hundreds of schemas, there would be hundreds of cache keys, rendering the cache much less performant. The new behavior is that the rendering is done in a similar manner as the "post compile" rendering added in 1.4 as part of :ticket:`4645`, :ticket:`4808`. Fixes: #5004 Change-Id: Ia5c89eb27cc8dc2c5b8e76d6c07c46290a7901b6
* Merge "Rework select(), CompoundSelect() in terms of CompileState"mike bayer2020-03-111-52/+131
|\
| * Rework select(), CompoundSelect() in terms of CompileStateMike Bayer2020-03-101-52/+131
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Continuation of I408e0b8be91fddd77cf279da97f55020871f75a9 - add an options() method to the base Generative construct. this will be where ORM options can go - Change Null, False_, True_ to be singletons, so that we aren't instantiating them and having to use isinstance. The previous issue with this was that they would produce dupe labels in SELECT statements. Apply the duplicate column logic, newly added in 1.4, to these objects as well as to non-apply-labels SELECT statements in general as a means of improving this. - create a revised system for generating ClauseList compilation constructs that simplfies up front creation to not actually use ClauseList; a simple tuple is rendered by the compiler using the same constrcution rules as what are used for ClauseList but without creating the actual object. Apply to Select, CompoundSelect, revise Update, Delete - Select, CompoundSelect get an initial CompileState implementation. All methods used only within compilation are moved here - refine update/insert/delete compile state to not require an outside boolean - refine and simplify Select._copy_internals - rework bind(), which is going away, to not use some of the internal traversal stuff - remove "autocommit", "for_update" parameters from Select, references #4643 - remove "autocommit" parameter from TextClause , references #4643 - add deprecation warnings for statement.execute(), engine.execute(), statement.scalar(), engine.scalar(). Fixes: #5193 Change-Id: I04ca0152b046fd42c5054ba10f37e43fc6e5a57b
* | Simplified module pre-loading strategy and made it linter friendlyFederico Caselli2020-03-071-4/+7
|/ | | | | | | | | | | | | | | | | Introduced a modules registry to register modules that should be lazily loaded in the package init. This ensures that they are in the system module cache, avoiding potential thread safety issues as when importing them directly in the function that uses them. The module registry is used to obtain these modules directly, ensuring that the all the lazily loaded modules are resolved at the proper time This replaces dependency_for decorator and the dependencies decorator logic, removing the need to pass the resolved modules as arguments of the decodated functions and removes possible errors caused by linters. Fixes: #4689 Fixes: #4656 Change-Id: I2e291eba4297867fc0ddb5d875b9f7af34751d01
* Decouple compiler state from DML objects; make cacheableMike Bayer2020-03-061-25/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Targeting select / insert / update / delete, the goal is to minimize overhead of construction and generative methods so that only the raw arguments passed are handled. An interim stage that converts the raw state into more compiler-ready state is added, which is analogous to the ORM QueryContext which will also be rolled in to be a similar concept, as is currently being prototyped in I19e05b3424b07114cce6c439b05198ac47f7ac10. the ORM update/delete BulkUD concept is also going to be rolled onto this idea. So while the compiler-ready state object, here called DMLState, looks a little thin, it's the base of a bigger pattern that will allow for ORM functionality to embed itself directly into the compiler, execution context, and result set objects. This change targets the DML objects, primarily focused on the values() method which is the most complex process. The work done by values() is minimized as much as possible while still being able to create a cache key. Additional computation is then offloaded to a new object ValuesState that is handled by the compiler. Architecturally, a big change here is that insert.values() and update.values() will generate BindParameter objects for the values now, which are then carefully received by crud.py so that they generate the expected names. This is so that the values() portion of these constructs is cacheable. for the "multi-values" version of Insert, this is all skipped and the plan right now is that a multi-values insert is not worth caching (can always be revisited). Using the coercions system in values() also gets us nicer validation for free, we can remove the NotAClauseElement thing from schema, and we also now require scalar_subquery() is called for an insert/update that uses a SELECT as a column value, 1.x deprecation path is added. The traversal system is then applied to the DML objects including tests so that they have traversal, cloning, and cache key support. cloning is not a use case for DML however having it present allows better validation of the structure within the tests. Special per-dialect DML is explicitly not cacheable at the moment, more as a proof of concept that third party DML constructs can exist as gracefully not-cacheable rather than producing an incomplete cache key. A few selected performance improvements have been added as well, simplifying the immutabledict.union() method and adding a new SQLCompiler function that can generate delimeter-separated clauses like WHERE and ORDER BY without having to build a ClauseList object at all. The use of ClauseList will be removed from Select in an upcoming commit. Overall, ClaustList is unnecessary for internal use and only adds overhead to statement construction and will likely be removed as much as possible except for explcit use of conjunctions like and_() and or_(). Change-Id: I408e0b8be91fddd77cf279da97f55020871f75a9
* Restore crud flags if visiting_cte is setMike Bayer2020-03-031-0/+1
| | | | | | | | | | Fixed bug where a CTE of an INSERT/UPDATE/DELETE that also uses RETURNING could then not be SELECTed from directly, as the internal state of the compiler would try to treat the outer SELECT as a DELETE statement itself and access nonexistent state. Fixes: #5181 Change-Id: Icba76f2148c8344baa1c04bac4ab6c6d24f23072
* Ensure all nested exception throws have a causeMike Bayer2020-03-021-5/+10
| | | | | | | | | | | | | | | Applied an explicit "cause" to most if not all internally raised exceptions that are raised from within an internal exception catch, to avoid misleading stacktraces that suggest an error within the handling of an exception. While it would be preferable to suppress the internally caught exception in the way that the ``__suppress_context__`` attribute would, there does not as yet seem to be a way to do this without suppressing an enclosing user constructed context, so for now it exposes the internally caught exception as the cause so that full information about the context of the error is maintained. Fixes: #4849 Change-Id: I55a86b29023675d9e5e49bc7edc5a2dc0bcd4751
* Discontinue dynamic __visit_name__Mike Bayer2020-03-011-0/+6
| | | | | | | | | Removed very antiquated logic that checks if __visit_name__ is a property. There's no need for this as the compiler can handle switching between implementations. Convert _compile_dispatch() to be fully inlined. Change-Id: Ic0c7247c2d7dfed93a27f09250a8ed6352370764
* Repair inline flagMike Bayer2020-02-221-1/+1
| | | | | | | | | | | | | | | In 9fca5d827d we attempted to deprecate the "inline=True" flag and add a generative inline() method, however failed to include any tests and the method was implemented incorrectly such that it would get overwritten with the boolean flag immediately. Rename the internal "inline" flag to "_inline" and add test support both for the method as well as deprecated support for the flag, including a fixture addition to assert the expected value of the flag as it generally does not affect the actual compiled SQL string. Change-Id: I0450049f17f1f0d91e22d27f1a973a2b6c0e59f7
View Lorry Controller Status