summaryrefslogtreecommitdiff
path: root/test/sql/test_compiler.py
Commit message (Collapse)AuthorAgeFilesLines
* Convert remaining ORM APIs to support 2.0 styleMike Bayer2020-07-111-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is kind of a mixed bag of all kinds to help get us to 1.4 betas. The documentation stuff is a work in progress. Lots of other relatively small changes to APIs and things. More commits will follow to continue improving the documentation and transitioning to the 1.4/2.0 hybrid documentation. In particular some refinements to Session usage models so that it can match Engine's scoping / transactional patterns, and a decision to start moving away from "subtransactions" completely. * add select().from_statement() to produce FromStatement in an ORM context * begin referring to select() that has "plugins" for the few edge cases where select() will have ORM-only behaviors * convert dynamic.AppenderQuery to its own object that can use select(), though at the moment it uses Query to support legacy join calling forms. * custom query classes for AppenderQuery are replaced by do_orm_execute() hooks for custom actions, a separate gerrit will document this * add Session.get() to replace query.get() * Deprecate session.begin->subtransaction. propose within the test suite a hypothetical recipe for apps that rely on this pattern * introduce Session construction level context manager, sessionmaker context manager, rewrite the whole top of the session_transaction.rst documentation. Establish context manager patterns for Session that are identical to engine * ensure same begin_nested() / commit() behavior as engine * devise all new "join into an external transaction" recipe, add test support for it, add rules into Session so it just works, write new docs. need to ensure this doesn't break anything * vastly reduce the verbosity of lots of session docs as I dont think people read this stuff and it's difficult to keep current in any case * constructs like case(), with_only_columns() really need to move to *columns, add a coercion rule to just change these. * docs need changes everywhere I look. in_() is not in the Core tutorial? how do people even know about it? Remove tons of cruft from Select docs, etc. * build a system for common ORM options like populate_existing and autoflush to populate from execution options. * others? Change-Id: Ia4bea0f804250e54d90b3884cf8aab8b66b82ecf
* Add future=True to create_engine/Session; unify select()Mike Bayer2020-07-081-573/+495
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several weeks of using the future_select() construct has led to the proposal there be just one select() construct again which features the new join() method, and otherwise accepts both the 1.x and 2.x argument styles. This would make migration simpler and reduce confusion. However, confusion may be increased by the fact that select().join() is different Current thinking is we may be better off with a few hard behavioral changes to old and relatively unknown APIs rather than trying to play both sides within two extremely similar but subtly different APIs. At the moment, the .join() thing seems to be the only behavioral change that occurs without the user taking any explicit steps. Session.execute() will still behave the old way as we are adding a future flag. This change also adds the "future" flag to Session() and session.execute(), so that interpretation of the incoming statement, as well as that the new style result is returned, does not occur for existing applications unless they add the use of this flag. The change in general is moving the "removed in 2.0" system further along where we want the test suite to fully pass even if the SQLALCHEMY_WARN_20 flag is set. Get many tests to pass when SQLALCHEMY_WARN_20 is set; this should be ongoing after this patch merges. Improve the RemovedIn20 warning; these are all deprecated "since" 1.4, so ensure that's what the messages read. Make sure the inforamtion link is on all warnings. Add deprecation warnings for parameters present and add warnings to all FromClause.select() types of methods. Fixes: #5379 Fixes: #5284 Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51 References: #5159
* Rename Table.tometadata to to_metadataGord Thompson2020-06-291-1/+1
| | | | | | | | | | | | | | | | Renamed the :meth:`_schema.Table.tometadata` method to :meth:`_schema.Table.to_metadata`. The previous name remains with a deprecation warning. Updated the "decorate" utility function to support decoration of functions that include non-builtins as default values. Moves test for deprecated "databases" package into test/dialect/test_deprecations.py Fixes: #5413 Fixes: #5426 Change-Id: I6ed899871c935f9e46360127c17ccb7cf97cea6e
* Turn on caching everywhere, add loggingMike Bayer2020-06-101-0/+137
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A variety of caching issues found by running all tests with statement caching turned on. The cache system now has a more conservative approach where any subclass of a SQL element will by default invalidate the cache key unless it adds the flag inherit_cache=True at the class level, or if it implements its own caching. Add working caching to a few elements that were omitted previously; fix some caching implementations to suit lesser used edge cases such as json casts and array slices. Refine the way BaseCursorResult and CursorMetaData interact with caching; to suit cases like Alembic modifying table structures, don't cache the cursor metadata if it were created against a cursor.description using non-positional matching, e.g. "select *". if a table re-ordered its columns or added/removed, now that data is obsolete. Additionally we have to adapt the cursor metadata _keymap regardless of if we just processed cursor.description, because if we ran against a cached SQLCompiler we won't have the right columns in _keymap. Other refinements to how and when we do this adaption as some weird cases were exposed in the Postgresql dialect, a text() construct that names just one column that is not actually in the statement. Fixed that also as it looks like a cut-and-paste artifact that doesn't actually affect anything. Various issues with re-use of compiled result maps and cursor metadata in conjunction with tables being changed, such as change in order of columns. mappers can be cleared but the class remains, meaning a mapper has to use itself as the cache key not the class. lots of bound parameter / literal issues, due to Alembic creating a straight subclass of bindparam that renders inline directly. While we can update Alembic to not do this, we have to assume other people might be doing this, so bindparam() implements the inherit_cache=True logic as well that was a bit involved. turn on cache stats in logging. Includes a fix to subqueryloader which moves all setup to the create_row_processor() phase and elminates any storage within the compiled context. This includes some changes to create_row_processor() signature and a revising of the technique used to determine if the loader can participate in polymorphic queries, which is also applied to selectinloading. DML update.values() and ordered_values() now coerces the keys as we have tests that pass an arbitrary class here which only includes __clause_element__(), so the key can't be cached unless it is coerced. this in turn changed how composite attributes support bulk update to use the standard approach of ClauseElement with annotations that are parsed in the ORM context. memory profiling successfully caught that the Session from Query was getting passed into _statement_20() so that was a big win for that test suite. Apparently Compiler had .execute() and .scalar() methods stuck on it, these date back to version 0.4 and there was a single test in the PostgreSQL dialect tests that exercised it for no apparent reason. Removed these methods as well as the concept of a Compiler holding onto a "bind". Fixes: #5386 Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
* Merge "Improve rendering of core statements w/ ORM elements"mike bayer2020-06-011-0/+24
|\
| * Improve rendering of core statements w/ ORM elementsMike Bayer2020-05-311-0/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch contains a variety of ORM and expression layer tweaks to support ORM constructs in select() statements, without the 1.3.x requiremnt in Query that a full _compile_context() + new select() is needed in order to get a working statement object. Includes such tweaks as the ability to implement aliased class of an aliased class, as we are looking to fully support ACs against subqueries, as well as the ability to access anonymously-labeled ColumnProperty expressions within subqueries by naming the ".key" of the label after the property key. Some tuning to query.join() as well as ORMJoin internals to allow things to work more smoothly. Change-Id: Id810f485c5f7ed971529489b84694e02a3356d6d
* | Add support for "real" sequences in mssqlGord Thompson2020-05-291-3/+3
|/ | | | | | | | | | | | | | | | | Added support for "CREATE SEQUENCE" and full :class:`.Sequence` support for Microsoft SQL Server. This removes the deprecated feature of using :class:`.Sequence` objects to manipulate IDENTITY characteristics which should now be performed using ``mssql_identity_start`` and ``mssql_identity_increment`` as documented at :ref:`mssql_identity`. The change includes a new parameter :paramref:`.Sequence.data_type` to accommodate SQL Server's choice of datatype, which for that backend includes INTEGER and BIGINT. The default starting value for SQL Server's version of :class:`.Sequence` has been set at 1; this default is now emitted within the CREATE SEQUENCE DDL for all backends. Fixes: #4235 Fixes: #4633 Change-Id: I6aa55c441e8146c2f002e2e201a7f645e667b916
* Render table hints in generic SQLMike Bayer2020-05-271-0/+35
| | | | | | | | | | | | Added :meth:`.Select.with_hint` output to the generic SQL string that is produced when calling ``str()`` on a statement. Previously, this clause would be omitted under the assumption that it was dialect specific. The hint text is presented within brackets to indicate the rendering of such hints varies among backends. Fixes: #5353 References: #4667 Change-Id: I01d97d6baa993e495519036ec7ecd5ae62856c16
* Convert execution to move through SessionMike Bayer2020-05-251-6/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the ORM execution flow with a single pathway through Session.execute() for all queries, including Core and ORM. Currently included is full support for ORM Query, Query.from_statement(), select(), as well as the baked query and horizontal shard systems. Initial changes have also been made to the dogpile caching example, which like baked query makes use of a new ORM-specific execution hook that replaces the use of both QueryEvents.before_compile() as well as Query._execute_and_instances() as the central ORM interception hooks. select() and Query() constructs alike can be passed to Session.execute() where they will return ORM results in a Results object. This API is currently used internally by Query. Full support for Session.execute()->results to behave in a fully 2.0 fashion will be in later changesets. bulk update/delete with ORM support will also be delivered via the update() and delete() constructs, however these have not yet been adapted to the new system and may follow in a subsequent update. Performance is also beginning to lag as of this commit and some previous ones. It is hoped that a few central functions such as the coercions functions can be rewritten in C to re-gain performance. Additionally, query caching is now available and some subsequent patches will attempt to cache more of the per-execution work from the ORM layer, e.g. column getters and adapters. This patch also contains initial "turn on" of the caching system enginewide via the query_cache_size parameter to create_engine(). Still defaulting at zero for "no caching". The caching system still needs adjustments in order to gain adequate performance. Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
* Unify Query and select() , move all processing to compile phaseMike Bayer2020-05-241-2/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | Convert Query to do virtually all compile state computation in the _compile_context() phase, and organize it all such that a plain select() construct may also be used as the source of information in order to generate ORM query state. This makes it such that Query is not needed except for its additional methods like from_self() which are all to be deprecated. The construction of ORM state will occur beyond the caching boundary when the new execution model is integrated. future select() gains a working join() and filter_by() method. as we continue to rebase and merge each commit in the steps, callcounts continue to bump around. will have to look at the final result when it's all in. References: #5159 References: #4705 References: #4639 References: #4871 References: #5010 Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
* Add 'schema' parameter to tableDylan Modesitt2020-05-101-0/+74
| | | | | | | | | | | | Added a "schema" parameter to the :func:`_expression.table` construct, allowing ad-hoc table expressions to also include a schema name. Pull request courtesy Dylan Modesitt. Fixes: #5309 Closes: #5310 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5310 Pull-request-sha: ce85681050500186678131f948b6ea277a65dc17 Change-Id: I32015d593e1ee1121c7426fbffdcc565d025fad1
* Propose Result as immediate replacement for ResultProxyMike Bayer2020-05-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As progress is made on the _future.Result, including breaking it out such that DBAPI behaviors are local to specific implementations, it becomes apparent that the Result object is a functional superset of ResultProxy and that basic operations like fetchone(), fetchall(), and fetchmany() behave pretty much exactly the same way on the new object. Reorganize things so that ResultProxy is now referred to as LegacyCursorResult, which subclasses CursorResult that represents the DBAPI-cursor version of Result, making use of a multiple inheritance pattern so that the functionality of Result is also available in non-DBAPI contexts, as will be necessary for some ORM patterns. Additionally propose the composition system for Result that will form the basis for ORM-alternative result systems such as horizontal sharding and dogpile cache. As ORM results will soon be coming directly from instances of Result, these extensions will instead build their own ResultFetchStrategies that perform the special steps to create composed or cached result sets. Also considering at the moment not emitting deprecation warnings for fetchXYZ() methods; the immediate issue is Keystone tests are calling upon it, but as the implementations here are proving to be not in any kind of conflict with how Result works, there's not too much issue leaving them around and deprecating at some later point. References: #5087 References: #4395 Fixes: #4959 Change-Id: I8091919d45421e3f53029b8660427f844fee0228
* Deprecate ``DISTINCT ON`` when not targeting PostgreSQLFederico Caselli2020-04-201-0/+7
| | | | | | | | | Deprecate usage of ``DISTINCT ON`` in dialect other than PostgreSQL. Previously this was silently ignored. Deprecate old usage of string distinct in MySQL dialect Fixes: #4002 Change-Id: I38fc64aef75e77748083c11d388ec831f161c9c9
* Try to measure new style caching in the ORM, take twoMike Bayer2020-04-011-1/+111
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf Trying to see if by making the cache key memoized, we still can have the older "identity" form of caching which is the cheapest of all, at the same time as the newer "cache key each time" version that is not nearly as cheap; but still much cheaper than no caching at all. Also needed is a per-execution update of _keymap when we invoke from a cached select, so that Column objects that are anonymous or otherwise adapted will match up. this is analogous to the adaption of bound parameters from the cache key. Adds test coverage for the keymap / construct_params() changes related to caching. Also hones performance to a large extent for statement construction and cache key generation. Also includes a new memoized attribute approach that vastly simplifies the previous approach of "group_expirable_memoized_property" and finally integrates cleanly with _clone(), _generate(), etc. no more hardcoding of attributes is needed, as well as that most _reset_memoization() calls are no longer needed as the reset is inherent in a _generate() call; this also has dramatic performance improvements. Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
* String compiler can now literal compile datetime objectsFederico Caselli2020-03-291-0/+16
| | | | | | | | | | Add ability to literal compile a :class:`DateTime`, :class:`Date` or :class:"Time" when using the string dialect for debugging purposes. This change does not impact real dialect implementation that retain their current behavior. Fixes: #5052 Change-Id: Ia3fad2be905c6d35b0106b9a2388c7508f067e90
* Convert schema_translate to a post compileMike Bayer2020-03-241-4/+45
| | | | | | | | | | | | | | | | Revised the :paramref:`.Connection.execution_options.schema_translate_map` feature such that the processing of the SQL statement to receive a specific schema name occurs within the execution phase of the statement, rather than at the compile phase. This is to support the statement being efficiently cached. Previously, the current schema being rendered into the statement for a particular run would be considered as part of the cache key itself, meaning that for a run against hundreds of schemas, there would be hundreds of cache keys, rendering the cache much less performant. The new behavior is that the rendering is done in a similar manner as the "post compile" rendering added in 1.4 as part of :ticket:`4645`, :ticket:`4808`. Fixes: #5004 Change-Id: Ia5c89eb27cc8dc2c5b8e76d6c07c46290a7901b6
* Rework select(), CompoundSelect() in terms of CompileStateMike Bayer2020-03-101-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Continuation of I408e0b8be91fddd77cf279da97f55020871f75a9 - add an options() method to the base Generative construct. this will be where ORM options can go - Change Null, False_, True_ to be singletons, so that we aren't instantiating them and having to use isinstance. The previous issue with this was that they would produce dupe labels in SELECT statements. Apply the duplicate column logic, newly added in 1.4, to these objects as well as to non-apply-labels SELECT statements in general as a means of improving this. - create a revised system for generating ClauseList compilation constructs that simplfies up front creation to not actually use ClauseList; a simple tuple is rendered by the compiler using the same constrcution rules as what are used for ClauseList but without creating the actual object. Apply to Select, CompoundSelect, revise Update, Delete - Select, CompoundSelect get an initial CompileState implementation. All methods used only within compilation are moved here - refine update/insert/delete compile state to not require an outside boolean - refine and simplify Select._copy_internals - rework bind(), which is going away, to not use some of the internal traversal stuff - remove "autocommit", "for_update" parameters from Select, references #4643 - remove "autocommit" parameter from TextClause , references #4643 - add deprecation warnings for statement.execute(), engine.execute(), statement.scalar(), engine.scalar(). Fixes: #5193 Change-Id: I04ca0152b046fd42c5054ba10f37e43fc6e5a57b
* Decouple compiler state from DML objects; make cacheableMike Bayer2020-03-061-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Targeting select / insert / update / delete, the goal is to minimize overhead of construction and generative methods so that only the raw arguments passed are handled. An interim stage that converts the raw state into more compiler-ready state is added, which is analogous to the ORM QueryContext which will also be rolled in to be a similar concept, as is currently being prototyped in I19e05b3424b07114cce6c439b05198ac47f7ac10. the ORM update/delete BulkUD concept is also going to be rolled onto this idea. So while the compiler-ready state object, here called DMLState, looks a little thin, it's the base of a bigger pattern that will allow for ORM functionality to embed itself directly into the compiler, execution context, and result set objects. This change targets the DML objects, primarily focused on the values() method which is the most complex process. The work done by values() is minimized as much as possible while still being able to create a cache key. Additional computation is then offloaded to a new object ValuesState that is handled by the compiler. Architecturally, a big change here is that insert.values() and update.values() will generate BindParameter objects for the values now, which are then carefully received by crud.py so that they generate the expected names. This is so that the values() portion of these constructs is cacheable. for the "multi-values" version of Insert, this is all skipped and the plan right now is that a multi-values insert is not worth caching (can always be revisited). Using the coercions system in values() also gets us nicer validation for free, we can remove the NotAClauseElement thing from schema, and we also now require scalar_subquery() is called for an insert/update that uses a SELECT as a column value, 1.x deprecation path is added. The traversal system is then applied to the DML objects including tests so that they have traversal, cloning, and cache key support. cloning is not a use case for DML however having it present allows better validation of the structure within the tests. Special per-dialect DML is explicitly not cacheable at the moment, more as a proof of concept that third party DML constructs can exist as gracefully not-cacheable rather than producing an incomplete cache key. A few selected performance improvements have been added as well, simplifying the immutabledict.union() method and adding a new SQLCompiler function that can generate delimeter-separated clauses like WHERE and ORDER BY without having to build a ClauseList object at all. The use of ClauseList will be removed from Select in an upcoming commit. Overall, ClaustList is unnecessary for internal use and only adds overhead to statement construction and will likely be removed as much as possible except for explcit use of conjunctions like and_() and or_(). Change-Id: I408e0b8be91fddd77cf279da97f55020871f75a9
* Discontinue dynamic __visit_name__Mike Bayer2020-03-011-8/+9
| | | | | | | | | Removed very antiquated logic that checks if __visit_name__ is a property. There's no need for this as the compiler can handle switching between implementations. Convert _compile_dispatch() to be fully inlined. Change-Id: Ic0c7247c2d7dfed93a27f09250a8ed6352370764
* Result initial introductionMike Bayer2020-02-211-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This builds on cc718cccc0bf8a01abdf4068c7ea4f3 which moved RowProxy to Row, allowing Row to be more like a named tuple. - KeyedTuple in ORM is replaced with Row - ResultSetMetaData broken out into "simple" and "cursor" versions for ORM and Core, as well as LegacyCursor version. - Row now has _mapping attribute that supplies full mapping behavior. Row and SimpleRow both have named tuple behavior otherwise. LegacyRow has some mapping features on the tuple which emit deprecation warnings (e.g. keys(), values(), etc). the biggest change for mapping->tuple is the behavior of __contains__ which moves from testing of "key in row" to "value in row". - ResultProxy breaks into ResultProxy and FutureResult (interim), the latter has the newer APIs. Made available to dialects using execution options. - internal reflection methods and most tests move off of implicit Row mapping behavior and move to row._mapping, result.mappings() method using future result - a new strategy system for cursor handling replaces the various subclasses of RowProxy - some execution context adjustments. We will leave EC in but refined things like get_result_proxy() and out parameter handling. Dialects for 1.4 will need to adjust from get_result_proxy() to get_result_cursor_strategy(), if they are using this method - out parameter handling now accommodated by get_out_parameter_values() EC method. Oracle changes for this. external dialect for DB2 for example will also need to adjust for this. - deprecate case_insensitive flag for engine / result, this feature is not used mapping-methods on Row are deprecated, and replaced with Row._mapping.<meth>, including: row.keys() -> use row._mapping.keys() row.items() -> use row._mapping.items() row.values() -> use row._mapping.values() key in row -> use key in row._mapping int in row -> use int < len(row) Fixes: #4710 Fixes: #4878 Change-Id: Ieb9085e9bcff564359095b754da9ae0af55679f0
* Create initial future package, RemovedIn20WarningMike Bayer2020-02-121-3/+3
| | | | | | | | | | | | | | | | | Reorganization of Select() is the first major element of the 2.0 restructuring. In order to start this we need to first create the new Select constructor and apply legacy elements to the old one. This in turn necessitates starting up the RemovedIn20Warning concept which itself need to refer to "sqlalchemy.future", so begin to establish this basic framework. Additionally, update the DML constructors with the newer no-keyword style. Remove the use of the "pending deprecation" and fix Query.add_column() deprecation which was not acting as deprecated. Fixes: #4845 Fixes: #4648 Change-Id: I0c7a22b2841a985e1c379a0bb6c94089aae6264c
* Deprecate empty or_() and and_()Federico Caselli2020-01-251-3/+19
| | | | | | | | | | | | | | | | | | | | | | | Creating an :func:`.and_` or :func:`.or_` construct with no arguments or empty ``*args`` will now emit a deprecation warning, as the SQL produced is a no-op (i.e. it renders as a blank string). This behavior is considered to be non-intuitive, so for empty or possibly empty :func:`.and_` or :func:`.or_` constructs, an appropriate default boolean should be included, such as ``and_(True, *args)`` or ``or_(False, *args)``. As has been the case for many major versions of SQLAlchemy, these particular boolean values will not render if the ``*args`` portion is non-empty. As there are some internal cases where an empty and_() construct is used in order to build an optional WHERE expression, a private utility function is added to suit this use case. Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Fixes: #5054 Closes: #5062 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5062 Pull-request-sha: 5ca2f27281977d74e390148c0fb8deaa0e0e4ad9 Change-Id: I599b9c8befa64d9a59a35ad7dd84ff400e3aa647
* Use expanding IN for all literal value IN expressionsMike Bayer2019-12-221-55/+151
| | | | | | | | | | | | | | | | | | | | | | | | The "expanding IN" feature, which generates IN expressions at query execution time which are based on the particular parameters associated with the statement execution, is now used for all IN expressions made against lists of literal values. This allows IN expressions to be fully cacheable independently of the list of values being passed, and also includes support for empty lists. For any scenario where the IN expression contains non-literal SQL expressions, the old behavior of pre-rendering for each position in the IN is maintained. The change also completes support for expanding IN with tuples, where previously type-specific bind processors weren't taking effect. As part of this change, a more explicit separation between "literal execute" and "post compile" bound parameters is being made; as the "ansi bind rules" feature is rendering bound parameters inline, as we now support "postcompile" generically, these should be used here, however we have to render literal values at execution time even for "expanding" parameters. new test fixtures etc. are added to assert everything goes to the right place. Fixes: #4645 Change-Id: Iaa2b7bfbfaaf5b80799ee17c9b8507293cba6ed1
* Do the CompoundSelect check for number of columns in the compile phaseMike Bayer2019-12-161-4/+2
| | | | | | | | Starting to go forward with the general idea of moving more of Core / ORM construction into the compile phase. Bigger initiatives like the refactor of Query will follow onto this. Change-Id: I0f364d3182e21e32ed85ef34cfd11fd9d11cf653
* Include DISTINCT in error message for label referenceMike Bayer2019-12-061-1/+2
| | | | | | | | Needed to add tests to ensure this label reference is handled correctly, so also modified the exception message to be more clear if someone has this error within distinct(). Change-Id: I6e685e46ae336596272d14366445ac224c18d92c
* create second level deduping when use_labels is turned onMike Bayer2019-10-071-5/+174
| | | | | | | | | | | | | | | | | | | | | As of #4753 we allow duplicate columns. This creates some new problems that there can be duplicate columns in a subquery which are then not addressible on the outside because they are ambiguous (Postgresql has this behavior at least). Additionally it creates situations where we are making an anon label of an anon label which is leaking into the query. New logic for generating anon labels handles this situation and also alters the .c collection of a subquery such that we are only getting the first column from the derived selectable that has that name, the subsequent ones have a new deduping label with two underscores and are not exposed in .c. The dedupe logic when rendering the columns will handle duplicate label names for different columns, vs. the same column repeated, as separate cases. Fixes: #4892 Change-Id: I929fbd8da14bcc239e0481c24bbd9b5ce826e8fa
* Deprecate textual column matching in RowMike Bayer2019-10-041-12/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Deprecate query.instances() without a context Deprecate string alias with contains_eager() Deprecated the behavior by which a :class:`.Column` can be used as the key in a result set row lookup, when that :class:`.Column` is not part of the SQL selectable that is being selected; that is, it is only matched on name. A deprecation warning is now emitted for this case. Various ORM use cases, such as those involving :func:`.text` constructs, have been improved so that this fallback logic is avoided in most cases. Calling the :meth:`.Query.instances` method without passing a :class:`.QueryContext` is deprecated. The original use case for this was that a :class:`.Query` could yield ORM objects when given only the entities to be selected as well as a DBAPI cursor object. However, for this to work correctly there is essential metadata that is passed from a SQLAlchemy :class:`.ResultProxy` that is derived from the mapped column expressions, which comes originally from the :class:`.QueryContext`. To retrieve ORM results from arbitrary SELECT statements, the :meth:`.Query.from_statement` method should be used. Note there is a small bump in test_zoomark because the column._label is being calculated for each of those columns within baseline_3_properties, as it is now part of the result map. This label can't be calculated when the column is attached to the table because it needs to have all the columns present to do this correctly. Another approach here would be to pre-load the _label before the test runs however the zoomark tests don't have an easy place for this to happen and it's not really worth it. Fixes: #4877 Fixes: #4719 Change-Id: I9bd29e72e6dce7c855651d69ba68d7383469acbc
* Run row value processors up frontMike Bayer2019-10-011-1/+3
| | | | | | | | | | | | as part of a larger series of changes to generalize row-tuples, RowProxy becomes plain Row and is no longer a "proxy"; the DBAPI row is now copied directly into the Row when constructed, result handling occurs at once. Subsequent changes will break out Row into a new version that behaves fully a tuple. Change-Id: I2ffa156afce5d21c38f28e54c3a531f361345dd5
* Unify generation between Core and ORM queryMike Bayer2019-09-261-1/+1
| | | | | | | | | | | | | | generation is to be enhanced to include caching functionality, so ensure that Query and all generative in Core (e.g. select, DML etc) are using the same generations system. Additionally, deprecate Select.append methods and state Select methods independently of their append versions. Mutability of expression objects is a special case only when generating new objects during a visit. Fixes: #4637 Change-Id: I3dfac00d5e0f710c833b236f7a0913e1ca24dde4
* Strip special chars in anonymized bind namesMike Bayer2019-09-051-0/+54
| | | | | | | | | | | | | | | | | | | | | | Characters that interfere with "pyformat" or "named" formats in bound parameters, namely ``%, (, )`` and the space character, as well as a few other typically undesirable characters, are stripped early for a :func:`.bindparam` that is using an anonymized name, which is typically generated automatically from a named column which itself includes these characters in its name and does not use a ``.key``, so that they do not interfere either with the SQLAlchemy compiler's use of string formatting or with the driver-level parsing of the parameter, both of which could be demonstrated before the fix. The change only applies to anonymized parameter names that are generated and consumed internally, not end-user defined names, so the change should have no impact on any existing code. Applies in particular to the psycopg2 driver which does not otherwise quote special parameter names, but also strips leading underscores to suit Oracle (but not yet leading numbers, as some anon parameters are currently entirely numeric/underscore based); Oracle in any case continues to quote parameter names that include special characters. Fixes: #4837 Change-Id: I21cb654c3e4ef786114160b8b4295242720bf3f9
* Render LIMIT/OFFSET conditions after compile on select dialectsMike Bayer2019-08-301-460/+501
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added new "post compile parameters" feature. This feature allows a :func:`.bindparam` construct to have its value rendered into the SQL string before being passed to the DBAPI driver, but after the compilation step, using the "literal render" feature of the compiler. The immediate rationale for this feature is to support LIMIT/OFFSET schemes that don't work or perform well as bound parameters handled by the database driver, while still allowing for SQLAlchemy SQL constructs to be cacheable in their compiled form. The immediate targets for the new feature are the "TOP N" clause used by SQL Server (and Sybase) which does not support a bound parameter, as well as the "ROWNUM" and optional "FIRST_ROWS()" schemes used by the Oracle dialect, the former of which has been known to perform better without bound parameters and the latter of which does not support a bound parameter. The feature builds upon the mechanisms first developed to support "expanding" parameters for IN expressions. As part of this feature, the Oracle ``use_binds_for_limits`` feature is turned on unconditionally and this flag is now deprecated. - adds limited support for "unique" bound parameters within a text() construct. - adds an additional int() check within the literal render function of the Integer datatype and tests that non-int values raise ValueError. Fixes: #4808 Change-Id: Iace97d544d1a7351ee07db970c6bc06a19c712c6
* Label simple column transformations as the column nameMike Bayer2019-08-281-6/+6
| | | | | | | | | | | | | | | | | | | | Additional logic has been added such that certain SQL expressions which typically wrap a single database column will use the name of that column as their "anonymous label" name within a SELECT statement, potentially making key-based lookups in result tuples more intutive. The primary example of this is that of a CAST expression, e.g. ``CAST(table.colname AS INTEGER)``, which will export its default name as "colname", rather than the usual "anon_1" label, that is, ``CAST(table.colname AS INTEGER) AS colname``. If the inner expression doesn't have a name, then the previous "anonymous label" logic is used. When using SELECT statements that make use of :meth:`.Select.apply_labels`, such as those emitted by the ORM, the labeling logic will produce ``<tablename>_<inner column name>`` in the same was as if the column were named alone. The logic applies right now to the :func:`.cast` and :func:`.type_coerce` constructs as well as some single-element boolean expressions. Fixes: #4449 Change-Id: Ie3b73470e3bea53f2386cd86514cdc556491564e
* Support tuple IN VALUES for SQLite, othersMike Bayer2019-07-191-0/+19
| | | | | | | | | | | | Added support for composite (tuple) IN operators with SQLite, by rendering the VALUES keyword for this backend. As other backends such as DB2 are known to use the same syntax, the syntax is enabled in the base compiler using a dialect-level flag ``tuple_in_values``. The change also includes support for "empty IN tuple" expressions for SQLite when using "in_()" between a tuple value and an empty set. Fixes: #4766 Change-Id: I416e1af29b31d78f9ae06ec3c3a48ef6d6e813f5
* Allow duplicate columns in from clauses and selectablesMike Bayer2019-07-111-10/+41
| | | | | | | | | | | | | | | | | | The :func:`.select` construct and related constructs now allow for duplication of column labels and columns themselves in the columns clause, mirroring exactly how column expressions were passed in. This allows the tuples returned by an executed result to match what was SELECTed for in the first place, which is how the ORM :class:`.Query` works, so this establishes better cross-compatibility between the two constructs. Additionally, it allows column-positioning-sensitive structures such as UNIONs (i.e. :class:`.CompoundSelect`) to be more intuitively constructed in those cases where a particular column might appear in more than one place. To support this change, the :class:`.ColumnCollection` has been revised to support duplicate columns as well as to allow integer index access. Fixes: #4753 Change-Id: Ie09a8116f05c367995c1e43623c51e07971d3bf0
* SelectBase no longer a FromClauseMike Bayer2019-07-061-162/+226
| | | | | | | | | | | | | | | | | | | | As part of the SQLAlchemy 2.0 migration project, a conceptual change has been made to the role of the :class:`.SelectBase` class hierarchy, which is the root of all "SELECT" statement constructs, in that they no longer serve directly as FROM clauses, that is, they no longer subclass :class:`.FromClause`. For end users, the change mostly means that any placement of a :func:`.select` construct in the FROM clause of another :func:`.select` requires first that it be wrapped in a subquery first, which historically is through the use of the :meth:`.SelectBase.alias` method, and is now also available through the use of :meth:`.SelectBase.subquery`. This was usually a requirement in any case since several databases don't accept unnamed SELECT subqueries in their FROM clause in any case. See the documentation in this change for lots more detail. Fixes: #4617 Change-Id: I0f6174ee24b9a1a4529168e52e855e12abd60667
* Implement new ClauseElement role and coercion systemMike Bayer2019-05-181-95/+85
| | | | | | | | | | | | | | | | | | | | A major refactoring of all the functions handle all detection of Core argument types as well as perform coercions into a new class hierarchy based on "roles", each of which identify a syntactical location within a SQL statement. In contrast to the ClauseElement hierarchy that identifies "what" each object is syntactically, the SQLRole hierarchy identifies the "where does it go" of each object syntactically. From this we define a consistent type checking and coercion system that establishes well defined behviors. This is a breakout of the patch that is reorganizing select() constructs to no longer be in the FromClause hierarchy. Also includes a rename of as_scalar() into scalar_subquery(); deprecates automatic coercion to scalar_subquery(). Partially-fixes: #4617 Change-Id: I26f1e78898693c6b99ef7ea2f4e7dfd0e8e1a1bd
* Remove all remaining text() coercions and ensure identifiers are safeMike Bayer2019-02-061-0/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fully removed the behavior of strings passed directly as components of a :func:`.select` or :class:`.Query` object being coerced to :func:`.text` constructs automatically; the warning that has been emitted is now an ArgumentError or in the case of order_by() / group_by() a CompileError. This has emitted a warning since version 1.0 however its presence continues to create concerns for the potential of mis-use of this behavior. Note that public CVEs have been posted for order_by() / group_by() which are resolved by this commit: CVE-2019-7164 CVE-2019-7548 Added "SQL phrase validation" to key DDL phrases that are accepted as plain strings, including :paramref:`.ForeignKeyConstraint.on_delete`, :paramref:`.ForeignKeyConstraint.on_update`, :paramref:`.ExcludeConstraint.using`, :paramref:`.ForeignKeyConstraint.initially`, for areas where a series of SQL keywords only are expected.Any non-space characters that suggest the phrase would need to be quoted will raise a :class:`.CompileError`. This change is related to the series of changes committed as part of :ticket:`4481`. Fixed issue where using an uppercase name for an index type (e.g. GIST, BTREE, etc. ) or an EXCLUDE constraint would treat it as an identifier to be quoted, rather than rendering it as is. The new behavior converts these types to lowercase and ensures they contain only valid SQL characters. Quoting is applied to :class:`.Function` names, those which are usually but not necessarily generated from the :attr:`.sql.func` construct, at compile time if they contain illegal characters, such as spaces or punctuation. The names are as before treated as case insensitive however, meaning if the names contain uppercase or mixed case characters, that alone does not trigger quoting. The case insensitivity is currently maintained for backwards compatibility. Fixes: #4481 Fixes: #4473 Fixes: #4467 Change-Id: Ib22a27d62930e24702e2f0f7c74a0473385a08eb
* Add deprecation warnings to all deprecated APIsMike Bayer2019-01-231-8/+0
| | | | | | | | | | | | | | | A large change throughout the library has ensured that all objects, parameters, and behaviors which have been noted as deprecated or legacy now emit ``DeprecationWarning`` warnings when invoked. As the Python 3 interpreter now defaults to displaying deprecation warnings, as well as that modern test suites based on tools like tox and pytest tend to display deprecation warnings, this change should make it easier to note what API features are obsolete. See the notes added to the changelog and migration notes for further details. Fixes: #4393 Change-Id: If0ea11a1fc24f9a8029352eeadfc49a7a54c0a1b
* Post black reformattingMike Bayer2019-01-061-71/+74
| | | | | | | | | | | | | Applied on top of a pure run of black -l 79 in I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes resolves all remaining flake8 conditions for those codes we have enabled in setup.cfg. Included are resolutions for all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
* Run black -l 79 against all source filesMike Bayer2019-01-061-1568/+1974
| | | | | | | | | | | | | | This is a straight reformat run using black as is, with no edits applied at all. The black run will format code consistently, however in some cases that are prevalent in SQLAlchemy code it produces too-long lines. The too-long lines will be resolved in the following commit that will resolve all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
* Move CRUDTest, InlineDefaultTest from test_compilerMike Bayer2018-12-011-255/+0
| | | | | | | | | | test_compiler is mostly related to SELECT statements as well as smaller SQL elements. While it still has some DDL related tests, move out all the remaining insert/update tests into the already present test_insert.py, test_update.py Fixes: #2630 Change-Id: I4167618543fd1235d12d1717c8c629d2374b325a
* Add Sequence to StrSQLCompilerMike Bayer2018-11-101-0/+12
| | | | | | | | | | Added :class:`.Sequence` to the "string SQL" system that will render a meaningful string expression (``"<next sequence value: my_sequence>"``) when stringifying without a dialect a statement that includes a "sequence nextvalue" expression, rather than raising a compilation error. Fixes: #4144 Change-Id: Ia910f0e22008a7cde7597365954ede324101cf4d
* Allow dialects to customize group by clause compilationSamuel Chou2018-09-201-0/+37
| | | | | | | | | | | Refactored :class:`.SQLCompiler` to expose a :meth:`.SQLCompiler.group_by_clause` method similar to the :meth:`.SQLCompiler.order_by_clause` and :meth:`.SQLCompiler.limit_clause` methods, which can be overridden by dialects to customize how GROUP BY renders. Pull request courtesy Samuel Chou. Change-Id: I0a7238e55032558c27a0c56a72907c7b883456f1 Pull-request: https://github.com/zzzeek/sqlalchemy/pull/474
* Add missing range_ / rows parameters to additional over() methodsMike Bayer2018-08-191-0/+44
| | | | | | | | | | | | Added missing window function parameters :paramref:`.WithinGroup.over.range_` and :paramref:`.WithinGroup.over.rows` parameters to the :meth:`.WithinGroup.over` and :meth:`.FunctionFilter.over` methods, to correspond to the range/rows feature added to the "over" method of SQL functions as part of :ticket:`3049` in version 1.1. Fixes: #4322 Change-Id: I77dcdac65c699a4b52a3fc3ee09a100ffb4fc20e
* Ensure all visit_sequence accepts **kw argsMike Bayer2018-04-041-1/+14
| | | | | | | | | | Fixed issue where the compilation of an INSERT statement with the "literal_binds" option that also uses an explicit sequence and "inline" generation, as on Postgresql and Oracle, would fail to accommodate the extra keyword argument within the sequence processing routine. Change-Id: Ibdab7d340aea7429a210c9535ccf1a3e85f074fb Fixes: #4231
* Make column-level collation quoting dialect-specificMike Bayer2018-01-121-0/+19
| | | | | | | | | | | | | Fixed regression in 1.2 where newly repaired quoting of collation names in :ticket:`3785` breaks SQL Server, which explicitly does not understand a quoted collation name. Whether or not mixed-case collation names are quoted or not is now deferred down to a dialect-level decision so that each dialect can prepare these identifiers directly. Change-Id: Iaf0a8123d9bf4711219e320896bb28c5d2649304 Fixes: #4154
* Allow delete where clause to refer multiple tables.inytar2017-12-051-1/+1
| | | | | | | | | | | | | | | | | | | | | Implemented "DELETE..FROM" syntax for Postgresql, MySQL, MS SQL Server (as well as within the unsupported Sybase dialect) in a manner similar to how "UPDATE..FROM" works. A DELETE statement that refers to more than one table will switch into "multi-table" mode and render the appropriate "USING" or multi-table "FROM" clause as understood by the database. Pull request courtesy Pieter Mulder. For SQL syntaxes see: Postgresql: https://www.postgresql.org/docs/current/static/sql-delete.html MySQL: https://dev.mysql.com/doc/refman/5.7/en/delete.html#multiple-table_syntax MSSQL: https://docs.microsoft.com/en-us/sql/t-sql/statements/delete-transact-sql Sybase: http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc00801.1510/html/iqrefso/X315721.htm Co-authored by: Mike Bayer <mike_mp@zzzcomputing.com> Change-Id: I6dfd57b49e44a095d076dc493cd2360bb5d920d3 Pull-request: https://github.com/zzzeek/sqlalchemy/pull/392 Fixes: #959
* Allow for multiple FOLLOWING/PRECEDING in a window rangeMike Bayer2017-08-251-12/+18
| | | | | | | | | | | Altered the range specification for window functions to allow for two of the same PRECEDING or FOLLOWING keywords in a range by allowing for the left side of the range to be positive and for the right to be negative, e.g. (1, 3) is "1 FOLLOWING AND 3 FOLLOWING". Change-Id: I7d3a6c641151bb49219104968d18dac2266f3db8 Fixes: #4053
* Add new "expanding" feature to bindparam()Mike Bayer2017-04-071-0/+12
| | | | | | | | | | | | | | | | Added a new kind of :func:`.bindparam` called "expanding". This is for use in ``IN`` expressions where the list of elements is rendered into individual bound parameters at statement execution time, rather than at statement compilation time. This allows both a single bound parameter name to be linked to an IN expression of multiple elements, as well as allows query caching to be used with IN expressions. The new feature allows the related features of "select in" loading and "polymorphic in" loading to make use of the baked query extension to reduce call overhead. This feature should be considered to be **experimental** for 1.2. Fixes: #3953 Change-Id: Ie708414a3ab9c0af29998a2c7f239ff7633b1f6e
* Add "empty in" strategies; default to "static"Mike Bayer2017-03-141-3/+5
| | | | | | | | | | | | | | | The longstanding behavior of the :meth:`.Operators.in_` and :meth:`.Operators.not_in_` operators emitting a warning when the right-hand condition is an empty sequence has been revised; a new flag :paramref:`.create_engine.empty_in_strategy` allows an empty "IN" expression to generate a simple boolean expression, or to invoke the previous behavior of dis-equating the expression to itself, with or without a warning. The default behavior is now to emit the simple boolean expression, allowing an empty IN to be evaulated without any performance penalty. Change-Id: I65cc37f2d7cf65a59bf217136c42fee446929352 Fixes: #3907
View Lorry Controller Status