summaryrefslogtreecommitdiff
path: root/test/sql/test_operators.py
Commit message (Collapse)AuthorAgeFilesLines
* Fix coercion issue for tuple bindparamsmike bayer2023-02-151-0/+8
| | | | | | | | | | | | Fixed issue where element types of a tuple value would be hardcoded to take on the types from a compared-to tuple, when the comparison were using the :meth:`.ColumnOperators.in_` operator. This was inconsistent with the usual way that types are determined for a binary expression, which is that the actual element type on the right side is considered first before applying the left-hand-side type. Fixes: #9313 Change-Id: Ia8874c09682a6512fcf4084cf14481024959c461
* apply self_group to all elements of multi-expressionMike Bayer2023-02-091-24/+80
| | | | | | | | | | | Fixed critical regression in SQL expression formulation in the 2.0 series due to :ticket:`7744` which improved support for SQL expressions that contained many elements against the same operator repeatedly; parenthesis grouping would be lost with expression elements beyond the first two elements. Fixes: #9271 Change-Id: Ib6ed5b71efe0f6816dab75bda622297fc89e3b49
* Dedicated bitwise operatorsjazzthief2023-02-061-0/+71
| | | | | | | | | | | | | Added a full suite of new SQL bitwise operators, for performing database-side bitwise expressions on appropriate data values such as integers, bit-strings, and similar. Pull request courtesy Yegor Statkevich. Fixes: #8780 Closes: #9204 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/9204 Pull-request-sha: a4541772a6a784f9161ad78ef84d2ea7a62fa8de Change-Id: I4c70e80f9548dcc1b4e3dccd71bd59d51d3ed46e
* typing updatesMike Bayer2023-01-201-7/+22
| | | | | | | | | | | | | | | | | | | | | | | | The :meth:`_sql.ColumnOperators.in_` and :meth:`_sql.ColumnOperators.not_in_` are typed to include ``Iterable[Any]`` rather than ``Sequence[Any]`` for more flexibility in argument type. The :func:`_sql.or_` and :func:`_sql.and_` from a typing perspective require the first argument to be present, however these functions still accept zero arguments which will emit a deprecation warning at runtime. Typing is also added to support sending the fixed literal ``False`` for :func:`_sql.or_` and ``True`` for :func:`_sql.and_` as the first argument only, however the documentation now indicates sending the :func:`_sql.false` and :func:`_sql.true` constructs in these cases as a more explicit approach. Fixed typing issue where iterating over a :class:`_orm.Query` object was not correctly typed. Fixes: #9122 Fixes: #9123 Fixes: #9125 Change-Id: I500e3e1b826717b3dd49afa1e682c3c8279c9226
* Implement ScalarValueFederico Caselli2022-11-261-141/+106
| | | | | | | | | | | | | | Added :class:`_expression.ScalarValues` that can be used as a column element allowing using :class:`_expression.Values` inside IN clauses or in conjunction with ``ANY`` or ``ALL`` collection aggregates. This new class is generated using the method :meth:`_expression.Values.scalar_values`. The :class:`_expression.Values` instance is now coerced to a :class:`_expression.ScalarValues` when used in a ``IN`` or ``NOT IN`` operation. Fixes: #6289 Change-Id: Iac22487ccb01553684b908e54d01c0687fa739f1
* Try running pyupgrade on the codeFederico Caselli2022-11-161-9/+7
| | | | | | | | command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>" pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not exists in sqlalchemy fixtures Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
* implement icontains, istartswith, iendswith operatorsMatias Martinez Rebori2022-09-081-0/+534
| | | | | | | | | | | | | | | | | | | | Added long-requested case-insensitive string operators :meth:`_sql.ColumnOperators.icontains`, :meth:`_sql.ColumnOperators.istartswith`, :meth:`_sql.ColumnOperators.iendswith`, which produce case-insensitive LIKE compositions (using ILIKE on PostgreSQL, and the LOWER() function on all other backends) to complement the existing LIKE composition operators :meth:`_sql.ColumnOperators.contains`, :meth:`_sql.ColumnOperators.startswith`, etc. Huge thanks to Matias Martinez Rebori for their meticulous and complete efforts in implementing these new methods. Fixes: #3482 Closes: #8496 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8496 Pull-request-sha: 7287e2c436959fac4fef022f359fcc73d1528211 Change-Id: I9fcdd603716218067547cc92a2b07bd02a2c366b
* use concat() directly for contains, startswith, endswithMike Bayer2022-07-171-0/+82
| | | | | | | | | | | | | | | Adjusted the SQL compilation for string containment functions ``.contains()``, ``.startswith()``, ``.endswith()`` to force the use of the string concatenation operator, rather than relying upon the overload of the addition operator, so that non-standard use of these operators with for example bytestrings still produces string concatenation operators. To accommodate this, needed to add a new _rconcat operator function, which is private, as well as a fallback in concat_op() that works similarly to Python builtin ops. Fixes: #8253 Change-Id: I2b7f56492f765742d88cb2a7834ded6a2892bd7e
* use plainto_tsquery for PG matchMike Bayer2022-05-221-3/+12
| | | | | | | | | | | | | | | | The :meth:`.Operators.match` operator now uses ``plainto_tsquery()`` for PostgreSQL full text search, rather than ``to_tsquery()``. The rationale for this change is to provide better cross-compatibility with match on other database backends. Full support for all PostgreSQL full text functions remains available through the use of :data:`.func` in conjunction with :meth:`.Operators.bool_op` (an improved version of :meth:`.Operators.op` for boolean operators). Additional doc updates here apply to 1.4 so will backport these out to a separate commit. Fixes: #7086 Change-Id: I1946075daf5d9c558e85f73f1bf852604b3b1b8c
* update for flake8-future-imports 0.0.5Mike Bayer2022-05-141-1/+0
| | | | | | | | a whole bunch of errors were apparently blocked by 0.0.4 being installed. Fixes: #8020 Change-Id: I22a0faeaabe03de501897893391946d677c2df7e
* use bindparam_type in BinaryElementImpl._post_coercion if availableMike Bayer2022-05-011-0/+19
| | | | | | | | | | Fixed an issue where using :func:`.bindparam` with no explicit data or type given could be coerced into the incorrect type when used in expressions such as when using :meth:`.ARRAY.comparator.any` and :meth:`.ARRAY.comparator.all`. Fixes: #7979 Change-Id: If7779e713c9a3a5fee496b66e417cfd3fca5b1f9
* properly type array element in any() / all()Mike Bayer2022-04-221-8/+8
| | | | | | | | | | Fixed bug in :class:`.ARRAY` datatype in combination with :class:`.Enum` on PostgreSQL where using the ``.any()`` method to render SQL ANY(), given members of the Python enumeration as arguments, would produce a type adaptation failure on all drivers. Fixes: #6515 Change-Id: Ia1e3b4e10aaf264ed436ce6030d105fc60023433
* implement multi-element expression constructsMike Bayer2022-04-131-4/+189
| | | | | | | | | | | | | | | | | | | | | | Improved the construction of SQL binary expressions to allow for very long expressions against the same associative operator without special steps needed in order to avoid high memory use and excess recursion depth. A particular binary operation ``A op B`` can now be joined against another element ``op C`` and the resulting structure will be "flattened" so that the representation as well as SQL compilation does not require recursion. To implement this more cleanly, the biggest change here is that column-oriented lists of things are broken away from ClauseList in a new class ExpressionClauseList, that also forms the basis of BooleanClauseList. ClauseList is still used for the generic "comma-separated list" of things such as Tuple and things like ORDER BY, as well as in some API endpoints. Also adds __slots__ to the TypeEngine-bound Comparator classes. Still can't really do __slots__ on ClauseElement. Fixes: #7744 Change-Id: I81a8ceb6f8f3bb0fe52d58f3cb42e4b6c2bc9018
* dont use exception catches for warnings; modernize xdist detectionMike Bayer2022-01-221-5/+5
| | | | | | | | | | | | | | | | | Improvements to the test suite's integration with pytest such that the "warnings" plugin, if manually enabled, will not interfere with the test suite, such that third parties can enable the warnings plugin or make use of the ``-W`` parameter and SQLAlchemy's test suite will continue to pass. Additionally, modernized the detection of the "pytest-xdist" plugin so that plugins can be globally disabled using PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 without breaking the test suite if xdist were still installed. Warning filters that promote deprecation warnings to errors are now localized to SQLAlchemy-specific warnings, or within SQLAlchemy-specific sources for general Python deprecation warnings, so that non-SQLAlchemy deprecation warnings emitted from pytest plugins should also not impact the test suite. Fixes: #7599 Change-Id: Ibcf09af25228d39ee5a943fda82d8a9302433726
* Merge "Fix various source comment/doc typos" into mainmike bayer2022-01-071-2/+2
|\
| * Fix various source comment/doc typosluz paz2021-12-291-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ### Description Found via `codespell -q 3 -L ba,crate,datas,froms,gord,hist,inh,nd,selectin,strat,ue` Also added codespell to the pep8 tox env ### Checklist This pull request is: - [x] A documentation / typographical error fix - Good to go, no issue or tests are needed Closes: #7338 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7338 Pull-request-sha: 0deac2219396bc0eba7da53eb3a80932edbf2dd7 Change-Id: Icd61db31c8dc655d4a39d8a304194804d08555fe
* | implement python_impl to custom_op for basic ORM evaluator extensibilityMike Bayer2022-01-041-0/+23
|/ | | | | | | | | | | | | | | Added new parameter :paramref:`_sql.Operators.op.python_impl`, available from :meth:`_sql.Operators.op` and also when using the :class:`_sql.Operators.custom_op` constructor directly, which allows an in-Python evaluation function to be provided along with the custom SQL operator. This evaluation function becomes the implementation used when the operator object is used given plain Python objects as operands on both sides, and in particular is compatible with the ``synchronize_session='evaluate'`` option used with :ref:`orm_expression_update_delete`. Fixes: #3162 Change-Id: If46ba6a0e303e2180a177ba418a8cafe9b42608e
* consider truediv as truediv; support floordiv operatorMike Bayer2021-12-261-16/+77
| | | | | | | | | | | | | | | | | | | | | Implemented full support for "truediv" and "floordiv" using the "/" and "//" operators. A "truediv" operation between two expressions using :class:`_types.Integer` now considers the result to be :class:`_types.Numeric`, and the dialect-level compilation will cast the right operand to a numeric type on a dialect-specific basis to ensure truediv is achieved. For floordiv, conversion is also added for those databases that don't already do floordiv by default (MySQL, Oracle) and the ``FLOOR()`` function is rendered in this case, as well as for cases where the right operand is not an integer (needed for PostgreSQL, others). The change resolves issues both with inconsistent behavior of the division operator on different backends and also fixes an issue where integer division on Oracle would fail to be able to fetch a result due to inappropriate outputtypehandlers. Fixes: #4926 Change-Id: Id54cc018c1fb7a49dd3ce1216d68d40f43fe2659
* Warn when caching is disabled / documentMike Bayer2021-12-061-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds new warnings for all elements that don't indicate their caching behavior, including user-defined ClauseElement subclasses and third party dialects. it additionally adds new documentation to discuss an apparent performance degradation in 1.4 when caching is disabled as a result in the significant expense incurred by ORM lazy loaders, which in 1.3 used BakedQuery so were actually cached. As a result of adding the warnings, a fair degree of lesser used SQL expression objects identified that they did not define caching behavior so would have been producing ``[no key]``, including PostgreSQL constructs ``hstore`` and ``array``. These have been amended to use inherit cache where appropriate. "on conflict" constructs in PostgreSQL, MySQL, SQLite still explicitly don't generate a cache key at this time. The change also adds a test for all constructs via assert_compile() to assert they will not generate cache warnings. Fixes: #7394 Change-Id: I85958affbb99bfad0f5efa21bc8f2a95e7e46981
* Clean up most py3k compatFederico Caselli2021-11-241-4/+5
| | | | Change-Id: I8172fdcc3103ff92aa049827728484c8779af6b7
* Remove object in class definitionFederico Caselli2021-11-221-2/+2
| | | | | References: #4600 Change-Id: I2a62ddfe00bc562720f0eae700a497495d7a987a
* change the POSTCOMPILE/ SCHEMA symbols to not conflict w mssql quotingMike Bayer2021-11-091-17/+19
| | | | | | | | | | | | | | | | | | Adjusted the compiler's generation of "post compile" symbols including those used for "expanding IN" as well as for the "schema translate map" to not be based directly on plain bracketed strings with underscores, as this conflicts directly with SQL Server's quoting format of also using brackets, which produces false matches when the compiler replaces "post compile" and "schema translate" symbols. The issue created easy to reproduce examples both with the :meth:`.Inspector.get_schema_names` method when used in conjunction with the :paramref:`_engine.Connection.execution_options.schema_translate_map` feature, as well in the unlikely case that a symbol overlapping with the internal name "POSTCOMPILE" would be used with a feature like "expanding in". Fixes: #7300 Change-Id: I6255c850b140522a4aba95085216d0bca18ce230
* First round of removal of python 2Federico Caselli2021-11-011-3/+1
| | | | | References: #4600 Change-Id: I61e35bc93fe95610ae75b31c18a3282558cd4ffe
* Remove deprecated dialects and driversFederico Caselli2021-10-311-17/+0
| | | | | Fixes: #7258 Change-Id: I3577f665eca04f2632b69bcb090f0a4ec9271db9
* repair any_() / all_() "implicit flip" behavior for NoneMike Bayer2021-10-041-1/+49
| | | | | | | | | | | | | | Fixed an inconsistency in the any_() / all_() functions / methods where the special behavior these functions have of "flipping" the expression such that the "ANY" / "ALL" expression is always on the right side would not function if the comparison were against the None value, that is, "column.any_() == None" should produce the same SQL expression as "null() == column.any_()". Added more docs to clarify this as well, plus mentions that any_() / all_() generally supersede the ARRAY version "any()" / "all()". Fixes: #7140 Change-Id: Ia5d55414ba40eb3fbda3598931fdd24c9b4a4411
* Modernize tests - select(whereclause)Gord Thompson2021-07-041-18/+24
| | | | Change-Id: I306cfbea9920b35100e3087dcc21d7ffa6c39c55
* Update black flak8 and zimportsFederico Caselli2021-05-121-1/+1
| | | | Change-Id: I488c9557eda390e4a88319affd4c8813ee274f80
* set bindparam.expanding in coercion againMike Bayer2021-05-101-6/+46
| | | | | | | | | | | | | | | | | Adjusted the logic added as part of :ticket:`6397` in 1.4.12 so that internal mutation of the :class:`.BindParameter` object occurs within the clause construction phase as it did before, rather than in the compilation phase. In the latter case, the mutation still produced side effects against the incoming construct and additionally could potentially interfere with other internal mutation routines. In order to solve the issue of the correct operator being present on the BindParameter.expand_op, we necessarily have to expand the BinaryExpression._negate() routine to flip the operator on the BindParameter also. Fixes: #6460 Change-Id: I1e53a9aeee4de4fc11af51d7593431532731561b
* Merge "don't cache TypeDecorator by default"mike bayer2021-05-061-0/+5
|\
| * don't cache TypeDecorator by defaultMike Bayer2021-05-061-0/+5
| | | | | | | | | | | | | | | | | | | | The :class:`.TypeDecorator` class will now emit a warning when used in SQL compilation with caching unless the ``.cache_ok`` flag is set to ``True`` or ``False``. ``.cache_ok`` indicates that all the parameters passed to the object are safe to be used as a cache key, ``False`` means they are not. Fixes: #6436 Change-Id: Ib1bb7dc4b124e38521d615c2e2e691e4915594fb
* | Parenthesize for empty not inMike Bayer2021-05-051-29/+84
|/ | | | | | | | | | | | | Fixed regression caused by the "empty in" change just made in :ticket:`6397` 1.4.12 where the expression needs to be parenthesized for the "not in" use case, otherwise the condition will interfere with the other filtering criteria. also amends StrSQLCompiler to use the newer "empty IN" style for its compilation process. Fixes: #6428 Change-Id: I182a552fc0d3065a9e38c0f4ece2deb143735c36
* Fit literal compile of empty in on a tupleFederico Caselli2021-04-161-4/+37
| | | | | | | | Fixed regression where an empty in statement on a tuple would result in an error when compiled with the option ``literal_binds=True``. Fixes: #6290 Change-Id: Ic0dff8f4a874cccdb201b6d9dcd3c2e7b7884cbb
* Infer types in BindParameter when expanding=TrueMike Bayer2021-04-081-0/+37
| | | | | | | | | | | | | Enhanced the "expanding" feature used for :meth:`_sql.ColumnOperators.in_` operations to infer the type of expression from the right hand list of elements, if the left hand side does not have any explicit type set up. This allows the expression to support stringification among other things. In 1.3, "expanding" was not automatically used for :meth:`_sql.ColumnOperators.in_` expressions, so in that sense this change fixes a behavioral regression. Fixes: #6222 Change-Id: Icdfda1e2c226a21896cafd6d8f251547794451c2
* Ensure Grouping._with_binary_element_type() maintains classMike Bayer2021-04-021-0/+16
| | | | | | | | | | Fixed regression where use of the :meth:`.Operators.in_` method with a :class:`_sql.Select` object against a non-table-bound column would produce an ``AttributeError``, or more generally using a :class:`_sql.ScalarSelect` that has no datatype in a binary expression would produce invalid state. Fixes: #6181 Change-Id: I1ddea433b3603fdab8f489bff571b512a6ffc66b
* Apply percent sign escaping to op(), custom_op()Mike Bayer2021-03-091-0/+69
| | | | | | | | | | | Fixed bug where the "percent escaping" feature that occurs with dialects that use the "format" or "pyformat" bound parameter styles was not enabled for the :meth:`.Operations.op` and :meth:`.Operations.custom_op` methods, for custom operators that use percent signs. The percent sign will now be automatically doubled based on the paramstyle as necessary. Fixes: #6016 Change-Id: I285c5fc082481c2ee989edf1b02a83a6087ea26a
* Replace with_labels() and apply_labels() in ORM/CoreGord Thompson2021-01-261-3/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Replace :meth:`_orm.Query.with_labels` and :meth:`_sql.GenerativeSelect.apply_labels` with explicit getters and setters ``get_label_style`` and ``set_label_style`` to accommodate the three supported label styles: ``LABEL_STYLE_DISAMBIGUATE_ONLY`` (default), ``LABEL_STYLE_TABLENAME_PLUS_COL``, and ``LABEL_STYLE_NONE``. In addition, for Core and "future style" ORM queries, ``LABEL_STYLE_DISAMBIGUATE_ONLY`` is now the default label style. This style differs from the existing "no labels" style in that labeling is applied in the case of column name conflicts; with ``LABEL_STYLE_NONE``, a duplicate column name is not accessible via name in any case. For legacy ORM queries using :class:`_query.Query`, the table-plus-column names labeling style applied by ``LABEL_STYLE_TABLENAME_PLUS_COL`` continues to be used so that existing test suites and logging facilities see no change in behavior by default, however this style of labeling is no longer required for SQLAlchemy queries to function, as result sets are commonly matched to columns using a positional approach since SQLAlchemy 1.0. Within test suites, all use of apply_labels() / use_labels now uses the new methods. New tests added to test/sql/test_deprecations.py nad test/orm/test_deprecations.py to cover just the old apply_labels() method call. Tests in ORM that made explicit use apply_labels()/ etc. where it isn't needed for the ORM to work correctly use default label style now. Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Fixes: #4757 Change-Id: I5fdcd2ed4ae8c7fe62f8be2b6d0e8f66409b6a54
* reinvent xdist hooks in terms of pytest fixturesMike Bayer2021-01-131-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To allow the "connection" pytest fixture and others work correctly in conjunction with setup/teardown that expects to be external to the transaction, remove and prevent any usage of "xdist" style names that are hardcoded by pytest to run inside of fixtures, even function level ones. Instead use pytest autouse fixtures to implement our own r"setup|teardown_test(?:_class)?" methods so that we can ensure function-scoped fixtures are run within them. A new more explicit flow is set up within plugin_base and pytestplugin such that the order of setup/teardown steps, which there are now many, is fully documented and controllable. New granularity has been added to the test teardown phase to distinguish between "end of the test" when lock-holding structures on connections should be released to allow for table drops, vs. "end of the test plus its teardown steps" when we can perform final cleanup on connections and run assertions that everything is closed out. From there we can remove most of the defensive "tear down everything" logic inside of engines which for many years would frequently dispose of pools over and over again, creating for a broken and expensive connection flow. A quick test shows that running test/sql/ against a single Postgresql engine with the new approach uses 75% fewer new connections, creating 42 new connections total, vs. 164 new connections total with the previous system. As part of this, the new fixtures metadata/connection/future_connection have been integrated such that they can be combined together effectively. The fixture_session(), provide_metadata() fixtures have been improved, including that fixture_session() now strongly references sessions which are explicitly torn down before table drops occur afer a test. Major changes have been made to the ConnectionKiller such that it now features different "scopes" for testing engines and will limit its cleanup to those testing engines corresponding to end of test, end of test class, or end of test session. The system by which it tracks DBAPI connections has been reworked, is ultimately somewhat similar to how it worked before but is organized more clearly along with the proxy-tracking logic. A "testing_engine" fixture is also added that works as a pytest fixture rather than a standalone function. The connection cleanup logic should now be very robust, as we now can use the same global connection pools for the whole suite without ever disposing them, while also running a query for PostgreSQL locks remaining after every test and assert there are no open transactions leaking between tests at all. Additional steps are added that also accommodate for asyncio connections not explicitly closed, as is the case for legacy sync-style tests as well as the async tests themselves. As always, hundreds of tests are further refined to use the new fixtures where problems with loose connections were identified, largely as a result of the new PostgreSQL assertions, many more tests have moved from legacy patterns into the newest. An unfortunate discovery during the creation of this system is that autouse fixtures (as well as if they are set up by @pytest.mark.usefixtures) are not usable at our current scale with pytest 4.6.11 running under Python 2. It's unclear if this is due to the older version of pytest or how it implements itself for Python 2, as well as if the issue is CPU slowness or just large memory use, but collecting the full span of tests takes over a minute for a single process when any autouse fixtures are in place and on CI the jobs just time out after ten minutes. So at the moment this patch also reinvents a small version of "autouse" fixtures when py2k is running, which skips generating the real fixture and instead uses two global pytest fixtures (which don't seem to impact performance) to invoke the "autouse" fixtures ourselves outside of pytest. This will limit our ability to do more with fixtures until we can remove py2k support. py.test is still observed to be much slower in collection in the 4.6.11 version compared to modern 6.2 versions, so add support for new TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that will run the suite for fewer backends under Python 2. For Python 3 pin pytest to modern 6.2 versions where performance for collection has been improved greatly. Includes the following improvements: Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would be raised rather than :class:`.exc.TimeoutError`. Also repaired the :paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using the async engine, which previously would ignore the timeout and block rather than timing out immediately as is the behavior with regular :class:`.QueuePool`. For asyncio the connection pool will now also not interact at all with an asyncio connection whose ConnectionFairy is being garbage collected; a warning that the connection was not properly closed is emitted and the connection is discarded. Within the test suite the ConnectionKiller is now maintaining strong references to all DBAPI connections and ensuring they are released when tests end, including those whose ConnectionFairy proxies are GCed. Identified cx_Oracle.stmtcachesize as a major factor in Oracle test scalability issues, this can be reset on a per-test basis rather than setting it to zero across the board. the addition of this flag has resolved the long-standing oracle "two task" error problem. For SQL Server, changed the temp table style used by the "suite" tests to be the double-pound-sign, i.e. global, variety, which is much easier to test generically. There are already reflection tests that are more finely tuned to both styles of temp table within the mssql test suite. Additionally, added an extra step to the "dropfirst" mechanism for SQL Server that will remove all foreign key constraints first as some issues were observed when using this flag when multiple schemas had not been torn down. Identified and fixed two subtle failure modes in the engine, when commit/rollback fails in a begin() context manager, the connection is explicitly closed, and when "initialize()" fails on the first new connection of a dialect, the transactional state on that connection is still rolled back. Fixes: #5826 Fixes: #5827 Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
* Apply underscore naming to several more operatorsjonathan vanasco2020-10-301-22/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | The operator changes are: * `isfalse` is now `is_false` * `isnot_distinct_from` is now `is_not_distinct_from` * `istrue` is now `is_true` * `notbetween` is now `not_between` * `notcontains` is now `not_contains` * `notendswith` is now `not_endswith` * `notilike` is now `not_ilike` * `notlike` is now `not_like` * `notmatch` is now `not_match` * `notstartswith` is now `not_startswith` * `nullsfirst` is now `nulls_first` * `nullslast` is now `nulls_last` Because these are core operators, the internal migration strategy for this change is to support legacy terms for an extended period of time -- if not indefinitely -- but update all documentation, tutorials, and internal usage to the new terms. The new terms are used to define the functions, and the legacy terms have been deprecated into aliases of the new terms. Fixes: #5435 Change-Id: Ifbd7cb1cdda5981990243c4fc4b4ff467dc132ac
* upgrade to black 20.8b1Mike Bayer2020-09-281-3/+3
| | | | | | | It's better, the majority of these changes look more readable to me. also found some docstrings that had formatting / quoting issues. Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
* Rename Core expression isnot, not_in_jonathan vanasco2020-09-141-8/+12
| | | | | | | | | | | | | | | | | | | Several operators are renamed to achieve more consistent naming across SQLAlchemy. The operator changes are: * `isnot` is now `is_not` * `not_in_` is now `not_in` Because these are core operators, the internal migration strategy for this change is to support legacy terms for an extended period of time -- if not indefinitely -- but update all documentation, tutorials, and internal usage to the new terms. The new terms are used to define the functions, and the legacy terms have been deprecated into aliases of the new terms. Fixes: #5429 Change-Id: Ia1e66e7a50ac35d3f6260d8bf6ba3ce8087cbad2
* Update select usage to use the new 1.4 formatFederico Caselli2020-09-081-75/+62
| | | | | | | | | | | | | | | | This change includes mainly that the bracketed use within select() is moved to positional, and keyword arguments are removed from calls to the select() function. it does not yet fully address other issues such as keyword arguments passed to the table.select(). Additionally, allows False / None to both be considered as "disable" for all of select.correlate(), select.correlate_except(), query.correlate(), which establishes consistency with passing of ``False`` for the legact select(correlate=False) argument. Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
* Merge "internal test framework files for standardization of is_not/not_in;"mike bayer2020-08-301-3/+3
|\
| * internal test framework files for standardization of is_not/not_in;jonathan vanasco2020-08-291-3/+3
| | | | | | | | | | | | this is safe for 1.3.x Change-Id: Icba38fdc20f5d8ac407383a4278ccb346e09af38
* | Add support for regular expression on supported backend.Federico Caselli2020-08-271-0/+175
|/ | | | | | | | | | | | Two operations have been defined: * :meth:`~.ColumnOperators.regexp_match` implementing a regular expression match like function. * :meth:`~.ColumnOperators.regexp_replace` implementing a regular expression string replace function. Fixes: #1390 Change-Id: I44556846e4668ccf329023613bd26861d5c674e6
* Deliver straight BinaryExpr w/ no negate for any() / all()Mike Bayer2020-08-181-0/+18
| | | | | | | | | Adjusted the :meth:`_types.ARRAY.Comparator.any` and :meth:`_types.ARRAY.Comparator.all` methods to implement a straight "NOT" operation for negation, rather than negating the comparison operator. Fixes: #5518 Change-Id: I87ee9278c321aafe51a679fcfcbb5fbb11307fda
* Create a real type for Tuple() and handle appropriately in compilerMike Bayer2020-08-171-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | Improved the :func:`_sql.tuple_` construct such that it behaves predictably when used in a columns-clause context. The SQL tuple is not supported as a "SELECT" columns clause element on most backends; on those that do (PostgreSQL, not surprisingly), the Python DBAPI does not have a "nested type" concept so there are still challenges in fetching rows for such an object. Use of :func:`_sql.tuple_` in a :func:`_sql.select` or :class:`_orm.Query` will now raise a :class:`_exc.CompileError` at the point at which the :func:`_sql.tuple_` object is seen as presenting itself for fetching rows (i.e., if the tuple is in the columns clause of a subquery, no error is raised). For ORM use,the :class:`_orm.Bundle` object is an explicit directive that a series of columns should be returned as a sub-tuple per row and is suggested by the error message. Additionally ,the tuple will now render with parenthesis in all contexts. Previously, the parenthesization would not render in a columns context leading to non-defined behavior. As part of this change, Tuple receives a dedicated datatype which appears to allow us the very desirable change of removing the bindparam._expanding_in_types attribute as well as ClauseList._tuple_values (which might already have not been needed due to #4645). Fixes: #5127 Change-Id: Iecafa0e0aac2f1f37ec8d0e1631d562611c90200
* Add future=True to create_engine/Session; unify select()Mike Bayer2020-07-081-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several weeks of using the future_select() construct has led to the proposal there be just one select() construct again which features the new join() method, and otherwise accepts both the 1.x and 2.x argument styles. This would make migration simpler and reduce confusion. However, confusion may be increased by the fact that select().join() is different Current thinking is we may be better off with a few hard behavioral changes to old and relatively unknown APIs rather than trying to play both sides within two extremely similar but subtly different APIs. At the moment, the .join() thing seems to be the only behavioral change that occurs without the user taking any explicit steps. Session.execute() will still behave the old way as we are adding a future flag. This change also adds the "future" flag to Session() and session.execute(), so that interpretation of the incoming statement, as well as that the new style result is returned, does not occur for existing applications unless they add the use of this flag. The change in general is moving the "removed in 2.0" system further along where we want the test suite to fully pass even if the SQLALCHEMY_WARN_20 flag is set. Get many tests to pass when SQLALCHEMY_WARN_20 is set; this should be ongoing after this patch merges. Improve the RemovedIn20 warning; these are all deprecated "since" 1.4, so ensure that's what the messages read. Make sure the inforamtion link is on all warnings. Add deprecation warnings for parameters present and add warnings to all FromClause.select() types of methods. Fixes: #5379 Fixes: #5284 Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51 References: #5159
* Turn on caching everywhere, add loggingMike Bayer2020-06-101-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A variety of caching issues found by running all tests with statement caching turned on. The cache system now has a more conservative approach where any subclass of a SQL element will by default invalidate the cache key unless it adds the flag inherit_cache=True at the class level, or if it implements its own caching. Add working caching to a few elements that were omitted previously; fix some caching implementations to suit lesser used edge cases such as json casts and array slices. Refine the way BaseCursorResult and CursorMetaData interact with caching; to suit cases like Alembic modifying table structures, don't cache the cursor metadata if it were created against a cursor.description using non-positional matching, e.g. "select *". if a table re-ordered its columns or added/removed, now that data is obsolete. Additionally we have to adapt the cursor metadata _keymap regardless of if we just processed cursor.description, because if we ran against a cached SQLCompiler we won't have the right columns in _keymap. Other refinements to how and when we do this adaption as some weird cases were exposed in the Postgresql dialect, a text() construct that names just one column that is not actually in the statement. Fixed that also as it looks like a cut-and-paste artifact that doesn't actually affect anything. Various issues with re-use of compiled result maps and cursor metadata in conjunction with tables being changed, such as change in order of columns. mappers can be cleared but the class remains, meaning a mapper has to use itself as the cache key not the class. lots of bound parameter / literal issues, due to Alembic creating a straight subclass of bindparam that renders inline directly. While we can update Alembic to not do this, we have to assume other people might be doing this, so bindparam() implements the inherit_cache=True logic as well that was a bit involved. turn on cache stats in logging. Includes a fix to subqueryloader which moves all setup to the create_row_processor() phase and elminates any storage within the compiled context. This includes some changes to create_row_processor() signature and a revising of the technique used to determine if the loader can participate in polymorphic queries, which is also applied to selectinloading. DML update.values() and ordered_values() now coerces the keys as we have tests that pass an arbitrary class here which only includes __clause_element__(), so the key can't be cached unless it is coerced. this in turn changed how composite attributes support bulk update to use the standard approach of ClauseElement with annotations that are parsed in the ORM context. memory profiling successfully caught that the Session from Query was getting passed into _statement_20() so that was a big win for that test suite. Apparently Compiler had .execute() and .scalar() methods stuck on it, these date back to version 0.4 and there was a single test in the PostgreSQL dialect tests that exercised it for no apparent reason. Removed these methods as well as the concept of a Compiler holding onto a "bind". Fixes: #5386 Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
* Refine IN and scalar subquery coercionsMike Bayer2020-06-011-0/+125
| | | | | | | | | | | | | | | | | Ensure IN emits a warning when it coerces a FromClause into a select(), however that it continues to allow the scalar_subquery() coercion to be automatic, particularly since it's not clear that "col IN (select)" is necessarily "scalar" in the case of tuples. Convert the "scalar_subquery()" warning emitted in other cases to be a warning, rather than a deprecation warning. I can't imagine taking this coercion out as it is intuitive and is always going to happen; we just would like to note that an implicit coercion is occurring. Fixes: #5369 Change-Id: I748f01f40bc85c64e2776f9b88ef35641fa8fb5c
* Rework select(), CompoundSelect() in terms of CompileStateMike Bayer2020-03-101-7/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Continuation of I408e0b8be91fddd77cf279da97f55020871f75a9 - add an options() method to the base Generative construct. this will be where ORM options can go - Change Null, False_, True_ to be singletons, so that we aren't instantiating them and having to use isinstance. The previous issue with this was that they would produce dupe labels in SELECT statements. Apply the duplicate column logic, newly added in 1.4, to these objects as well as to non-apply-labels SELECT statements in general as a means of improving this. - create a revised system for generating ClauseList compilation constructs that simplfies up front creation to not actually use ClauseList; a simple tuple is rendered by the compiler using the same constrcution rules as what are used for ClauseList but without creating the actual object. Apply to Select, CompoundSelect, revise Update, Delete - Select, CompoundSelect get an initial CompileState implementation. All methods used only within compilation are moved here - refine update/insert/delete compile state to not require an outside boolean - refine and simplify Select._copy_internals - rework bind(), which is going away, to not use some of the internal traversal stuff - remove "autocommit", "for_update" parameters from Select, references #4643 - remove "autocommit" parameter from TextClause , references #4643 - add deprecation warnings for statement.execute(), engine.execute(), statement.scalar(), engine.scalar(). Fixes: #5193 Change-Id: I04ca0152b046fd42c5054ba10f37e43fc6e5a57b
View Lorry Controller Status