| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
| |
Added reflection support in the Oracle dialect to expression based indexes
and the ordering direction of index expressions.
Fixes: #9597
Change-Id: I40e163496789774e9930f46823d2208c35eab6f8
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Various performance improvements to Row instanciation
- avoid passing processors if they are all None
- improve processor logic in cython
- improve tuplegetter using slices when contiguous indexes are used
Some timing follow.
In particular [base_]row_new_proc that tests using processors has
a 25% improvement compared to before in cython.
Looking at the [b]row_new_proc_none that test a list of processors
all None, this has 50% improvement in cython when passing the none list,
but in this patch it would usually be disabled by passing None, so the
performance gain is actually 90%, since it would run the case
[base_]row_new.
Tuplegetter is a bit faster in the single item get and when getting
sequential indexes (like indexes 1,2,3,4) at the cost of a bit
longer creation time in python, cython is mostly the same.
Current times
| python | cython | cy / py |
base_row_new | 0.639817400 | 0.118265500 | 0.184842582 |
row_new | 0.680355100 | 0.129714600 | 0.190657202 |
base_row_new_proc | 3.076538900 | 1.488428600 | 0.483799701 |
row_new_proc | 3.119700100 | 1.532197500 | 0.491136151 |
brow_new_proc_none | 1.917702300 | 0.475511500 | 0.247958977 |
row_new_proc_none | 1.956253300 | 0.497803100 | 0.254467609 |
tuplegetter_one | 0.152512600 | 0.148523900 | 0.973846751 |
tuplegetter_many | 0.184394100 | 0.184511500 | 1.000636680 |
tuplegetter_seq | 0.154832800 | 0.156270100 | 1.009282917 |
tuplegetter_new_one | 0.523730000 | 0.343402200 | 0.655685563 |
tuplegetter_new_many| 0.738924400 | 0.420961400 | 0.569694816 |
tuplegetter_new_seq | 1.062036900 | 0.495462000 | 0.466520514 |
Parent commit times
| python | cython | cy / py |
base_row_new | 0.643890800 | 0.113548300 | 0.176347138 |
row_new | 0.674885900 | 0.124391800 | 0.184315304 |
base_row_new_proc | 3.072020400 | 2.017367000 | 0.656690626 |
row_new_proc | 3.109943400 | 2.048359400 | 0.658648450 |
brow_new_proc_none | 1.967133700 | 1.006326000 | 0.511569702 |
row_new_proc_none | 1.960814900 | 1.025217800 | 0.522852922 |
tuplegetter_one | 0.197359900 | 0.205999000 | 1.043773330 |
tuplegetter_many | 0.196575900 | 0.194888500 | 0.991416038 |
tuplegetter_seq | 0.192723900 | 0.205635000 | 1.066992729 |
tuplegetter_new_one | 0.534644500 | 0.414311700 | 0.774929322 |
tuplegetter_new_many| 0.479376500 | 0.417448100 | 0.870814694 |
tuplegetter_new_seq | 0.481580200 | 0.412697900 | 0.856966088 |
Change-Id: I2ca1f49dca2beff625c283f1363c29c8ccc0c3f7
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Improved :class:`_engine.Row` implementation to optimize
``__getattr__`` performance.
The serialization of a :class:`_engine.Row` to pickle has changed with
this change. Pickle saved by older SQLAlchemy versions can still be loaded,
but new pickle saved by this version cannot be loaded by older ones.
Fixes: #9678
Closes: #9668
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/9668
Pull-request-sha: 86b8ccd1959dbd91b1208f7a648a91f217e1f866
Change-Id: Ia85c26a59e1a57ba2bf0d65578c6168f82a559f2
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds the very small plugin flake8-import-single which
will prevent us from having an import with more than one symbol
on a line.
Flake8 by itself prevents this pattern with E401:
import collections, os, sys
However does not do anything with this:
from sqlalchemy import Column, text
Both statements have the same issues generating merge artifacts
as well as presenting a manual decision to be made. While
zimports generally cleans up such imports at the top level, we
don't enforce zimports / pre-commit use.
the plugin finds the same issue for imports that are inside of
test methods. We shouldn't usually have imports in test methods
so most of them here are moved to be top level.
The version is pinned at 0.1.5; the project seems to have no
activity since 2019, however there are three 0.1.6dev releases
on pypi which stopped in September 2019, they seem to be
experiments with packaging. The source for 0.1.5
is extremely simple and only reveals one method to flake8
(the run() method).
Change-Id: Icea894e43bad9c0b5d4feb5f49c6c666d6ea6aa1
|
| |
|
|
|
|
|
|
| |
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures
Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The migration20 guide had a lot of "whats new" stuff in it,
which is moved to the whatsnew20 document.
Also moved most of the content regarding the reflection rewrite
into the whatsnew20 document and added some performance comparisons
to 1.4.
The sectioning for these two documents, as has happened for some
other documents I'm working on, has been modified to only use the
double-equals operator at the top. This is strictly because my
vscode seems to be able to show me an .rst outline in the side panel
if I do it this way, and otherwise it's blank.
Change-Id: I1c89af8b6fd47a3c14f33becc6f9ee06b1743a0f
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds support for comments on named constraints, including `ForeignKeyConstraint`, `PrimaryKeyConstraint`, `CheckConstraint`, `UniqueConstraint`, solving the [Issue 5667](https://github.com/sqlalchemy/sqlalchemy/issues/5667).
Supports only PostgreSQL backend.
### Description
Following the example of [Issue 1546](https://github.com/sqlalchemy/sqlalchemy/issues/1546), supports comments on constraints. Specifically, enables comments on _named_ ones — as I get it, PostgreSQL prohibits comments on unnamed constraints.
Enables setting the comments for named constraints like this:
```
Table(
'example', metadata,
Column('id', Integer),
Column('data', sa.String(30)),
PrimaryKeyConstraint(
"id", name="id_pk", comment="id_pk comment"
),
CheckConstraint('id < 100', name="cc1", comment="Id value can't exceed 100"),
UniqueConstraint(['data'], name="uc1", comment="Must have unique data field"),
)
```
Provides the DDL representation for constraint comments and routines to create and drop them. Class `.Inspector` reflects constraint comments via methods like `get_check_constraints` .
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- [ ] A short code fix
- [x] A new feature implementation
- Solves the issue 5667.
- The commit message includes `Fixes: 5667`.
- Includes tests based on comment reflection.
**Have a nice day!**
Fixes: #5667
Closes: #7742
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7742
Pull-request-sha: 42a5d3c3e9ccf9a9d5397fd007aeab0854f66130
Change-Id: Ia60f578595afdbd6089541c9a00e37997ef78ad3
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Rearchitected the schema reflection API to allow some dialects to make use
of high performing batch queries to reflect the schemas of many tables at
once using much fewer queries. The new performance features are targeted
first at the PostgreSQL and Oracle backends, and may be applied to any
dialect that makes use of SELECT queries against system catalog tables to
reflect tables (currently this omits the MySQL and SQLite dialects which
instead make use of parsing the "CREATE TABLE" statement, however these
dialects do not have a pre-existing performance issue with reflection. MS
SQL Server is still a TODO).
The new API is backwards compatible with the previous system, and should
require no changes to third party dialects to retain compatibility;
third party dialects can also opt into the new system by implementing
batched queries for schema reflection.
Along with this change is an updated reflection API that is fully
:pep:`484` typed, features many new methods and some changes.
Fixes: #4379
Change-Id: I897ec09843543aa7012bcdce758792ed3d415d08
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
These are small bits where cache_anon_map in particular
is part of the cache key generation scheme which is a key
target for cython.
changing such a tiny element of the cache key gen is
doing basically nothing yet, as the cython impl is
mostly the exact same speed as the python one. I guess for
cython to be effective we'd need to redo the whole cache key
generation and possibly not use the same kinds of structures,
which might not be very easy to do.
Additionally, some cython runtime import errors are being
observed on jenkins, add an upfront check to the test suite
to indicate if the expected build succeeded when REQUIRE_SQLALCHEMY_CEXT
is set.
Running case CacheAnonMap
Running python .... Done
Running cython .... Done
| python | cython | cy / py |
test_get_anon_non_present| 0.301266758 | 0.231203834 | 0.767438915 |
test_get_anon_present| 0.300919362 | 0.227336695 | 0.755473803 |
test_has_key_non_present| 0.152725077 | 0.133191719 | 0.872101171 |
test_has_key_present| 0.152689778 | 0.133673095 | 0.875455428 |
Running case PrefixAnonMap
Running python .. Done
Running cython .. Done
| python | cython | cy / py |
test_apply_non_present| 0.358715744 | 0.335245703 | 0.934572034 |
test_apply_present | 0.354434996 | 0.338579782 | 0.955266229 |
Change-Id: I0d3f1dd285c044afc234479141d831b2ee0455be
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Re-implement c version immutabledict / processors / resultproxy / utils with cython.
Performance is in general in par or better than the c version
Added a collection module that has cython version of OrderedSet and IdentitySet
Added a new test/perf file to compare the implementations.
Run ``python test/perf/compiled_extensions.py all`` to execute the comparison test.
See results here: https://docs.google.com/document/d/1nOcDGojHRtXEkuy4vNXcW_XOJd9gqKhSeALGG3kYr6A/edit?usp=sharing
Fixes: #7256
Change-Id: I2930ef1894b5048210384728118e586e813f6a76
Signed-off-by: Federico Caselli <cfederico87@gmail.com>
|
| |
|
|
| |
Change-Id: I8172fdcc3103ff92aa049827728484c8779af6b7
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Removed the warning that emits from the :class:`_types.Numeric` type about
DBAPIs not supporting Decimal values natively. This warning was oriented
towards SQLite, which does not have any real way without additional
extensions or workarounds of handling precision numeric values more than 15
significant digits as it only uses floating point math to represent
numbers. As this is a known and documented limitation in SQLite itself, and
not a quirk of the pysqlite driver, there's no need for SQLAlchemy to warn
for this. The change does not otherwise modify how precision numerics are
handled. Values can continue to be handled as ``Decimal()`` or ``float()``
as configured with the :class:`_types.Numeric`, :class:`_types.Float` , and
related datatypes, just without the ability to maintain precision beyond 15
significant digits when using SQLite, unless alternate representations such
as strings are used.
Fixes: #7299
Change-Id: Ic570f8107177dec3ddbe94c7b43f40057b03276a
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A variety of caching issues found by running
all tests with statement caching turned on.
The cache system now has a more conservative approach where
any subclass of a SQL element will by default invalidate
the cache key unless it adds the flag inherit_cache=True
at the class level, or if it implements its own caching.
Add working caching to a few elements that were
omitted previously; fix some caching implementations
to suit lesser used edge cases such as json casts
and array slices.
Refine the way BaseCursorResult and CursorMetaData
interact with caching; to suit cases like Alembic
modifying table structures, don't cache the
cursor metadata if it were created against a
cursor.description using non-positional matching,
e.g. "select *". if a table re-ordered its columns
or added/removed, now that data is obsolete.
Additionally we have to adapt the cursor metadata
_keymap regardless of if we just processed
cursor.description, because if we ran against
a cached SQLCompiler we won't have the right
columns in _keymap.
Other refinements to how and when we do this
adaption as some weird cases
were exposed in the Postgresql dialect,
a text() construct that names just one column that
is not actually in the statement. Fixed that
also as it looks like a cut-and-paste artifact
that doesn't actually affect anything.
Various issues with re-use of compiled result maps
and cursor metadata in conjunction with tables being
changed, such as change in order of columns.
mappers can be cleared but the class remains, meaning
a mapper has to use itself as the cache key not the class.
lots of bound parameter / literal issues, due to Alembic
creating a straight subclass of bindparam that renders
inline directly. While we can update Alembic to not
do this, we have to assume other people might be doing
this, so bindparam() implements the inherit_cache=True
logic as well that was a bit involved.
turn on cache stats in logging.
Includes a fix to subqueryloader which moves all setup to
the create_row_processor() phase and elminates any storage
within the compiled context. This includes some changes
to create_row_processor() signature and a revising of the
technique used to determine if the loader can participate
in polymorphic queries, which is also applied to
selectinloading.
DML update.values() and ordered_values() now coerces the
keys as we have tests that pass an arbitrary class here
which only includes __clause_element__(), so the
key can't be cached unless it is coerced. this in turn
changed how composite attributes support bulk update
to use the standard approach of ClauseElement with
annotations that are parsed in the ORM context.
memory profiling successfully caught that the Session
from Query was getting passed into _statement_20()
so that was a big win for that test suite.
Apparently Compiler had .execute() and .scalar() methods
stuck on it, these date back to version 0.4 and there
was a single test in the PostgreSQL dialect tests
that exercised it for no apparent reason. Removed
these methods as well as the concept of a Compiler
holding onto a "bind".
Fixes: #5386
Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch replaces the ORM execution flow with a
single pathway through Session.execute() for all queries,
including Core and ORM.
Currently included is full support for ORM Query,
Query.from_statement(), select(), as well as the
baked query and horizontal shard systems. Initial
changes have also been made to the dogpile caching
example, which like baked query makes use of a
new ORM-specific execution hook that replaces the
use of both QueryEvents.before_compile() as well
as Query._execute_and_instances() as the central
ORM interception hooks.
select() and Query() constructs alike can be passed to
Session.execute() where they will return ORM
results in a Results object. This API is currently
used internally by Query. Full support for
Session.execute()->results to behave in a fully
2.0 fashion will be in later changesets.
bulk update/delete with ORM support will also
be delivered via the update() and delete()
constructs, however these have not yet been adapted
to the new system and may follow in a subsequent
update.
Performance is also beginning to lag as of this
commit and some previous ones. It is hoped that
a few central functions such as the coercions
functions can be rewritten in C to re-gain
performance. Additionally, query caching
is now available and some subsequent patches
will attempt to cache more of the per-execution
work from the ORM layer, e.g. column getters
and adapters.
This patch also contains initial "turn on" of the
caching system enginewide via the query_cache_size
parameter to create_engine(). Still defaulting at
zero for "no caching". The caching system still
needs adjustments in order to gain adequate performance.
Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
|
| |
|
|
|
|
|
|
|
|
|
| |
A few small mistakes led to huge callcounts. Additionally,
the warn-on-get behavior which is attempting to warn for
deprecated access in SQLAlchemy 2.0 is very expensive; it's not clear
if its feasible to have this warning or to somehow alter how it
works.
Fixes: #5340
Change-Id: I73bdd2d7b6f1b25cc0222accabd585cf761a5af4
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Execution of literal sql string is deprecated in the
:meth:`.Connection.execute` and a warning is raised when used stating
that it will be coerced to :func:`.text` in a future release.
To execute a raw sql string the new connection method
:meth:`.Connection.exec_driver_sql` was added, that will retain the previous
behavior, passing the string to the DBAPI driver unchanged.
Usage of scalar or tuple positional parameters in :meth:`.Connection.execute`
is also deprecated.
Fixes: #4848
Fixes: #5178
Change-Id: I2830181054327996d594f7f0d59c157d477c3aa9
|
| |
|
|
|
|
|
|
|
|
|
| |
This is a very useful assertion which prevents unused variables
from being set up allows code to be more readable and sometimes
even more efficient. test suites seem to be where the most
problems are and there do not seem to be documentation examples
that are using this, or at least the linter is not taking effect
within rst blocks.
Change-Id: I2b3341d8dd14da34879d8425838e66a4b9f8e27d
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Applied on top of a pure run of black -l 79 in
I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes
resolves all remaining flake8 conditions for those codes
we have enabled in setup.cfg.
Included are resolutions for all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a straight reformat run using black as is, with no edits
applied at all.
The black run will format code consistently, however in
some cases that are prevalent in SQLAlchemy code it produces
too-long lines. The too-long lines will be resolved in the
following commit that will resolve all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
|
| | |
|
| |
|
|
|
|
|
|
| |
tested using pycodestyle version 2.2.0
Fixes: #3885
Change-Id: I5df43adc3aefe318f9eeab72a078247a548ec566
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/343
|
| |
|
|
|
|
|
|
| |
simple but unusual system allows for a dramatic savings in Python
overhead for the construction and processing of orm :class:`.Query`
objects, from query construction up through rendering of a string
SQL statement.
fixes #3054
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
didn't match. use straight memoized_props here for now, add a perf test to check it
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
- went through examples/ and cleaned out excess list() calls
|
| |
|
|
|
|
| |
- other cleanup
- don't need compat.decimal, that approach never panned out. hopefully
outside libs aren't pulling it in, they shouldn't be
|
| |
|
|
| |
:func:`.create_engine` as well as :class:`.String` is removed.
|
| |
|
|
|
|
|
| |
become an externally usable package but still remains within the main sqlalchemy parent package.
in this system, we use kind of an ugly hack to get the noseplugin imported outside of the
"sqlalchemy" package, while still making it available within sqlalchemy for usage by
third party libraries.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
inheriting mappers to be GC'ed when the
class itself is deferenced. The mapper
must not have its own table (i.e.
single table inh only) without polymorphic
attributes in place.
This allows for the use case of
creating a temporary subclass of a declarative
mapped class, with no table or mapping
directives of its own, to be garbage collected
when dereferenced by a unit test.
[ticket:2526]
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
now accepts aliased() class constructs as well
as with_polymorphic constructs, and works with
query.join(), any(), has(), and also
eager loaders subqueryload(), joinedload(),
contains_eager()
[ticket:2438] [ticket:1106]
- a rewrite of the query path system to use an
object based approach for more succinct usage. the system
has been designed carefully to not add an excessive method overhead.
- [feature] select() features a correlate_except()
method, auto correlates all selectables except those
passed. Is needed here for the updated any()/has()
functionality.
- remove some old cruft from LoaderStrategy, init(),debug_callable()
- use a namedtuple for _extended_entity_info. This method should
become standard within the orm internals
- some tweaks to the memory profile tests, number of runs can
be customized to work around pysqlite's very annoying behavior
- try to simplify PropertyOption._get_paths(), rename to _process_paths(),
returns a single list now. overall works more completely as was needed
for of_type() functionality
|
| |
|
|
|
|
|
| |
classes to a new test.lib.fixtures module
- move testing.TestBase to test.lib.fixtures
- massive search and replace
|
| |
|
|
|
|
|
| |
all referncing tests to not use globals
- tests that deal with pickle specifically load the fixture classes
from test.lib.pickleable, which gets some more classes added
- removed weird sa05 pickling tests that don't matter
|
| | |
|
| | |
|
| |
|
|
|
| |
cascade_iterator() should probably not yield the "instance" at all
and only deal in states. 30-40K methods taken off the orm2010 test.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- add --with-cdecimal flag to tests, monkeypatches cdecimal in
- fix mssql/pyodbc.py to not use private '_int' accessor in decimal conversion
routines
- pyodbc version 2.1.8 is needed for cdecimal in any case as
previous versions also called '_int', 2.1.8 adds the same string
logic as our own dialect, so that logic is skipped for modern
pyodbc version
- make the imports for "Decimal" consistent across the whole lib. not sure
yet how we should be importing "Decimal" or what the best way forward
is that would allow a clean user-invoked swap of cdecimal; for now,
added docs suggesting a global monkeypatch - the two decimal libs
are not compatible with each other so any chance of mixing produces
serious issues. adding adapters to DBAPIs tedious and adds in-python
overhead. suggestions welcome on how we should be doing
Decimal/cdecimal.
|
| | |
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
- Added "map_to()" method to SqlSoup, which is a "master"
method which accepts explicit arguments for each aspect of
the selectable and mapping, including a base class per
mapping. [ticket:1975]
- Mapped selectables used with the map(), with_labels(),
join() methods no longer put the given argument into the
internal "cache" dictionary. Particularly since the
join() and select() objects are created in the method
itself this was pretty much a pure memory leaking behavior.
|
| |/ |
|
| | |
|
| |
|
|
|
|
|
| |
- with m2m we have to go back to the previous approach of having both sides of
the DP fire off, tracking each pair of objects. history may not be consistently present
in one side or the other
- this revealed a whole lot of issues with self-referential m2m, which are fixed
|
