summaryrefslogtreecommitdiff
path: root/test/engine/test_pool.py
Commit message (Collapse)AuthorAgeFilesLines
...
* Post black reformattingMike Bayer2019-01-061-9/+20
| | | | | | | | | | | | | Applied on top of a pure run of black -l 79 in I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes resolves all remaining flake8 conditions for those codes we have enabled in setup.cfg. Included are resolutions for all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
* Run black -l 79 against all source filesMike Bayer2019-01-061-284/+263
| | | | | | | | | | | | | | This is a straight reformat run using black as is, with no edits applied at all. The black run will format code consistently, however in some cases that are prevalent in SQLAlchemy code it produces too-long lines. The too-long lines will be resolved in the following commit that will resolve all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
* Add LIFO for connection poolingTaem Park2018-09-181-0/+77
| | | | | | | | | | | | Added new "lifo" mode to :class:`.QueuePool`, typically enabled by setting the flag :paramref:`.create_engine.pool_use_lifo` to True. "lifo" mode means the same connection just checked in will be the first to be checked out again, allowing excess connections to be cleaned up from the server side during periods of the pool being only partially utilized. Pull request courtesy Taem Park. Change-Id: Idb5e299c5082b3e6b547bd03022acf65fdc34f35 Pull-request: https://github.com/zzzeek/sqlalchemy/pull/467
* Refactor pool.py into a packageMike Bayer2018-07-091-1/+1
| | | | | | | | | | | for better separation of concerns split out the core pooling constructs from the implementations. also makes more room for new pool classes should we decide to add them. The DBProxy feature is never used so remove it from documentation and split it out. Change-Id: I04c9d7ae1882b17f3cf5c37ed05b781b05ef88fa Fixes: #4255
* Prevent double-checkins and guard during reset-on-return invalidationsMike Bayer2018-05-151-0/+34
| | | | | | | | | | | | | | | | Fixed connection pool issue whereby if a disconnection error were raised during the connection pool's "reset on return" sequence in conjunction with an explicit transaction opened against the enclosing :class:`.Connection` object (such as from calling :meth:`.Session.close` without a rollback or commit, or calling :meth:`.Connection.close` without first closing a transaction declared with :meth:`.Connection.begin`), a double-checkin would result, which could then lead towards concurrent checkouts of the same connection. The double-checkin condition is now prevented overall by an assertion, as well as the specific double-checkin scenario has been fixed. Change-Id: If5bb6941e36326846b14918c33ebfdd5604f642e Fixes: #4252
* Invalidate on failed connect handlerMike Bayer2018-03-281-0/+29
| | | | | | | | | | | | | Fixed bug in connection pool where a connection could be present in the pool without all of its "connect" event handlers called, if a previous "connect" handler threw an exception; note that the dialects themselves have connect handlers that emit SQL, such as those which set transaction isolation, which can fail if the database is in a non-available state, but still allows a connection. The connection is now invalidated first if any of the connect handlers fail. Change-Id: I61d6f4827a98ab8455f1c3e1c55d046eeccec09a Fixes: #4225
* Ensure weakref finalize_fairy operates upon the current connectionMike Bayer2018-02-081-0/+34
| | | | | | | | | | | | | | | Fixed a fairly serious connection pool bug where a connection that is acquired after being refreshed as a result of a user-defined :class:`.DisconnectionError` or due to the 1.2-released "pre_ping" feature would not be correctly reset if the connection were returned to the pool by weakref cleanup (e.g. the front-facing object is garbage collected); the weakref would still refer to the previously invalidated DBAPI connection which would have the reset operation erroneously called upon it instead. This would lead to stack traces in the logs and a connection being checked into the pool without being reset, which can cause locking issues. Change-Id: Iabd9f3a63a1d0207d0de0054a6ced3560818cf9c Fixes: #4184
* Disable the singletonthreadpool test cleanup no gcMike Bayer2018-01-091-4/+10
| | | | | | | | | | | | This test often fails on CI under load and reveals a small race condition within SingleonThreadPool's "cleanup" system. As this pool's "cleanup" system is pretty much never used in any case since this pool is no longer used for SQLite file databases, this race should have little impact, especially since the occasional non-closed connection will get closed out when it's GC'ed in any case. Change-Id: I68aefc0f9bbfc708c95af0497844e65366fc8429
* - keep trying to identify the race here. can reproduce locallyMike Bayer2017-04-271-11/+11
| | | | | | now and it seems like mock might not be doing the right thing. Change-Id: I5c108d82631c9217da54a8ace68d7728c3e204d8
* - try to work around a race that can occur in STP whenMike Bayer2017-04-271-2/+11
| | | | | | used in this intentionally broken scenario Change-Id: I88ea6fa710da2189e6d47e2d12f5f0fd6f6bb7d4
* Spelling fixesVille Skyttä2016-10-081-1/+1
|
* Additions to support HAAlchemy pluginMike Bayer2016-09-161-0/+75
| | | | | | | | | | | | | | | | | | | - add a connect=True key to connection record to support pre-loading of _ConnectionRecord objects - ensure _ConnectionRecord.close() leaves the record in a good state for reopening - add _ConnectionRecord.record_info for persistent storage - add "in_use" accessor based on fairy_ref being present or not - allow for the exclusions system and SuiteRequirements to be usable without the full plugin_base setup. - move some Python-env requirements to the importable requirements.py module. - allow starttime to be queried - add additional events for engine plugins - have "dialect" be a first-class parameter to the pool, ensure the engine strategy supplies it up front Change-Id: Ibf549f7a1766e49d335cd6f5e26bacfaef9a8229
* - dont use id() to test identity as these can be recycledMike Bayer2016-06-151-12/+14
| | | | Change-Id: Ie4cb4924909d55c5962f66e36cd5325e8e8f0538
* - flake8Mike Bayer2016-06-151-106/+128
| | | | Change-Id: I74e369d1d71c98cb0af58dde80b7d535d17cbdf7
* - use a mock here, do away with timing problems for good. need to do thisMike Bayer2016-06-021-12/+18
| | | | | | for the rest of the timing tests Change-Id: I06a815b1500222126a4dbc2a9a5da3ef7847e321
* - Added connection pool events :meth:`ConnectionEvents.close`,Mike Bayer2016-03-251-0/+60
| | | | | :meth:`.ConnectionEvents.detach`, :meth:`.ConnectionEvents.close_detached`.
* - Fixed critical issue whereby the pool "checkout" event handlerMike Bayer2015-07-221-1/+94
| | | | | | | | | | | | may be called against a stale connection without the "connect" event handler having been called, in the case where the pool attempted to reconnect after being invalidated and failed; the stale connection would remain present and would be used on a subsequent attempt. This issue has a greater impact in the 1.0 series subsequent to 1.0.2, as it also delivers a blanked-out ``.info`` dictionary to the event handler; prior to 1.0.2 the ``.info`` dictionary is still the previous one. fixes #3497
* - Fixed bug where in the case that a pool checkout event handler is usedMike Bayer2015-05-141-17/+48
| | | | | | | | | | | | | | | | | and the database can no longer be connected towards, that the checkout handler failure is caught, the attempt to re-acquire the connection also raises an exception, but the underlying connection record is not immediately re-checked in before the exception is propagated outwards, having the effect that the checked-out record does not close itself until the stack trace it's associated with is garbage collected, preventing that record from being used for a new checkout until we leave the scope of the stack trace. This can lead to confusion in the specific case of when the number of current stack traces in memory exceeds the number of connections the pool can return, as the pool will instead begin to raise errors about no more checkouts available, rather than attempting a connection again. The fix applies a checkin of the record before re-raising. fixes #3419
* - revise the last commit with a more traditional approachMike Bayer2015-04-301-6/+1
| | | | using descriptors; ensure that mock.patch() honors descriptor setters
* - work the wrapping of the "creator" to be as resilient toMike Bayer2015-04-301-0/+55
| | | | | old / new style, direct access, and ad-hoc patching and unpatching as possible
* - New features added to support engine/pool plugins with advancedMike Bayer2015-04-301-2/+96
| | | | | | | | | | | | | | | | | | | | | | | | | functionality. Added a new "soft invalidate" feature to the connection pool at the level of the checked out connection wrapper as well as the :class:`._ConnectionRecord`. This works similarly to a modern pool invalidation in that connections aren't actively closed, but are recycled only on next checkout; this is essentially a per-connection version of that feature. A new event :class:`.PoolEvents.soft_invalidate` is added to complement it. fixes #3379 - Added new flag :attr:`.ExceptionContext.invalidate_pool_on_disconnect`. Allows an error handler within :meth:`.ConnectionEvents.handle_error` to maintain a "disconnect" condition, but to handle calling invalidate on individual connections in a specific manner within the event. - Added new event :class:`.DialectEvents.do_connect`, which allows interception / replacement of when the :meth:`.Dialect.connect` hook is called to create a DBAPI connection. Also added dialect plugin hooks :meth:`.Dialect.get_dialect_cls` and :meth:`.Dialect.engine_created` which allow external plugins to add events to existing dialects using entry points. fixes #3355
* Merge remote-tracking branch 'origin/pr/163' into pr163Mike Bayer2015-04-121-12/+64
|\
| * PEP8 cleanup in /test/enginepr/163Eric Streeper2015-03-201-12/+64
| |
* | - add some teardown for connections that are held open after testsMike Bayer2015-04-081-6/+18
|/ | | | in some cases, interfering with tests that check pool._refs
* - take out the iterator approach here as it does not support concurrent accessMike Bayer2014-08-071-8/+9
|
* - ensure all tests are named test_*Mike Bayer2014-07-301-4/+5
|
* - add support for tags, including include/exclude support.Mike Bayer2014-07-271-5/+13
| | | | simplify tox again now that we can exclude tests more easily
* - more pg8000 tests passingMike Bayer2014-07-251-1/+1
|
* - work on fixing some race-condition failures:Mike Bayer2014-03-261-4/+5
| | | | | | | | | | | 1. make sure pool._invalidate() sets the timestamp up before invalidating the target connection. we can otherwise show how the conn.invalidate() + pool._invalidate() can lead to an extra connection being made. 2. to help with that, soften up the check on connection.invalidate() when connection is already closed. a warning is fine here 3. add a mutex to test_max_overflow() when we connect, because the way we're using mock depends on an iterator, that needs to be synchronized
* take out accidental PG dialect use hereMike Bayer2014-03-241-1/+1
|
* - A major improvement made to the mechanics by which the :class:`.Engine`Mike Bayer2014-03-221-10/+93
| | | | | | | | | | | | recycles the connection pool when a "disconnect" condition is detected; instead of discarding the pool and explicitly closing out connections, the pool is retained and a "generational" timestamp is updated to reflect the current time, thereby causing all existing connections to be recycled when they are next checked out. This greatly simplifies the recycle process, removes the need for "waking up" connect attempts waiting on the old pool and eliminates the race condition that many immediately-discarded "pool" objects could be created during the recycle operation. fixes #2985
* - Fixed a critical regression caused by :ticket:`2880` where the newlyMike Bayer2014-02-191-0/+38
| | | | | | concurrent ability to return connections from the pool means that the "first_connect" event is now no longer synchronized either, thus leading to dialect mis-configurations under even minimal concurrency situations.
* - :class:`.Connection` now associates a newMike Bayer2014-01-121-0/+90
| | | | | | | | | | | | | | | | | | | | | | :class:`.RootTransaction` or :class:`.TwoPhaseTransaction` with its immediate :class:`._ConnectionFairy` as a "reset handler" for the span of that transaction, which takes over the task of calling commit() or rollback() for the "reset on return" behavior of :class:`.Pool` if the transaction was not otherwise completed. This resolves the issue that a picky transaction like that of MySQL two-phase will be properly closed out when the connection is closed without an explicit rollback or commit (e.g. no longer raises "XAER_RMFAIL" in this case - note this only shows up in logging as the exception is not propagated within pool reset). This issue would arise e.g. when using an orm :class:`.Session` with ``twophase`` set, and then :meth:`.Session.close` is called without an explicit rollback or commit. The change also has the effect that you will now see an explicit "ROLLBACK" in the logs when using a :class:`.Session` object in non-autocommit mode regardless of how that session was discarded. Thanks to Jeff Dairiki and Laurence Rowe for isolating the issue here. [ticket:2907]
* - add new event PoolEvents.invalidate(). allows interception of invalidationMike Bayer2014-01-121-0/+32
| | | | | | | | | | | | | | | events including auto-invalidation, which is useful both for tests here as well as detecting failure conditions within the "reset" or "close" cases. - rename the argument for PoolEvents.reset() to dbapi_connection and connection_record to be consistent with everything else. - add new documentation sections on invalidation, including auto-invalidation and the invalidation process within the pool. - add _ConnectionFairy and _ConnectionRecord to the pool documentation. Establish docs for common _ConnectionFairy/_ConnectionRecord methods and accessors and have PoolEvents docs refer to _ConnectionRecord, since it is passed to all events. Rename a few _ConnectionFairy methods that are actually private to pool such as _checkout(), _checkin() and _checkout_existing(); there should not be any external code calling these
* - The :class:`.QueuePool` has been enhanced to not block new connectionMike Bayer2013-12-061-0/+82
| | | | | | | | attempts when an existing connection attempt is blocking. Previously, the production of new connections was serialized within the block that monitored overflow; the overflow counter is now altered within it's own critical section outside of the connection process itself. [ticket:2880]
* - Made a slight adjustment to the logic which waits for a pooledMike Bayer2013-12-061-6/+13
| | | | | | | | | | | connection to be available, such that for a connection pool with no timeout specified, it will every half a second break out of the wait to check for the so-called "abort" flag, which allows the waiter to break out in case the whole connection pool was dumped; normally the waiter should break out due to a notify_all() but it's possible this notify_all() is missed in very slim cases. This is an extension of logic first introduced in 0.8.0, and the issue has only been observed occasionally in stress tests.
* make sure thread.join() is used completely hereMike Bayer2013-11-301-4/+8
|
* - apply a timeout to all join() calls for test_poolMike Bayer2013-11-171-5/+10
| | | | - use thread.join() for waiters_handled test
* try to get this to pass on slow environmnetsMike Bayer2013-10-211-1/+1
|
* - Removed some now unneeded version checks [ticket:2829] courtesy alex gaynorMike Bayer2013-09-221-1/+0
|
* try seeing if just clearing this ahead of time helpsMike Bayer2013-07-271-0/+1
|
* further tweakingMike Bayer2013-07-131-3/+2
|
* fix the scoping hereMike Bayer2013-07-131-2/+4
|
* Fixed bug where :class:`.QueuePool` would lose the correctMike Bayer2013-07-041-1/+50
| | | | | | checked out count if an existing pooled connection failed to reconnect after an invalidate or recycle event. Also in 0.8.3. [ticket:2772]
* - refactor pool a bit so that intent between ↵Mike Bayer2013-07-021-3/+4
| | | | | | | | | | | | | | ConnectionRecord/ConnectionFairy is clear; make sure that the DBAPI connection passed to the reset-on-return events/dialect hooks is also a "fairy", so that dictionaries like "info" are available. [ticket:2770] - rework the execution_options system so that the dialect is given the job of making any immediate adjustments based on a set event. move the "isolation level" logic to use this new system. Also work things out so that even engine-level execution options can be used for things like isolation level; the dialect attaches a connect-event handler in this case to handle the task. - to support this new system as well as further extensibiltiy of execution options add events engine_connect(), set_connection_execution_options(), set_engine_execution_options()
* mark the tests that seem to be segfauling py3k+coverage so that we can ↵Mike Bayer2013-06-301-0/+7
| | | | | | exclude them at the nose command line
* - replace most explicitly-named test objects called "Mock..." withMike Bayer2013-06-301-64/+85
| | | | | | | | | | | | actual mock objects from the mock library. I'd like to use mock for new tests so we might as well use it in obvious places. - use unittest.mock in py3.3 - changelog - add a note to README.unittests - add tests_require in setup.py - have tests import from sqlalchemy.testing.mock - apply usage of mock to one of the event tests. we can be using this approach all over the place.
* Preserve reset_on_return when recreating a Pool.pr/6Eevee2013-06-101-1/+2
|
* merge defaultMike Bayer2013-05-051-17/+20
|\
| * - cleanupMike Bayer2013-05-051-17/+20
| | | | | | | | - move the timeout here to 14 seconds as jenkins still chokes occasionally
View Lorry Controller Status