summaryrefslogtreecommitdiff
path: root/oslo_concurrency
Commit message (Collapse)AuthorAgeFilesLines
* Fix fair internal lock used from eventlet.spawn_nBalazs Gibizer2022-09-052-19/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The fasteners lib in version 0.15.0 removed the threading.current_thread workaround for eventlet[1] because eventlet seemed to fixed the current_thread issues tracked in [2]. However the fix for [2] only fixed half of the problem. The threading.current_thread call works if it is called from thread created by eventlet.spawn. However if the thread is created with eventlet.spawn_n then threading.current_thread is still broken and returns the ID of the python native thread. The fasteners' ReaderWriterLock depends heavily on threading.current_thread to decide which thread holds a lock and to allow re-entry of that thread. This leads to the situation that multiple threads created from spawn_n could take the same ReaderWriterLock at the same time. The fair internal lock in oslo.concurrency uses ReaderWriterLock and as a result such lock is broken for threads created with spawn_n. Note that this issue was raised with eventlet in [3] when the nova team detected it via a direct usage of ReaderWriterLock in the nova test code. As [3] did not lead to a solution in eventlet nova implemented a nova local fix for the test code in [4]. However now we detected that oslo.concurrency is affected by this issue as well. This patch restores the workaround that was removed by [1]. Note that a fasteners issue [5] also opened to restore the workaround[1]. [1] https://github.com/harlowja/fasteners/commit/467ed75ee1e9465ebff8b5edf452770befb93913 [2] https://github.com/eventlet/eventlet/issues/172 [3] https://github.com/eventlet/eventlet/issues/731 [4] https://review.opendev.org/c/openstack/nova/+/813114 [5] https://github.com/harlowja/fasteners/issues/96 Closes-Bug: #1988311 Change-Id: Ia873bcc6b07121c9bd0b94c593567d537b4c1112
* Prove that spawn_n with fair lock is brokenBalazs Gibizer2022-09-051-0/+63
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The fasteners lib in version 0.15.0 removed the threading.current_thread workaround for eventlet[1] because eventlet seemed to fixed the current_thread issues tracked in [2]. However the fix for [2] only fixed half of the problem. The threading.current_thread call works if it is called from thread created by eventlet.spawn. However if the thread is created with eventlet.spawn_n then threading.current_thread is still broken and returns the ID of the python native thread. The fasteners' ReaderWriterLock depends heavily on threading.current_thread to decide which thread holds a lock and to allow re-entry of that thread. This leads to the situation that multiple threads created from spawn_n could take the same ReaderWriterLock at the same time. The fair internal lock in oslo.concurrency uses ReaderWriterLock and as a result such lock is broken for threads created with spawn_n. Note that this issue was raised with eventlet in [3] when the nova team detected it via a direct usage of ReaderWriterLock in the nova test code. As [3] did not lead to a solution in eventlet nova implemented a nova local fix for the test code in [4]. However now we detected that oslo.concurrency is affected by this issue as well. This patch adds tests to show the scope of the problem. Note that the coverage tox target is changed to explicitly enable native threading otherwise it runs eventlet specific tests in a native environment. Also note that [5] was opened to reintroduce the workaround[1] in fasteners. [1] https://github.com/harlowja/fasteners/commit/467ed75ee1e9465ebff8b5edf452770befb93913 [2] https://github.com/eventlet/eventlet/issues/172 [3] https://github.com/eventlet/eventlet/issues/731 [4] https://review.opendev.org/c/openstack/nova/+/813114 [5] https://github.com/harlowja/fasteners/issues/96 Related-Bug: #1988311 Change-Id: Ibc193c855b49b95b46ebd2aac82ea89e33f885f0
* Log when waiting to acquire lockGorka Eguileor2022-07-071-2/+6
| | | | | | | | | | | | | | | | | We currently have no log entry when we start trying to acquire a lock. In general this is ok, but there are cases where it can be problematic, for example if we have a deadlock situation or if a lock takes a very long time to be acquired. In those scenarios looking at the logs we would see the operation proceed normally and suddenly go completely silent without knowing that it's waiting for a lock to be freed. This patch adds a debug log message right before trying to acquire the lock so we can detect those situations. Change-Id: I1354dfb98b0927ae167802ecc4ab1d34f6b4d720
* Imported Translations from ZanataOpenStack Proposal Bot2022-06-211-2/+6
| | | | | | | For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: Ic6a8cb3efdddaa809b1381a7d9b96e50b7e9da9f
* Merge "Allow python_exec kwarg to be None"Zuul2022-04-051-3/+3
|\
| * Allow python_exec kwarg to be NoneBrian Rosmaita2022-03-011-3/+3
| | | | | | | | | | | | | | | | | | | | | | processutils.execute() is documented to take a python_exec kwarg and default to using sys.executable if that argument isn't set. It is convenient (and more intuitive) for this behavior to also occur if the argument is present with value None. Closes-bug: #1962603 Related-bug: #1962581 Change-Id: I2e1f187feaf4bf9fbfaf04fce78efa0cba49fc07
* | Fix RequiredOptError when missing lock_pathThiago Brito2022-03-241-1/+1
|/ | | | | | | | | | | | When lock_file is not present on the [oslo_concurrency] section of the config file, we are getting a misleading error saying that it is not preset on the DEFAULT group. This commit fixes it by providing the group arg to the RequiredOptError constructor. Closes-Bug: #1966320 Signed-off-by: Thiago Brito <thiago.brito@windriver.com> Change-Id: Idee6987739917c7eae4faee15a66085a9ef7f6d2
* Add support for non-blocking locks4.5.0Romain Poupin2021-06-242-10/+136
| | | | | | | | | | | | | | | | This adds support for a non-blocking behavior of the lock : the context-based lock now raises an AcquireLockFailedException if the lock can't be acquired on the first try. At a higher layer, we just caught this exception inside the 'synchronized' decorator to prevent calling the wrapped function. In which case, we then trace this acquisition failure. For now at least, disabling blocking is not supported when using fair locks because the ReaderWriterLock.write_lock() provided by the fasteners module doesn't implements this behavior. Change-Id: I409da79007c9ba4fb8585da881e3d56998b0b98b
* Remove references to 'sys.version_info'dengzhaosen2021-04-271-6/+2
| | | | | | We support Python 3.6 as a minimum now, making these checks no-ops. Change-Id: I1ff344d345b56246e6a86f7761ef900e42daefa2
* Monkey patch original current_thread _active in processutilsRodolfo Alonso Hernandez2020-09-301-0/+7
| | | | | | | | | Monkey patch the original current_thread to use the up-to-date _active global variable. This solution is based on that documented at: https://github.com/eventlet/eventlet/issues/592 Change-Id: I0a2c1e0d8a8cad99d68100d25e88e0d3a2eb8f5c Related-Bug: #1863021
* Add support for timeout to processutils.execute4.2.0Dmitry Tantsur2020-06-242-5/+31
| | | | | | This is a standard Python feature since 3.3. Change-Id: Ib13af5aab0ebbae532f1e46309ad6023ca94d6b9
* Merge "Don't warn on lock removal if file doesn't exist"Zuul2020-06-082-5/+22
|\
| * Don't warn on lock removal if file doesn't existGorka Eguileor2020-03-182-5/+22
| | | | | | | | | | | | | | | | | | | | | | There may be cases were library users want to request the removal of an external lock that may not have been created and don't want to pollute the logs when the file is not present. We raise the log level from info to warning, but we don't log it if the file didn't exist, since the end result the caller wanted is there. Change-Id: I5ce8be34c9f2c4c59ea99dabc6760c3300f743a3
* | Merge "Stop to use the __future__ module."Zuul2020-06-032-4/+0
|\ \
| * | Stop to use the __future__ module.Hervé Beraud2020-06-022-4/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The __future__ module [1] was used in this context to ensure compatibility between python 2 and python 3. We previously dropped the support of python 2.7 [2] and now we only support python 3 so we don't need to continue to use this module and the imports listed below. Imports commonly used and their related PEPs: - `division` is related to PEP 238 [3] - `print_function` is related to PEP 3105 [4] - `unicode_literals` is related to PEP 3112 [5] - `with_statement` is related to PEP 343 [6] - `absolute_import` is related to PEP 328 [7] [1] https://docs.python.org/3/library/__future__.html [2] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html [3] https://www.python.org/dev/peps/pep-0238 [4] https://www.python.org/dev/peps/pep-3105 [5] https://www.python.org/dev/peps/pep-3112 [6] https://www.python.org/dev/peps/pep-0343 [7] https://www.python.org/dev/peps/pep-0328 Change-Id: I11cc9e15a26cfc663296d6a99a5705d764654b3f
* | | Merge "Monkey patch original current_thread _active"4.1.0Zuul2020-06-031-0/+6
|\ \ \ | |/ / |/| |
| * | Monkey patch original current_thread _activeArthur Dayne2020-05-081-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | Monkey patch the original current_thread to use the up-to-date _active global variable. This solution is based on that documented at: https://github.com/eventlet/eventlet/issues/592 Change-Id: I876e2825da628d51c9977a4eec0c2ef05aed3229 Closes-Bug: #1863021
* | | Remove six usageAndreas Jaeger2020-05-094-101/+50
|/ / | | | | | | | | | | Remove the usage of six, the python2-python3 compatibility lib. Change-Id: I56e479d0749504d65be1cf2b5235d454b0b90a1a
* | Imported Translations from ZanataOpenStack Proposal Bot2020-04-291-2/+6
| | | | | | | | | | | | | | For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I87ec1f7fc4fb7a70b748905d5dad4e7ffc132aa8
* | Use unittest.mock instead of third party mock4.0.2Sean McGinnis2020-03-312-2/+2
|/ | | | | | | | Now that we no longer support py27, we can use the standard library unittest.mock module instead of the third party mock lib. Change-Id: I792a17f4b2667713503e6bc10cfabed9905d8535 Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
* Merge "Fix remove_lock test"Zuul2019-12-201-22/+29
|\
| * Fix remove_lock testGorka Eguileor2019-12-201-22/+29
| | | | | | | | | | | | | | | | | | | | Current external lock removal unit tests are not really testing the removal as they don't bother to check that we are calling os.remove This patch fixes this situation by mocking the appropriate methods and checking that they are being called (or not) as expected. Change-Id: I25efc1638106431ee36e760c0c213a1a675ec908
* | Spiff up docs for *_with_prefixEric Fried2019-11-041-11/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | Following up on comments in [1], clarify and unify the docs for the following lockutils methods: - lock_with_prefix - synchronized_with_prefix - remove_external_lock_file_with_prefix [1] I4e723ee3be1e57c543684390b607c84388c6e930 Change-Id: I4179e8732dba7367bd0c835cbd11939ca7b8cc42
* | Add lock_with_prefix convenience utilityEric Fried2019-09-172-0/+32
|/ | | | | | | | | | | | | | | There's a convenience wrapper around the lockutils.synchronized decorator that lets oslo.concurrency consumers set up a prefix to use for all lock files (presumably to obviate collisions with other consumers when using jejune lock names, like 'lock'). This commit adds an equivalent wrapper around lockutils.lock, the context manager counterpart to the lockutils.synchronized decorator. Note that the unit test added for lock_with_prefix is pretty bare; but it follows the precedent set by the existing tests. Future commits should make all these tests more thorough/robust. Change-Id: I4e723ee3be1e57c543684390b607c84388c6e930
* Some test cleanupEric Fried2019-09-173-142/+87
| | | | | | | | | | | | | | - Tests are based on oslotest.base.BaseTestCase, which uses the NestedTempFile fixture, which uses the TempDir fixture, which adds a cleanup routine to remove the base temporary directory it creates. There's therefore no need for tests to clean up this directory, so all the code that does that is removed. - The eventlet incarnation of tests was trying to make use of the `env` external without whitelisting it, resulting in an ugly red deprecation warning. This commit adds `env` to whitelist_externals in [testenv]. - A handful of typos in the Beowulf quote are corrected. Change-Id: I91cc52e00e0a918dadd2a3a771bd322b0f165ed2
* Replace git.openstack.org URLs with opendev.org URLsjacky062019-05-051-2/+2
| | | | | | | | 1. Thorough replacement of git.openstack.org URLs with their opendev.org counterparts. 2. dict_object.keys() is not required for *in* operator Change-Id: Ie1d0c828522ae92b01f28e7017fd7fd42c2e91e7
* Update hacking versionZhijunWei2019-01-211-5/+3
| | | | | | Use latest release 1.1.0 and compatible changes w.r.t pep8 Change-Id: Iec0c74ab679d6046b12f1d16ecd5cb6dfa74b3ed
* Merge "Add support for fair locks"3.29.0Zuul2018-11-052-6/+99
|\
| * Add support for fair locksChris Friesen2018-11-022-6/+99
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds support for a "fair" variant of the lock. When there are multiple entities within a single process that are blocked waiting for the lock the fair lock will ensure that they acquire the lock in FIFO order. For now at least, when fair locks are in use we don't support the "semaphores" argument. If external locks are enabled, the inter-process ordering will be determined by the underlying OS lock ordering and process scheduling. Change-Id: I37577becff4978bf643c65fa9bc2d78d342ea35a
* | ignore warning from bandit for using shell=Doug Hellmann2018-08-161-1/+1
| | | | | | | | | | | | | | | | | | The functions in this library are meant to wrap subprocess, and sometimes we do want to allow passing shell=True, so mark that line with nosec to ignore B604. Change-Id: Iab676449e543a4fdabee62efa9283940d30e3398 Signed-off-by: Doug Hellmann <doug@doughellmann.com>
* | Switch to stestrVu Cong Tuan2018-07-041-1/+1
| | | | | | | | | | | | | | | | | | | | According to Openstack summit session [1], stestr is maintained project to which all Openstack projects should migrate. Let's switch to stestr as other projects have already moved to it. [1] https://etherpad.openstack.org/p/YVR-python-pti Change-Id: I753f32ecf3275cf49d8c93bf648a6a26bc6da8e7
* | Switch pep8 job to python 3Ben Nemec2018-04-121-2/+2
| | | | | | | | | | | | | | | | pep8 under python 3 is more strict than under python 2, so to make sure our projects meet the more strict standards we want to run the pep8 jobs using python 3 by default. Change-Id: Ice3cffa36e9167f2f1f30337372cf9187af5e02e
* | Imported Translations from ZanataOpenStack Proposal Bot2018-03-011-4/+4
| | | | | | | | | | | | | | For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I38a631e8a24e8807dc59b14f368ab21caa02aba5
* | Mask passwords only when command execution failsprashkre2018-02-222-3/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | At many places, processutils.ssh_execute() is being invoked to run a command over ssh and output returned is parsed to get appropriate information. In this flow, unsanitized output is being expected where processutils.ssh_execute() was invoked but found that output like volume details(containing "password" string in its name) is being masked away with strutils.mask_password(stdout) even though no error occured during command execution. This is regression issue from patch[0]. In this fix, stdout and stderr in processutils.ssh_execute() will be masked only when ProcessExecutionError exception is thrown i.e. command execution failed due to some reasons. [0] https://github.com/openstack/oslo.concurrency/commit/ ae9e05bfc3d7ec867bd6ec78c301f11c2bdd0d5f Change-Id: I2ce344330905eef437ef3f89a2a01169a30df8ab Closes-Bug: #1482382
* | Add python_exec kwarg to processutils.execute()3.25.0Matthew Treinish2018-01-092-1/+19
|/ | | | | | | | | | | | | | | | | | | | | This commit adds a new kwarg to the process_utils.execute() function to specify the python executable to use when launching python to check prlimits. This is necessary when processutils.execute() is called from inside an API server running with uwsgi. In this case sys.executable is uwsgi (because uwsgi links libpython.so and is actually the interpreter) This doesn't work with the execute() function because it assumes the cpython interpreter CLI is used for the arguments it uses to call the prlimits module. To workaround this and enable API servers that may run under uwsgi to use this those applications can simply pass in an executable to use. Longer term it might be better to migrate the prlimits usage to call multiprocessing instead of subprocessing python. But that would require a more significant rewrite of both processutils and prlimit to facilitate that. Change-Id: I0ae60f0b4cc3700c783f6018e837358f0e053a09 Closes-Bug: #1712463
* Minor correction to docstrings3.22.0Brian Rosmaita2017-08-301-4/+6
| | | | | | | | Noticed that the generated docs were displaying the parameter to trycmd() in a misleading way, fixed the syntax there and for get_worker_count(). Change-Id: If989523be81e917ed62f525f2431f368dcf13c7a
* Windows: ensure exec calls don't block other greenthreadsLucian Petrut2017-08-162-3/+61
| | | | | | | | | | | | | | | | | | eventlet.green.subprocess is not actually greenthread friendly on Windows. It just uses the native subprocess.Popen in this case. For this reason, exec calls do not yield on Windows, blocking other greenthreads. This change avoids this issue by wrapping the 'communicate' call using eventlet.tpool. We're also ensuring that subprocess.Popen uses *native* threads internally in order to avoid deadlocks when passing data through stdin. Change-Id: Ic25fd1b61b5498f16e6049cbbe0877492f8aab4d Closes-Bug: #1709586
* Add debug log to indicate when external lock is takenGary Kotton2017-07-261-0/+3
| | | | | | | | | | In a case when a number of threads are concurrently accessing a resource that has a external lock there is no indication when one of the threads has taken the external lock. This patch adds in a debug log to try and add some clarity. Change-Id: Ibd7cfd3b39370b1a54f2ab50c75c6b92a876ae1a
* Merge "Remove log translations"3.21.0Jenkins2017-07-122-12/+2
|\
| * Remove log translationsloooosy2017-07-112-12/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | Log messages are no longer being translated. This removes all use of the _LE, _LI, and _LW translation markers to simplify logging and to avoid confusion with new contributions. See: http://lists.openstack.org/pipermail/openstack-i18n/2016-November/002574.html http://lists.openstack.org/pipermail/openstack-dev/2017-March/113365.html Change-Id: Iec2ef31f8543d51c8b6e1aed53f950800813c30b
* | Imported Translations from ZanataOpenStack Proposal Bot2017-07-125-105/+13
|/ | | | | | | For more information about this automatic import see: http://docs.openstack.org/developer/i18n/reviewing-translation-import.html Change-Id: Ia11093f4fd2fa12077a202872b3ed6ce79e34e83
* Check for SubprocessError by name on Python 3.xYuriy Taraday2017-06-291-6/+11
| | | | | | | | | | | | With eventlet SubprocessError raised by Popen seem to have different class from subprocess.SubprocessError accessible from test. I don't have proper solution for this, it seems somewhere old SubprocessError is cached and then eventlet overrides it, so it is not visible from test method context. This workaround seems to be good enough to unblock gate. Change-Id: If5ae0911e14671e05aca5e393c5cc183b72703d6 Closes-Bug: #1688201
* Using fixtures.MockPatch instead of mockpatch.Patchgecong19732017-03-091-4/+3
| | | | | | This module has been deprecated in favor of fixtures.MockPatch. Change-Id: Iee59fa82e727e0575e8b6b6afe042231aa8fa5b8
* pbr.version.VersionInfo needs package name (oslo.xyz and not oslo_xyz)Davanum Srinivas2017-02-061-1/+1
| | | | | | | | | | | | We need to pass into VersionInfo what one would expect from running: setup.py --name Right now we pass in say oslo_context and pbr fails if there is no .git in the python source tree Closes-Bug: #1662266 Change-Id: I6bb3594ad625e0538cc8c654d679886b8302719a
* Automatically convert process_input to bytes3.18.03.17.0Davanum Srinivas2017-01-122-2/+16
| | | | | | | | | | | | | | | | On py35, if process_input is a string we end with an Error: TypeError: memoryview: a bytes-like object is required, not 'str' A whole lot of code in os-brick/cinder/neutron currently need to be fixed. Hardest problem is tracking down every instance of this problem and trying to test/fix it at source. See example of cinder/os-brick/oslo.rootwrap/oslo.privep code problem here: http://logs.openstack.org/43/418643/2/check/gate-rally-dsvm-py35-cinder-nv/6efa7b5/logs/screen-n-cpu.txt.gz?level=ERROR It's just better to fix it in one spot and pass the correct thing to subprocess.communicate Change-Id: I88d1510a7ba4c020f73452f0b80e996c22b1edf1
* [TrivialFix] Replace 'assertTrue(a in b)' with 'assertIn(a, b)'howardlee2016-10-211-7/+7
| | | | Change-Id: Ib61f2f2c695f79c8eb0fa47a2d2d1bc66ccaeb7f
* Change assertTrue(isinstance()) by optimal assertgecong19732016-10-081-6/+6
| | | | | | | | | Some of tests use different method of assertTrue(isinstance(A, B)) or assertEqual(type(A), B). The correct way is to use assertIsInstance(A, B) provided by testtools Change-Id: Iad2ee275837c28b40c1062386db5a29969ee98dc
* Ignore prlimit argument on WindowsLucian Petrut2016-09-292-5/+32
| | | | | | | | | | | | The built in 'resource' module is used in order to apply process resource limits. This is not available on Windows. For the moment, we'll ignore this argument and log a warning message. In the future, we may provide a portable implementation for this feature. Change-Id: I6a82916eb68fbf8737b45a65f71cae1f835d12d1 Closes-Bug: #1627766
* Fix external lock tests on WindowsDenis Buliga2016-08-081-42/+55
| | | | | | | | | | | | | | | At the moment, those tests use fcntl to acquire file locks, which will fail on Windows. This change addresses this issue by adding some platform checks, using the appropriate functions when handling file locks. Note that we've avoided running private methods from fasteners. Also, we now avoid using os.fork, which won't work on Windows. Instead, we spawn new processes using multiprocessing.Process, using queues for communication between processes. Change-Id: I9839b9033c814280f6d1b53c5ed2643fd2bf8bf8
* Fix parameters of assertEqual are misplacedyanheven2016-07-232-39/+41
| | | | | | | | | | | | Many assertEqual sentences don't follow assertEqual(expected, actual), These misplaces have 2 impacts: 1, giving confusing messages when some tests failed. 2, mislead other developers, new test modules may follow these wrong pattern. This patch fix all of them. Change-Id: I00bb4801c208d0eb8078e8a34ae52f9bcfe0b1a6 Closes-Bug: #1604213
View Lorry Controller Status