summaryrefslogtreecommitdiff
path: root/taskflow/tests
Commit message (Collapse)AuthorAgeFilesLines
* Merge "Fix parsing of zookeeper jobboard backend options"Zuul2023-04-192-0/+111
|\
| * Fix parsing of zookeeper jobboard backend optionsGregory Thiemonge2023-01-122-0/+111
| | | | | | | | | | | | | | | | Fix the zookeeper backend options when values are passed as strings, a "False" string is now treated as the False boolean. Closes-Bug: #1999174 Change-Id: I048faf06d89ebf980efe0598e647f2ec89f65ada
* | Merge "Change StrictRedis usage to Redis"Zuul2023-02-171-1/+1
|\ \ | |/ |/|
| * Change StrictRedis usage to RedisTobias Urdin2022-09-291-1/+1
| | | | | | | | | | | | | | The StrictRedis class is only an alias for Redis in >= 3.0.0 Change-Id: Ief27531f120a50805053c214cb61bb4151678d70
* | Fix test_while_is_not with python 3.115.1.0Gregory Thiemonge2022-12-051-11/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The test_while_is_not function relied on strings and literals to test the 'while_is_not' function. But the while_is_not function uses the 'is' operator to test the equivalency of 2 objects. This triggers a bug with python 3.11 where using 'is' with literals is not advised (it is not recommended since python 3.8 [0]). The test now uses objects from a specific class to evaluate the while_is_not function. [0] https://docs.python.org/3/whatsnew/3.8.html#changes-in-python-behavior Change-Id: I38a0135aaf73e25aa20a11c0685d5c2a7b587a07
* | Merge "Quote string representations"5.0.0Zuul2022-06-134-8/+8
|\ \
| * | Quote string representationsMichael Johnson2022-06-104-8/+8
| |/ | | | | | | | | | | | | | | networkx 2.8.3 will raise errors if certain characters, such as a colon, in the node names and the string is not quoted. This patch double quotes flow, node, and task string representations to make sure there are not issues with these characters occuring in the names. Story: 2010083 Change-Id: Ib0941cddf14dde5d6b9f97fe0224d6e6f3975226
* | Remove sixTakashi Kajinami2022-05-1816-100/+61
|/ | | | | | | | This library no longer supports Python 2, thus usage of six can be removed. This also removes workaround about pickle library used in Python 2 only. Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
* Use LOG.warning instead of deprecated LOG.warnTakashi Kajinami2021-11-291-1/+1
| | | | | | | | | The LOG.warn method is deprecated[1] and the LOG.warning method should be used instead. [1] https://docs.python.org/3/library/logging.html#logging.warning Change-Id: I4321a489c56eb1aa650e776ee35d8f88d4d8910c
* Use unittest.mock instead of mockHervé Beraud2021-04-271-1/+1
| | | | | | | | The mock third party library was needed for mock support in py2 runtimes. Since we now only support py36 and later, we can use the standard lib unittest.mock module instead. Change-Id: Ib169e3deb7ddb2bc93a206ebec4043552281aa7f
* Add sentinel redis supportAnn Taraday2020-08-051-0/+34
| | | | | | | | | | | | | Redis client has an ability to connect to Redis server using Sentinel[1] (especially important for Redis clusters), but this ability was missing here. Allow to pass 'sentinel' variable to Redis conf, extend _make_client to use sentinel for that case. [1] - https://github.com/andymccurdy/redis-py#sentinel-support Change-Id: Ia8cc98e701435fd0231da3724f5d7108fc4f96f4
* Switch from unittest2 compat methods to Python 3.x methodsmelissaml2020-07-053-49/+49
| | | | | | | | | | | | | | | | | With the removal of Python 2.x we can remove the unittest2 compat wrappers and switch to assertCountEqual instead of assertItemsEqual We have been able to use them since then, because testtools required unittest2, which still included it. With testtools removing Python 2.7 support [3][4], we will lose support for assertItemsEqual, so we should switch to use assertCountEqual. [1] - https://bugs.python.org/issue17866 [2] - https://hg.python.org/cpython/rev/d9921cb6e3cd [3] - testing-cabal/testtools#286 [4] - testing-cabal/testtools#277 Change-Id: Iaa8251a1e9965a00fe99b7a740a104c011260340
* Update TaskFlow for networkx 2.xMichael Johnson2019-10-183-27/+26
| | | | | | | | | | | The networkx 2.x series has been out for two years now and supports python 3.6 and greater[1]. This patch updates TaskFlow to require a minimum of networkx 2.1. It also updates the code to support recent deprecation expiration introduced in the 2.4 release. [1] https://networkx.github.io/documentation/stable/news.html Change-Id: Ife31d353ba80824ebc63c8b21ee90943badc8da3
* Fix code to support networkx > 1.0Michal Arbet2018-07-112-8/+8
| | | | | | | | | | | | | With the release of NetworkX 2.0 the reporting API was moved to view/iterator model. Many methods were moved from reporting lists or dicts to iterating over the information. Methods that used to return containers now return views and methods that returned iterators have been removed in networkx. Because of this change in NetworkX 2.0 , taskflow code have to be changed also to support networkx > 2.0 Change-Id: I23c226f37bd85c1e38039fbcb302a2d0de49f333 Closes-Bug: #1778115
* Fix invalid json unit testBen Nemec2018-02-081-1/+5
| | | | | | | | | | Recent versions of oslo.serialization have made it possible to dump exceptions to JSON, which broke a unit test in taskflow that assumed exceptions were unserializable. This change switches to an explicitly unserializable class for that test. Change-Id: If6d19bc9fcf1f1813cb087d42dc7ba6a61c71b3d Closes-Bug: 1748241
* Update URLs in documents according to document migrationChangBo Guo(gcb)2017-07-131-1/+1
| | | | Change-Id: I9ca92fdcec388e02462332e04fe7c1bf8b5f64b8
* Fix process based executor task proxying-back eventsJoshua Harlow2017-07-112-0/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Let's dive into what the problem is here. First a description of what happens to a task that is to be executed in a external (but local) process via the process executor mechanism. When a task is about to be sent to execute in the external (but local) process its first cloned, this is mainly done so that its notification callbacks can be altered in a safe manner (ie not altering the original task object to do this) and that clone has its notifier emptied out. What replaces the clone's notifier callbacks though is a new object (that has a __call__ method so it looks like just another callback) that will send messages to the parent process (the one that has the engine in it) over a secure(ish) channel whenever the local task triggers its notifier notify() method. This allows for callbacks in the parent process to get triggered because once the messages recieved the original tasks notifier object has its notify() method called (therefore those callbacks do not really know the task they are getting messages from is executing out of process). The issue though is that if the ANY(*) event type is registered due to how it works in the notifier is that if the child/cloned notifier has the ANY event type registered and the cloned task calls notify() with a specific event this will cause the ANY callback (in the clone) to transmit a message *and* it will cause the *specific* event callback to also transmit a message back to the parent process. On the engine process side it will get 2 messages and trigger the callbacks 3 times (twice for the specific event callback because how the local notifier has the ANY callback registered and one more time when the local process also sends the same event based on its registration of the ANY event in the child process). This is not what is expected (the message rcved on the engine process should only trigger one callback to get triggered if the engine process task has no ANY callback registered or two engine process callbacks to get triggered if the engine process task has the ANY callback registered). Closes-Bug: #1537948 Change-Id: I271bf1f23ad73df6c177cf00fd902c4881ba44ae
* Replace assertRaisesRegexp with assertRaisesRegexVu Cong Tuan2017-06-0312-100/+100
| | | | | | | | | | assertRaisesRegexp was renamed to assertRaisesRegex in Py3.2 For more details, please check: https://docs.python.org/3/library/ unittest.html#unittest.TestCase.assertRaisesRegex Change-Id: I89cce19e80b04074aab9f49a76c7652acace78b3 Closes-Bug: #1436957
* do not allow redis job reclaim by same ownerRick van de Loo2017-05-131-0/+23
| | | | | | | | | | | | | | | Running a nonblocking conductor or two conductors on the same host will re-execute the same job multiple times with the current implementation of 'claim' for the redis jobboard backend. This is different from the ZooKeeper jobboard backend, there the same owner of a job is not allowed to reclaim the job again (https://github.com/openstack/taskflow/blob/master/taskflow/jobs/backends/impl_zookeeper.py#L554). If the same owner is allowed to reclaim the job again there can be no concurrent execution on the same owner because all jobs will be re-claimed and re-executed by the same owner every pass as long as it's on the jobboard. To reproduce this behavior: - Use the redis jobboard backend - Create a flow with a task that sleeps 10 seconds in the execute method - Post that flow as a job - Run a nonblocking conductor It will claim and execute the same job multiple times in a loop until the first worker is finished and consumes the job. After this change it will not re-execute the same job multiple times. Change-Id: I4f6c364211500e510fc496f23b03ce056771417d
* python3.0 has deprecated LOG.warnxhzhf2017-03-072-13/+13
| | | | | | | | python3.0 has deprecated LOG.warn https://docs.python.org/3/library/logging.html#logging.warning Closes-Bug: #1650843 Change-Id: Ib8473cbdb84547f385e619fe08ed723b1ee7bcdb
* Use assertIs(Not)None to check for Nonehowardlee2016-11-072-6/+6
| | | | | | | | | | | | | [H203] Use assertIs(Not)None to check for None (off by default) Unit test assertions tend to give better messages for more specific assertions. As a result, assertIsNone(...) is preferred over assertEqual(None, ...) and assertIs(None, ...), and assertIsNotNone(...) is preferred over assertNotEqual(None, ...) and assertIsNot(None, ...). Off by default. Trivial fix Change-Id: I5b6ac7d99f0689843eb98cb3e9b9b10531322640
* Using assertIsNone() instead of assertIs(None, ..)qinchunhua2016-10-101-2/+2
| | | | | | | | | | Following OpenStack Style Guidelines[1]: http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises [H203] Unit test assertions tend to give better messages for more specific assertions. As a result, assertIsNone(...) is preferred over assertEqual(None, ...) and assertIs(..,None). Change-Id: I52ecf3f4945c4cb99f6350afca1c51d88b16bb8d
* Some classes not define __ne__() built-in functionJi-Wei2016-08-311-0/+3
| | | | | | | | | | Some classes defines __eq__() built-in function, but does not define __ne__() built-in function, so self.assertEqual works but self.assertNotEqual does not work at all in this test case in python2. This patch fixes it. Change-Id: I3e4f213081268bad44583a63a84795d39094117f Closes-Bug: #1586268
* Replace assertEqual(None, *) with assertIsNone in testsweiweigu2016-07-124-8/+8
| | | | | | | | Replace assertEqual(None, *) with assertIsNone in tests to have more clear messages in case of failure. Change-Id: I74452af6d840bcf612fd3bb2521db9134460dd63 Closes-Bug: #1280522
* Merge "Make tests less dependent on transient state"Jenkins2016-07-081-3/+2
|\
| * Make tests less dependent on transient stateGreg Hill2016-06-021-3/+2
| | | | | | | | | | | | | | | | | | Having to update the endpoint count every time you add a test class is really obnoxious and leads to a ton of pointless rebasing. Now we just check that it finds at least the task it knows about and call that good. Change-Id: I96b8c6cd6cbc1fdc58dee4b18cab5699e3daa844
* | Merge "Fix some misspellings in the function name and descriptions"Jenkins2016-07-082-3/+3
|\ \
| * | Fix some misspellings in the function name and descriptionsPablo Iranzo Gómez2016-06-302-3/+3
| |/ | | | | | | Change-Id: I7e3451feb94b1f25b00c5e7b197bb6b527548306
* | Add tests to verify kwargs behavior on revert validationGreg Hill2016-06-091-23/+41
|/ | | | | | | | | | | A bug was reported where a task's revert method was throwing an 'extra args' exception even though it had a **kwargs to slurp up any extra args. I added some tests to duplicate the behavior but the tests all pass. I'm hoping pushing this up will cause CI to reproduce the bug so I can diagnose it further. Change-Id: I9f4d0f065f11370f7981e93349f473df2bb1015d Closes-Bug: #1589848
* Merge "Add a simple sanity test for pydot outputting"Jenkins2016-05-311-0/+23
|\
| * Add a simple sanity test for pydot outputtingJoshua Harlow2016-05-201-0/+23
| | | | | | | | | | | | | | | | Depends-On: Ia633ccd1dca94f70b05ae4376a1c3a3f252a9923 Related-Bug: #1561656 Change-Id: I930a1ee9c1d17a12328d920bbe02ea0b77947295
* | Instead of a multiprocessing queue use sockets via asyncoreJoshua Harlow2016-05-245-13/+153
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For a local process based executor usage currently to ensure that task emitted notifications are proxied we use the multi processing library and use its queue concept. This sadly creates a proxy process that gets associated, and this proxy process handles the queue and messages sent to and from it. Instead of doing this we can instead just create a temporary local socket using a random socket and have tasks (which are running in different processes) use that to communicate back any emitted notifications instead (and we can use the asyncore module to handle the emitted notifications since it handles the lower level socket reading, polling and dispatching). To ensure that the socket created is somewhat secure we use a similar process as the multi-processing library uses where we sign all messages with a hmac that uses a one time key that only the main process and the child process know about (and reject any messages that do not validate using this key). Change-Id: Iff9180054bf14495e5667af00ae2fafbdbc23791
* | Remove deprecated things for 2.0 releaseJoshua Harlow2016-05-103-29/+18
| | | | | | | | Change-Id: Id9adbc50bd51adc77ce88f698ad0ea2ee63fc5e2
* | Merge "Allow for specifying green threaded to parallel engine"1.32.0Jenkins2016-05-062-5/+2
|\ \
| * | Allow for specifying green threaded to parallel engineJoshua Harlow2016-05-032-5/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently when a string is passed to the parallel engine it will only know how to create a process or a native thread based executor. The futurist library also supports making a green thread based executor, so support creating it. This will save glance some code that they have to create a executor based on different options (one of those is a green option). Change-Id: I15c164a38b4445d28eb6062aed6c56cce0e0364b
* | | Make conductor.stop stop the running engine gracefullyGreg Hill2016-05-033-1/+60
|/ / | | | | | | | | | | | | | | | | | | | | | | Previously stopping the conductor would only prevent it from accepting new jobs. This change makes it abort the current job by telling the engine to stop processing tasks after the current ones finish. This allows for conductors to gracefully exit when receiving a kill signal (although the signal handling is not done automatically yet). Change-Id: Ie6ddcbb2df4508ad1e3f6698c6f4cb2fc26a278f
* | Merge "Add rundimentary and/or non-optimized job priorities"Jenkins2016-04-281-0/+1
|\ \
| * | Add rundimentary and/or non-optimized job prioritiesJoshua Harlow2016-01-091-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Allow for providing job priority on job posting and order iteration by priority (this does not store in any special way different priorities under different keys/paths, but it could be done that way in the future). This is backwards compat in that all old jobs posted without a priority assume a normal/default priority. Change-Id: Ie670023d624f4442a16a1278a3aee0d88b102842
* | | Merge "Ensure that the engine finishes up even under sent-in failures"Jenkins2016-04-281-0/+30
|\ \ \
| * | | Ensure that the engine finishes up even under sent-in failuresJoshua Harlow2016-01-201-0/+30
| | | | | | | | | | | | | | | | Change-Id: Ic16c854d285398c688f132697c3bb7e637feb9a8
* | | | Merge "Add some basic/initial engine statistics"Jenkins2016-04-281-1/+1
|\ \ \ \
| * | | | Add some basic/initial engine statisticsJoshua Harlow2016-02-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It can be quite nice to expose a basic set of metrics about the internals of an engine, including the time in each state, and how long the engine is active for and likely more in the future. To start add a engine statistics property and gather some basic timing data and place this data into this property for access (and/or introspection) by users. Part of blueprint gather-engine-statistics Change-Id: Ibc3c78755bd8ae779b52fc4772519f243a521576
* | | | | Merge "Use a automaton machine for WBE request state machine"Jenkins2016-04-281-2/+2
|\ \ \ \ \
| * | | | | Use a automaton machine for WBE request state machineJoshua Harlow2016-02-211-2/+2
| | |_|_|/ | |/| | | | | | | | | | | | | Change-Id: Ib26ff4418ab1128a578519be964c4d39cbd1d2f4
* | | | | Allow for revert to have a different argument list from executeGreg Hill2016-02-264-1/+65
|/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Also allows for people to create Atom's with a different rebind or requires structure for the revert method, if desired. Implements blueprint: seperate-revert-args Change-Id: Ie7d13c8000ef08ff303481d486d1ba1cfbdeea44
* | | | Add WBE worker expiryJoshua Harlow2016-02-142-5/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a worker hasn't responded to a notification request for a given amount of time remove it from being a useable worker that we can match task submissions to. Change-Id: I596bccc1c42f83ee79136dd27bc87039154ff7b1
* | | | Some WBE protocol/executor cleanupsJoshua Harlow2016-02-144-32/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove some of the usage of @property as none of these objects are publicly exposed (or have docstrings on them) to save some space/lines of code that aren't really adding any benefit. Use less **kwargs when we know exactly what the keyword arguments will or will not be. Being explicit makes it easier to understand these functions (vs not knowing what the arguments can or can't be). Removes base worker finder because right now we only have one implementation (at some point we will have two) but we can just wait to add a base class until then. Change-Id: I7107ff6b77a355b4c5d301948355fb6386605388
* | | | Remove need for separate notify threadJoshua Harlow2016-02-141-3/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of having a periodic notification thread that will drop messages to try to find workers we can just have this same work be done in the periodically called on_wait callback that is already used for expiring and matching workers to new/updated workers. This avoids having one more thread that doesn't do all that much (and activating it during waiting calls will be often enough to achieve its goal in life). Change-Id: If80233d13d914f2ed3665001a27627b78e6ee780
* | | | Fix for WBE sporadic timeout of tasksJoshua Harlow2016-02-052-88/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes the sporadic of tasks that would happen under certain circumstances. What happened was that a new worker notification would be sent to a callback while at the same time a task submission would come in and there would be a small race period where the task would insert itself into the requests cache while the callback was processing. So to work around this the whole concept of a requests cache was revamped and now the WBE executor just maintains its own local dictionary of ongoing requests and accesses it safely. During the on_wait function that is periodically called by kombu the previous expiry of work happens but now any requests that are pending are matched to any new workers that may have appeared. This avoids the race (and ensures that even if a new worker is found but a submission is in progress that the duration until that submission happens will only be until the next on_wait call happens). Related-Bug: #1431097 Change-Id: I98b0caeedc77ab2f7214847763ae1eb0433d4a78
* | | | Merge "Handle cases where exc_args can't be serialized as JSON in the WBE"Jenkins2016-02-032-0/+18
|\ \ \ \
View Lorry Controller Status