summaryrefslogtreecommitdiff
path: root/taskflow/engines
Commit message (Collapse)AuthorAgeFilesLines
* Replace abc.abstractproperty with property and abc.abstractmethodljhuang2022-08-031-2/+4
| | | | | | | | | Replace abc.abstractproperty with property and abc.abstractmethod, as abc.abstractproperty has been deprecated since python3.3[1] [1]https://docs.python.org/3.8/whatsnew/3.3.html?highlight=deprecated#abc Change-Id: I1bcecd99d8856c26621a5304d9f7f01f8f111918
* Merge "Quote string representations"5.0.0Zuul2022-06-131-1/+2
|\
| * Quote string representationsMichael Johnson2022-06-101-1/+2
| | | | | | | | | | | | | | | | networkx 2.8.3 will raise errors if certain characters, such as a colon, in the node names and the string is not quoted. This patch double quotes flow, node, and task string representations to make sure there are not issues with these characters occuring in the names. Story: 2010083 Change-Id: Ib0941cddf14dde5d6b9f97fe0224d6e6f3975226
* | Remove sixTakashi Kajinami2022-05-1813-76/+36
|/ | | | | | | | This library no longer supports Python 2, thus usage of six can be removed. This also removes workaround about pickle library used in Python 2 only. Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
* Merge "Use LOG.warning instead of deprecated LOG.warn"Zuul2021-12-211-2/+2
|\
| * Use LOG.warning instead of deprecated LOG.warnTakashi Kajinami2021-11-291-2/+2
| | | | | | | | | | | | | | | | | | The LOG.warn method is deprecated[1] and the LOG.warning method should be used instead. [1] https://docs.python.org/3/library/logging.html#logging.warning Change-Id: I4321a489c56eb1aa650e776ee35d8f88d4d8910c
* | Fix minor typo in ActionEngine exception messageEric Harney2021-12-021-1/+1
|/ | | | Change-Id: I66b9937df5c58561792328e08f836dc05927e7f1
* Merge "Update TaskFlow for networkx 2.x"3.8.0Zuul2019-11-156-16/+16
|\
| * Update TaskFlow for networkx 2.xMichael Johnson2019-10-186-16/+16
| | | | | | | | | | | | | | | | | | | | | | The networkx 2.x series has been out for two years now and supports python 3.6 and greater[1]. This patch updates TaskFlow to require a minimum of networkx 2.1. It also updates the code to support recent deprecation expiration introduced in the 2.4 release. [1] https://networkx.github.io/documentation/stable/news.html Change-Id: Ife31d353ba80824ebc63c8b21ee90943badc8da3
* | Fix python3.8 hmac compatibilityEric Harney2019-09-031-1/+2
|/ | | | | | | | | The "digestmod" parameter must be passed in for Python 3.8. md5 was chosen here because this is what was implicitly used for Python < 3.8. Change-Id: I17eca0692c4a89ff90cfa9d4ef3947d2c9ee23e7
* set default python to python3Doug Hellmann2018-04-171-1/+1
| | | | | | | | | | | | | | | | | | Set the default python to python3 except for the py27 environment. We have to set that explicitly to override the new default. Have py27 and py3* jobs run same commands by removing the commands entry for py27. The pep8 rules under python 3 are different, so we need to fix a couple of issues with blank space. The E721 rule is ignored entirely because we are not yet using a version of flake8 that supports ignoring individual rules on specific lines, although the instructions for that are added so that when we upgrade flake8 to a newer version we can remove the global rule. Change-Id: Id236b0632a83a188958130d04f1e9c476c5ace65 Signed-off-by: Doug Hellmann <doug@doughellmann.com>
* Merge "Fix process based executor task proxying-back events"Jenkins2017-07-111-9/+35
|\
| * Fix process based executor task proxying-back eventsJoshua Harlow2017-07-111-9/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Let's dive into what the problem is here. First a description of what happens to a task that is to be executed in a external (but local) process via the process executor mechanism. When a task is about to be sent to execute in the external (but local) process its first cloned, this is mainly done so that its notification callbacks can be altered in a safe manner (ie not altering the original task object to do this) and that clone has its notifier emptied out. What replaces the clone's notifier callbacks though is a new object (that has a __call__ method so it looks like just another callback) that will send messages to the parent process (the one that has the engine in it) over a secure(ish) channel whenever the local task triggers its notifier notify() method. This allows for callbacks in the parent process to get triggered because once the messages recieved the original tasks notifier object has its notify() method called (therefore those callbacks do not really know the task they are getting messages from is executing out of process). The issue though is that if the ANY(*) event type is registered due to how it works in the notifier is that if the child/cloned notifier has the ANY event type registered and the cloned task calls notify() with a specific event this will cause the ANY callback (in the clone) to transmit a message *and* it will cause the *specific* event callback to also transmit a message back to the parent process. On the engine process side it will get 2 messages and trigger the callbacks 3 times (twice for the specific event callback because how the local notifier has the ANY callback registered and one more time when the local process also sends the same event based on its registration of the ANY event in the child process). This is not what is expected (the message rcved on the engine process should only trigger one callback to get triggered if the engine process task has no ANY callback registered or two engine process callbacks to get triggered if the engine process task has the ANY callback registered). Closes-Bug: #1537948 Change-Id: I271bf1f23ad73df6c177cf00fd902c4881ba44ae
* | turn on warning-is-error in doc buildDoug Hellmann2017-07-081-2/+4
|/ | | | | Change-Id: Id87a1b17c1986d0e30629fed13bdfbecef4a3603 Signed-off-by: Doug Hellmann <doug@doughellmann.com>
* Merge "python3.0 has deprecated LOG.warn"Jenkins2017-05-034-29/+31
|\
| * python3.0 has deprecated LOG.warnxhzhf2017-03-074-29/+31
| | | | | | | | | | | | | | | | python3.0 has deprecated LOG.warn https://docs.python.org/3/library/logging.html#logging.warning Closes-Bug: #1650843 Change-Id: Ib8473cbdb84547f385e619fe08ed723b1ee7bcdb
* | Fix py35 test failureChangBo Guo(gcb)2017-05-031-1/+1
|/ | | | | | | | dict.keys() returns object dict_keys on Python 3, can't work well with kombu recently, convert it to list before futhur processing. Closes-Bug: #1687855 Change-Id: I9dcfadd8f16eccce06f4cde3b2a6669f28a3f8bc
* Protect storage better against external concurrent accessJoshua Harlow2017-01-091-44/+50
| | | | | | | | | Lock down the various state machine action handling functions so that they are ensured correct (and consistent) access to the storage layer when they are modifiying and/or reading it. Change-Id: Ie893a44aa963ab515f19e77f9904f49c843cb4e5
* Rename engine analyzer to be named selectorJoshua Harlow2017-01-035-67/+67
| | | | | | | | | | | | | This moves out the engine next to run (or revert) selection process to a single class that just does this and moves out the common functions the analyzer class provided to the runtime object (which all components can access). This makes it easier to adjust the selection algorithm in different ways. Change-Id: I091c69297a7bff60729791d3ca6c3fae14d6eea5
* Merge "Add logging around metadata, ignore tallying + history"Jenkins2016-10-112-24/+47
|\
| * Add logging around metadata, ignore tallying + historyJoshua Harlow2016-08-012-24/+47
| | | | | | | | | | | | | | | | This can be useful to look at for debugging purposes when trying to understand how (and if) a atom is being ignored by a set of deciders. Change-Id: I550a2f7f361e516c255683d725b2f3a8130a0a86
* | Some classes not define __ne__() built-in functionJi-Wei2016-08-311-0/+3
|/ | | | | | | | | | Some classes defines __eq__() built-in function, but does not define __ne__() built-in function, so self.assertEqual works but self.assertNotEqual does not work at all in this test case in python2. This patch fixes it. Change-Id: I3e4f213081268bad44583a63a84795d39094117f Closes-Bug: #1586268
* remove unused LOGji-xuepeng2016-07-102-6/+0
| | | | | | This is to remove unused LOG to keep code clean. Change-Id: Ie44659f3b7af9612f127f7d46bb2c1481b6a634f
* Merge "Use the full 'get_execute_failures' vs the shortname"Jenkins2016-07-081-6/+6
|\
| * Use the full 'get_execute_failures' vs the shortnameJoshua Harlow2016-05-281-6/+6
| | | | | | | | | | | | | | | | The usage of the full name is easier to read and easier to understand in the surronding code so prefer to use it. Change-Id: I1c43704ae7dfdba90217ce9a1c60cef8e2ce8632
* | Fix some misspellings in the function name and descriptionsPablo Iranzo Gómez2016-06-303-6/+6
| | | | | | | | Change-Id: I7e3451feb94b1f25b00c5e7b197bb6b527548306
* | Split revert/execute missing args messagesGreg Hill2016-05-261-9/+12
|/ | | | | | | | | To make it more obvious when a revert method is missing a dependency, we now identify which method was missing the required paramater. Change-Id: I24b0d0b01dbd239b92ba444f6ef44273c5ac25d3 Closes-Bug: 1581616
* Instead of a multiprocessing queue use sockets via asyncoreJoshua Harlow2016-05-243-412/+724
| | | | | | | | | | | | | | | | | | | | | | For a local process based executor usage currently to ensure that task emitted notifications are proxied we use the multi processing library and use its queue concept. This sadly creates a proxy process that gets associated, and this proxy process handles the queue and messages sent to and from it. Instead of doing this we can instead just create a temporary local socket using a random socket and have tasks (which are running in different processes) use that to communicate back any emitted notifications instead (and we can use the asyncore module to handle the emitted notifications since it handles the lower level socket reading, polling and dispatching). To ensure that the socket created is somewhat secure we use a similar process as the multi-processing library uses where we sign all messages with a hmac that uses a one time key that only the main process and the child process know about (and reject any messages that do not validate using this key). Change-Id: Iff9180054bf14495e5667af00ae2fafbdbc23791
* Remove deprecated things for 2.0 releaseJoshua Harlow2016-05-102-131/+32
| | | | Change-Id: Id9adbc50bd51adc77ce88f698ad0ea2ee63fc5e2
* Merge "Allow for specifying green threaded to parallel engine"1.32.0Jenkins2016-05-062-0/+27
|\
| * Allow for specifying green threaded to parallel engineJoshua Harlow2016-05-032-0/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | Currently when a string is passed to the parallel engine it will only know how to create a process or a native thread based executor. The futurist library also supports making a green thread based executor, so support creating it. This will save glance some code that they have to create a executor based on different options (one of those is a green option). Change-Id: I15c164a38b4445d28eb6062aed6c56cce0e0364b
* | Attempt to cancel active futures when suspending is underwayJoshua Harlow2016-05-031-9/+40
|/ | | | | | | | | | | | | | Instead of waiting on futures to complete that may still be in an executors backlog (aka, not even started yet) we can avoid even executing (and waiting on these to finish) by forcing them to be cancelled and handling this when analyzing the futures result (by doing nothing with the cancelled result of a future). This saves some time when doing suspension with executors that have a very small pool of workers that actually run tasks/atoms. Change-Id: Ie79466c44ac6af5ff8936f192ebce4428aa12d98
* Merge "Have the storage class provide a 'change_flow_state' method"Jenkins2016-04-281-13/+9
|\
| * Have the storage class provide a 'change_flow_state' methodJoshua Harlow2015-08-151-13/+9
| | | | | | | | | | | | | | | | | | | | Instead of needing the engine method to lock, fetch, check and then update the flow state it is more appropriate if the storage class does all these actions itself (therefore hiding this change from the engine code, making the engine code that much smaller). Change-Id: I2a289c2bcabe76728fa8eb26265ce168abf81b7c
* | Merge "Ensure that the engine finishes up even under sent-in failures"Jenkins2016-04-281-0/+7
|\ \
| * | Ensure that the engine finishes up even under sent-in failuresJoshua Harlow2016-01-201-0/+7
| | | | | | | | | | | | Change-Id: Ic16c854d285398c688f132697c3bb7e637feb9a8
* | | Merge "Add some basic/initial engine statistics"Jenkins2016-04-283-21/+87
|\ \ \
| * | | Add some basic/initial engine statisticsJoshua Harlow2016-02-013-21/+87
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It can be quite nice to expose a basic set of metrics about the internals of an engine, including the time in each state, and how long the engine is active for and likely more in the future. To start add a engine statistics property and gather some basic timing data and place this data into this property for access (and/or introspection) by users. Part of blueprint gather-engine-statistics Change-Id: Ibc3c78755bd8ae779b52fc4772519f243a521576
* | | | Merge "Spice up WBE banner and add simple worker __main__ entrypoint"Jenkins2016-04-281-71/+49
|\ \ \ \
| * | | | Spice up WBE banner and add simple worker __main__ entrypointJoshua Harlow2016-02-081-71/+49
| | | | | | | | | | | | | | | | | | | | Change-Id: Ifdf275a623352aa3e42fbf0aa9a4394b64b54337
* | | | | Merge "Use a automaton machine for WBE request state machine"Jenkins2016-04-282-38/+90
|\ \ \ \ \
| * | | | | Use a automaton machine for WBE request state machineJoshua Harlow2016-02-212-38/+90
| | | | | | | | | | | | | | | | | | | | | | | | Change-Id: Ib26ff4418ab1128a578519be964c4d39cbd1d2f4
* | | | | | Merge "Allow for revert to have a different argument list from execute"Jenkins2016-04-283-10/+21
|\ \ \ \ \ \
| * | | | | | Allow for revert to have a different argument list from executeGreg Hill2016-02-263-10/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Also allows for people to create Atom's with a different rebind or requires structure for the revert method, if desired. Implements blueprint: seperate-revert-args Change-Id: Ie7d13c8000ef08ff303481d486d1ba1cfbdeea44
* | | | | | | Add the ability to skip resolving from activatingJoshua Harlow2016-04-042-10/+25
|/ / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It can be useful for debugging a failure to be able to stop when a failure is detected so that the failure can be examined in more detail. This adds the ability to have a 'never_resolve' engine option which will skip such resolution attempts for these types of scenarios. Change-Id: If2386cba970e14c7168a6ff2d4fd4786bcd8e989
* | | | | | Refactor Atom/BaseTask/Task/Retry class hierarchyGreg Hill2016-02-182-2/+2
|/ / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Moved common argument mapping logic to Atom * Removed BaseTask * Removed duplicated logic from subclasses Change-Id: I275745adb80cecb0c35c1230eac76436bf3b0157
* | | | | Add WBE worker expiryJoshua Harlow2016-02-144-7/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a worker hasn't responded to a notification request for a given amount of time remove it from being a useable worker that we can match task submissions to. Change-Id: I596bccc1c42f83ee79136dd27bc87039154ff7b1
* | | | | Some WBE protocol/executor cleanupsJoshua Harlow2016-02-143-107/+69
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove some of the usage of @property as none of these objects are publicly exposed (or have docstrings on them) to save some space/lines of code that aren't really adding any benefit. Use less **kwargs when we know exactly what the keyword arguments will or will not be. Being explicit makes it easier to understand these functions (vs not knowing what the arguments can or can't be). Removes base worker finder because right now we only have one implementation (at some point we will have two) but we can just wait to add a base class until then. Change-Id: I7107ff6b77a355b4c5d301948355fb6386605388
* | | | | Remove need for separate notify threadJoshua Harlow2016-02-142-46/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of having a periodic notification thread that will drop messages to try to find workers we can just have this same work be done in the periodically called on_wait callback that is already used for expiring and matching workers to new/updated workers. This avoids having one more thread that doesn't do all that much (and activating it during waiting calls will be often enough to achieve its goal in life). Change-Id: If80233d13d914f2ed3665001a27627b78e6ee780
* | | | | Don't bother scanning for workers if no new messages arrivedJoshua Harlow2016-02-112-7/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If the worker finder has not gotten any new notification messages letting it know about new (or updated) workers we can just skip trying to match existing waiting work to workers as without messages being processed the match will still not work (as the worker data doesn't change without those messages being processed). Change-Id: I41d50c676f04f85c49a03d9d503da1955af45f7d
View Lorry Controller Status