summaryrefslogtreecommitdiff
path: root/taskflow/utils
Commit message (Collapse)AuthorAgeFilesLines
* Merge "Fix parsing of zookeeper jobboard backend options"Zuul2023-04-191-8/+21
|\
| * Fix parsing of zookeeper jobboard backend optionsGregory Thiemonge2023-01-121-8/+21
| | | | | | | | | | | | | | | | Fix the zookeeper backend options when values are passed as strings, a "False" string is now treated as the False boolean. Closes-Bug: #1999174 Change-Id: I048faf06d89ebf980efe0598e647f2ec89f65ada
* | Merge "Change StrictRedis usage to Redis"Zuul2023-02-171-4/+4
|\ \ | |/ |/|
| * Change StrictRedis usage to RedisTobias Urdin2022-09-291-4/+4
| | | | | | | | | | | | | | The StrictRedis class is only an alias for Redis in >= 3.0.0 Change-Id: Ief27531f120a50805053c214cb61bb4151678d70
* | Merge "Adapt to new jsonschema versions"Zuul2022-11-211-7/+8
|\ \ | |/
| * Adapt to new jsonschema versionsCorey Bryant2022-08-031-7/+8
| | | | | | | | | | | | | | | | | | | | | | | | This change provides fixes that were dectected by unit test failures with new jsonschema (and py310). The types argument has been removed in favor of providing a type_checker to jsonschema.validators.extend: https://github.com/python-jsonschema/jsonschema/issues/577 Closes-Bug: #1983412 Change-Id: I86f12b3d264320308e7f4841910fc21a6e8b3fa9
* | Remove sixTakashi Kajinami2022-05-187-69/+27
|/ | | | | | | | This library no longer supports Python 2, thus usage of six can be removed. This also removes workaround about pickle library used in Python 2 only. Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
* Replace deprecated import of ABCs from collections4.6.2Takashi Kajinami2021-07-171-2/+2
| | | | | | | | ABCs in collections should be imported from collections.abc and direct import from collections is deprecated since Python 3.3. Closes-Bug: #1936667 Change-Id: Ia6316894a67cc0ea7ee0d2110565b45546c813e7
* Drop use of deprecated collections classesSean McGinnis2020-04-141-3/+3
| | | | | | | | | | | Many classes have moved from collections to collections.abc. The original imports work, but emit deprecation warnings. Now that we are py3-only, we can drop any references to the older deprecated locations. Change-Id: I24b13c8654a33d99bf687a44a36cfdace39e3866 Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
* Zookeeper backend SSL supportussuri-em4.1.0Ann Kamyshnikova2020-03-061-0/+13
| | | | | | | | | | | | | | | | | | Bump kazoo version to 2.6.0 that supports SSL. Allow to configure ssl config options via conf * keyfile * keyfile_password * certfile * use_ssl * verify_certs https://kazoo.readthedocs.io/en/latest/api/client.html#kazoo.client.KazooClient This should be done in conformance with Redis jobboard backend, that already allows SSL settings. Change-Id: Ic2670c3c4caa93f69bc2d541c3701df2b6037d2e
* Update TaskFlow for networkx 2.xMichael Johnson2019-10-181-5/+0
| | | | | | | | | | | The networkx 2.x series has been out for two years now and supports python 3.6 and greater[1]. This patch updates TaskFlow to require a minimum of networkx 2.1. It also updates the code to support recent deprecation expiration introduced in the 2.4 release. [1] https://networkx.github.io/documentation/stable/news.html Change-Id: Ife31d353ba80824ebc63c8b21ee90943badc8da3
* Create KazooClient with taskflow loggerTheodoros Tsioutsias2018-08-161-0/+5
| | | | | | | | | KazooClient should use the taskflow logger and not the default one. Using the default logger means that the user cannot control its log level Change-Id: Id9549028522955ece0ae9f675c5ecc1c2fd816de Closes-Bug: #1785036
* Update various links in docsVu Cong Tuan2018-07-171-8/+8
| | | | | | Avoid redirects, s/http/https/. Change-Id: Id03bd49f250b48109967a0e04b92b75925f08932
* Fix code to support networkx > 1.0Michal Arbet2018-07-111-0/+5
| | | | | | | | | | | | | With the release of NetworkX 2.0 the reporting API was moved to view/iterator model. Many methods were moved from reporting lists or dicts to iterating over the information. Methods that used to return containers now return views and methods that returned iterators have been removed in networkx. Because of this change in NetworkX 2.0 , taskflow code have to be changed also to support networkx > 2.0 Change-Id: I23c226f37bd85c1e38039fbcb302a2d0de49f333 Closes-Bug: #1778115
* python3.0 has deprecated LOG.warnxhzhf2017-03-071-2/+2
| | | | | | | | python3.0 has deprecated LOG.warn https://docs.python.org/3/library/logging.html#logging.warning Closes-Bug: #1650843 Change-Id: Ib8473cbdb84547f385e619fe08ed723b1ee7bcdb
* Remove vim header from source fileshowardlee2016-11-091-2/+0
| | | | | | | | Trivial fix [H106] Don't put vim configuration in source files (off by default). Change-Id: I7e9e9930fd6f135074078637f5af6f403e75ff78
* Fix broken linkAndreas Jaeger2016-11-031-1/+1
| | | | | | | | http://docs.openstack.org/developer/taskflow/utils.html points to http://docs.openstack.org/developer/taskflow/kazoo.readthedocs.org/ - fix the broken URL. Change-Id: I6f6650cef94cea9d6f138925e5e8d2ed6ea177c3
* Use method ensure_tree from oslo.utilsChangBo Guo(gcb)2016-09-251-19/+0
| | | | | | Oslo.utils provides same function and just use it. Change-Id: Iac245d4d98c41edea5294a4d8842db69a42b3794
* Instead of a multiprocessing queue use sockets via asyncoreJoshua Harlow2016-05-241-0/+8
| | | | | | | | | | | | | | | | | | | | | | For a local process based executor usage currently to ensure that task emitted notifications are proxied we use the multi processing library and use its queue concept. This sadly creates a proxy process that gets associated, and this proxy process handles the queue and messages sent to and from it. Instead of doing this we can instead just create a temporary local socket using a random socket and have tasks (which are running in different processes) use that to communicate back any emitted notifications instead (and we can use the asyncore module to handle the emitted notifications since it handles the lower level socket reading, polling and dispatching). To ensure that the socket created is somewhat secure we use a similar process as the multi-processing library uses where we sign all messages with a hmac that uses a one time key that only the main process and the child process know about (and reject any messages that do not validate using this key). Change-Id: Iff9180054bf14495e5667af00ae2fafbdbc23791
* Remove deprecated things for 2.0 releaseJoshua Harlow2016-05-102-159/+0
| | | | Change-Id: Id9adbc50bd51adc77ce88f698ad0ea2ee63fc5e2
* Merge "Spice up WBE banner and add simple worker __main__ entrypoint"Jenkins2016-04-281-0/+107
|\
| * Spice up WBE banner and add simple worker __main__ entrypointJoshua Harlow2016-02-081-0/+107
| | | | | | | | Change-Id: Ifdf275a623352aa3e42fbf0aa9a4394b64b54337
* | Allow cachedproperty to avoid lockingJoshua Harlow2016-02-081-12/+21
|/ | | | | | | | | Under some cirumstances it is ok to recompute a value when raced to by multiple threads, for these cases allow the cachedproperty to avoid needing and using a lock. Change-Id: Icb71cb5ea280185cfb120671be900435dabcd50b
* Retrieve the store from flowdetails as well, if it existsGreg Hill2016-01-251-2/+9
| | | | | | | | Gives users a more permanent way to provide an initial set of arguments to a flow. Change-Id: Ib9c3d60882548120d467a645bbac9be78408bac3 Implements: blueprint flow-details-keep-store
* Allow for alterations in decider 'area of influence'Joshua Harlow2016-01-091-31/+0
| | | | | | | | Christmas came early. Closes-Bug: #1479466 Change-Id: I931d826690c925f022dbfffe9afb7bf41345b1d0
* Fix wrong usage of iter_utils.unique_seenJoshua Harlow2016-01-091-8/+9
| | | | | | | | | | This wasn't actually using the right data to derive uniqueness, so fix it to actually use the right entry to determine what to skip or what not to skip. Closes-Bug: #1525379 Change-Id: Ic23b6d03877f7714f6a3fb74adac0ba58cd97f0d
* Merge "Add ability of job poster/job iterator to wait for jobs to complete"Jenkins2016-01-091-0/+31
|\
| * Add ability of job poster/job iterator to wait for jobs to completeJoshua Harlow2015-12-161-0/+31
| | | | | | | | | | | | | | | | It is quite useful for someone who has posted a job to be able to easily wait for its completion (up to a given timeout) so add a wait method onto the job class to allow for this to be possible. Change-Id: Id3a7c724020962591e323da0febfd0c71d1acc50
* | Merge "Use the retrying lib. to do basic sqlalchemy engine validation"Jenkins2016-01-091-24/+0
|\ \
| * | Use the retrying lib. to do basic sqlalchemy engine validationJoshua Harlow2016-01-051-24/+0
| |/ | | | | | | | | | | | | | | Instead of a custom loop and backoff just use the retrying library to do this same code and delegate the loop complexity to it instead. Change-Id: Iaf02cc728d2a2cfc7077300e03d7ef25522717b7
* | Use shared util helper for driver name + config extractionJoshua Harlow2015-12-221-0/+13
|/ | | | Change-Id: I43465b8f5868e64bdf38d2873417a8a4a403a23b
* Merge "Change name of misc.ensure_dict to misc.safe_copy_dict"Jenkins2015-12-151-2/+7
|\
| * Change name of misc.ensure_dict to misc.safe_copy_dictJoshua Harlow2015-12-141-2/+7
| | | | | | | | Change-Id: I2ff3b9bb4d0b8143163e4bf370f10c74e79dbd31
* | Move validation of compiled unit out of compiler1.25.0Joshua Harlow2015-11-201-13/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | Instead of having the compiler do any validation on the graph it has created instead have the compiler just compile and have the engine that uses that compiled result do any post compilation validation instead. This makes it more clear that the compiler just compiles a flow (and tasks and nested flows) into a graph, and that is all that it does. Change-Id: I96a35d732dc2be9fc8bc8dc6466256a19ac2df6d
* | Merge "Move 'fill_iter' to 'iter_utils.fill'"Jenkins2015-11-181-7/+58
|\ \ | |/ |/|
| * Move 'fill_iter' to 'iter_utils.fill'Joshua Harlow2015-10-261-7/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This is better placed in the iterator utility module as it acts on iterables and provides its own iterator that fills up to (and potentially beyond) a provided iterator. Also adds some nice tests to make sure it keeps on working as expected; as well as tests for other parts of iter_utils to ensure they keep on working as expected as well. Change-Id: Ica90816cbdedfd87f3861a111d7a852655c1fb74
* | Add optional 'defer_reverts' behaviorGreg Hill2015-11-171-0/+8
| | | | | | | | | | | | | | | | | | | | | | This makes it possible to REVERT a subflow and have it also revert the parent flow if the parent flow doesn't have its own retry strategy. We will probably want to make this new behavior the default or only behavior in a future release. Change-Id: Iea5ac366380ba7396a87d0185703549fb0c2f825
* | Add a executor backed conductor and have existing impl. use itJoshua Harlow2015-11-122-0/+26
|/ | | | | | | | | | | | | | | | | | | This adds a executor backed job dispatching base class and has the existing blocking executor use it by running jobs and dispatching jobs into a sync executor. It also allows for dispatching jobs into a thread executor, or other executor via a new '_executor_factory' method that can generate executors (it can be overriden in the non-blocking executor to provide your own executors instances). This does alter the behavior in that now that jobs are dispatched into an executor we no longer can immediatly know if a job was dispatched and raised an exception or whether it will raise an exception in the future, so we now alter the 'local_dispatched' to just be a boolean that is used to determine if any dispatches happened (failure or not). Change-Id: I485770e8f4c85d3833892a453c9fb5168d8f0407
* feat: add max_dispatches arg to conductor's runSriram Madapusi Vasudevan2015-10-151-0/+15
| | | | | | | | | | | | | - This will cause the conductor to only do 'n' number of dispatches, after which it stops dispatching jobs. - This will allow the code that call the conductor, to monitor conductor.dispatching, and make a decision on what is to be done with it. Eg: Decomission a conductor, restart the conductor etc - Backward Compatible. Change-Id: I3386c7050806806b5ee44a74ba93e50515a5ab7b
* Simplify flow action engine compilation1.22.0Joshua Harlow2015-10-011-0/+13
| | | | | | | | | | | | | | Instead of the added complexity of discarding flow nodes we can simplify the compilation process by just retaining them and jumping over them in further iteration and graph and tree runtime usage. This change moves toward a model that does just this, which makes it also easier to in the future use the newly added flow graph nodes to do meaningful things (like use them as a point to change which flow_detail is used). Change-Id: Icb1695f4b995a0392f940837514774768f222db4
* Merge "Extend and improve failure logging"Jenkins2015-09-141-0/+8
|\
| * Extend and improve failure loggingJoshua Harlow2015-08-071-0/+8
| | | | | | | | | | | | | | | | | | Add the ability to show the intention, result, and state of predecessors of the atom that has failed so that more contextual information can be shown in the logs that are written. Change-Id: Ic77c0d4e94a147e54da74976c1d148aef82eccb3
* | Merge "Replace the tree 'pformat()' recursion with non-recursive variant"Jenkins2015-07-311-0/+42
|\ \ | |/
| * Replace the tree 'pformat()' recursion with non-recursive variantJoshua Harlow2015-07-271-0/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | This adjusted variant creates the same output but is hopefully easier to understand and follow than the recursive version. This version is also not limited by the python stack limit which is a general good thing to avoid being limited by. It also adds a bunch of tests to make sure the format is as expected under various tree structures. Change-Id: I2ae42c7c1bf72794800929250bcf6ccbe658230b
* | Bump futurist and remove waiting code in taskflowJoshua Harlow2015-07-251-88/+0
|/ | | | Change-Id: Ifc9780aa129a4a2804cead301a519895c2bfc0b5
* Merge "Remove **most** usage of taskflow.utils in examples"Jenkins2015-07-211-75/+0
|\
| * Remove **most** usage of taskflow.utils in examplesJoshua Harlow2015-07-171-75/+0
| | | | | | | | | | | | | | | | | | | | It appears folks are using the taskflow.utils code in there own code-bases (likely taking it from the examples) which we do not want to encourage, so remove the usage of **most** of taskflow.utils code from the examples so that people are less likely to copy/paste/reference it. Change-Id: I0ce3c520de347e3e746e7912aa1366a515458424
* | Merge "Remove kazoo hack/fix for issue no longer needed"Jenkins2015-07-211-7/+1
|\ \
| * | Remove kazoo hack/fix for issue no longer neededJoshua Harlow2015-06-291-7/+1
| | | | | | | | | | | | | | | | | | Depends-On: I5cf30d2952850de140f4bcc8bb3eac100ee8001e Change-Id: Ifdf6ec863a3596b6b5e2e58ea383112484b47c26
* | | Merge "Use encodeutils for exception -> string function"Jenkins2015-07-211-0/+35
|\ \ \
View Lorry Controller Status