| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Set the default python to python3 except for the py27 environment. We
have to set that explicitly to override the new default.
Have py27 and py3* jobs run same commands by removing the commands entry
for py27.
The pep8 rules under python 3 are different, so we need to fix a
couple of issues with blank space. The E721 rule is ignored entirely
because we are not yet using a version of flake8 that supports
ignoring individual rules on specific lines, although the instructions
for that are added so that when we upgrade flake8 to a newer version
we can remove the global rule.
Change-Id: Id236b0632a83a188958130d04f1e9c476c5ace65
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
|
| |
|
|
|
|
|
|
|
| |
When a worker hasn't responded to a notification
request for a given amount of time remove it from
being a useable worker that we can match task submissions
to.
Change-Id: I596bccc1c42f83ee79136dd27bc87039154ff7b1
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of having the dispatcher target be a tuple
or a single callback, have it be an actual type with
comments as to what the types fields are and how they
are used.
This makes it more obvious as to what those fields are
and how they are used so that it is easier to understand
how the WBE engine and components work.
Change-Id: I5541ccd5a7aa6ae73ed9adceeac2c0524e2a1dc9
|
| |
|
|
|
|
|
|
|
|
|
| |
Instead of having a nominally useful single-threaded storage
unit that uses a dummy r/w lock and a multi-threaded storage
unit just have the storage unit by default protect itself from
multi-threading calls being used on it (with the appropriate
reader/writer locks being activated to make this work
correctly).
Change-Id: Ib6879edb465156a8e54fd5b4002550d1cec49137
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Link the engine/worker 'retry_options' to the proxy which has
linkage to what the defaults are and to where more information
can be gathered about the option (which really gets sent in to
kombu) so that users can more easily understand it.
Also removes showing the docs about the following as its
more of an internal API/class not meant for public consumption:
* Module: taskflow.engines.worker_based.executor
Change-Id: Ib16ee10765ec6b9a0af320bd6818d9649c2485b1
|
| |
|
|
|
|
|
|
|
| |
Use the kombu provided ensure() decorator/wrapper along
with sensible default settings to ensure that retries are
attempted when kombu detects recoverable connection or
recoverable channel errors have occurred.
Change-Id: If47f72d02561d0b5d556ac386869a6122c8b871d
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
To enable a parallel process executor to be used without
having to pass in a futures executor allow for the executor
option to be a string 'processes' (or similar) that will cause
the default parallel process executor to be used automatically.
Also allow for a 'threads' string that ensure a parallel thread
executor is used to match the ability to uses processes.
This also adjusts the WBE engine to have a similar executor fetching
function (which in the WBE case now validates a provided executor to
be of the desired type).
Change-Id: I54a82584c32c697922507b4f6e01ea7b8acc73c6
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since we support various executors (threaded and distributed)
the next best executor when a threaded executor will not perform
and a distributed one requires to much setup is a local process
based one so it would be great to support this where we can.
Things that are currently (likely never) not going to work:
* Non-pickleable/non-copyable tasks
* Tasks that return non-pickleable/non-copyable results
* Tasks that use non-pickleable/non-copyable args/kwargs
Part of blueprint process-executor
Change-Id: I966ae01d390c7217b858db3feb2db949ce5c08d1
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
To avoid having one set of options coming from `engine_conf`
and another set of options coming from `kwargs` and another set
coming from `engine_conf` if it is a URI just start to shift
toward `engine_conf` being deprecated and `engine` being a string
type only (or a URI with additional query parameters) and having
any additional **kwargs that are provided just get merged into the
final engine options.
This adds a new helper function that handles all these various
options and adds in a keyword argument `engine` that will be shifted
to in a future version (in that future version we can also then
remove the `engine_conf` and just stick to a smaller set of option
mechanisms).
It also adjusts all examples to use this new and more easier to
understand format and adjusts tests, conductor interface to use
this new more easily understandable style of getting an engine.
Change-Id: Ic7617057338e0c63775cf38a24643cff6e454950
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
To enable longer (or shorter) timeouts for a WBE submitted
request to transition out of the (PENDING, WAITING) states
allow the transition timeout that was previously set to 60
seconds to be provided as a WBE configuration option (it
still defaults to the previously fixed 60 seconds when it
is not provided).
Fixes bug 1356002
Change-Id: Idf384217004a334df03e2fff9150309fdfe08005
|
| |
|
|
|
|
|
|
|
|
| |
Some of the time these attributes are types but other
times they are functions, to avoid the confusion of naming
these with a '_cls' postfix move to having a '_factory' postfix
since these attributes generate other objects (which is what
factories do).
Change-Id: I73edd0c794223d719fbfbd0608c985cb335c8c26
|
| |
|
|
|
|
|
|
|
|
| |
A worker task executor is a good candidate for reuse since
it maintains worker knowledge that is valuable to be retained
across engine runs (tasks on which workers for example). In order
for it to be reused we need a way for the WBE to be able to
receive and reuse a previously existing executor.
Change-Id: Ia9a8f4c544b74e12e2cbd6bd941945da1111499c
|
| |
|
|
|
|
|
|
|
| |
Begin to rework the twiki docs into a developer doc,
moving items from the twiki to rst and adjusting examples
to make a unified place where the worker model can be
documented.
Change-Id: I43ed828be33351b9fb6606317011e7204f61a136
|
| |
|
|
|
|
|
|
|
|
|
| |
* Added the `Notify` message type. Used to notify workers
about executor has been started and to receive workers
information back (topic and tasks list);
* Implemented `WorkersCache` to interact with workers
information easly;
* Got rid of the engine `workers_info` parameter.
Change-Id: I4a810b1ddb0b04c11d12b47afc5f9cdf77d070be
|
| |
|
|
| |
Change-Id: I06eada275398adc3523c02a99d9e4650e2dd1637
|
|
|
* Implemented Worker to be started on remote host for
handling tasks request.
* Implemented WorkerTaskExecutor that proxies tasks
requests to remote workers.
* Implemented Proxy that is used for consuming and
publishing messages by Worker and Executor.
* Added worker-based engine and worker task executor.
* Added kombu dependency to requirements.
* Added worker-based flow example.
* Added unit-tests for worker-based flow components.
Implements: blueprint worker-based-engine
Change-Id: I8c6859ba4a1a56c2592e3d67cdfb8968b13ee99c
|
