| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| ... | |
| |
|
| |
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
|
| |
|
|
|
|
|
| |
* Removed deprecated (object) inheritance
Add new py38,py39 versions to tox, removed deprecated py27,py34
Replace enum internal function with Enum class
* fix
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(#1372)
* cleanup jobs that are not really running due to zombie workers
* remove registry entries for zombie jobs
* return only the job ids on cleanup
* test zombie job cleanup
* format code
* rename variable to explain that second element in tuple is expiry, not score
* remove worker_key
* detect zombie jobs using old heartbeats
* reuse get_expired_job_ids
* set score using current_timestamp
* test idle jobs using stale heartbeats
* extract timeout into variable
* move heartbeats into StartedJobRegistry
* use registry.heartbeat in tests
* remove heartbeats when job removed from StartedJobRegistry
* remove idle and expired jobs from both wip and heartbeats set
* send heartbeat_ttl to registry.add
* typo
* revert everything 😶
* only keep job heartbeats as score (and get rid of job timeouts as scores
* calculate heartbeat_ttl in an overrideable function + override it in SimpleWorker + move storing StartedJobRegistry scores to job.heartbeat()
* set heartbeat to monitoring interval for infinite timeouts
* track elapsed_execution_time as part of worker
* reset current job working time when work on a job is done
* persisting the job working time as part of monitoring
|
| |
|
| |
fix #1448
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
* Newer pip install command from git
Updated cutting edge pip install command to use HTTPS rather than insecure git+git protocol, as recommended by pip documentation. This allows the command to work with pip >= 21.0.1
* Updated pip command in README
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
strategies (#1425)
* implemented round-robin and random access to queues
* added tests for RoundRobinQueue
* reverted change in gitignore
* removed linebreak
* added tests for random queues
* added documentation for round robin and random queues
* moved round robin strategy to worker
* reverted changes to queue.py
* reverted changes to workers.md
* reverted changes to test_queue
* added tests for RoundRobinWorker and RandomWorker
* added doc for round robin and random workers
* removed f-strings for backward compatibility
* corrected a mistake
* minor changes (code style)
* now using _ordered_queues instead of queues for reordering queues
|
| |
|
|
|
| |
Job.fetch_dependencies (#1411)
Co-authored-by: Jahn Thomas Fidje <jahn-thomas.fidje@telenor.com>
|
| |
|
|
|
|
|
| |
* Add a "stopped" JobStatus and the machinery to properly handle it
fixes #1389
* Apply requested changes
|
| |
|
|
|
|
|
|
|
| |
* add asyncio runner
* add asyncio runner
* fix for old version
* fix tests
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Also accept lists and tuples as value of `depends_on`.
* The elements of the lists/tuples may be either Jobs or Job IDs.
* A single Job / Job ID is still accepted as well.
* Represent _all_ job dependencies in `Job.to_dict()`.
We now represent the entire list, instead of just the first element.
* Fix some doctext regarding plurality of dependencies.
* Add unit tests for job dependencies.
* One unit test establishes a pattern for checking execution order as affected by dependencies.
* Another unit test applies this pattern to the new capability to name multiple dependencies.
* Add unit test for new `depends_on` input formats.
Also test that these are properly persisted.
* Repair `Job.restore()`.
Need to convert bytes back to strings when reloading `dependency_ids`.
* Maintain backwards compat. in `Job.to_dict()`.
Keep the old `dependency_id` (singular) key.
* Provide coverage for new test fixture.
* Simplify some code.
Cut some superfluous `as_text()` calls left over from an earlier commit.
* Check for `dependency_id` in `Job.restore()` for backwd. compat.
Also eliminate use of `as_text()` here, in favor of `.decode()`.
* Switch to snake case instead of camel case.
* Eliminate some usages of `as_text()`.
Also cut some `print` statements.
* Cleanup.
* Accept arbitrary iterables for `Job`'s `depends_on` kwarg.
Instead of requiring a list or tuple, we now make use of `ensure_list()`.
* Add test fixtures.
* Provide a system to get two workers working simultaneously, using `multiprocessing`.
* Define a simple job that just says whether its dependencies are met.
* In `rpush`, make an option to record the name of the worker.
* Improve unit tests on execution order with dependencies.
These now actually have two workers going, which makes a more thorough test.
* Add unit test examining `Job.dependencies_are_met()` at execution time.
* Redesign dependency execution order unit tests.
* Simplify assertions.
* Improve doctext and formatting.
* Move fixture tests to new, dedicated module `test_fixtures.py`.
* Use `enqueue` instead of `enqueue_call` in new unit tests.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If a UNIX socket path is passed to the constructor of the Redis client,
`redis.client.Redis`, the value of keyword argument `unix_socket_path`
is passed to the constructor of `UnixDomainSocketConnection` with the
key `path`.
When RQ's scheduler creates its own Redis connection, it instantiates
class `redis.client.Redis` with keyword arguments obtained from the
connection pool. If the pooled connection is a
`UnixDomainSocketConnection`, its keyword arguments contain `path`, as
given on instantiation. This results in a `TypeError: __init__() got an
unexpected keyword argument 'path'`.
This change renames the key back to `unix_socket_path` before the
keyword arguments dictionary is used to instantiate
`redis.client.Redis`.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
* Added --serializer option to cli, finishing off PR #1381 and fix #1357
* Update documentation
* Update documentation
* Modified help message
Co-authored-by: f0cker <dturner@trustwave.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* add exception catch for redis connection failure
* Add test for connection recovery
* add exponential backoff
* limit worker max connection wait time to 60 seconds
* fix undefined class variable
* fix string formatting issue while printing error log
* cap max connection wait time:better code style
Co-authored-by: corynezin <cory.nezin@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
* clean_worker_registry cleans in batches to prevent submitting too much data to redis at once when there are a large number of invalid keys
* Address code review comments
Rename MAX_REMOVABLE_KEYS to MAX_KEYS
* Fix tests
Co-authored-by: Joel Harris <combolations@gmail.com>
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
* Ensure that the custom serializer defined is passed into the job fetch calls
* add serializer as argument to fetch_many and dequeue_any methods
* add worker test for custom serializer
* move json serializer to serializers.py
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
* Quick and dirty set up of SSL
* copy connection kwargs in scheduler
* fix
* chmod the cert
* Skip SSL tests in CI
|
| |
|
|
|
| |
The former link is broken since http://flask.pocoo.org/snippets has been
taken down. See https://github.com/pallets/website/issues/41 for more
info.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
* Added send_stop_job_command().
* send_stop_job_command now accepts just connection and job_id
* Document send_job_job_command
* Updated test coverage
|
| |
|
|
|
|
|
|
|
|
|
| |
* Added job.worker_name
* Fix compatibility with Redis server 3.x
* Document job.worker_name
* Removed some Python 2 compatibility stuff.
* Remove unused codes
|
| | |
|
| |\ |
|
| | |
| |
| |
| |
| | |
* Fixed an issue where scheduler lock is not release when running worker in burst mode
* Remove unused import
|
| |/ |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
* tests: updated github worklow for tests to use requirements.txt and dev-requirements.txt
* build: updated dev-requirements.txt
Co-authored-by: Ruslan Mullakhmetov <ruslan@twentythree.net>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* feat: added job heartbeat to track whether job is actually executing
heartbeat might be needed in cases when worker was hardkilled or the whole VM/docker was forcibly rebooted.
* fixed tests
* fixed test coverage issue
* chore: renamed job.heartbeat stuff according to review feedback
* chore: pipelined worker heartbeat and job heartbeat
* docs: documented job.heartbeat property
* fixes after review
* docs: updated last_heartbeat description
* chore: review
Co-authored-by: Ruslan Mullakhmetov <ruslan@twentythree.net>
|
| |
|
|
|
|
|
|
|
|
|
| |
* Added a way to send shutdown command via pubsub
* Added kill-horse command
* Added kill horse command
* Added send_kill_horse_command() and send_shutdown_command()
* Document worker commands
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* tests: added ability to run tests in Docker
useful to run full test suit on Mac
* tests: minor improvement in dockerfile for tests
* tests: typo in Dockerfile
* tests: updated dev requirements
Co-authored-by: Ruslan Mullakhmetov <ruslan@twentythree.net>
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* scheduler: now operates with chunks of jobs
* scheduler: set default chunk_size for ScheduledJobRegistry.get_jobs_to_schedule
* scheduler: fixed missing indent
* scheduler: added test for get_jobs_to_schedule() with chunk_size parameter
* scheduler: fixed test for passing python 3.5 (no f-strings)
* scheduler: fixed chunk_size in test make it lighter to run
|
| |
|
|
|
|
|
|
|
|
|
| |
* feat: avoided "zombie" processes after killing work horse by setting work horse process group and killing this group
* fixed tests
* tests: added test to check that all workhorse subprocesses are killed
* tests: updated guthub run tests dependencies since they are not using (dev-)requirements.txt
Co-authored-by: Ruslan Mullakhmetov <ruslan@twentythree.net>
|
| |
|
|
|
|
|
| |
* docker folder along with Dockerfile has been added
* fix #1340
Co-authored-by: orkhan.mamedov <orkhan.mamedov@sravni.ru>
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
| |
Fixes: #1328
|
