summaryrefslogtreecommitdiff
path: root/docs
Commit message (Collapse)AuthorAgeFilesLines
* Update cli.py to support custom loggers (#1906)HEADmasterjuur2023-05-181-0/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update cli.py to support custom loggers Allows a config.py file (via rq worker --config) to support things like this for logfmt logging: DICT_CONFIG = { "version": 1, "formatters": {"logfmt": { "()": "logfmter.Logfmter", "keys": ["level","when","pid"], "mapping": {"level":"levelname","when":"asctime","pid":"process"}, "datefmt": "%Y-%m-%dT%H:%M:%S%z" }}, "handlers": {"console": {"class": "logging.StreamHandler","formatter": "logfmt" }}, "loggers": { "root": {"handlers":["console"], "level": "INFO"}, "rq": {"handlers":["console"], "level": "INFO", "propagate": False}, } } * added simple test and documentation for DICT_CONFIG * further attempt to get testing right for dictConfig * move import to correct location * fix * remove meaningless options.get() usage * linting checks and added missing test config file
* Update linting configuration (#1915)Rob Hudson2023-05-171-1/+1
| | | | | | | | | * Update linting configuration This removes flake8 in favor of ruff, which also provides isort support, and updates all files to be black, isort, and ruff compliant. This also adds black and ruff checks to the tox and Github linting workflow. * Tweak the code coverage config and calls
* Worker pool (#1874)Selwin Ong2023-05-011-30/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * First stab at implementating worker pool * Use process.is_alive() to check whether a process is still live * Handle shutdown signal * Check worker loop done * First working version of `WorkerPool`. * Added test for check_workers() * Added test for pool.start() * Better shutdown process * Comment out test_start() to see if it fixes CI * Make tests pass * Make CI pass * Comment out some tests * Comment out more tests * Re-enable a test * Re-enable another test * Uncomment check_workers test * Added run_worker test * Minor modification to dead worker detection * More test cases * Better process name for workers * Added back pool.stop_workers() when signal is received * Cleaned up cli.py * WIP on worker-pool command * Fix test * Test that worker pool ignores consecutive shutdown signals * Added test for worker-pool CLI command. * Added timeout to CI jobs * Fix worker pool test * Comment out test_scheduler.py * Fixed worker-pool in burst mode * Increase test coverage * Exclude tests directory from coverage.py * Improve test coverage * Renamed `Pool(num_workers=2) to `Pool(size=2)` * Revert "Renamed `Pool(num_workers=2) to `Pool(size=2)`" This reverts commit a1306f89ad0d8686c6bde447bff75e2f71f0733b. * Renamed Pool to WorkerPool * Added a new TestCase that doesn't use LocalStack * Added job_class, worker_class and serializer arguments to WorkerPool * Use parse_connection() in WorkerPool.__init__ * Added CLI arguments for worker-pool * Minor WorkerPool and test fixes * Fixed failing CLI test * Document WorkerPool
* Added changelogSelwin Ong2023-05-011-2/+6
|
* improve the docs on in-flight jobs of killed/lost workers (#1840)luto2023-04-011-3/+5
| | | | | | | * improve the docs on in-flight jobs of killed/lost workers * improve the docs on in-flight jobs of killed/lost workers * document abandoned job behavior before 0.14
* Fix error in example in the documentation (#1870)Joris2023-04-011-4/+4
| | | The solution that @rpkak proposes works. Closes #1524
* Consolidate job callbacks execution (#1851)Rony Lutsky2023-03-211-0/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * consolidate job failure callback execution * fix * success callback as well * merge fix * create Callback class and change how callbacks serde * deprecation * add tests * string format * pr fix * revert serialization changes * fix timeout typing * add documentation * add test * fix a bug * fix tests * move job heartbeat call to worker and make sure there is always a callback timeout * fix test * fix test
* Remove `use_connection` (#1859)lowercase002023-03-071-50/+17
| | | | | * feat: remove use_connection * fix: clear old test
* Update arguments passed to the Sentinel Object when created from the ↵Joachim Burket2023-03-071-5/+15
| | | | | | | | | | | | | | | | | settings (#1850) * Updated arguments passed to the Sentinel Object when created from the settings - added `USERNAME` key - added `CONNECTION_KWARGS` key to allow passing additionals arguments to the Redis connections - updated the documentation * added missing comma * tests(helpers): Added tests for Sentinel --------- Co-authored-by: Joachim Burket <joachim.burket@hopitalvs.ch>
* Update index.md (#1858)Tianci Hu Marrero2023-03-071-1/+1
| | | fixed minor duplicate article ("the its config var")
* New dequeue strategy (#1806)lowercase002023-03-051-6/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * New dequeue strategy This implements a new parameter `dequeue_strategy` that should replace the `RoundRobinWorker` and `RandomWorker`. Changes includes: feature, docs, tests, deprecation warning. * Fix dequeue strategy name * Black & Fix warning * feat: tests, warnings, refactor naming * feat: improve worker check * fix: revert to str subclass * fix: dequeue strategy into bootstrap * org: move DequeueStrategy to worker * refactor: round robin naming * fix: naming * fix: type annotation * fix: typo * refactor: remove kwarg from worker's init * fix: typo * move `dequeue_strategy` from `bootstrap()` into `work()`
* Add failure callback call to started job registry cleanup (#1824)Rony Lutsky2023-03-041-0/+8
| | | | | | | | | | | | | | | * Add started job registry cleanup job failure callback call * WIP - need to fix test * fix test * rename, tests and docs * better log message * use class name * Update registry.py
* docs: scheduler safe import (#1835)lowercase002023-03-031-1/+22
| | | | | * docs: scheduler safe import * docs: rollback main block
* (Docs) Updated patterns - heroku (#1841)Tianci Hu Marrero2023-02-271-10/+11
| | | | | | | | | * Update index.md * Update index.md * Update index.md import herokuWorker instead.
* docs: add warning of `decode_responses` (#1833)lowercase002023-02-261-0/+14
|
* Enhance worker termination logic (#1729)Rony Lutsky2023-02-171-0/+25
| | | | | | | | | | | | | | | | | | | | | | | * enhance worker termination logic and allow passing custom exc_info in failure callback * handle ret_val None * fix unbound variable * typing * docs * Update exceptions.md * rename * typing * rename * Update exceptions.md * revert test change
* docs: fix typos (#1813)lowercase002023-02-131-5/+5
|
* Docs: Updated testing instructions (#1812)lowercase002023-02-132-2/+35
| | | | | * docs: update rq repo address * docs: detailed test instructions
* Enhanced Redis Connection Reliability (#1753)lowercase002023-02-021-0/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * Enhanced Redis Connection Reliability The Redis connection may fail for several reasons. As the connection can be (1) explicitly passed to the worker or (2) implicity set, this will improve the Connection configuration by setting a timeout to the socket, and adding an ExponentialBackoff Retry logic. * Simpler Connection logic * Add simple retry logic to Redis Connection Error * Make retry exponential, add keepalive & socket_connect_timeout * Handles configuration on Redis' connection pool * Simplifies timeout exception logic * Fix burst bug, add test * Add docs related to `socket_timeout`, improve compatibility with older RedisPy versions * Fixes * New timeout private method * Fix timeout
* Fix worker example (#1765)lowercase002023-01-301-9/+5
| | | | | | | * Fix worker example * Clean worker example * Improve docs
* #1733 Typography (#1734)Krzysztof Jeziorny2022-11-211-125/+72
| | | | | html font size removed, body font size set in rem w/o weight; font stacks defined; code formatted
* Multiple results using Redis Streams (#1725)Selwin Ong2022-11-192-7/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * WIP job results * Result can now be saved * Successfully saved and restored result * result.save() should accept pipeline * Successful results are saved * Failures are now saved properly too. * Added test for Result.get_latest() * Checkpoint * Got Result.all() to work * Added Result.count(), Result.delete() * Backward compatibility for job.result and job.exc_info * Added some typing * More typing stuff * Fixed typing in job.py * More typing updates * Only keep the last 10 results * Documented job.results() * Got results test to pass * Don't run test_results.py on Redis server < 5.0 * Fixed mock import on some Python versions * Remove Redis 3 from test matrix * Jobs should never use the new Result implementation if server is < 5.0 * Results should only be created is Redis stream is supported. * Added back Redis 3 to test matrix * Fixed job.supports_redis_streams * Fixed worker test * Updated docs.
* Add feature to enqueue dependents at the front of queues (#1696)Jahn Thomas Fidje2022-09-231-6/+15
| | | | | | | | | | | | | * Add feature to enqueue dependents at the front of queues * Add documentation for the Dependency(enqueue_at_front=...) parameter * docs: Add `enqueue_at_front` to list of parameters for Dependency * test: Update dependency test to not rely on Redis ordering * refactor: Save enqueue_at_front boolean in job.meta instead of separate instance attr * fix: Made enqueue_at_front an instance attribute instead of putting it inside meta
* doc: default queues order is a priority order (#1704)Alex Garel2022-09-221-0/+4
| | | | | | | | | * doc: default queues order is a priority order In introduction, we say that queue order matters, but we are not very specific. Added a paragraph to explicit that. * doc: fix typo
* Dependency with failures (#1681)Selwin Ong2022-07-311-1/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * added Dependency class with allow_failures * Requested changes * Check type before setting `job.dependency_allow_fail` within `Job.create` * Set `job.dependency_allow_fail` within `Job.create` * Added test to ensure persistence of `dependency_allow_fail` * Removed typing and allow mixed list of ints and Job objects * Convert dependency_allow_fail boolean to integer during serialization to avoid redis DataError * Updated `test_multiple_dependencies_are_accepted_and_persisted` test to include `Dependency` cases * Adding placeholder test to test actual behavior of new `Dependency` usage in `depends_on` * Updated `test_job_dependency` to include cases using `Dependency` * Added dependency_allow_fail logic to `Job.restore` * Renamed `dependency_allow_fail` to a simpler `allow_failure` * Update docs to add section about the new `Dependency` class and use-case * Updated `Job.dependencies_are_met` logic to take `FAILED` and `STOPPED` jobs into account when `allow_failure=True` * Updated `test_job_dependency` test. Still failing with `Dependency` case. * Fix `allow_failure` type coercion in `Job.restore` * Re-arrange tests, so default `Dependency.allow_failure` is before explicit `allow_failure=True` * Fixed Dependency, so it works correctly when allow_failure=True * Attempt to execute pipeline prior to queueing a failed job's dependents. test_create_and_cancel_job_enqueue_dependents_in_registry test now passes. * Added `Depedency` test utilizing multiple dependencies * Removed irrelevant on_success and on_failure keyword arguments in example * Replaced use of long_running_job * Add test to verify `Dependency.jobs` contraints * Suppress connection error in handle_job_failure * test_dependencies have passed * All tests pass if enqueue_dependents called without pipeline.watch() * All tests now pass * Removed print statements * Cleanup Dependency implementation * Renamed job.allow_failure to job.allow_dependency_failures Co-authored-by: mattchan <mattchan@tencent.com> Co-authored-by: Mike Hill <mhilluniversal@gmail.com>
* docs: Fix a few typos (#1679)Tim Gates2022-07-312-4/+4
| | | | | | | | | | | | | | | | | There are small typos in: - docs/docs/exceptions.md - docs/docs/jobs.md - rq/queue.py - tests/fixtures.py - tests/test_job.py Fixes: - Should read `slightly` rather than `slighty`. - Should read `requeuing` rather than `requeueing`. - Should read `implementers` rather than `implementors`. - Should read `definition` rather than `defition`. - Should read `canceled` rather than `canceld`. Signed-off-by: Tim Gates <tim.gates@iress.com>
* Fix syntax error in docs example (#1664)Pavel Levchuk2022-05-301-1/+1
| | | Adds a missing colon in function arguments
* Update exceptions.md (#1655)urmikab2022-05-161-0/+1
| | | Added the import statement for Job (line 18)
* fix typo (#1640)Rajesh Khadka2022-04-021-1/+1
|
* Docs: don't break links at the header (#1636)Mateusz Soszyński2022-04-021-0/+1
|
* Cross platform simple worker (#1629)Michael Hill2022-03-101-1/+20
| | | | | | | | | | | | | | | | | | | | | * Added CrossPlatformDeathPenalty that doesn't rely on signals * Updated `SimpleWorker`'s `death_penalty_class` to utilize `CrossPlatformDeathPenalty` to allow use on Windows machines * Changed `CrossPlatformDeathPenalty` to `TimerDeathPenalty` * Removed overridden `death_penalty_class` in `SimpleWorker` until feature matures * Added section in testing.md explaining how to utilize `SimpleWorker` on Windows OS * Replaced usage of chatting with .format for python 3.5 compatibility * Add tests for new timeout feature * Explicitly set defaults.CALLBACK_TIMEOUT * Implemented cross-thread method of raising errors by using ctypes * Finished writing tests for new TimerDeathPenalty
* Replace highlight tag in docs (#1600)Selwin Ong2021-12-077-34/+34
|
* add REDIS_SSL_CERT_REQS cli parameter (#1495)mgcdanny2021-10-271-4/+31
| | | | | | | * add REDIS_SSL_CERT_REQS cli parameter * update docs Co-authored-by: dan <dan@betterfin.com>
* Fix link Round Robin / Random worker classes (#1561)luto2021-09-131-1/+1
|
* Add option to enqueue a job's dependents when canceling (#1549)Josh Cohen2021-08-262-0/+14
| | | | | * Add option to enqueue a jobs dependents when canceling * Address @selwin's review
* job.cancel() puts job into CanceledJobRegistry. (#1546)Selwin Ong2021-08-222-3/+33
| | | | | * job.cancel() puts job into CanceledJobRegistry. * Improve test coverage
* job: add get_meta() function (#1536)Paul Spooren2021-08-211-1/+5
| | | | | | | | | | | | | | | | | | | | | | * job: add get_meta() function The newly introduced function returns meta data stored for the job. This is required since job.meta stays an empty dict until the job is finished or failed. With the new function it's possible to store arbiatraty states/stages of the job and allow the user to track progress. A long running job may return custom stages like `downloading_data`, `unpacking_data`, `processing_data`, etc. This may allow better interfaces since users can track progress. Signed-off-by: Paul Spooren <mail@aparcar.org> * docs: add missing `refresh` arg to get_status() This was previously missing. Signed-off-by: Paul Spooren <mail@aparcar.org>
* Allows enqueueing by the cli (#1466)rpkak2021-08-201-0/+74
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Allows enqueueing by the cli #372 * schedule support * `_` to `-` * fix flake8 * echo job-id * Some improvements - Description as in python jobs - return result - quiet mode - allows `--boolean` and `--integer` - raises errors if not used correctly * added tests * add schedule tests * add retry test * use click exceptions * add error test * add job_func test * change messages https://github.com/rq/rq/pull/1466#discussion_r640211128 https://github.com/rq/rq/pull/1466#discussion_r640210850 * Use different format for arguments View https://github.com/rq/rq/pull/1466#discussion_r650510889 * Add file support Usage: @filename * ast.literal_eval support with `#` instead of `:` * func -> function Makes error messages more readable * click Error * print function string * add docs * increase seconds in test * Update `parse_function_arg` Add `ParsingMode` enum (https://github.com/rq/rq/pull/1466#discussion_r656676114) Change error messages (https://github.com/rq/rq/pull/1466#discussion_r656676800, https://github.com/rq/rq/pull/1466#discussion_r656677082) * `#` to `%` `#` is the letter for a comment in bash * Add some tests (https://github.com/rq/rq/pull/1466#discussion_r656674539, https://github.com/rq/rq/pull/1466#discussion_r656676543) * Add some tests * docs: Add some examples * catch all literal_eval exceptions There are some edge cases with other exceptions * remove job_func (https://github.com/rq/rq/pull/1466#pullrequestreview-690110118) * edit docs https://github.com/rq/rq/pull/1466#pullrequestreview-695758691 * format examples * format examples `queue.enqueue(path.to.func, args=['abc'])` to `queue.enqueue(path.to.func, 'abc')` https://github.com/rq/rq/pull/1466#discussion_r673615464 * add examples https://github.com/rq/rq/pull/1466#discussion_r673658933 * add doc test https://github.com/rq/rq/pull/1466#discussion_r673659124 * Update index.md * Update test_cli.py * Update test_cli.py * Add version info Co-authored-by: rpkak <rpkak@users.noreply.github.com>
* Success and failure callbacks (#1480)Selwin Ong2021-06-222-19/+71
| | | | | | | | | | | | | | | * Support enqueueing with on_success_callback * Got success callback to execute properly * Added on_failure callback * Document success and failure callbacks * More Flake8 fixes * Mention that options can also be passed in via environment variables * Increase coverage on test_callbacks.py
* Make `Queue.enqueue`, `Queue.enqueue_call`, ↵Josh Cohen2021-06-201-0/+29
| | | | | | | | | | | | | | | | | | | | | | | | | `Queue.enqueue_at``Queue.parse_args` accept `pipeline` arg, add `Queue.enqueue_many` method (#1485) * Make `enqueue_*` and `Queue.parse_args` accept `pipeline` arg * undo bad docs * Fix lints in new code * Implement enqueue_many, refactor dependency setup out to method * Make name consistant * Refactor enqueue_many, update tests, add docs * Refactor out enqueueing from dependency setup * Move new docs to queue page * Fix section header * Fix section header again * Update rq version to 1.9.0
* Docs: Add scheduled to job.get_status() (#1493)rpkak2021-06-161-1/+1
|
* Add chat page (#1474)rpkak2021-06-123-0/+24
| | | | | * Add chat page * no js / iframe
* Typo: Than -> Then (#1479)Joseph2021-05-301-1/+1
| | | Co-authored-by: mgjo5899 <mgjo5899@gmail.com>
* add `/` (#1475)rpkak2021-05-271-1/+1
|
* Fix mistake in doc of Random dequeuing strategy (#1443)Antonino Sabetta2021-04-201-1/+1
|
* Newer pip install command from git (#1439)James Cobb2021-04-011-1/+1
| | | | | | | * Newer pip install command from git Updated cutting edge pip install command to use HTTPS rather than insecure git+git protocol, as recommended by pip documentation. This allows the command to work with pip >= 21.0.1 * Updated pip command in README
* Workers dequeuing jobs from queues using both Round-Robin and Random ↵Biel Cardona2021-03-311-0/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | strategies (#1425) * implemented round-robin and random access to queues * added tests for RoundRobinQueue * reverted change in gitignore * removed linebreak * added tests for random queues * added documentation for round robin and random queues * moved round robin strategy to worker * reverted changes to queue.py * reverted changes to workers.md * reverted changes to test_queue * added tests for RoundRobinWorker and RandomWorker * added doc for round robin and random workers * removed f-strings for backward compatibility * corrected a mistake * minor changes (code style) * now using _ordered_queues instead of queues for reordering queues
* Add a "stopped" JobStatus and the machinery to properly handle it (#1394)Daniel Alley2021-02-091-12/+14
| | | | | | | * Add a "stopped" JobStatus and the machinery to properly handle it fixes #1389 * Apply requested changes
* Document that --serializer CLI argument is only available in 1.8.0Selwin Ong2021-01-191-4/+2
|
* Added --serializer option to cli, finishing off PR #1381 and fix #1357 (#1395)f0cker2021-01-191-5/+7
| | | | | | | | | | | * Added --serializer option to cli, finishing off PR #1381 and fix #1357 * Update documentation * Update documentation * Modified help message Co-authored-by: f0cker <dturner@trustwave.com>
View Lorry Controller Status