| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Use the `no_schedule` option to speed up job dequeuing. This optimization
allows dequeuing jobs more efficiently if these conditions are met:
1) Job IDs start with a random prefix
2) No time-based scheduling is used
Both of those can be true for views job IDs can be generated such that
signature comes before the db name part, which is what this commit does.
The way the optimization works, is random IDs are generating in pending jobs
range, then, a key selection is used to pick either a job before or after
it. That reduces each dequeue attempt to just 1 read instead of reading up to
1000 jobs.
|
| |
|
|
|
|
| |
Under load accept loop can blow up with timeout error from
`erlfdb:wait(...)`(https://github.com/apache/couchdb-erlfdb/blob/master/src/erlfdb.erl#L255)
so guard against it just like we do for fdb transaction timeout (1031) errors.
|
| |
|
|
|
|
| |
Update db handles right away as soon we db verison is checked. This ensures
concurrent requests will get access to the current handle as soon as possible
and may avoid doing extra version checks and re-opens.
|
| |
|
|
|
|
| |
Check metadata versions to ensure newer handles are not clobbered. The same
thing is done for removal, `maybe_remove/1` removes handle only if there isn't
a newer handle already there.
|
| |
|
|
| |
And also against too many conflicts during overload
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
Let users specify the maximum document count for the _bulk_docs requests. If
the document count exceeds the maximum it would return a 413 HTTP error. This
would also signal the replicator to try to bisect the _bulk_docs array into
smaller batches.
|
| |
|
|
| |
Observed a number of timeouts with the previous default
|
| |
|
|
|
| |
In a constrained CI environment transactions could retry multiple times so we
cannot rely on precisely counting erlfdb:transactional/2 calls.
|
| |
|
|
|
|
|
|
|
|
| |
Increase couch_views job timeout by 20 seconds. This will set a larger jitter
when multiple nodes concurrently check and re-equeue jobs. It would reduce the
chance of them bumping into each other and conflicting.
If they do conflict in activity monitor, catch the error and emit an error log.
We gain some more robustness under load for a longer timeout for jobs with
workers that have suddenly died getting re-enqueued.
|
| |\
| |
| | |
Fix handling of limit query parameter
|
| | | |
|
| |\ \
| | |
| | | |
Improve log of permanently deleting databases
|
| |/ / |
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Interactive (regular) requests are split into smaller transactions, so
larger updates won't fail with either timeout so or transaction too large
FDB errors.
* Non-interactive (replicated) requests can now batch their updates in a few
transaction and gain extra performance.
Batch size is configurable:
```
[fabric]
update_docs_batch_size = 5000000
```
|
| |
|
|
|
|
|
| |
Sometimes this test fails on Jenkins but doesn't fail locally. The attempted
fix is to make sure to simply retry a few times for the number of children in
the supervisor to be the expected values. Also extend the timeout to 15
seconds.
|
| |\
| |
| | |
Pagination API
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
| |
|
|
|
|
| |
Add some longer timeouts and fix a race condition in db cleanup tests
(Thanks to @jdoane for the patch)
|
| |\
| |
| | |
Background database deletion
|
| |/
|
|
|
|
|
|
| |
allow background job to delete soft-deleted database according to
specified criteria to release space. Once database is hard-deleted,
the data can't be fetched back.
Co-authored-by: Nick Vatamaniuc<vatamane@apache.org>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Previously we always returned `false` because the result from
`couch_jobs:get_job_state` was expected to be just `Status`, but it is `{ok,
Status}`. That part is now explicit so we account for every possible job state
and would fail on a clause match if we get something else there.
Moved `job_state/2` function to `couch_view_jobs` to avoid duplicating the
logic on how to calculate job_id and keep it all in one module.
Tests were updated to explicitly check for each state job state.
|
| | |
|
| | |
|
| | |
|
| |\
| |
| | |
Re-enable ExUnit tests
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| |
|
|
|
| |
On CI creating a 100 dbs in a row was too much to do in 5 seconds so bump it to
15.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In its current incarnation, the so-called "simple lifecycle" test is
prone to numerous failures in the CI system [1], doubtless because it's
riddled with race conditions. The original author makes many assumptions
about how quickly an (actual, unmocked) FDB instance will respond to a
request.
The primary goal is to stop failing CI builds, while other
considerations include: keeping the run time of the test as low as
possible, keeping the code coverage high, and documenting the known
races.
Specifically:
- Increase the `stale` and `expired` times by a factor of 5 to decrease
sensitivity to poor FDB performance.
- Change default timer from `erlang:system_time/1` to `os:timestamp` on
the assumption that the latter is less prone to warping [2].
- Decrease the period of the cache server reaper by half to increase
accuracy of eviction time.
- Inline and modify the `test_util:wait` code to make the timer
explicit, and emphasize that `timer:delay/1` only works with millisecond
resolution.
- Don't fail the test if it can't get a fresh lookup immediately after
insertion, but let it continue on to the next race, at least to the
point of expiration and deletion, which continue to be asserted.
- Factor `Timeout` and `Interval` to allow declarations near the other
hard-coded parameters.
- Move cache server `Opts` into `setup/0` and eliminate `start_link/0`.
- Double the overall test timeout to 20 seconds.
This has soaked for hundreds of runs on a 5 year old laptop, but the
real test is the CI system.
Should this test continue to fail CI builds, additional improvements
could include mocking the timer and/or FDB layer to eliminate the
variability of an integrated system.
[1] https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FPullRequests/detail/PR-2813/10/pipeline
[2] http://erlang.org/doc/apps/erts/time_correction.html#terminology
|
| |
|
|
|
|
|
| |
And an extra level of error checking to erlfdb:set_option since it could fail
if we forget to update erlfdb dependency or fdb server version is too old. That
operation can fail with an error:badarg which is exactly how list_to_integer
fails and result in a confusing log message.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With the latest erlfdb release v1.1.0 we have the ability to set default
transaction options on the database handle. Once set, those are inherited by
every transaction started from that handle.
Use this feature to give advanced users a way to experiment with various
transaction options. Descriptions of those options in the default.ini file have
been mostly a copy and paste from the fdb_c_option.g.h file from the client
library.
In addition, specify some safer default values for transaction timeouts (1min)
and retry limit (100). These quite conservative and are basically something
less that "infinity". In the future these may be adjusted lower.
|
| |
|
|
| |
https://github.com/apache/couchdb-erlfdb/releases/tag/v1.1.0
|
| | |
|
| |\
| |
| | |
Add native encryption support
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A new application, aegis, is introduced to provide strong at-rest
protection of CouchDB data (where possible).
Currently we encrypt the following values (if enabled):
1. Document content
2. Attachment content
3. Index values
Things not encrypted:
1. _all_docs
2. _changes
3. doc id
4. doc rev
5. Index keys
6. All other metadata
Co-Authored-By: Eric Avdey <eiri@apache.org>
Co-Authored-By: Robert Samuel Newson <rnewson@apache.org>
|
