| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Interactive (regular) requests are split into smaller transactions, so
larger updates won't fail with either timeout so or transaction too large
FDB errors.
* Non-interactive (replicated) requests can now batch their updates in a few
transaction and gain extra performance.
Batch size is configurable:
```
[fabric]
update_docs_batch_size = 5000000
```
|
| |
|
|
|
|
|
| |
Sometimes this test fails on Jenkins but doesn't fail locally. The attempted
fix is to make sure to simply retry a few times for the number of children in
the supervisor to be the expected values. Also extend the timeout to 15
seconds.
|
| |\
| |
| | |
Pagination API
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
| |
|
|
|
|
| |
Add some longer timeouts and fix a race condition in db cleanup tests
(Thanks to @jdoane for the patch)
|
| |\
| |
| | |
Background database deletion
|
| |/
|
|
|
|
|
|
| |
allow background job to delete soft-deleted database according to
specified criteria to release space. Once database is hard-deleted,
the data can't be fetched back.
Co-authored-by: Nick Vatamaniuc<vatamane@apache.org>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Previously we always returned `false` because the result from
`couch_jobs:get_job_state` was expected to be just `Status`, but it is `{ok,
Status}`. That part is now explicit so we account for every possible job state
and would fail on a clause match if we get something else there.
Moved `job_state/2` function to `couch_view_jobs` to avoid duplicating the
logic on how to calculate job_id and keep it all in one module.
Tests were updated to explicitly check for each state job state.
|
| | |
|
| | |
|
| | |
|
| |\
| |
| | |
Re-enable ExUnit tests
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| |
|
|
|
| |
On CI creating a 100 dbs in a row was too much to do in 5 seconds so bump it to
15.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In its current incarnation, the so-called "simple lifecycle" test is
prone to numerous failures in the CI system [1], doubtless because it's
riddled with race conditions. The original author makes many assumptions
about how quickly an (actual, unmocked) FDB instance will respond to a
request.
The primary goal is to stop failing CI builds, while other
considerations include: keeping the run time of the test as low as
possible, keeping the code coverage high, and documenting the known
races.
Specifically:
- Increase the `stale` and `expired` times by a factor of 5 to decrease
sensitivity to poor FDB performance.
- Change default timer from `erlang:system_time/1` to `os:timestamp` on
the assumption that the latter is less prone to warping [2].
- Decrease the period of the cache server reaper by half to increase
accuracy of eviction time.
- Inline and modify the `test_util:wait` code to make the timer
explicit, and emphasize that `timer:delay/1` only works with millisecond
resolution.
- Don't fail the test if it can't get a fresh lookup immediately after
insertion, but let it continue on to the next race, at least to the
point of expiration and deletion, which continue to be asserted.
- Factor `Timeout` and `Interval` to allow declarations near the other
hard-coded parameters.
- Move cache server `Opts` into `setup/0` and eliminate `start_link/0`.
- Double the overall test timeout to 20 seconds.
This has soaked for hundreds of runs on a 5 year old laptop, but the
real test is the CI system.
Should this test continue to fail CI builds, additional improvements
could include mocking the timer and/or FDB layer to eliminate the
variability of an integrated system.
[1] https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FPullRequests/detail/PR-2813/10/pipeline
[2] http://erlang.org/doc/apps/erts/time_correction.html#terminology
|
| |
|
|
|
|
|
| |
And an extra level of error checking to erlfdb:set_option since it could fail
if we forget to update erlfdb dependency or fdb server version is too old. That
operation can fail with an error:badarg which is exactly how list_to_integer
fails and result in a confusing log message.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With the latest erlfdb release v1.1.0 we have the ability to set default
transaction options on the database handle. Once set, those are inherited by
every transaction started from that handle.
Use this feature to give advanced users a way to experiment with various
transaction options. Descriptions of those options in the default.ini file have
been mostly a copy and paste from the fdb_c_option.g.h file from the client
library.
In addition, specify some safer default values for transaction timeouts (1min)
and retry limit (100). These quite conservative and are basically something
less that "infinity". In the future these may be adjusted lower.
|
| |
|
|
| |
https://github.com/apache/couchdb-erlfdb/releases/tag/v1.1.0
|
| | |
|
| |\
| |
| | |
Add native encryption support
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A new application, aegis, is introduced to provide strong at-rest
protection of CouchDB data (where possible).
Currently we encrypt the following values (if enabled):
1. Document content
2. Attachment content
3. Index values
Things not encrypted:
1. _all_docs
2. _changes
3. doc id
4. doc rev
5. Index keys
6. All other metadata
Co-Authored-By: Eric Avdey <eiri@apache.org>
Co-Authored-By: Robert Samuel Newson <rnewson@apache.org>
|
| |
|
|
|
|
|
| |
Currently, GET `/_session` reports the `authentication_db` of the
obsolete admin port 5986. This updates it to report the actual db used
for authentication, provided it is configured. Otherwise, it omits
`authentication_db` entirely from session info.
|
| |
|
|
|
|
|
|
| |
Adds a fold_docs function that will do a parallel fetch for the supplied
Doc Ids. This is used for _all_docs?keys=["id1", "id2"].
This uses a queue for fetching the revs and another queue for fetching
the doc bodies. These queues will be drained if the future queue gets
to large.
|
| |
|
|
|
| |
Use a common `get_revs_future` and `get_revs_wait` for fetching
winning revs and all revs.
|
| | |
|
| |\
| |
| | |
Fix typo in error message
|
| |/ |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
By default, transactions are used to check metadata, and possibly
reopen the db, to get a current db handle. However, if a `max_age`
option is provided and db handle was checked less than `max_age`
milliseconds ago, use properties from that cached handle instead.
The main use of this feature be in pluggable authorization handlers
where it might be necessary to inspect the security doc multiple times
for the same request before a final decision is made.
`revs_limit/1` was updated as well, mainly for consistency since it is
almost identical to `get_security/1`.
|
| |\
| |
| | |
Merge keys from rebar.config
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
This change allows creation of local `src/couch/rebar.config` and `rebar.config`
files to set additional configuration options. This is useful for:
- disabling deprecation warnings `{nowarn_deprecated_function, MFAs}`
- control debugging in eunit tests
- `DEBUG` - `{eunit_compile_opts, [{d, DEBUG, true}]}`
- `NODEBUG` - `{eunit_compile_opts, [{d, NODEBUG, true}]}`
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
When an existing key is inserted with different timestamps, the primary
key is the same but the primary value is different from the existing one.
Currently, this results in a new expiry key being inserted, but the old
one is not deleted and lingers until it is removed by the inexorable
advance of time via the `remove_expired` server messages.
This checks whether there's already primary key for the inserted key,
and if so, cleans up the existing expiry key before proceeding with the
insert.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
- Rename `clear_expired_range` to `clear_range_to`
- Move `EXPIRING_CACHE` layer prefix into fabric2.hrl
- Move primary key setting to just after key & value calculations
- Factor out `get_val/2` to lookup a key from FDB and unpack the value
- Factor out `prefixes/2`
- Factor out `fold_range/5`
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
* report changes stats intermittently with boolean market
Stats are reported at the end of a request. With changes feeds,
sometimes the request can be long or forever. This commit allows
stats to be reported intermittently via a configurable time in seconds.
The report function can return a boolean whether stats was reported
so that a reset may not necessarily be needed.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Currently, the size of binary chunks used for values is fixed at the FDB
imposed limit of 100kB, although they recommend using 10KB [1], (also
note they subtly change units).
This makes that value configurable, allowing e.g. benchmarks to compare
performance of runs with varying chunk size. The cost is a ~10µs config
lookup penalty each time data needs to be chunked.
[1] https://www.foundationdb.org/files/record-layer-paper.pdf
|
| |\ \
| |/
|/|
| |
| | |
Integrate emilio - erang linter
Merging it on the grounds of CI pass and +1 in the original PR.
|
| |/ |
|
| |
|
|
| |
Add tests for view cleanup.
|
