| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
| |
Enhance it to behave closer to lists:map such that exceptions with
stacktraces will are re-thrown. Also use a macro to avoid the
stacktrace deprecation warning. This also eliminates the need to
validate its results as we can use EUnit asserts directly in the
executor function.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* There was a good amount of duplication between `_db_crud_tests` and
`_changes_fold_tests`, so make a common test utility module so both suites can
use.
* Clean up test names. Previously some were named `tx_too_long` but since the
official FDB error is `transaction_too_old` rename them to match a bit
better.
* `list_dbs_info` implementation queue of 100 futures to parallelize fetching.
So its test was update to create more than 100 dbs. Creating 100 dbs took
about 3 seconds so add a small parallel map (pmap) utility function to help
with that.
|
| |
|
|
|
|
|
|
|
| |
Previously those endpoints would break when transactions time-out and are
retried. To fix it we re-use the mechanism from changes feeds.
There is a longer discussion about this on the mailing list:
https://lists.apache.org/thread.html/r02cee7045cac4722e1682bb69ba0ec791f5cce025597d0099fb34033%40%3Cdev.couchdb.apache.org%3E
|
| |
|
|
|
|
|
|
| |
Seeing log errors like the following:
CRASH REPORT Process epep_fdb_decision_cache (<0.633.0>) with 0 neighbors exited with reason: {bad_info,{#Ref<0.2506675824.3127640065.49278>,ready}} at gen_server:handle_common_reply/8(line:726) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_expiring_cache_server,init,['Argument__1']}, ancestors: [epep_sup,<0.596.0>], message_queue_len: 0, messages: [], links: [<0.614.0>], dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.8.15449617>,next => #Fun<..>,...},...}},...], trap_exit: false, status: running, heap_size: 2586, stack_size: 27, reductions: 7493102
This should handle those errors, and prevent the crashes.
|
| |
|
|
| |
Use `fabric2_db:is_clustered/1` instead of `couch_db:is_clustered`
|
| |
|
|
|
|
|
| |
* couch_jobs resubmit updates job data
When a job is either pending or finished and the job is resubmitted
with new data the job data is updated.
|
| |
|
|
|
| |
Fixes an issue where the first k/v was skipped if the startkey_docid
was included.
|
| |
|
|
|
|
|
|
|
|
| |
* Register with the fabric2_index module
* Provide a build_indices/2 callback
* In the callback attempt to parse an `#mrst{}` indexing context and then
trigger an indexing job. We don't wait for the job to finish but instead
rely on `couch_job`'s max worker limit to keep concurrency in check.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The main logic is as follows:
- After doc update transaction(s) are completed in the `fabric2_db` module,
call the `fabric2_index:db_updated(Db)` function.
- `fabric2_index:db_updated(Db)` inserts a `{DbName, Timestamp}` tuple into
one of the sharded ets tables. The tuple is inserted using `ets:insert_new`,
which ensures only the first entry succeeds and others will be ignored,
until that entry is cleared.
- Each ets table in `fabric2_index` has a simple monitor process that
periodically scans that table. If it finds databases which have been
updated, it notifies all the indices which have registered with
`fabric2_index` to build indices.
There are config settings to disable index auto-updating and to adjust the
delay interval, and the resolution. The interval specifies how long to wait
since the first time the db was modified. The resolution interval specifies how
often to check the ets tables.
Just like in the original ken, design documents can have an `"autoupdate":
false` option to disable auto-updating that design document only.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Previously even when the scheduled time was the same the job was still deleted
and re-inserted into the pending queue. Now we perform the re-enqueing
operation only if the scheduled time has changed. So if the whole operation is
run in its own transaction, the transaction will now become a read-only
transaction.
This optimization should come in handy with the indexing auto-builder, for
example, where multiple nodes might try to re-enqueue the job and then only the
first would succeed, and the rest will perform a quick one row read and not do
any writes.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
These tests exercise that logic when updating rows in a map index. It
works by generating two states and a transition between those states.
That is, the set of rows that exist in the index, and the set of rows
that exist after an update along with how many rows should be kept in
common.
This is a fairly exhaustive search of the state space in these
transitions. A full run takes approximately 60s on my laptop so I have
limited the set of tests to a random selection of 100 single and multi
transition tests (i.e., updating either one or five documents in a
single view build iteration).
|
| |
|
|
|
| |
This saves us a key/value pair for any document that is not included as
part of a view.
|
| |
|
|
|
|
|
|
|
| |
If a map function emits duplicate keys for a document this stores
multiple rows in the map index differentiated by a `DupeId` counter.
Previously we were attempting to save some work avoiding clearing ranges
for keys that would be overwritten. However, if a document update causes
fewer duplicates to be emitted for the same key we left orphaned rows in
the index.
|
| |
|
|
|
| |
This moves index meta information to its own key space so that we can
scan all existing indexes efficiently.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Both a performance and style cleanup. This reduces the number of ets
calls by roughly `2 * 2 * $num_docs * $num_views`. The first factor of
two is because this avoids re-calculating the size twice for both the id
and map indexes. The second factor of two is because we have to lookup
two different settings.
Stylistically this also moves the logic out of the fdb modules. Keeping
key/value structure and update logic is already pretty complicated so we
should avoid mixing external application logic into those modules as
much as possible for our sanity.
The behavior is slightly changed vs the original patch as well.
Originally only the view that contained the errant key or value was
affected. However, pre-existing behavior where a document fails to be
mapped correctly resulted in it being excluded from all view indexes
defined in the design document. That behavior has been restored.
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
view_sig -> view_seq
|
| | |
|
| |
|
|
|
| |
The Reason term in the exception is no longer wrapped with an `{error,
Reason}` tuple since its now an `error` exception.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This test adds coverage for the 4 fdb_to_revinfo compatibility clauses,
and will help ensure any additional future clauses will not break
backwards compatibility.
Module coverage without this test:
fabric2_fdb : 92%
Module coverage with this test:
fabric2_fdb : 94%
|
| |
|
|
|
|
|
|
|
|
| |
A bug was introduced in https://github.com/apache/couchdb/commit/eb1a09e114dafc55fa9511d477b9ada4350134eb#diff-32274bcb4785f432811085d2e08c3580L1227
when `CURR_REV_FORMAT` was bumped from `1->2`, but `?CURR_REV_FORMAT` in
compatibility clauses going from format version `0->1` were left
unchanged, when those values of `?CURR_REV_FORMAT` should have changed
to `1`.
This fixes the compatibility clauses for rev format versions from `0->1`
|
| |
|
|
|
|
|
|
| |
Currently, a 5-tuple `Val` parameter crashes fdb_to_revinfo with a
function_clause error.
This adds a compatible function clause, and also a catchall clause to
log an error and throw an informative exception.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since we started re-using the main changes feed transaction to open the docs
https://github.com/apache/couchdb/commit/1bceb552594af404961e4ab8e6f88cffa1548f69,
we also started to pass the transactional db handle to the ddoc cache. However
ddoc cache uses a `spawn_monitor` to open docs, and since our transaction
objects are tied to the owner process the open was failing with a `badarg`
error from erlfdb.
This could be seen by running the replication elixir tests:
```
make elixir tests=test/elixir/test/replication_test.exs:214
```
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The reason why previous attempt failed is because it overrode [important logic in
`process_url/2`](https://github.com/myfreeweb/httpotion/blob/v3.1.2/lib/httpotion.ex#L34:L35):
```
def process_url(url, options) do
process_url(url)
|> prepend_protocol
|> append_query_string(options)
end
```
This PR fixes the problem by adding the `prepend_protocol`
and `append_query_string`. It also refactor the way base_url
is passed around.
|
| |
|
|
|
| |
We started to emit that in CouchDB 4.x for temporary views and possibly other
endpoints.
|
| |
|
|
|
| |
It's possible for other couch_epi plugins to interfere with this test,
so mock `couch_epi:decide/5` to always return `no_decision`.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, if the metadata key is bumped in a transaction, the same
transaction could not be used to add jobs with `couch_jobs`. That's because
metadata is a versionstamped value, and when set, it cannot be read back until
that transaction has committed. In `fabric2_fdb` there is a process dict key
that is set which declares that metadata was already read, which happens before
any db update, however `couch_jobs` uses it's own caching mechanism and doesn't
know about that pdict key.
Ideally we'd implement a single `couch_fdb` module to be shared between
`couch_jobs` and `fabric2_db` but until then it maybe simpler to just let
`couch_jobs` use its own metadata key. This way, it doesn't get invalidated or
bumped every time dbs get recreated or design docs are updated. The only time
it would be bumped is if the FDB layer prefix changed at runtime.
|
| |
|
|
|
|
|
|
| |
We already handle them in couch_jobs_type_monitor so let's do it in
`couch_jobs:wait_pending` as well. Recent fixes in FDB 6.2 didn't completely
fix the issue and ther are still spurious 1009 errors dumped in the logs. They
seem to be benign as far as couch_jobs operation goes as type monitor code
already showed, so let's not pollute the logs with them.
|
| |
|
|
|
|
|
| |
Previously each doc was read in a separate transaction. It turns out that size
limits do not apply to read-only transactions so we don't have to worry about
that here. Also transaction restart are already implemented so we don't have to
worry about timeout either.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
Versionstamp sequences should always be binaries when retrieved from a
rev info map.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This tracks the number of bytes that would be required to store the
contents of a database as flat files on disk. Currently the following
items are tracked:
* Doc ids
* Revisions
* Doc body as JSON
* Attachment names
* Attachment type
* Attachment length
* Attachment md5s
* Attachment headers
* Local doc id
* Local doc revision
* Local doc bodies
|
| |
|
|
|
| |
I accidentally ported part of the old couch_att test suite into an
actual "feature" that's not actually accessible through any API.
|
| |
|
|
|
| |
Index builder performs writes in the same transaction as the changes
feed so we can't use iterators as they disable writes.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously changes feeds would fail if they streamed data for more than five
seconds. This was because of the FoundationDB's transaction time limit. After
the timeout fired, an 1007 (transaction_too_long) error was raised, and
transaction was retried. The emitted changes feed would often crash or simple
hang because the HTTP state would be garbled as response data was re-sent over
the same socket stream again.
To fix the issue introduce a new `{restart_tx, true}` option for
`fold_range/4`. This option sets up a new transaction to continue iterating
over the range from where the last one left off.
To avoid data being resent in the response stream, user callback functions must
first read all the data they plan on sending during that callback, send it out,
and then after that it must not do any more db reads so as not to trigger a
`transaction_too_old` error.
|
| |
|
|
|
| |
Previously only `POST` with a list of keys was supported. The new `GET`
support just dumps all database info blobs in a single ordered response.
|
| |
|
|
|
|
| |
This API allows for listing all database info blobs in a single request.
It accepts the same parameters as `_all_dbs` for controlling pagination
of results and so on.
|
| | |
|
| |
|
|
|
| |
This way we can expose the total view size for a database in the dbinfo
JSON blob.
|
| |
|
|
| |
Attachment names should be binaries
|
| |
|
|
|
|
| |
* Encode startkey/endkey for all_docs
Encodes the startkey/endkey so that if a startkey is not binary it will return the expected results.
|
| | |
|
| | |
|
| | |
|
