| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
These were only used in one place so not a whole lot of reason to move
them out of the handle_deleted_dbs_info function clauses. Solely a style
thing.
Though in the undelete bit I did do rename `source_timestamp` to
`timestamp` which I think we decided to do on the PR discussion.
|
| | |
|
| |
|
|
|
| |
The diff makes it look like the other functions were moved which is a
bit confusing. The end result is the same though.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
Previously we are using the DbName to set DbPrefix for clarity. In order
to support soft-deletion while providing efficient value for DbPrefix
allocation, we use value allocated with erlfdb_hca for DbPrefix.
|
| |
|
|
|
|
| |
* add info endpoint
This commit adds the info endpoint for design docs stored in fdb.
|
| |
|
|
|
|
|
|
| |
The e520294c7ee3f55c3e8cc7d528ff37a5a93c800f commit inadvertently
changed the `fabric2_fdb:refresh/1` head matching to accept db
instances with no names. `couch_jobs` uses those but they are not
cached in fabric2_server. So here we return to the previous matching
rule where contexts without names don't get refreshed from the cache.
|
| |
|
|
|
|
|
|
|
| |
Check that on a transaction restarts `database_does_not_exist` error is
thrown properly if database was re-created.
Also we forgot to properly unload the mocked erlfdb module in
`tx_too_old_mock_erlfdb/0` so we make sure to do that, otherwise it
has a chance of messing up subsequent tests.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously it was possible for a database to be re-created while a `Db` handle
was open and the `Db` handle would continue operating on the new db without any
error.
To avoid that situation ensure instance UUID is explicitly checked during open
and reopen calls. This includes checking it after the metadata is loaded in
`fabric2_fdb:open/2` and when fetching the handle from the cache.
Also, create a `{uuid, UUID}` option to allow specifying a particular instance
UUID when opening a database. If that instance doesn't exist raise a
`database_does_not_exist` error.
|
| |\
| |
| | |
Add additional get_doc spans
|
| |/ |
|
| |
|
|
|
|
|
|
| |
* Use the `row/3` helper function in a few more places
* Make a `run_query/3` function to shorten all the query calls
* A few minor emilio suggesions (whitespace, comma issues, ...)
|
| |
|
|
|
|
|
|
| |
Previously transactions could time-out, retry and re-emit the same data. Use
the same mechanism as the _list_dbs and _changes feeds to fix it. Additional
detail in the mailing list discussion:
https://lists.apache.org/thread.html/r02cee7045cac4722e1682bb69ba0ec791f5cce025597d0099fb34033%40%3Cdev.couchdb.apache.org%3E
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`list_dbs_info/3` maintains a queue of up to 100 futures which are used to
concurrently fetch data. Previously, if the transaction was reset, and the
accumulator inside the fold may have had futures from a previous transaction,
which have gotten their results yet, they threw a transaction_canceled (1025)
error.
To fix this, if we're in a read-only transaction, we return the tx object in
the opaque db info record. Then, if `erlfdb:wait/1` throws a transaction
canceled error, we re-fetch the future from the now restarted transaction.
Potentially, the transaction may also time-out while the futures queues is
drained after the main range fold has finished already. Handle that case by
reseting the transaction and then re-fetching the futures. To avoid an infinite
loop we allow up to 2 retries only.
This approach is not the most optimal but simpler as it hides the complexity
inside the fabric2_fdb module where we already handle these conditions. It
means that every 5 or so seconds we might have to refetch less than 100 extra
futures from the queue (as some or all may have gotten their results back
already).
|
| | |
|
| |
|
|
|
|
|
|
| |
* Made a silly error building the accumulator list:
`[Row, Acc]` -> `[Row | Acc]`
* Left some debugging code in `list_dbs_tx_too_old` test
The test was supposed to setup only 1 failure in the user callback
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* There was a good amount of duplication between `_db_crud_tests` and
`_changes_fold_tests`, so make a common test utility module so both suites can
use.
* Clean up test names. Previously some were named `tx_too_long` but since the
official FDB error is `transaction_too_old` rename them to match a bit
better.
* `list_dbs_info` implementation queue of 100 futures to parallelize fetching.
So its test was update to create more than 100 dbs. Creating 100 dbs took
about 3 seconds so add a small parallel map (pmap) utility function to help
with that.
|
| |
|
|
|
|
|
|
|
| |
Previously those endpoints would break when transactions time-out and are
retried. To fix it we re-use the mechanism from changes feeds.
There is a longer discussion about this on the mailing list:
https://lists.apache.org/thread.html/r02cee7045cac4722e1682bb69ba0ec791f5cce025597d0099fb34033%40%3Cdev.couchdb.apache.org%3E
|
| |
|
|
|
|
|
|
| |
Seeing log errors like the following:
CRASH REPORT Process epep_fdb_decision_cache (<0.633.0>) with 0 neighbors exited with reason: {bad_info,{#Ref<0.2506675824.3127640065.49278>,ready}} at gen_server:handle_common_reply/8(line:726) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_expiring_cache_server,init,['Argument__1']}, ancestors: [epep_sup,<0.596.0>], message_queue_len: 0, messages: [], links: [<0.614.0>], dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.8.15449617>,next => #Fun<..>,...},...}},...], trap_exit: false, status: running, heap_size: 2586, stack_size: 27, reductions: 7493102
This should handle those errors, and prevent the crashes.
|
| |
|
|
| |
Use `fabric2_db:is_clustered/1` instead of `couch_db:is_clustered`
|
| |
|
|
|
|
|
| |
* couch_jobs resubmit updates job data
When a job is either pending or finished and the job is resubmitted
with new data the job data is updated.
|
| |
|
|
|
| |
Fixes an issue where the first k/v was skipped if the startkey_docid
was included.
|
| |
|
|
|
|
|
|
|
|
| |
* Register with the fabric2_index module
* Provide a build_indices/2 callback
* In the callback attempt to parse an `#mrst{}` indexing context and then
trigger an indexing job. We don't wait for the job to finish but instead
rely on `couch_job`'s max worker limit to keep concurrency in check.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The main logic is as follows:
- After doc update transaction(s) are completed in the `fabric2_db` module,
call the `fabric2_index:db_updated(Db)` function.
- `fabric2_index:db_updated(Db)` inserts a `{DbName, Timestamp}` tuple into
one of the sharded ets tables. The tuple is inserted using `ets:insert_new`,
which ensures only the first entry succeeds and others will be ignored,
until that entry is cleared.
- Each ets table in `fabric2_index` has a simple monitor process that
periodically scans that table. If it finds databases which have been
updated, it notifies all the indices which have registered with
`fabric2_index` to build indices.
There are config settings to disable index auto-updating and to adjust the
delay interval, and the resolution. The interval specifies how long to wait
since the first time the db was modified. The resolution interval specifies how
often to check the ets tables.
Just like in the original ken, design documents can have an `"autoupdate":
false` option to disable auto-updating that design document only.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Previously even when the scheduled time was the same the job was still deleted
and re-inserted into the pending queue. Now we perform the re-enqueing
operation only if the scheduled time has changed. So if the whole operation is
run in its own transaction, the transaction will now become a read-only
transaction.
This optimization should come in handy with the indexing auto-builder, for
example, where multiple nodes might try to re-enqueue the job and then only the
first would succeed, and the rest will perform a quick one row read and not do
any writes.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
These tests exercise that logic when updating rows in a map index. It
works by generating two states and a transition between those states.
That is, the set of rows that exist in the index, and the set of rows
that exist after an update along with how many rows should be kept in
common.
This is a fairly exhaustive search of the state space in these
transitions. A full run takes approximately 60s on my laptop so I have
limited the set of tests to a random selection of 100 single and multi
transition tests (i.e., updating either one or five documents in a
single view build iteration).
|
| |
|
|
|
| |
This saves us a key/value pair for any document that is not included as
part of a view.
|
| |
|
|
|
|
|
|
|
| |
If a map function emits duplicate keys for a document this stores
multiple rows in the map index differentiated by a `DupeId` counter.
Previously we were attempting to save some work avoiding clearing ranges
for keys that would be overwritten. However, if a document update causes
fewer duplicates to be emitted for the same key we left orphaned rows in
the index.
|
| |
|
|
|
| |
This moves index meta information to its own key space so that we can
scan all existing indexes efficiently.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Both a performance and style cleanup. This reduces the number of ets
calls by roughly `2 * 2 * $num_docs * $num_views`. The first factor of
two is because this avoids re-calculating the size twice for both the id
and map indexes. The second factor of two is because we have to lookup
two different settings.
Stylistically this also moves the logic out of the fdb modules. Keeping
key/value structure and update logic is already pretty complicated so we
should avoid mixing external application logic into those modules as
much as possible for our sanity.
The behavior is slightly changed vs the original patch as well.
Originally only the view that contained the errant key or value was
affected. However, pre-existing behavior where a document fails to be
mapped correctly resulted in it being excluded from all view indexes
defined in the design document. That behavior has been restored.
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
view_sig -> view_seq
|
| | |
|
| |
|
|
|
| |
The Reason term in the exception is no longer wrapped with an `{error,
Reason}` tuple since its now an `error` exception.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This test adds coverage for the 4 fdb_to_revinfo compatibility clauses,
and will help ensure any additional future clauses will not break
backwards compatibility.
Module coverage without this test:
fabric2_fdb : 92%
Module coverage with this test:
fabric2_fdb : 94%
|
| |
|
|
|
|
|
|
|
|
| |
A bug was introduced in https://github.com/apache/couchdb/commit/eb1a09e114dafc55fa9511d477b9ada4350134eb#diff-32274bcb4785f432811085d2e08c3580L1227
when `CURR_REV_FORMAT` was bumped from `1->2`, but `?CURR_REV_FORMAT` in
compatibility clauses going from format version `0->1` were left
unchanged, when those values of `?CURR_REV_FORMAT` should have changed
to `1`.
This fixes the compatibility clauses for rev format versions from `0->1`
|
| |
|
|
|
|
|
|
| |
Currently, a 5-tuple `Val` parameter crashes fdb_to_revinfo with a
function_clause error.
This adds a compatible function clause, and also a catchall clause to
log an error and throw an informative exception.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since we started re-using the main changes feed transaction to open the docs
https://github.com/apache/couchdb/commit/1bceb552594af404961e4ab8e6f88cffa1548f69,
we also started to pass the transactional db handle to the ddoc cache. However
ddoc cache uses a `spawn_monitor` to open docs, and since our transaction
objects are tied to the owner process the open was failing with a `badarg`
error from erlfdb.
This could be seen by running the replication elixir tests:
```
make elixir tests=test/elixir/test/replication_test.exs:214
```
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The reason why previous attempt failed is because it overrode [important logic in
`process_url/2`](https://github.com/myfreeweb/httpotion/blob/v3.1.2/lib/httpotion.ex#L34:L35):
```
def process_url(url, options) do
process_url(url)
|> prepend_protocol
|> append_query_string(options)
end
```
This PR fixes the problem by adding the `prepend_protocol`
and `append_query_string`. It also refactor the way base_url
is passed around.
|
| |
|
|
|
| |
We started to emit that in CouchDB 4.x for temporary views and possibly other
endpoints.
|
