| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
Update tests to use the new replicator. Also clean up redundancy and re-use
some of the newer macros from fabric2 (?TDEF_FE).
Make sure replicator tests are included in `make check`
|
| |
|
|
|
| |
Remove sections which don't apply anymore and describe briefly how frontend and
backend interact.
|
| |
|
|
|
| |
Update settings with defaults. Also comment out values which are already set to
default in the code.
|
| |
|
|
|
|
|
|
|
|
| |
Stich everything together: the backend, frontend and http handlers.
The supervisor `couch_replicator_sup` handles starting a set of fronted or
backend children. It may also start both or neither.
The HTTP layer for monitoring and creating jobs is simpler than before since
there is rpc and clustering involved.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Backend replicator modules execute replication jobs. The two main modules
reponsible for job management and execution are `couch_replicator_job` and
`couch_replicator_job_server`.
`couch_replicator_job`
- Is the main process of each replication job. When this process starts, it
waits in the `couch_jobs:accept/2` call. This may take an indefinite amount
of time. The spawned `couch_replicator_job` waiting in accept like that is
called internally an "acceptor". The main pattern of execution is multiple
acceptors are started, and after some of them accept jobs, they become
"workers".
- After it accepts a job, it parses the `couch_jobs` job data, which contains
the `Rep` object and calculates the replication ID from it. Replication ID
calculation may involve making a call to the source endpoint in order to
fetch the contents of the javascript filter. Then, the `Rep` object and the
replication ID is used to construct the internal `#rep_state{}` state record
of the `gen_server`.
- Multiple replication jobs may end up trying to run the same replication
(with the same replication ID) concurrently. To manage these types of
colisions, `check_ownership/3` function is called to determine if the
current replication is the correct `owner` of that replication. If it is
not, then the job maybe fail and exit.
- There is a periodic checkpoint timer which sends a `checkpoint` message. The
checkpoint frequency is calculated as the minimum of the `couch_jobs`
activity timeout and the configured checkpoint interval. During each
checkpoint attempt, there is a call to `couch_jobs:update/3` which updates
the job's data with latest state and ensure the job doesn't get re-enqueued
due to inactivity.
- If the job completes, then `couch_jobs:finish/3` is called and the
replication process exits `normal`. If the job crashes, there is a
consecutive error count field (`?ERROR_COUNT`) which, is used to calculate
the backoff penalty. There is an exponential backoff schedule, that starts
with the base value, then doubles, but only up to a maximum value. Both the
base and the maximum values are configurable with the
`min_backoff_penalty_sec` and `max_backoff_penalty_sec` settings
respecively. This is an improvement from before where the users could only
influence the maximum backoff penalty by reducing the number of failed
events kept by each job.
`couch_replicator_server`
- This is a module which spawns and keeps track of `couch_replicator_job`
processes.
- Periodically, every `interval_sec` seconds, it runs the `reschedule` function
which checks for pending jobs. If they are some, it will start acceptors to
run them. If those acceptors become workers, and if the total number of
workers goes above the `max_jobs` setting, the oldest `continuous` workers
will be stopped until the total number of jobs falls below `max_jobs` value.
- In addition to `max_jobs` limit, there is a `max_churn` limit which
determines up to how many job starts to allow during each scheduling
interval. As jobs are started, they reduce the available churn "budget" for
that cycle and after it goes below 0 no more jobs can start until the next
cycle.
- This module also performs transient job cleanup. After transient jobs stop
running previously they simply vanished but with this update they maybe
linger for at least `transient_job_max_age_sec` seconds.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The frontend is the part responsible for parsing replication parameters and
creating replication jobs. Most of that happens in the `couch_replicator`
module.
- `couch_replicator:replicate/2` is the main API for creating transient
replication jobs.
- Replication jobs from `_replicator` documents are updated from
`couch_replicator:after_*` callbacks. `after_db_create/2` besides being
called on db creation also gets called when a database is undeleted and
`add_jobs_from_db/1` function will attempt to parse them all.
`couch_replicator` exports monitoring functions `docs/2,3 and jobs/0,1`. Those
get called from HTTP handlers for `_scheduler/docs` and `_scheduler/jobs`
respectively.
For hands-on remsh access there some debuging functions such as:
- rescan_jobs/0,1 : Simulates a db being re-created so all the jobs are added
- reenqueue_jobs/0,1 : Deletes all the jobs from a db then re-adds them
- remove_jobs/0 : Removes all the replication jobs
- get_job_ids/0 : Read the RepId -> JobId mapping area
|
| |
|
|
|
|
|
|
|
|
| |
This is the `couch_jobs` abstraction module. All replicator calls to
`couch_jobs` should go through it. This module takes care of adding types to
some of the API calls, handles maintencence of the RepId -> JobId mappings when
jobs are added and removed, and some subscription logic.
`fabric2.hrl` include file is updated with the definition of the
`?REPLICATION_IDS` prefix where the RepId -> JobId keyspace lives.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This module is responsible for calculating replication IDs. It inspects all the
replication options which may affect the replication results and hashes them
into a single ID. CouchDB replicator tries to maintain compatibility with older
versions of itself so it keep tracks of how to calculate replication IDs used
by previous version of CouchDB. Replication ID calculation algorithms have
their own version, the latest one is at version 4.
One of the goals of this update is to not alter the replication ID algorithm
and keep it at version 4, such that for all the same parameters the replication
IDs should stay the same as they would be on CouchDB <= 3.x. That is why in
some internal function, options maps and binares are turned back into proplist
and tuples before hashing is performed. There is a unit tests which asserts
that the replication ID calcuated with this update matches what was calcuated
in CouchDB 3.x.
Internal representation of the replication ID has changed slighly. Previously
it was represented by a tuple of `{BaseId, ExtId}`, where `BaseId` was the ID
without any options such as `continuous` or `create_target`, and `ExtId` was
the concatenated list of those options. In most cases it was useful to operate
on the full ID and in only a few place the `BaseId` was needed. So the
calculation function was updated to return `{RepId, BaseId}` instead. `RepId`
is a binary that is the full relication ID (base + extensions) and `BaseId` is
just the base.
The function which calculated the base ID was updated to actually be called
`base_id/2` as opposed to `replication_id/2`.
Another update to the module is a function which calculates replication job
IDs. A `JobId` is used to identify replication jobs in the `couch_jobs` API. A
`JobId`, unlike a `RepId` never requires making a network round-trip to
calculate. For replications created from `_replicator` docs, `JobId` is defined
as the concatenation of the database instance UUID and document ID. For a
transient jobs it is calculated by hashing the source, target endpoint
parameters, replication options. In fact, it is almost the same as a
replication ID, with one important difference that the filter design doc name
and function name are used instead of the contents of the filter from the
source, so no network round-trip is necessary to calculate it.
|
| |
|
|
|
|
|
|
|
|
| |
The goal is to keep everything below the _api_wrap module level relatively
intact. To achieve that handle option maps in some places, or translate back to
a proplist or `#httpd{}` records in others.
The `couch_replicator_api:db_from_json/1` function is where endpoint map object
from a `Rep` object are translated into `#httpdb{}` records. Headers are
translated back to lists and ibrowse options into proplist with atom keys.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This module is in responsible for parsing either an HTTP `_replicate` request
body, or a _replicator doc into an internal `Rep` object (an Erlang map).
`parse_transient_rep/2` parses _replicate requests. It also handles
cancelations, where requests bodies look like ```{"id": ..., "cancel": true}```
instead of having all the expected parameters.
`parse_rep_doc/1` parses _replicator docs.
Parsing consists of 3 main parts:
- Parsing the endpoint definitions: source and target url, headers, TLS bits
and proxies
- Parsing options into an options map, possibly using defaults from config
parameters
- Parsing socket parameters. These now have a hard-coded allow-list as opposed
accepting all possible Erlang socket options.
The parsing function also double as validation function which gets called from
the _replicator's before_doc_update callback when users update replication
documents. They would get an immediate feedback if their replicationd document
is malformed.
Everything is turned into a map object. This object should be able to be
serialized and de-serialized to (from) JSON.
Since maps are used, add the definitions of some common fields
couch_replicator.hrl. Mistyping them should raise a compiler error.
couch_replicator_docs lost all of its parsing function and also functions which
update intermediate replication doc states (triggered and error). It still
handles function which relate to interacting with _replicator docs.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Remove unused functions and some function used only from one place like `sum_stats/2`.
* Update time functions to use the more modern `erlang:system_time/1` API.
* `parse_int_param/5` and `parse_replication_states/1` was moved from the old
_httpd_util module as they were they only ones need from there.
* `default_headers_map/0` Used to the default httpd record headers as a map
since part of the replication data will be kept as map object.
* `proplist_options/1` Some parts of the replicator, like _httpc and _api_wrap
still use proplist options, so this function can be used to translate
options as maps to a proplist version.
|
| |
|
|
| |
These modules are not used by the new replicator.
|
| |
|
|
|
|
|
|
|
|
| |
fold_jobs/4 is a generalized folding API which can be used to process all the
jobs of a particular type.
get_pending_count/2,3 can be used to get the count of pending jobs. It takes
the same options as accept, including a limit and `max_sched_time`. Just like
accept it reads the range of pending jobs as a snapshot to avoid generating
read conflicts.
|
| |
|
|
|
| |
Also, ensure to use the same options as other couch apps: force_utf8 and
dedupe_keys.
|
| |
|
|
|
|
|
|
|
|
| |
Previously the data was read from the parser in the transaction. If the
transaction had to retry, for example, because of a conflict, the parser would
have been drained and exited resulting the request failing with a 500
"mp_parser noproc" error.
Since FDB cannot handle transactions larger than 10MB opt to read the
attachment data into memory first, before the transaction starts.
|
| |
|
|
|
|
| |
`after_db_create/2` and `after_db_delete/2` are when databases are created and
deleted respectively. The callbacks are called with both the database name and
the database instance UUID values.
|
| |
|
| |
We need to call StartFun as it might add headers, etc.
|
| |
|
|
|
|
|
| |
Depending on the computer and ordering of tests couch_rate will
sometimes give a budget of 1000 or more. This leads the active tasks
test to grab the initial _active_task blob which contains
`"changes_done": 0` which fails the test.
|
| |
|
|
|
|
| |
This mostly helps with flaky tests where some jobs might complete before the
type monitor discovers this particular type, so opt to always re-scan and start
notification monitors when any type timeout is set.
|
| | |
|
| |\
| |
| | |
Make COPY doc return only one "ok"
|
| |/ |
|
| |
|
|
|
|
|
|
|
|
|
| |
When set, every response is sent once fully generated on the server
side. This increases memory usage on the nodes but simplifies error
handling for the client as it eliminates the possibility that the
response will be deliberately terminated midway through due to a
timeout.
The config value can be changed at runtime without impacting any
in-flight responses.
|
| |
|
|
|
| |
This keeps validation during tests but disables the validation during
production to avoid the overhead of collation.
|
| |
|
|
|
|
| |
Using lists:umerge/3 adds extra invocations of the collation algorithm
because its using `=<` semantics when ebtree collations are capable of
producing `lt, eq, gt` results.
|
| |
|
|
| |
Inner nodes of the B+Tree are now immutable so that they can be cached.
|
| |
|
|
|
| |
This allows looking up multiple keys simultaneously which reduces the
amount of overhead due to node serialization and collation.
|
| |
|
|
|
| |
This allows for batch insertion of keys in order to minimize node
serialization and collation costs.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This makes it compatible with CouchDB <= 3.x where we can create deleted
documents.
How to check:
```
$ http put $DB1/mydb
$ http put $DB1/mydb/foo _deleted:='true' a=b
{
"id": "foo",
"ok": true,
"rev": "1-ad7eb689fcae75e7a7edb57dc1f30939"
}
```
|
| |
|
|
|
|
|
| |
Empty maps maybe useful to initialize the data in some cases but we don't want
to emit an entry in the output with just an empty map.
While at it, add some tests to check the basics.
|
| |\
| |
| | |
Fix ordering of page_size based pagination for views
|
| |/
|
|
|
| |
The pagination relied on id of the document. However for views it should use
combination of key and id.
|
| |\
| |
| | |
clear jobs data in active area during removal
|
| |/
|
|
|
|
|
| |
During job removal, it was not cleared from the active area so
active_tasks would mistakenly believe the job still existed. When we
try to actually open the data it is not there and not_found error
would be issued.@nickva found this issue during replication work.
|
| |\
| |
| | |
Clear sensitive flag at end of public api functions
|
| |/ |
|
| |\
| |
| | |
Don't log client disconnects
|
| | | |
|
| |/
|
|
|
|
|
| |
Any error there would just be generating a case clause.
Remove the `{not_found, missing}` clause since it was accidentally matching on
the Rev string and the case was included in the `_Else` clause anyway.
|
| |
|
|
| |
Fixes the case where no writes are done for an index, the rater limiter
assumed it was a failure.
|
| |\
| |
| | |
Pluggable persist_fun
|
| |/ |
|
| |\
| |
| | |
Prototype/fdb layer ebtree enhance
|
| | | |
|
| |/ |
|
