<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/couchdb.git, branch prototype/fdb-layer</title>
<subtitle>github.com: apache/couchdb.git
</subtitle>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/couchdb.git/'/>
<entry>
<title>convert erlfdb_error 2101 to transaction_too_large (#3215)</title>
<updated>2020-10-20T12:49:29+00:00</updated>
<author>
<name>Peng Hui Jiang</name>
<email>jiangph@cn.ibm.com</email>
</author>
<published>2020-10-20T12:49:29+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/couchdb.git/commit/?id=7bdb432849f477d0c4451aaf9d41b35ab076e9f3'/>
<id>7bdb432849f477d0c4451aaf9d41b35ab076e9f3</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Update and clean up tests</title>
<updated>2020-09-15T20:13:46+00:00</updated>
<author>
<name>Nick Vatamaniuc</name>
<email>vatamane@apache.org</email>
</author>
<published>2020-08-28T08:36:18+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/couchdb.git/commit/?id=ae858196848cf9533dfa03a2006227481f47388d'/>
<id>ae858196848cf9533dfa03a2006227481f47388d</id>
<content type='text'>
Update tests to use the new replicator. Also clean up redundancy and re-use
some of the newer macros from fabric2 (?TDEF_FE).

Make sure replicator tests are included in `make check`
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Update tests to use the new replicator. Also clean up redundancy and re-use
some of the newer macros from fabric2 (?TDEF_FE).

Make sure replicator tests are included in `make check`
</pre>
</div>
</content>
</entry>
<entry>
<title>Update replicator's readme file</title>
<updated>2020-09-15T20:13:46+00:00</updated>
<author>
<name>Nick Vatamaniuc</name>
<email>vatamane@apache.org</email>
</author>
<published>2020-08-28T08:36:12+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/couchdb.git/commit/?id=99262909129602bceac82e7907ebfcafc9eba629'/>
<id>99262909129602bceac82e7907ebfcafc9eba629</id>
<content type='text'>
Remove sections which don't apply anymore and describe briefly how frontend and
backend interact.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Remove sections which don't apply anymore and describe briefly how frontend and
backend interact.
</pre>
</div>
</content>
</entry>
<entry>
<title>Update and cleanup default.ini replicator entries</title>
<updated>2020-09-15T20:13:46+00:00</updated>
<author>
<name>Nick Vatamaniuc</name>
<email>vatamane@apache.org</email>
</author>
<published>2020-08-28T08:36:05+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/couchdb.git/commit/?id=b718d3380cbc8dc5799f36e19e1a94b8c2e00c2d'/>
<id>b718d3380cbc8dc5799f36e19e1a94b8c2e00c2d</id>
<content type='text'>
Update settings with defaults. Also comment out values which are already set to
default in the code.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Update settings with defaults. Also comment out values which are already set to
default in the code.
</pre>
</div>
</content>
</entry>
<entry>
<title>Update replicator http handlers and supervisor</title>
<updated>2020-09-15T20:13:46+00:00</updated>
<author>
<name>Nick Vatamaniuc</name>
<email>vatamane@apache.org</email>
</author>
<published>2020-08-28T16:49:59+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/couchdb.git/commit/?id=aa8836434655b9b1b34c6d72035eb306266c484b'/>
<id>aa8836434655b9b1b34c6d72035eb306266c484b</id>
<content type='text'>
Stich everything together: the backend, frontend and http handlers.

The supervisor `couch_replicator_sup` handles starting a set of fronted or
backend children. It may also start both or neither.

The HTTP layer for monitoring and creating jobs is simpler than before since
there is rpc and clustering involved.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Stich everything together: the backend, frontend and http handlers.

The supervisor `couch_replicator_sup` handles starting a set of fronted or
backend children. It may also start both or neither.

The HTTP layer for monitoring and creating jobs is simpler than before since
there is rpc and clustering involved.
</pre>
</div>
</content>
</entry>
<entry>
<title>Update backend replicator modules</title>
<updated>2020-09-15T20:13:46+00:00</updated>
<author>
<name>Nick Vatamaniuc</name>
<email>vatamane@apache.org</email>
</author>
<published>2020-08-28T08:35:07+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/couchdb.git/commit/?id=7dd18993f25b107db05a9e99e01770c7f84bd33b'/>
<id>7dd18993f25b107db05a9e99e01770c7f84bd33b</id>
<content type='text'>
Backend replicator modules execute replication jobs. The two main modules
reponsible for job management and execution are `couch_replicator_job` and
`couch_replicator_job_server`.

`couch_replicator_job`

 - Is the main process of each replication job. When this process starts, it
   waits in the `couch_jobs:accept/2` call. This may take an indefinite amount
   of time. The spawned `couch_replicator_job` waiting in accept like that is
   called internally an "acceptor". The main pattern of execution is multiple
   acceptors are started, and after some of them accept jobs, they become
   "workers".

 - After it accepts a job, it parses the `couch_jobs` job data, which contains
   the `Rep` object and calculates the replication ID from it. Replication ID
   calculation may involve making a call to the source endpoint in order to
   fetch the contents of the javascript filter. Then, the `Rep` object and the
   replication ID is used to construct the internal `#rep_state{}` state record
   of the `gen_server`.

 - Multiple replication jobs may end up trying to run the same replication
   (with the same replication ID) concurrently. To manage these types of
   colisions, `check_ownership/3` function is called to determine if the
   current replication is the correct `owner` of that replication. If it is
   not, then the job maybe fail and exit.

 - There is a periodic checkpoint timer which sends a `checkpoint` message. The
   checkpoint frequency is calculated as the minimum of the `couch_jobs`
   activity timeout and the configured checkpoint interval. During each
   checkpoint attempt, there is a call to `couch_jobs:update/3` which updates
   the job's data with latest state and ensure the job doesn't get re-enqueued
   due to inactivity.

 - If the job completes, then `couch_jobs:finish/3` is called and the
   replication process exits `normal`. If the job crashes, there is a
   consecutive error count field (`?ERROR_COUNT`) which, is used to calculate
   the backoff penalty. There is an exponential backoff schedule, that starts
   with the base value, then doubles, but only up to a maximum value. Both the
   base and the maximum values are configurable with the
   `min_backoff_penalty_sec` and `max_backoff_penalty_sec` settings
   respecively. This is an improvement from before where the users could only
   influence the maximum backoff penalty by reducing the number of failed
   events kept by each job.

`couch_replicator_server`

 - This is a module which spawns and keeps track of `couch_replicator_job`
   processes.

 - Periodically, every `interval_sec` seconds, it runs the `reschedule` function
   which checks for pending jobs. If they are some, it will start acceptors to
   run them. If those acceptors become workers, and if the total number of
   workers goes above the `max_jobs` setting, the oldest `continuous` workers
   will be stopped until the total number of jobs falls below `max_jobs` value.

 - In addition to `max_jobs` limit, there is a `max_churn` limit which
   determines up to how many job starts to allow during each scheduling
   interval. As jobs are started, they reduce the available churn "budget" for
   that cycle and after it goes below 0 no more jobs can start until the next
   cycle.

 - This module also performs transient job cleanup. After transient jobs stop
   running previously they simply vanished but with this update they maybe
   linger for at least `transient_job_max_age_sec` seconds.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Backend replicator modules execute replication jobs. The two main modules
reponsible for job management and execution are `couch_replicator_job` and
`couch_replicator_job_server`.

`couch_replicator_job`

 - Is the main process of each replication job. When this process starts, it
   waits in the `couch_jobs:accept/2` call. This may take an indefinite amount
   of time. The spawned `couch_replicator_job` waiting in accept like that is
   called internally an "acceptor". The main pattern of execution is multiple
   acceptors are started, and after some of them accept jobs, they become
   "workers".

 - After it accepts a job, it parses the `couch_jobs` job data, which contains
   the `Rep` object and calculates the replication ID from it. Replication ID
   calculation may involve making a call to the source endpoint in order to
   fetch the contents of the javascript filter. Then, the `Rep` object and the
   replication ID is used to construct the internal `#rep_state{}` state record
   of the `gen_server`.

 - Multiple replication jobs may end up trying to run the same replication
   (with the same replication ID) concurrently. To manage these types of
   colisions, `check_ownership/3` function is called to determine if the
   current replication is the correct `owner` of that replication. If it is
   not, then the job maybe fail and exit.

 - There is a periodic checkpoint timer which sends a `checkpoint` message. The
   checkpoint frequency is calculated as the minimum of the `couch_jobs`
   activity timeout and the configured checkpoint interval. During each
   checkpoint attempt, there is a call to `couch_jobs:update/3` which updates
   the job's data with latest state and ensure the job doesn't get re-enqueued
   due to inactivity.

 - If the job completes, then `couch_jobs:finish/3` is called and the
   replication process exits `normal`. If the job crashes, there is a
   consecutive error count field (`?ERROR_COUNT`) which, is used to calculate
   the backoff penalty. There is an exponential backoff schedule, that starts
   with the base value, then doubles, but only up to a maximum value. Both the
   base and the maximum values are configurable with the
   `min_backoff_penalty_sec` and `max_backoff_penalty_sec` settings
   respecively. This is an improvement from before where the users could only
   influence the maximum backoff penalty by reducing the number of failed
   events kept by each job.

`couch_replicator_server`

 - This is a module which spawns and keeps track of `couch_replicator_job`
   processes.

 - Periodically, every `interval_sec` seconds, it runs the `reschedule` function
   which checks for pending jobs. If they are some, it will start acceptors to
   run them. If those acceptors become workers, and if the total number of
   workers goes above the `max_jobs` setting, the oldest `continuous` workers
   will be stopped until the total number of jobs falls below `max_jobs` value.

 - In addition to `max_jobs` limit, there is a `max_churn` limit which
   determines up to how many job starts to allow during each scheduling
   interval. As jobs are started, they reduce the available churn "budget" for
   that cycle and after it goes below 0 no more jobs can start until the next
   cycle.

 - This module also performs transient job cleanup. After transient jobs stop
   running previously they simply vanished but with this update they maybe
   linger for at least `transient_job_max_age_sec` seconds.
</pre>
</div>
</content>
</entry>
<entry>
<title>Update frontend replicator modules</title>
<updated>2020-09-15T20:13:46+00:00</updated>
<author>
<name>Nick Vatamaniuc</name>
<email>vatamane@apache.org</email>
</author>
<published>2020-08-28T08:34:21+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/couchdb.git/commit/?id=5b98e8a6c169449d1a3e362e52e86822ef350ed5'/>
<id>5b98e8a6c169449d1a3e362e52e86822ef350ed5</id>
<content type='text'>
The frontend is the part responsible for parsing replication parameters and
creating replication jobs. Most of that happens in the `couch_replicator`
module.

 - `couch_replicator:replicate/2` is the main API for creating transient
   replication jobs.

 - Replication jobs from `_replicator` documents are updated from
  `couch_replicator:after_*` callbacks. `after_db_create/2` besides being
  called on db creation also gets called when a database is undeleted and
  `add_jobs_from_db/1` function will attempt to parse them all.

`couch_replicator` exports monitoring functions `docs/2,3 and jobs/0,1`. Those
get called from HTTP handlers for `_scheduler/docs` and `_scheduler/jobs`
respectively.

For hands-on remsh access there some debuging functions such as:

 - rescan_jobs/0,1 : Simulates a db being re-created so all the jobs are added
 - reenqueue_jobs/0,1 : Deletes all the jobs from a db then re-adds them
 - remove_jobs/0 : Removes all the replication jobs
 - get_job_ids/0 : Read the RepId -&gt; JobId mapping area
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The frontend is the part responsible for parsing replication parameters and
creating replication jobs. Most of that happens in the `couch_replicator`
module.

 - `couch_replicator:replicate/2` is the main API for creating transient
   replication jobs.

 - Replication jobs from `_replicator` documents are updated from
  `couch_replicator:after_*` callbacks. `after_db_create/2` besides being
  called on db creation also gets called when a database is undeleted and
  `add_jobs_from_db/1` function will attempt to parse them all.

`couch_replicator` exports monitoring functions `docs/2,3 and jobs/0,1`. Those
get called from HTTP handlers for `_scheduler/docs` and `_scheduler/jobs`
respectively.

For hands-on remsh access there some debuging functions such as:

 - rescan_jobs/0,1 : Simulates a db being re-created so all the jobs are added
 - reenqueue_jobs/0,1 : Deletes all the jobs from a db then re-adds them
 - remove_jobs/0 : Removes all the replication jobs
 - get_job_ids/0 : Read the RepId -&gt; JobId mapping area
</pre>
</div>
</content>
</entry>
<entry>
<title>Introduce couch_replicator_jobs abstraction module</title>
<updated>2020-09-15T20:13:46+00:00</updated>
<author>
<name>Nick Vatamaniuc</name>
<email>vatamane@apache.org</email>
</author>
<published>2020-08-28T08:33:18+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/couchdb.git/commit/?id=3c9b7540cbb41225b35c89b741e0c5b83cdbf4e1'/>
<id>3c9b7540cbb41225b35c89b741e0c5b83cdbf4e1</id>
<content type='text'>
This is the `couch_jobs` abstraction module. All replicator calls to
`couch_jobs` should go through it. This module takes care of adding types to
some of the API calls, handles maintencence of the RepId -&gt; JobId mappings when
jobs are added and removed, and some subscription logic.

`fabric2.hrl` include file is updated with the definition of the
`?REPLICATION_IDS` prefix where the RepId -&gt; JobId keyspace lives.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This is the `couch_jobs` abstraction module. All replicator calls to
`couch_jobs` should go through it. This module takes care of adding types to
some of the API calls, handles maintencence of the RepId -&gt; JobId mappings when
jobs are added and removed, and some subscription logic.

`fabric2.hrl` include file is updated with the definition of the
`?REPLICATION_IDS` prefix where the RepId -&gt; JobId keyspace lives.
</pre>
</div>
</content>
</entry>
<entry>
<title>Update couch_replicator_ids</title>
<updated>2020-09-15T20:13:46+00:00</updated>
<author>
<name>Nick Vatamaniuc</name>
<email>vatamane@apache.org</email>
</author>
<published>2020-08-28T08:33:11+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/couchdb.git/commit/?id=276d19731bc5df73838f40efc126f1f709e04fbe'/>
<id>276d19731bc5df73838f40efc126f1f709e04fbe</id>
<content type='text'>
This module is responsible for calculating replication IDs. It inspects all the
replication options which may affect the replication results and hashes them
into a single ID. CouchDB replicator tries to maintain compatibility with older
versions of itself so it keep tracks of how to calculate replication IDs used
by previous version of CouchDB. Replication ID calculation algorithms have
their own version, the latest one is at version 4.

One of the goals of this update is to not alter the replication ID algorithm
and keep it at version 4, such that for all the same parameters the replication
IDs should stay the same as they would be on CouchDB &lt;= 3.x. That is why in
some internal function, options maps and binares are turned back into proplist
and tuples before hashing is performed. There is a unit tests which asserts
that the replication ID calcuated with this update matches what was calcuated
in CouchDB 3.x.

Internal representation of the replication ID has changed slighly. Previously
it was represented by a tuple of `{BaseId, ExtId}`, where `BaseId` was the ID
without any options such as `continuous` or `create_target`, and `ExtId` was
the concatenated list of those options. In most cases it was useful to operate
on the full ID and in only a few place the `BaseId` was needed. So the
calculation function was updated to return `{RepId, BaseId}` instead. `RepId`
is a binary that is the full relication ID (base + extensions) and `BaseId` is
just the base.

The function which calculated the base ID was updated to actually be called
`base_id/2` as opposed to `replication_id/2`.

Another update to the module is a function which calculates replication job
IDs. A `JobId` is used to identify replication jobs in the `couch_jobs` API. A
`JobId`, unlike a `RepId` never requires making a network round-trip to
calculate. For replications created from `_replicator` docs, `JobId` is defined
as the concatenation of the database instance UUID and document ID. For a
transient jobs it is calculated by hashing the source, target endpoint
parameters, replication options. In fact, it is almost the same as a
replication ID, with one important difference that the filter design doc name
and function name are used instead of the contents of the filter from the
source, so no network round-trip is necessary to calculate it.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This module is responsible for calculating replication IDs. It inspects all the
replication options which may affect the replication results and hashes them
into a single ID. CouchDB replicator tries to maintain compatibility with older
versions of itself so it keep tracks of how to calculate replication IDs used
by previous version of CouchDB. Replication ID calculation algorithms have
their own version, the latest one is at version 4.

One of the goals of this update is to not alter the replication ID algorithm
and keep it at version 4, such that for all the same parameters the replication
IDs should stay the same as they would be on CouchDB &lt;= 3.x. That is why in
some internal function, options maps and binares are turned back into proplist
and tuples before hashing is performed. There is a unit tests which asserts
that the replication ID calcuated with this update matches what was calcuated
in CouchDB 3.x.

Internal representation of the replication ID has changed slighly. Previously
it was represented by a tuple of `{BaseId, ExtId}`, where `BaseId` was the ID
without any options such as `continuous` or `create_target`, and `ExtId` was
the concatenated list of those options. In most cases it was useful to operate
on the full ID and in only a few place the `BaseId` was needed. So the
calculation function was updated to return `{RepId, BaseId}` instead. `RepId`
is a binary that is the full relication ID (base + extensions) and `BaseId` is
just the base.

The function which calculated the base ID was updated to actually be called
`base_id/2` as opposed to `replication_id/2`.

Another update to the module is a function which calculates replication job
IDs. A `JobId` is used to identify replication jobs in the `couch_jobs` API. A
`JobId`, unlike a `RepId` never requires making a network round-trip to
calculate. For replications created from `_replicator` docs, `JobId` is defined
as the concatenation of the database instance UUID and document ID. For a
transient jobs it is calculated by hashing the source, target endpoint
parameters, replication options. In fact, it is almost the same as a
replication ID, with one important difference that the filter design doc name
and function name are used instead of the contents of the filter from the
source, so no network round-trip is necessary to calculate it.
</pre>
</div>
</content>
</entry>
<entry>
<title>Handle option maps in lower level modules</title>
<updated>2020-09-15T20:13:46+00:00</updated>
<author>
<name>Nick Vatamaniuc</name>
<email>vatamane@apache.org</email>
</author>
<published>2020-08-28T08:33:05+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/couchdb.git/commit/?id=941cfc3d7b33cbfbf7e95eb7db388515d0595399'/>
<id>941cfc3d7b33cbfbf7e95eb7db388515d0595399</id>
<content type='text'>
The goal is to keep everything below the _api_wrap module level relatively
intact. To achieve that handle option maps in some places, or translate back to
a proplist or `#httpd{}` records in others.

The `couch_replicator_api:db_from_json/1` function is where endpoint map object
from a `Rep` object are translated into `#httpdb{}` records. Headers are
translated back to lists and ibrowse options into proplist with atom keys.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The goal is to keep everything below the _api_wrap module level relatively
intact. To achieve that handle option maps in some places, or translate back to
a proplist or `#httpd{}` records in others.

The `couch_replicator_api:db_from_json/1` function is where endpoint map object
from a `Rep` object are translated into `#httpdb{}` records. Headers are
translated back to lists and ibrowse options into proplist with atom keys.
</pre>
</div>
</content>
</entry>
</feed>
