| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This commit aims to improve Mango by reducing the data transferred to
the coordinator during query execution. It may reduce memory or CPU use
at the coordinator but that isn't the primary goal.
Currently, when documents are read at the shard level, they are compared
locally at the shard with the selector to ensure they match before they
are sent to the coordinator. This ensures we're not sending documents
across the network that the coordinator immediately discards, saving
bandwidth and coordinator processing. This commit further executes field
projection (`fields` in the query) at the shard level. This should
further save bandwidth, particularly for queries that project few fields
from large documents.
One item of complexity is that a query may request a quorum read of
documents, meaning that we need to do the document read at the
coordinator and not the shard, then perform the `selector` and `fields`
processing there rather than at the shard. To ensure that documents are
processed consistently whether at the shard or coordinator,
match_and_extract_doc/3 is added. There is still one orphan call outside
match_and_extract_doc/2 to extract/2 which supports cluster upgrade and
should later be removed.
Shard level processing is already performed in a callback, view_cb/2,
that's passed to fabric's view processing to run for each row in the
view result set. It's used for the shard local selector and fields
processing. To make it clear what arguments are destined for this
callback, the commit encapsulates the arguments, using viewcbargs_new/2
and viewcbargs_get/2.
As we push down more functionality to the shard, the context this
function needs to carry with it will increase, so having a record for it
will be valuable.
Supporting cluster upgrades:
The commit supports shard pushdown for Mango `fields` processing for
situations during rolling cluster upgrades.
In the state where the coordinator is speaking to an upgraded node, the
view_cb/2 needs to support being passed just the `selector` outside of
the new viewcbargs record. In this case, the shard will not process
fields, but the coordinator will.
In the situation where the coordinator is upgraded but the shard is not,
we need to send the selector to the shard via `selector` and also
execute the fields projection at the coordinator. Therefore we pass
arguments to view_cb/2 via both `selector` and `callback_args` and have
an apparently spurious field projection (mango_fields:extract/2) in the
code that receives back values from the shard ( factored out into
doc_member_and_extract).
Both of these affordances should only need to exist through one minor
version change and be removed thereafter -- if people are jumping
several minor versions of CouchDB in one go, hopefully they are prepared
for a bit of trouble.
Testing upgrade states:
As view_cb is completely separate from the rest of the cursor code,
we can first try out the branch's code using view_cb from `main`, and
then the other way -- the branch's view_cb with the rest of the file
from main. I did both of these tests successfully.
|
| | |
| |
| |
| |
| |
| |
| | |
I needed to understand the format of arguments to `match/2` when writing
the code to support projecting fields on the shard, so I wrote some code
to figure it out as a test. I figure this may be useful for future
work in this area, so push as commit.
|
| |/ |
|
| | |
|
| |
|
|
| |
Changes since 2.5.2: https://github.com/ferd/recon/compare/2.5.2...2.5.3
|
| |
|
|
| |
Backporting the `build-report` target from the *nix makefile
to the Windows pendant.
|
| |\
| |
| | |
docs(mango): match description of `$mod` with reality
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
| |
The remainder argument for the `$mod` operator can be zero, while
its documentation suggests otherwise. It actually covers a very
realistic use case where divisibility is expressed.
Neither related restrictions could be identified in the sources
[1] nor MongoDB forbids this [2]. Tests also seem to exercise this
specific case [3]. Thanks @iilyak for checking on these.
[1] https://github.com/apache/couchdb/blob/adf17140e81d0b74f2b2ecdea48fc4f702832eaf/src/mango/src/mango_selector.erl#L512:L513
[2] https://www.mongodb.com/docs/manual/reference/operator/query/mod/
[3] https://github.com/apache/couchdb/blob/0059b8f90e58e10b199a4b768a06a762d12a30d3/src/mango/test/03-operator-test.py#L58
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Previously, when replication jobs used _bulk_get they didn't upload design
docs individually like they do when not using _bulk_get.
Here were are preserving an already existing behavior which we had in the
replicator without _bulk_get usage for more than 2 years. It was introduced
here in #2426. Related to these issues #2415 and #2413.
Add tests to cover both attachments and ddoc cases. meck:num_calls/3 is helpful
as it allows to nicely assert which API function was called and how many times.
|
| |
|
|
| |
Thanks, LSP!
|
| |
|
|
|
|
|
|
|
|
|
| |
The total_rows property was decremented by one to account for the VDU
that was automatically added to the total. Now that a BDU has replaced
the VDU [1] total_rows is one less than it should be.
This removes the decrement so that total_rows equals the actual doc
count.
[1] https://github.com/apache/couchdb/pull/4274
|
| |
|
|
|
|
| |
./configure uses version 91 of Spidermonkey as default now. The previous
commit (9a92fbbe3130127703db004614d836348e13ad16) stated that the source
code of 1.8.5 is removed. This is not true, the code still exists, the
commit message wasn't updated when updating the previous PR.
|
| |
|
|
|
| |
Removes the old Spidermonkey 1.8.5 source code.
Spidermonkey 91 is set as the new default version.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With more recent OTP versions, this test has been observed to fail
with the following error:
chttpd_dbs_info_test:88: -dbs_info_test_/0-fun-22- (should_return_500_time_out_when_time_is_not_enough_for_get_dbs_info)...*failed*
in function chttpd_dbs_info_test:should_return_500_time_out_when_time_is_not_enough_for_get_dbs_info/1 (test/eunit/chttpd_dbs_info_test.erl, line 192)
in call from eunit_test:run_testfun/1 (eunit_test.erl, line 71)
in call from eunit_proc:run_test/1 (eunit_proc.erl, line 531)
in call from eunit_proc:with_timeout/3 (eunit_proc.erl, line 356)
in call from eunit_proc:handle_test/2 (eunit_proc.erl, line 514)
in call from eunit_proc:tests_inorder/3 (eunit_proc.erl, line 456)
in call from eunit_proc:with_timeout/3 (eunit_proc.erl, line 346)
in call from eunit_proc:run_group/2 (eunit_proc.erl, line 570)
**error:{badmatch,{error,socket_closed_remotely}}
output:<<"">>
This change waits until the expected 500 timeout error is received.
|
| |
|
|
|
|
|
|
|
|
| |
Previously we handled `{bad_request, Msg}` but not `{forbidden, Msg}`, so a
`before_doc_update` handler which wanted to emulate the VDU `forbidden`
response ended up crashing with an unhandled clause instead.
(Note: The replicator doc validation BDU wasn't affected as those BDU calls are
made directly from fabric since `*_replicator` shards are "special" system
dbs).
|
| |
|
|
| |
To correctly generate the configuration reference html output,
escape special html characters like '<' and '>' in section names.
|
| | |
|
| |
|
| |
Add Prometheus port to ./configure.ps1 to catch up with ./configure
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
We don't want to register replication jobs with the module name as that
prevents us from running more than one job at a time.
This was introduced when we removed the dependence on global registrations and
switched to using pg instead.
Even more embarrassing is we didn't have tests to check multiple replication
jobs running at the same time.
|
| |
|
|
| |
Just make it use the Erlang 24 version like all the other images
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\
| |
| | |
add find_bugs makefile target for undefined functions
|
| |/ |
|
| |
|
|
|
|
|
| |
This adds a test to cover the case when a replication doc contains a
user_ctx, and subsequently executes `get_json_value/3`.
See: https://github.com/apache/couchdb/pull/4343
|
| |\ |
|
| |/
|
|
| |
Closes https://github.com/apache/couchdb-fauxton/issues/1382
|
| |
|
| |
Remind committers to backport doc changes to other branches.
|
| |
|
|
|
|
| |
Update all 23,24,25 to latest patch versions
Use 24 as a base for the full CI run
|
| | |
|
| | |
|
| |
|
|
|
|
| |
Don't run erlfmt and elixir format checks
We already ran those during the pull request CI phase
|
| | |
|
| | |
|
| |
|
|
|
| |
This is to help unblock 3.3 windows build so make check doesn't choke on
re-formatting elixir files for Windows.
|
| | |
|
| | |
|
| |
|
|
| |
Instead of having it download its own every time
|
| |
|
|
|
|
|
| |
Latest jiffy apparently is blocking windows builds so for now let's
revert to the previous version. The previous version didn't quite work
right off the bat as enc doesn't load on Erlang 25 any longer. So
there is a new 1.0.9-2 release wiht enc rebuilt on Erlang 23.
|
| |
|
|
|
|
|
|
|
| |
At some point it broke - it looks like rebar3, erlfmt and python-black
usage ended up needing the network to install/update themselves.
Make sure rebar3 and erlfmt is in bin/ alongside rebar, and make
python-black is in a separate source formatting stage step in CI and
remove it from `make check`.
|
| |
|
|
| |
It's a deprecated feature but we want to make sure it still works.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
* Since the replicator optimization issues were a bit scattered, create a
single parent one as a highlight, and also run somewhat more realistic
benchmark tests.
* Add an image macro to keep the tradition going.
* Update docs section with a 5.3.0 PR which just merged.
* Add reference to the newly merged `couchjs -v` enhancement.
|
