| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
|\
| |
| | |
Do not git ignore src/ioq subfolder
|
|/
|
|
|
| |
We include ioq in the main repo currently so it doesn't make sense to
ignore it.
|
| |
|
|
|
|
|
| |
Create new config options in `couchdb` and `smoosh` sections to enable
finer control of compaction logging levels.
|
|
|
|
|
| |
These entries are logged because of exceptions, and so should be
at least warning level.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For non-existent ddocs, a new ddoc_cache_entry is spawned for each
call to ddoc_cache:open. Multiple calls to open the same non-existent
ddoc will create multiple cache entries with the same Key but
different PIDs. This can result in khash lookups returning `not_found`
as in the error below:
[error] 2021-12-10T02:25:21.622743Z node1@127.0.0.1 <0.18923.9> -------- gen_server ddoc_cache_lru terminated with reason: no match of right hand value not_found at ddoc_cache_lru:remove_key/2(line:308) <= ddoc_cache_lru:handle_info/2(line:219) <= gen_server:try_dispatch/4(line:637) <= gen_server:handle_msg/6(line:711) <= proc_lib:init_p_do_apply/3(line:249)
This checks the return values of `khash:lookup` and only proceeds to
delete keys if the results are other than `not_found`.
|
| |
|
|
|
|
|
|
|
|
| |
It's difficult to add rebar.config files per application when that
file name is ignored by git.
Also, ken and smoosh are now core CouchDB applications, so it's no
longer necessary to git ignore them.
|
|
|
|
|
|
| |
src/couch/src/test_util.erl:276:16: Warning: variable 'Timeout' is unused
src/test_util.erl:276:25: Warning: variable 'Delay' is unused
src/couchdb/src/couch/src/test_util.erl:276:32: Warning: variable 'Started' is unused
|
|\
| |
| | |
Fix smoosh enqueueing not found dbs and typo
|
|/ |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`couch_key_tree:stem/2`, as seen in
https://github.com/apache/couchdb/pull/3963, has a potential to consume quite a
bit of memory. Replacing sets with maps helped in that case however, since
stemming has a non-tail recursive section, there is a chance in future versions
of Erlang to experience the same behavior again.
As a safeguard, add a memory limit test by stemming a larger conflicted rev
tree while limiting the maximum process heap size. For that, use the nifty
`max_heap_size` process flag, which ensures a process is killed if it starts
using too much memory.
To reduce the flakiness, use a deterministic tree shape by using a hard-coded
seed value and leave a decent margin of error for the memory limit.
Ref: https://github.com/apache/couchdb/pull/3963
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
sets to due to a compiler bug in 22 consumes too much memory and is slower on Erlang 22+
Reproducer: https://gist.github.com/nickva/ddc499e6e05678faf20d344c6e11daaa
With sets:
```
couch_key_tree:gen_and_stem().
{8,6848.812683105469}
```
With maps:
```
couch_key_tree:gen_and_stem().
{0,544.000732421875}
```
|
|\
| |
| | |
Add smoosh queue persistence
|
|/ |
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
configurable
The code that forwards attachment data to cluster nodes via fabric has a
hard-coded timeout of five minutes for nodes to request the data. Making
this configurable lets us mitigate the impact of issue #3939 [1], which
causes requests to block if one of the nodes already has the given
attachment and doesn't end up requesting the data for it.
[1]: https://github.com/apache/couchdb/issues/3939
|
|
|
|
|
|
| |
`ibrowse` responses may crash the job process with a function clause in
`handle_info/2`. Ingore those with a note that aliases should hopefully fix
this issue in the future.
|
|\
| |
| | |
Debug for sharded index server
|
| | |
|
|/ |
|
|
|
|
| |
`replicated_edits` should be `replicated_changes`
|
|
|
|
|
|
|
| |
Couldn't reproduce flakiness locally but it failed in CI at least twice.
Try to extend the wait timeout and poll frequence, and set `commit_freq = 0`
for some tests.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Checking a few scenarios:
* _design/*/_info returns collation versions
* A 2.x view can upgraded to have a view info map
* A 3.2.1 view can be upgraded to have a view info map
* A view with multiple collator versions:
- Can be opened and tagged with the current libicu version
- Can be queried, written to and then queried again
- Will be submitted for compaction when open and on updates
- After compaction the view will have only one collator version
Includes two new fixture views:
* Version 3.2.1 view without the view info map field
* A view with a bogus (old) libicu version "1.1.1.1"
The bogus libicu version view is generated by overriding the collation
version API to return `[1, 1, 1, 1]` [1]. And it can be re-created
with this snippet [2]
[1]
```
update_collator_versions(#{} = ViewInfo) ->
Versions = maps:get(ucol_vs, ViewInfo, []),
- Ver = tuple_to_list(couch_ejson_compare:get_collator_version()),
+ % Ver = tuple_to_list(couch_ejson_compare:get_collator_version()),
+ Ver = [1, 1, 1, 1],
```
[2] `http` is the `httpie` command line HTTP client
```
http delete $DB/colltest1 && http put $DB/colltest1'?q=1' && http put $DB/colltest1/_design/colltest1ddoc views:='{"colltest1view":{"map":"function(doc){emit(doc.id, null);}"}}' && http put $DB/colltest1/d1 a=b && http put $DB/colltest1/d2 c=d
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, libicu collator versions were not tracked, and during major OS
version upgrades, it was possible to experience apparent data loss due to
collation order changes between libicu library versions. The view order
inconsistency would last until the view is compacted.
This commit introduces a view info map in the header which records the list of
libicu collator versions used by that view. The collator versions list is
checked and updated every time a view is opened.
The new view info map is re-using a previously removed view header field from
2.x views. The upgrade logic from 2.x to 3.x ignores that header field, and
this allows for transparent downgrading back to 3.2.1, and then upgrading back
to 3.2.1+ versions, all while keeping the same view signature.
If there is no collator version recorded in the view header, the first time the
view is opened, the header will be upgraded to record the current libicu
version. It's possible to avoid immediately writting the upgraded header and
instead delaying till the next view data update with this setting:
```
[view_upgrade]
commit_on_header_upgrade = false
```
By default it's toggled to `true`, meaning the view header will be written
immediately.
The list of collator version is returned in the _design/*/_info response. This
allows users to easily track the condition when the view is built or opened
with more than one libicu collator versions.
Views which have more than one collator versions are submitted for
re-compaction to the "upgrade_views" channel. This behavior is triggered both
on update (which is the typical smoosh trigger mechanism), and when opened.
Triggering on open is inteded to be used with read-only views, which may not be
updated after libicu upgrades, and so would perpetually emit inconsistent data.
Automatic re-compaction may be disabled with a config setting:
```
[view_upgrade]
compact_on_collator_upgrade = false
```
The default value is `true`.
|
|
|
|
|
|
|
|
|
|
| |
We already return the collation algorithm version and the libicu library version,
but since we're tracking the opaque collator versions in the view, it is
beneficial to show that to the user as well.
Thanks to Will Young for the idea [1]
[1] https://lists.apache.org/thread/rqfwrt4kszz79l3wxxtg6zwygz6my8p2
|
|
|
|
|
|
|
|
| |
get_collator_version/0 calls ucol_getVersion() C API [1]. It returns
an opaque sequence of 4 bytes which encodes both the base UCA version
and any tailorings which may affect collation order.
[1] https://unicode-org.github.io/icu-docs/apidoc/dev/icu4c/ucol_8h.html#a0f98dd01ba7a64069ade6f0fda13528d
|
|\
| |
| | |
Include index sig in _search_info response
|
|/ |
|
|\
| |
| | |
Add couch_mrview_debug:view_signature/2
|
|/ |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In a rare, unsupported scenario, a user can call _bulk_docs with
new_edits:false and a VDU that denies the update of certain documents.
When nodes are out of sync (under load and mem3_repl can't keep up),
an update may or may not update the revision tree.
When two nodes extend the revision tree and forbids the update,
but one node is behind, an {error, W, [{Doc, FirstReply} | Acc]}
is returned instead of {ok, _} or {accepted, _}. The _bulk_docs
request does not accept {error, _} which leads to a function_clause.
This fix changes the return value to:
{ok, W, [{Doc, {forbidden, Msg}} | Acc]}
when any of the nodes forbids the update.
Updating one document is also addressed so if the same issue occurs
during one document update, a 403 is still returned.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Remove leftover CentOS 6 RPM bits
* Stop publishing SM 1.8.5 package for Debian 10
We're using the native SM 60 package in the CI system. If you look
closely you'll see that we weren't actually successfully publishing that
package anyway, so removing this line is a no-op.
* Publish Debian 11 packages
* Publish packages on main, not master
* Apply labels directly to build steps
|
|\
| |
| | |
Execute chttpd_dbs_info_tests in clean database_dir
|
|/ |
|
|\
| |
| | |
Execute fabric_rpc_tests in clean database_dir
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The `fabric_rpc_tests` pollutes the state of `shards_db` which causes flakiness
of other tests. This PR fixes the problem by configuring temporary `database_dir`.
The important implementation detail is that we need to wait for all `couch_server`
processes to restart. Before initroduction of sharded couch server in the
https://github.com/apache/couchdb/pull/3366 this could be done as:
```erlang
test_util:with_process_restart(couch_server, fun() ->
config:set("couchdb", "database_dir", NewDatabaseDir)
end),
```
This method has to be updated to support sharded couch_server. Following auxilary
functions where added:
- `couch_server:names/0` - returns list of registered names of each
`couch_server` process
- `test_util:with_processes_restart/{2,4}` - waits all process to be restarted
returns `{Pids :: #{} | timeout, Res :: term()}`
- `test_util:with_couch_server_restart/1` - waits for all `couch_server` processes
to finish restart
The new way of configuring `database_dir` in test suites is:
```erlang
test_util:with_couch_server_restart(fun() ->
config:set("couchdb", "database_dir", NewDatabaseDir)
end),
```
|
| |
|
|
|
|
| |
In order to fix formatting issue outlined in https://github.com/apache/couchdb-documentation/pull/704
|
| |
|
| |
|
|
|
|
| |
Already fixed by @lostnet on main.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is one of those situations where you go in to make a small change,
see an opportunity for some refactoring, and get sucked into a rabbit
hole that leaves you wondering if you have any idea how computers
actually work. My initial goal was simply to update the Erlang version
used in our binary packages to a modern supported release. Along the
way I decided I wanted to figure out how to eliminate all the copypasta
we generate for making any change to this file, and after a few days of
hacking here we are. This rewrite has the following features:
* Updates to use Debian 11 (current stable) as the base image for
building releases and packaging repos.
* Defaults to Erlang 23 as the embedded Erlang version in packages. We
avoid Erlang 24 for now because Clouseau is not currently compatible.
* Dynamically generates the parallel build stages used to test and
package CouchDB on various OSes. This is accomplished through a bit
of scripted pipeline code that relies on two new methods defined at
the beginning of the Jenkinsfile, one for "native" builds on macOS
and FreeBSD and one for container-based builds. See comments in the
Jenkinsfile for additional details.
* Expands commands like `make check` into a series of steps to improve
visibility. The Jenkins UI will now show the time spent in each step
of the build process, and if a step (e.g. `make eunit`) fails it will
only expand the logs for that step by default instead of showing the
logs for the entire build stage. The downside is that if we do make
changes to the series of targets underneath `check` we need to
remember to update the Jenkinsfile as well.
* Starts per-stage timer _after_ agent is acquired. Previously builds could
fail with a 15m timeout when all they did was sit in the build queue.
This is a cherry-pick of 9b6454b with the following modifications:
- Drop the MINIMUM_ERLANG_VERSION to 20
- Drop the packaging ERLANG_VERSION to 23
- Add the weatherreport-test as a build step
- Add ARM and POWER back into the matrix using a new buildx-based
multi-architecture container image.
|
| |
|
|
|
|
|
|
|
|
| |
The previous response order of
{accepted, [{Doc1, {accepted, Doc2}}, {Doc2, {accepted, Doc1}}
looked a bit odd so update the order to look as expected.
|
|
|
|
|
|
|
|
|
|
|
|
| |
The code that generates suite.elixir will repreatedly strip the "test "
from the name of the test when writing the file, resulting in a mismatch
between the actual test name and what's in suite.elixir. You can see
this by searching for e.g. COUCHDB-497 in the suite file.
I tried using String.replace_prefix instead of String.replace_leading in
test_name() but that function seems to get called multiple times during
the test grouping. Simpler to just avoid naming the tests that way for
now.
|