summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* fix: consistent use of new config sections makes tests less flakeyfix/proxyauth-testJan Lehnardt2022-04-092-4/+2
|
* Merge pull request #3982 from apache/do-not-git-ignore-ioqNick Vatamaniuc2022-04-031-1/+0
|\ | | | | Do not git ignore src/ioq subfolder
| * Do not git ignore src/ioq subfolderNick Vatamaniuc2022-04-031-1/+0
|/ | | | | We include ioq in the main repo currently so it doesn't make sense to ignore it.
* Fix skip and limit parametersjiahuili2022-03-314-25/+85
|
* Implement configurable compaction log levelsJay Doane2022-03-297-18/+57
| | | | | Create new config options in `couchdb` and `smoosh` sections to enable finer control of compaction logging levels.
* Convert notice to warning for exceptionsJay Doane2022-03-292-2/+2
| | | | | These entries are logged because of exceptions, and so should be at least warning level.
* Handle `not_found` lookups removing ddoc cache keyJay Doane2022-03-291-10/+24
| | | | | | | | | | | | | For non-existent ddocs, a new ddoc_cache_entry is spawned for each call to ddoc_cache:open. Multiple calls to open the same non-existent ddoc will create multiple cache entries with the same Key but different PIDs. This can result in khash lookups returning `not_found` as in the error below: [error] 2021-12-10T02:25:21.622743Z node1@127.0.0.1 <0.18923.9> -------- gen_server ddoc_cache_lru terminated with reason: no match of right hand value not_found at ddoc_cache_lru:remove_key/2(line:308) <= ddoc_cache_lru:handle_info/2(line:219) <= gen_server:try_dispatch/4(line:637) <= gen_server:handle_msg/6(line:711) <= proc_lib:init_p_do_apply/3(line:249) This checks the return values of `khash:lookup` and only proceeds to delete keys if the results are other than `not_found`.
* Give smoosh eunit test coverageJay Doane2022-03-271-0/+2
|
* Simplify adding rebar.config per applicationJay Doane2022-03-271-3/+0
| | | | | | | | It's difficult to add rebar.config files per application when that file name is ignored by git. Also, ken and smoosh are now core CouchDB applications, so it's no longer necessary to git ignore them.
* Eliminate unused variable warningsJay Doane2022-03-271-1/+1
| | | | | | src/couch/src/test_util.erl:276:16: Warning: variable 'Timeout' is unused src/test_util.erl:276:25: Warning: variable 'Delay' is unused src/couchdb/src/couch/src/test_util.erl:276:32: Warning: variable 'Started' is unused
* Merge pull request #3970 from noahshaw11/fix-smoosh-enqueueing-not-found-dbsJay Doane2022-03-262-5/+19
|\ | | | | Fix smoosh enqueueing not found dbs and typo
| * Fix smoosh enqueueing not found dbs and typoncshaw2022-03-262-5/+19
|/
* Add test to ensure rev tree stemming doesn't take too much memoryNick Vatamaniuc2022-03-241-1/+47
| | | | | | | | | | | | | | | | | | `couch_key_tree:stem/2`, as seen in https://github.com/apache/couchdb/pull/3963, has a potential to consume quite a bit of memory. Replacing sets with maps helped in that case however, since stemming has a non-tail recursive section, there is a chance in future versions of Erlang to experience the same behavior again. As a safeguard, add a memory limit test by stemming a larger conflicted rev tree while limiting the maximum process heap size. For that, use the nifty `max_heap_size` process flag, which ensures a process is killed if it starts using too much memory. To reduce the flakiness, use a deterministic tree shape by using a hard-coded seed value and leave a decent margin of error for the memory limit. Ref: https://github.com/apache/couchdb/pull/3963
* Remove include couch_db_intncshaw2022-03-221-3/+2
|
* Opimtize key tree stemming by using maps instead of setsNick Vatamaniuc2022-03-211-5/+5
| | | | | | | | | | | | | | | | | | sets to due to a compiler bug in 22 consumes too much memory and is slower on Erlang 22+ Reproducer: https://gist.github.com/nickva/ddc499e6e05678faf20d344c6e11daaa With sets: ``` couch_key_tree:gen_and_stem(). {8,6848.812683105469} ``` With maps: ``` couch_key_tree:gen_and_stem(). {0,544.000732421875} ```
* Merge pull request #3766 from noahshaw11/add-smoosh-queue-persistenceiilyak2022-03-1517-89/+733
|\ | | | | Add smoosh queue persistence
| * Add smoosh queue persistencencshaw2022-03-1417-89/+733
|/
* Win32-SM91 support and fixesWill2022-03-123-15/+12
|
* 3.x: Add rebar3 and erlfmt install commands to configure.ps1 #3873Will2022-03-122-11/+52
|
* feat(couchjs): add support for SpiderMonkey 91esrWill Young2022-03-123-18/+37
|
* feat: make the timeout for receiving requests from attachment writers ↵James Coglan2022-03-082-2/+9
| | | | | | | | | | | | configurable The code that forwards attachment data to cluster nodes via fabric has a hard-coded timeout of five minutes for nodes to request the data. Making this configurable lets us mitigate the impact of issue #3939 [1], which causes requests to block if one of the nodes already has the given attachment and doesn't end up requesting the data for it. [1]: https://github.com/apache/couchdb/issues/3939
* Fix function clause in couch_replicator_scheduler_jobNick Vatamaniuc2022-03-041-1/+7
| | | | | | `ibrowse` responses may crash the job process with a function clause in `handle_info/2`. Ingore those with a note that aliases should hopefully fix this issue in the future.
* Merge pull request #3933 from cloudant/debug-for-sharded-index-serveriilyak2022-02-233-12/+372
|\ | | | | Debug for sharded index server
| * Add couch_index_debug.erl and restart functionsILYA Khlopotov2022-02-222-3/+328
| |
| * Fix couch_debug:print_linked_processes/1 to work with sharded couch_index_serverILYA Khlopotov2022-02-222-9/+44
|/
* Fix replicated_changes typo for purge doc updatesNick Vatamaniuc2022-02-181-1/+1
| | | | `replicated_edits` should be `replicated_changes`
* Fix flaky collator auto-upgrade testNick Vatamaniuc2022-02-151-9/+17
| | | | | | | Couldn't reproduce flakiness locally but it failed in CI at least twice. Try to extend the wait timeout and poll frequence, and set `commit_freq = 0` for some tests.
* View collation upgrade testsNick Vatamaniuc2022-02-116-18/+392
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Checking a few scenarios: * _design/*/_info returns collation versions * A 2.x view can upgraded to have a view info map * A 3.2.1 view can be upgraded to have a view info map * A view with multiple collator versions: - Can be opened and tagged with the current libicu version - Can be queried, written to and then queried again - Will be submitted for compaction when open and on updates - After compaction the view will have only one collator version Includes two new fixture views: * Version 3.2.1 view without the view info map field * A view with a bogus (old) libicu version "1.1.1.1" The bogus libicu version view is generated by overriding the collation version API to return `[1, 1, 1, 1]` [1]. And it can be re-created with this snippet [2] [1] ``` update_collator_versions(#{} = ViewInfo) -> Versions = maps:get(ucol_vs, ViewInfo, []), - Ver = tuple_to_list(couch_ejson_compare:get_collator_version()), + % Ver = tuple_to_list(couch_ejson_compare:get_collator_version()), + Ver = [1, 1, 1, 1], ``` [2] `http` is the `httpie` command line HTTP client ``` http delete $DB/colltest1 && http put $DB/colltest1'?q=1' && http put $DB/colltest1/_design/colltest1ddoc views:='{"colltest1view":{"map":"function(doc){emit(doc.id, null);}"}}' && http put $DB/colltest1/d1 a=b && http put $DB/colltest1/d2 c=d ```
* Track libicu collator versions in the view headerNick Vatamaniuc2022-02-116-26/+171
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, libicu collator versions were not tracked, and during major OS version upgrades, it was possible to experience apparent data loss due to collation order changes between libicu library versions. The view order inconsistency would last until the view is compacted. This commit introduces a view info map in the header which records the list of libicu collator versions used by that view. The collator versions list is checked and updated every time a view is opened. The new view info map is re-using a previously removed view header field from 2.x views. The upgrade logic from 2.x to 3.x ignores that header field, and this allows for transparent downgrading back to 3.2.1, and then upgrading back to 3.2.1+ versions, all while keeping the same view signature. If there is no collator version recorded in the view header, the first time the view is opened, the header will be upgraded to record the current libicu version. It's possible to avoid immediately writting the upgraded header and instead delaying till the next view data update with this setting: ``` [view_upgrade] commit_on_header_upgrade = false ``` By default it's toggled to `true`, meaning the view header will be written immediately. The list of collator version is returned in the _design/*/_info response. This allows users to easily track the condition when the view is built or opened with more than one libicu collator versions. Views which have more than one collator versions are submitted for re-compaction to the "upgrade_views" channel. This behavior is triggered both on update (which is the typical smoosh trigger mechanism), and when opened. Triggering on open is inteded to be used with read-only views, which may not be updated after libicu upgrades, and so would perpetually emit inconsistent data. Automatic re-compaction may be disabled with a config setting: ``` [view_upgrade] compact_on_collator_upgrade = false ``` The default value is `true`.
* Return the opaque collator version from _node/*/_versionsNick Vatamaniuc2022-02-112-9/+13
| | | | | | | | | | We already return the collation algorithm version and the libicu library version, but since we're tracking the opaque collator versions in the view, it is beneficial to show that to the user as well. Thanks to Will Young for the idea [1] [1] https://lists.apache.org/thread/rqfwrt4kszz79l3wxxtg6zwygz6my8p2
* Add get_collator_version/0 libicu NIF functionNick Vatamaniuc2022-02-113-2/+32
| | | | | | | | get_collator_version/0 calls ucol_getVersion() C API [1]. It returns an opaque sequence of 4 bytes which encodes both the base UCA version and any tailorings which may affect collation order. [1] https://unicode-org.github.io/icu-docs/apidoc/dev/icu4c/ucol_8h.html#a0f98dd01ba7a64069ade6f0fda13528d
* Merge pull request #3927 from apache/index_sigRobert Newson2022-02-112-2/+9
|\ | | | | Include index sig in _search_info response
| * Include index sig in _search_info responseRobert Newson2022-02-102-2/+9
|/
* Merge pull request #3929 from cloudant/add-couch_mrview_debugiilyak2022-02-101-0/+50
|\ | | | | Add couch_mrview_debug:view_signature/2
| * Add couch_mrview_debug:view_signature/2ILYA Khlopotov2022-02-101-0/+50
|/
* Fix new_edits:false and VDU function_clause (#3909)Tony Sun2022-02-091-3/+243
| | | | | | | | | | | | | | | | | | In a rare, unsupported scenario, a user can call _bulk_docs with new_edits:false and a VDU that denies the update of certain documents. When nodes are out of sync (under load and mem3_repl can't keep up), an update may or may not update the revision tree. When two nodes extend the revision tree and forbids the update, but one node is behind, an {error, W, [{Doc, FirstReply} | Acc]} is returned instead of {ok, _} or {accepted, _}. The _bulk_docs request does not accept {error, _} which leads to a function_clause. This fix changes the return value to: {ok, W, [{Doc, {forbidden, Msg}} | Acc]} when any of the nodes forbids the update. Updating one document is also addressed so if the same issue occurs during one document update, a 403 is still returned.
* Fix publication of nightly packages (#3925)Adam Kocoloski2022-02-051-23/+17
| | | | | | | | | | | | | | | * Remove leftover CentOS 6 RPM bits * Stop publishing SM 1.8.5 package for Debian 10 We're using the native SM 60 package in the CI system. If you look closely you'll see that we weren't actually successfully publishing that package anyway, so removing this line is a no-op. * Publish Debian 11 packages * Publish packages on main, not master * Apply labels directly to build steps
* Merge pull request #3923 from jiahuili430/81-fixup-dbs-infoiilyak2022-02-041-311/+216
|\ | | | | Execute chttpd_dbs_info_tests in clean database_dir
| * Execute chttpd_dbs_info_tests in clean database_dirjiahuili2022-02-031-311/+216
|/
* Merge pull request #3921 from cloudant/fix-flaky-test-in-fabric_rpciilyak2022-02-013-1/+51
|\ | | | | Execute fabric_rpc_tests in clean database_dir
| * Execute fabric_rpc_tests in clean database_dirILYA Khlopotov2022-02-013-1/+51
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The `fabric_rpc_tests` pollutes the state of `shards_db` which causes flakiness of other tests. This PR fixes the problem by configuring temporary `database_dir`. The important implementation detail is that we need to wait for all `couch_server` processes to restart. Before initroduction of sharded couch server in the https://github.com/apache/couchdb/pull/3366 this could be done as: ```erlang test_util:with_process_restart(couch_server, fun() -> config:set("couchdb", "database_dir", NewDatabaseDir) end), ``` This method has to be updated to support sharded couch_server. Following auxilary functions where added: - `couch_server:names/0` - returns list of registered names of each `couch_server` process - `test_util:with_processes_restart/{2,4}` - waits all process to be restarted returns `{Pids :: #{} | timeout, Res :: term()}` - `test_util:with_couch_server_restart/1` - waits for all `couch_server` processes to finish restart The new way of configuring `database_dir` in test suites is: ```erlang test_util:with_couch_server_restart(fun() -> config:set("couchdb", "database_dir", NewDatabaseDir) end), ```
* Clear junk test dbs before running chttpd_dbs_info_testsjiahuili2022-01-313-4/+8
|
* Bump docs to 3.2.1-1Nick Vatamaniuc2022-01-311-1/+1
| | | | In order to fix formatting issue outlined in https://github.com/apache/couchdb-documentation/pull/704
* Display 3.x build status in READMEAdam Kocoloski2022-01-251-2/+2
|
* Update mix.lockAdam Kocoloski2022-01-251-2/+2
|
* Properly test for SM60 on ARM buildsAdam Kocoloski2022-01-251-2/+2
| | | | Already fixed by @lostnet on main.
* Refactor Jenkins to dynamically generate stagesAdam Kocoloski2022-01-252-586/+278
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is one of those situations where you go in to make a small change, see an opportunity for some refactoring, and get sucked into a rabbit hole that leaves you wondering if you have any idea how computers actually work. My initial goal was simply to update the Erlang version used in our binary packages to a modern supported release. Along the way I decided I wanted to figure out how to eliminate all the copypasta we generate for making any change to this file, and after a few days of hacking here we are. This rewrite has the following features: * Updates to use Debian 11 (current stable) as the base image for building releases and packaging repos. * Defaults to Erlang 23 as the embedded Erlang version in packages. We avoid Erlang 24 for now because Clouseau is not currently compatible. * Dynamically generates the parallel build stages used to test and package CouchDB on various OSes. This is accomplished through a bit of scripted pipeline code that relies on two new methods defined at the beginning of the Jenkinsfile, one for "native" builds on macOS and FreeBSD and one for container-based builds. See comments in the Jenkinsfile for additional details. * Expands commands like `make check` into a series of steps to improve visibility. The Jenkins UI will now show the time spent in each step of the build process, and if a step (e.g. `make eunit`) fails it will only expand the logs for that step by default instead of showing the logs for the entire build stage. The downside is that if we do make changes to the series of targets underneath `check` we need to remember to update the Jenkinsfile as well. * Starts per-stage timer _after_ agent is acquired. Previously builds could fail with a 15m timeout when all they did was sit in the build queue. This is a cherry-pick of 9b6454b with the following modifications: - Drop the MINIMUM_ERLANG_VERSION to 20 - Drop the packaging ERLANG_VERSION to 23 - Add the weatherreport-test as a build step - Add ARM and POWER back into the matrix using a new buildx-based multi-architecture container image.
* fixup get _dbs_info corner casejiahuili2022-01-251-11/+15
|
* Fix doc order in fabric_doc_update testsNick Vatamaniuc2022-01-201-5/+5
| | | | | | | | The previous response order of {accepted, [{Doc1, {accepted, Doc2}}, {Doc2, {accepted, Doc1}} looked a bit odd so update the order to look as expected.
* Do not start test name with "test "Adam Kocoloski2022-01-202-2/+2
| | | | | | | | | | | | The code that generates suite.elixir will repreatedly strip the "test " from the name of the test when writing the file, resulting in a mismatch between the actual test name and what's in suite.elixir. You can see this by searching for e.g. COUCHDB-497 in the suite file. I tried using String.replace_prefix instead of String.replace_leading in test_name() but that function seems to get called multiple times during the test grouping. Simpler to just avoid naming the tests that way for now.