| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
| |
We blow up supervisors with timeouts and other errors so let's keep them
restarting for now
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
couch_jobs activity monitor is responsible for checking jobs which have not
been updated often enough by their workers and re-enqueuing them. Previously,
when the number of jobs grew high enough, couch_jobs could fail to either
iterate through all the jobs and timeout with a 1007, or try to re-enqueue too
many jobs such that the sum of the commit data would end up being larger than
the 10MB FDB limit.
couch_jobs notifier is in charge of notifying subscribers when job state
changes. If the jobs are updated, it would notify if it noticed updates,
otherwise it would notify if jobs switched to a new state (running -> pending,
running -> finished, etc). Previously, if there were too many jobs and/or the
cluster was overloaded, it was possible for the notifier to consistently fail
with timeouts.
To fix both issues introduce batching with the batch size dynamically adjusted
based on load. When consecutive errors occur the batch size will shrink
exponentially down to 1 row per transaction. Then, with each success, the batch
will grow linearly by a fixed amount. This auto-configurable behavior should
provide optimal behavior during overload and during normal operating
conditions.
For tests, since there are already tests which test enqueuing and subscription,
use the same tests but make sure they are run while errors are periodically
generated. That's accomplished with the help of `meck:loop/1` meck return
specification.
|
|
|
|
|
|
| |
This macro can be used to simplify retryable error checks throughout couch_jobs
app. It checks for erlfdb retryable errors (1007, 1009, etc), for the 1031
(`transaction_timed_out`) error and for `{timeout, _}`.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Also, add a clause for the variant without a txid part.
Previously, `next_vs/1` could overflow the batch or the txid field. The range
of values for both those is [0..16#FFFF], so the correct check before
incrementing each field should be `< 16#FFFF` instead of `=< 16#FFFF`. Since
we're dealing with bytes and in other places in the file we use 16#FFFF for max
values in the versionstamp fields, switch to hex constants.
The tests were included in the fabric2_changes_fold_tests module as next_vs is
relevant for the _changes feed since_seq calculation.
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We now host our CI containers directly under the Apache Docker Hub
org.
In addition, the newly rebuilt buster-erlang-all image has 4 Erlang
releases in it, corresponding to the latest version available in each
supported major release today:
* 20.3.8.26 (against which our 3.2 binaries will be built)
* 21.3.8.22
* 22.3.4.17
* 23.3.1
This PR changes our PR builds to run against all 4 of these versions.
|
|
|
|
|
|
|
|
|
|
|
| |
This patch introduces a macro and inserts it everywhere we catch errors
and then generatre a stacktrace.
So far the only thing that is a little bit ugly is that in two places,
I had to add a header include dependency on couch_db.erl where those
modules didn’t have any ties to couchdb/* before, alas. I’d be willing
to duplicate the macros in those modules, if we don’t want the include
dependency.
|
|
|
|
|
|
|
|
|
|
| |
* Remove non-existent applications
* Most importantly, start running all the unit test. This should include 500+ new couch tests.
* Noticed `local` application was flaky and periodically timing out in CI.
Since it's a transitive dependency of jaeger, let's skip running it for now.
It's a bit in the same category as brcypt, meck and hyper.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Update the couch_views include paths
* Exclude non-existent applications from setup logic
* Do not run tests agains the backdoor port
* Do not run tests checking for non-existent system dbs
* Since "new" couch_att attachment format has changed a bit and encoding is
not `identity` and `md5` is not `<<>>` any longer, some tests had to be
updated to set those explicitly.
|
|
|
|
|
|
|
|
|
|
| |
This is mostly a bulk search and replace to update fabric, couch_views, chttpd,
mango and couch_replicator to use either the new included file or the new
utility functions in couch_views.
The `couch_views_http:transform_row/2` function was brought from the
removed`fabric_view` module. It's used in only one place to it was copied there
directly.
|
|
|
|
|
|
|
|
| |
couch_auth_cache only handles reading server admin credentials from config files and returns the auth design doc (used in chttpd_auth_cache).
Node local `_user` docs logic has been removed. Validation to check
for _conflicts is also not needed as the "docs" proplists created from
the config server admin section don't have conflicts.
|
|
|
|
|
|
|
|
| |
`normalize_dbname/1` is not needed as database names do not have the `.couch`
suffix, and we don't have shard paths any more. For validation, send the
`DbName` to the `fabric2_db_plugin` as both the real DbName and the
"normalized" one. This is mostly to avoid changing the plugin interface for now
and should be eventually updated (in a separate PR).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* `couch_mrview_util` functions ended up mostly in `couch_views_util`.
* `couch_mrview` validatation functions ended up in `couch_views_validate`
module.
* `couch_mrview_http` functions moved to `couch_views_http_util`. The reason
they didn't end up in `couch_views_http` is because a lot of the functions
there have the exact same names as the ones in `couch_views_http_util`.
There is quite a bit of duplication involved but that is left for another
refactoring in the future. The general flow of control goes from chttpd ->
couch_views_http -> couch_views_http_util.
Most of the changes are just copy and paste with the exception of the
`ddoc_to_mrst/2` function. Previously, there were two almost identical copies
-- one in `couch_mrview_util` and another in `couch_views_util`. Both were used
by different parts of the code. The difference was the couch_views one
optionally disabled reduce functions, and replaced their body with the
`disabled` atom, while the one in `couch_mrview` didn't. Trying to unify them
such that only the `couch_views` one is used, resulted in the inability to
write design documents on server which have custom reduce disabled. That may be
a better behavior, however that should be updated in a separate PR and possibly
a mailing list discussion. So in order to preserve the exisiting behavior,
couch_eval was update to not fail in `try_compile` when design documents are
disabled.
Patches to the rest of the code to update the include path and use the new
utility functions will be updated in a separate commit.
|
|
|
|
|
|
| |
Try to minimize changes and cheated a bit by returning `false` from
`is_text_service_available()`. Also keeping in mind that we'd probably want
this functionality in the future.
|
|
|
|
| |
Un-used defines and records are removed
|
|
|
|
| |
It was replaced by fabric2_db_plugin
|
|
|
|
| |
Also remove the with_db/2 function as it's not used any longer
|
| |
|
|
|
|
|
| |
Remove ioq call from `couch_os_process:prompt/2` and
`couch_js_os_process:prompt/2`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Remove all the code related to opening and caching of databases.
However, couch_server did a few other things such as:
* Parse and maintain the CouchDB version
* Return the server "uuid" value
* The gen_server was monitoring config updates and hashing admin passwords
when they were updated
It was a 50/50 decision to move that functionality out to other modules
completely or keep it where it is. Since it wasn't just a single thing, and the
overall PR was getting rather large, opted to pair the exisiting code to the
minimum, and then later we can do another round of cleanup and find a better
place for those features.
|
|
|
|
|
|
|
|
| |
The main change is to remove `validate_docid/1,2` and use
`fabric2_db:validate_docid/1` instead.
`with_ejson_body` is also not needed as request bodies are parsed
to ejson and fabric2_fdb also deserializes to ejson.
|
|
|
|
|
|
|
| |
Remove shard handling from `couch_flags`.
`couch_db:normalize_dbname/1` call is not necessary as db names are not shards
and do not have the `.couch` extension any more.
|
|
|
|
|
|
| |
Remove logic handling index directories and couch_files. Definitely not a
comprehensive cleanup. We should probably have a separate PR to pick out what
would be useful for `main`.
|
|
|
|
| |
Index server is gone and replaced by couch_views + couch_jobs applications.
|
|
|
|
|
|
| |
Many of those tests check attachemnt upgrades from older 1-3.x formats, check
the `follows` functionality which is not supported in FDB (we don't stream
attachments but buffer them outside transactions), or import couch_bt_engine.
|
|
|
|
|
| |
`couch_task_status` is removed. `_active_tasks` endpoint is driven by the
`couch_jobs` application.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Backend (5986) port is closed. Requests to `/_node/_local/_config` and
a few other `_*` endpoints continues to work. Backend db access now
returns error code 410 (not_supported). Previously, it was accessing
couch_server and couch_files to create 3.x style local dbs.
Url handlers are updated to return `not_supported` for features which are not
coming back, and `not_implemented` for features which haven't been implemented
yet.
`parse_copy_destination_header/1` is the only function from `couch_httpd_db`
that's still needed, so it was moved to `chttpd_util` module.
`couch_httpd_db` handled `/_uuid` requests and that handler was moved to
`chttpd_misc` module.
"Welcome" endpoint (/) was updated to not call `clouseau_rpc:connected/0`.
Request handling in `couch_httpd` was removed, so most of the file is
now a bunch of utility functions mostly.
|
|
|
|
|
|
|
|
|
|
|
| |
Remove default.ini and local.ini sections related to applications and
functionality which is no longer there.
One odd change is the commented out`partitioned` flag. The reason is because
the "flags" logic is kept, and seems to be functional, however partitioned flag
is not functional. `couch_flags` loads flags using `list_to_existing_atom/1`
and since `partitioned` is not an existing atom it continuously emits crash
logs from the `_up` point related to that.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Affected applications fall in a few categories:
1. Applications which are not used any more:
* rexi
* mem3
* ddoc_cache
* smoosh
* ken
* ioq
* dreyfus
* couch_event
* snappy
* kash
* couch_plugins
2. Applications we intend to keep for 4.x, but in their current form they rely
on 3.x architecture (mem3, couch_file, clustering). These applications when
they are ready to be implemented should be copied from 3.x as they will be more
recent there:
* couch_peruser
* global_changes
* setup
3. Applications used only for utility functions, those function will be moved
to other applications:
* couch_mrview
* couch_index
4. Applications where some modules have been removed and some stayed:
* couch:
- removed: couch_db_*, couch_btree, couch_file, couch_lru, etc.
- kept: couch_doc, key tree and some others stayed
- couch_server: stripped down only to functions which are still used,
serving uuids, returning couch version and hashing passwords. It's a
candidate to update in a separate PR to split that functionality to other
modules (couch_passwords, fabric2_server).
* fabric:
- removed: all fabric_rpc modules
- kept: fabric2_* modules
* mango:
- removed clouseau pings and check for "text" service
|
| |
|
|\
| |
| | |
Prometheus endpoint
|
| | |
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| | |
We add an option to spawn a new mochiweb_http server to allow for an
additional port for scraping which does not require authentication.
The default ports are 17986, 27986, 37986 across 3 nodes.
|
|/
|
|
|
|
|
|
|
|
|
|
| |
This will be a new app add a _prometheus endpoint which will
return metrics information that adheres to the format described at
https://prometheus.io/.
Initial implementation of new _prometheus endpoint. A gen_server
waits for scraping calls while polling couch_stats:fetch and
other system info. The return value is constructed to adhere to
prometheus format and returned as text/plain. The format code
was originally written by @davisp.
|
| |
|
| |
|
|
|
|
| |
Increase timeout for continuous filtered changes with doc ids and for eventsource changes
|
| |
|
|
|
|
|
|
| |
To bring in a future `timeout` fix:
* https://github.com/apache/couchdb-erlfdb/commit/55022a566df3d98a543148ce7a0e2eb6773909ae
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This continues improvements to retryable error handling started in
https://github.com/apache/couchdb/pull/3460. Here we add the same logic we
already have for the `erlfdb:wait/2` call in
https://github.com/apache/couchdb/blob/main/src/couch_jobs/src/couch_jobs_type_monitor.erl#L55-L57
to the `get_vs_and_watch/1` section.
couch_jobs_type_monitor is meant to be linked to and run in a continuous loop
as long as the parent process is alive. If FDB becomes unavailable the main
process which we linked to or other main component (the whole application)
should crash and fail as opposed to the type monitor itself. Still, to avoid
running in a tight loop we use the holdoff interval to sleep a bit before
recursing. The typical values of the holdoff is around 50-100 msec.
|
|
|
|
|
|
| |
This makes it that the only way to validate a design doc is
through couch_mrview:validate_ddoc so that the correct couch_eval
will be used.
|
|
|
|
|
|
| |
Change the validate ddoc check to use couch_eval.
Also add in some extra functions in couch_eval so that the try_compile
will work.
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, mango tests with objects as keys were failing on CentOS 6 and
CentOS 7. The reason for the failures was that old libicu collation algorithms
didn't consider the `<<255,255,255,255>>` as the highest sortable string as
CouchDB intends it to be. Later versions of libicu, at least as old as 59,
started to do that
https://www.unicode.org/reports/tr35/tr35-collation.html#tailored_noncharacter_weights.
However, as long as we support CentOS 7 we can fix the issue by explicitly
checkign for the highest marker.
|
|\
| |
| | |
Configurable filter fields
|
| | |
|