| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
| |
This is a library for creating an FDB backed key value cache, where
each entry has a `stale` and `expires` time associated with it. Once
the current time exceeds the `expires` time, the entry is automatically
removed. The `stale` time can be used to indicate that a refresh is
necessary, while still returning a non-expired value. It is potentially
useful for implementing e.g. caches to external systems of record, such
as OAuth 2.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This fixes attachment handling to properly remove attachment data when
it is no longer referenced by the document's revision tree.
As implemented this accounts for the possibility that multiple revisions
may reference a given attachment id. However, due to how the current
revision tree removes revisions aggressively it's not currently possible
for multiple leaf revisions to share an underlying attachment. This is
because when attempting to use a stub attachment when replicating in a
conflict we will encounter the `missing_stub` error because the previous
shared revision has already been removed.
|
| | |
|
| |
|
|
|
| |
This change ensures that all test names are visible and follows a single
unified pattern for each test module.
|
| |
|
|
|
| |
Retry building a failing index for a set number of retries. If it
never completes, then return the error to the user.
|
| |
|
|
|
| |
If the view is already built, read the view status and view
in the same transaction. This will improve view read latency.
|
| | |
|
| |
|
|
|
|
| |
This adds operation names to all valid HTTP end points. This covers all
of `make elixir` except for seven requests that are testing specific
error conditions in URL and Methods for various endpoints.
|
| |
|
|
|
|
| |
The implementation follows the RFC [1]
[1]: https://github.com/apache/couchdb-documentation/blob/master/rfcs/013-node-types.md
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, membership check was disabled when fetching the security doc. That
was not correct, as membership should be checked before every db operation,
including when fetching the security doc itself.
Also, most of the security tests relied on patching the user context in the
`Db` handle then calling `check_*` functions. Those functions however call
`get_security/1` before doing the actual check, and in some cases, like when
checking for admin, the failure was coming from the membership check in
`get_security/1` instead. Also some tests were going through the regular
request path of opening a new db by name. In order, make the tests more
uniform, switch all the tests to apply the tested `UserCtx` in the open call.
|
| |
|
|
|
|
| |
Forgot to push this in the previous PR so made a new commit.
https://github.com/apache/couchdb/pull/2300#discussion_r346592418
|
| |
|
|
|
|
|
|
|
|
|
| |
Previously `set_config/3` needed keys and values to be transalted to binaries,
now that is done inside the function. It's a bit more consistent as binary
config values and encodings are better encapsulated in the `fabric2_fdb`
module.
Since `set_config` does, it made sense to update get_config as well. There, it
turns out it was used only to load configuration setting after a db open, so
the function was renamed to `load_config` and was made private.
|
| |
|
|
|
|
|
|
|
| |
Previously, a stale db handle could be re-used across a few separate
transactions. That would result in the database getting re-opened before every
one of those operations.
To prevent that from happening, check the cache before the transaction starts,
and if there is a newer version of the db handle and use that.
|
| |
|
|
|
|
|
|
|
|
|
| |
Previously we checked security properties in a separate transaction, after
opening the db or fetching it from the cache. To avoid running an extra
transaction move the check inside the main transaction right after the metadata
check runs. That ensure it will be consistent and it won't be accidentally
missed as all operations run the `ensure_current` metadata check.
Also remove the special `get_config/2` function in `fabric2_fdb` for getting
revs limit and security properties and just read them directly from the db map.
|
| |
|
|
| |
Users should be able to replicate their partitioned dbs to the new environment.
|
| |\
| |
| | |
Pass contexts to fabric2_db functions
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Since the db structure returned from fabric2_db:open and fabric2_db:create
includes `user_ctx` there is no need to pass it explicitly
in every `fabric2_db` call. This means we can simplify few things:
- Don't pass user_ctx in `chttpd_db:db_req/2` since we pass db already
- Don't have to use `db_open_options` in `chttpd_changes`
- Don't have to pass `user_ctx` to `fabric2_db:open_doc` and `fabric2_db:update_doc`
|
| | |
| |
| |
| |
| | |
If we don't explicitly bail out of running the job it will loop
indefinitely in the couch_jobs retry logic.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Need to use FDB version 6.1+ and erlfdb version that has this commit:
https://github.com/cloudant-labs/couchdb-erlfdb/commit/7718a3d7e1994e1384c56d39fae5cad3d8c6c4b3
Since fabric2.hrl is a public include now, use that in couch_jobs to avoid
redefining a bunch of things.
|
| | | |
|
| | |
| |
| |
| |
| |
| | |
Since we are dealing with upgrades and both versions start out as binaries,
make sure we add extra belts and suspenders to surface any issues with encoding
errors.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Previously local docs were not chunkified and it was possible for replications
which checkpointed a few dozen times to create local documents above the 100KB
limit.
Documents are chunkiefied according to the same scheme as the regular docs --
rev values are in a main `?DB_LOCAL_DOCS` subspace, and doc body chunks in a
separate `?DB_LOCAL_DOC_BODIES` subspace that looks like:
{?DB_LOCAL_DOC_BODIES, DocId, ChunkId} = BinaryChunk
where `ChunkId` is an incrementing integer and BinaryChunk is a 100KB chunk of
the term_to_binary of the body.
We also go to some lengths to read and silently upgrade docs written with the
old encoding. Upgrades happen on doc writes as a first step, to ensure stats
update logic is not affected.
|
| |\ \
| | |
| | | |
Remove old clause which is no longer used
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The history of `send_error(_Req, {already_sent, Resp, _Error})`
clause is bellow:
- it was added on [2009/04/18](https://svn.apache.org/viewvc/couchdb/trunk/src/couchdb/couch_httpd.erl?r1=762574&r2=765819&diff_format=h)
- we triggered that clause [in couch_httpd:do](https://svn.apache.org/viewvc/couchdb/trunk/src/couchdb/couch_httpd.erl?revision=642432&view=markup#l88)
- at that time we were using inets webserver [see use of `httpd_response/3`](https://svn.apache.org/viewvc/couchdb/trunk/src/couchdb/couch_httpd.erl?revision=642432&view=markup#l170)
- The inets OTP codebase uses `already_sent` messages [here](https://github.com/erlang/otp/blob/50214f02501926fee6ec286efa68a57a47c2e531/lib/inets/src/http_server/httpd_response.erl#L220)
It should be safe to remove this clause because we are not using inets anymore
and search of `already_sent` term across all dependencies doesn't return any results.
|
| |\ \
| |/
|/| |
Remove compiler warning
|
| |/ |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
FDB's metadata version key allows more efficient metadata invalidation
(see https://github.com/apple/foundationdb/pull/1213). To take advantage of
that feature update the caching logic to check the metadata version first, and
if it is current, skip checking the db version altogether.
When db version is bumped we now update the metadata version as well.
There is a bit of a subtlety when the metadata version is stale. In that case
we check the db version, and if that is current, we still don't reopen the
database, instead we continue with the transaction. Then, after the transaction
succeeds, we update the cached metadata version for that db handle. Next client
would get the updated db metadata, it will be current, and they won't need to
check the db version. If the db version is stale as well, then we throw a
`reopen` exception and the handle gets removed from the cache and reopened.
Note: this commit doesn't actually use the new metadata version key, it still
uses the old plain key. That update will be a separate commit where we also start
setting a new API version (610) and will only work on FDB version 6.1.x
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
To trace FDB transactions:
1. Enable tracing in erlfdb application environment:
`network_options = [{trace_enable, ...}]`
OR with an environment variable:
`FDB_NETWORK_OPTION_TRACE_ENABLE = ""`
2. Set `[fabric] fdb_trace=true` configuration value
3. Add the `x-couchdb-fdb-trace:true` header to each request that should be
traced.
The transaction name is set to the nonce value, which is already used by
CouchDB to track API requests.
Only transactions started from the main request process will be traced. So if a
process is spawned without inheriting the `erlfdb_trace` process dict key, that
transaction will not be traced.
|
| | |
|
| |
|
|
|
|
| |
It was suggested in another PR's discussion:
https://github.com/apache/couchdb/pull/2107#pullrequestreview-274431487
|
| |
|
|
| |
set_type_timeout takes seconds as the argument but we gave it milliseconds
|
| |
|
|
|
| |
This check fails if Clouseau isn't present. Though we don't need
Clouseau to perform the check so just avoid it.
|
| | |
|
| |
|
|
|
| |
These are ported over from the existing couch Eunit suite and updated to
be less racey hopefully.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
This commit is mostly a copy paste of the existing modules in the
`couch` application. For now I've left the build of the `couchjs`
executable in `couch/priv` to avoid having to do the work of moving the
build config over. I had contemplated just referencing the modules as
they current exist but decided this would prepare us a bit better for
when we eventually remove the old modules.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
We test that a newly deleted document is replicated to the target and it bumps
the deled doc count but doesn't change doc count.
Another thing to test is that an already deleted document is replicated in
where its revision path was extended on the source then gets replicated to the
target. In that case neither del doc count not doc count are bumped.
|
| |
|
|
|
|
|
|
|
|
|
| |
Do not decrement `doc_count` stat if document was previously missing, or if it
was already deleted.
Deleted documents could be brought in by replication. In that case, if there
were more replicated documents than the current `doc_count`, the `doc_count`
could even underflow the 64 bit unsigned integer range and end up somewhere in
the vicinity of 2^64. The counter, of course, would still be incorrect even if
it didn't underflow, the huge value would just make the issue more visible.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This brings this to parity with master `couch_db:open/2` logic:
https://github.com/apache/couchdb/blob/master/src/couch/src/couch_db.erl#L166
There are two separate cases that have to be handled:
1) Db was already opened and cached. In that case, call `check_is_member(Db)`
which reads the security from the Db to ensure we don't authorize against a
stale security doc. Otherwise, the delay could be as long as the last write
that went through that node. A potential future optimization could be to have
a timestamp and only get the new security context if last refresh hasn't
happened recently.
2) Db was not cached, and was just opened. To avoid running another two read
transactions to get the security doc after the main transaction finished, call
a version of check_is_member which gets the security doc passed in as an
argument.
As a bonus, `check_is_members(Db, SecDoc)` version ends up saving one extra
security read since we don't read twice in is_member and is_admin calls.
`delete/2` was updated to pass ?ADMIN_CTX to `open/2` since it only cares about
getting a `database_does_not_exist` error thrown. There is a check for server
admin at the HTTP API level that would care of authentication / authorization.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
There are two fixes:
1) In `fabric2_fdb:get_config/1`, Db was matched before and after
`ensure_current/1`. Only the db prefix path was used, which doesn't normally
change, but it's worth fixing it anyway.
2) We used a cached version of the security document outside the transaction.
Now we force it go through a transaction to call `fabric2_fdb:get_config/1`
which call `ensure_current/1`. When done, we also update the cached Db handle.
Do the same thing for revs_limit even thought it is not as critical as
security.
|
| | |
|
| |\
| |
| | |
Support `--extra_args` parameter in `dev/run`
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Sometimes there is a need to specify additional arguments for the beam process we start from dev/run.
In particular the feature is handy for:
- changing emulator flags
- simulate OOM via available RAM restrictions
- enable module loading tracing
- configure number of schedulers
- modify applications configuration
- run customization script to add extra development deps (such as automatic code reload)
Historically developers had to edit dev/run to do it.
This PR adds an ability to specify additional arguments via `--extra_args` argument.
In order to run customization script create `customization.erl` which exports `start/0` and run it using:
```
dev/run --extra_args='-run customization'
```
|
