summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Wrap lines to 80 chars and remove trailing whitespacemango-docs-formattingJay Doane2020-02-053-90/+270
|
* background indexing for mangoGarren Smith2020-02-0516-56/+877
|
* fix loading doc body in mango_idx:listGarren Smith2020-02-037-15/+25
|
* Refactor mango indexer hookJay Doane2020-02-032-33/+43
|
* able to add/delete/update mango fdb indexesGarren Smith2020-01-307-166/+252
|
* range query fixes from testsGarren Smith2020-01-2912-415/+448
|
* index and _all_docs queries workingGarren Smith2020-01-298-393/+439
|
* initial creation of fdb startkey/endkeyGarren Smith2020-01-279-138/+223
|
* very rough indexing and return docsGarren Smith2020-01-2712-855/+1006
|
* mango crud index definitionsGarren Smith2020-01-275-24/+69
|
* change mango test auth to match elixirGarren Smith2020-01-272-4/+4
|
* add crude mango hook and indexer setupGarren Smith2020-01-275-4/+215
|
* Change map indexes to be stored in one rowGarren Smith2020-01-223-114/+17
| | | | | Changes map indexes to store the original key and value in a single FDB row.
* Merge pull request #2451 from cloudant/tracing-externaliilyak2020-01-212-5/+10
|\ | | | | Add `external` tag to opentrace events
| * Add `external` tag to opentrace eventsILYA Khlopotov2020-01-142-5/+10
|/ | | | | | | | | | | | | | | | | | | | | | | | | This PR adds an ability to selectively enable opentracing for HTTP requests with X-B3-... headers. This is helpful in following cases: - tracing all requests with X-B3-... headers `all = (#{external := E}) when E == true -> true` - tracing all requests to specific database with X-B3-... headers ``` all = (#{external := E, 'db.name' := Db}) when E == true andalso Db == <<"foo">> -> true ``` - tracing requests to specific endpoint with X-B3-... headers ``` db.design.view.read = (#{external := E, 'design.id' := Name}) when E == true andalso Name == <<"bar">> -> true ``` I want to remind that we support following X-B3-... headers: - X-B3-TraceId - X-B3-SpanId - X-B3-ParentSpanId - B3 which is in the following format <TraceId>-<SpanId>-<1 | 0>-<ParentSpanId>
* Improve transaction name setting when tracing FDB transactionsNick Vatamaniuc2019-12-181-1/+4
| | | | | | | | | Previously the per-request nonce value was set as the transaction name and so in the trace logs multiple transactions ended up having the same `TransactionID` which was pretty confusing. To fix the issue, append a transaction ID to the name. The ID is guaranteed to be unique for the life of the VM node.
* Merge pull request #2331 from cloudant/expiring-cacheJay Doane2019-12-109-0/+507
|\ | | | | Expiring cache
| * Expiring cacheexpiring-cacheJay Doane2019-12-109-0/+507
|/ | | | | | | | | | This is a library for creating an FDB backed key value cache, where each entry has a `stale` and `expires` time associated with it. Once the current time exceeds the `expires` time, the entry is automatically removed. The `stale` time can be used to indicate that a refresh is necessary, while still returning a non-expired value. It is potentially useful for implementing e.g. caches to external systems of record, such as OAuth 2.
* Delete attachments when no longer referencedPaul J. Davis2019-12-105-20/+428
| | | | | | | | | | | | | This fixes attachment handling to properly remove attachment data when it is no longer referenced by the document's revision tree. As implemented this accounts for the possibility that multiple revisions may reference a given attachment id. However, due to how the current revision tree removes revisions aggressively it's not currently possible for multiple leaf revisions to share an underlying attachment. This is because when attempting to use a stub attachment when replicating in a conflict we will encounter the `missing_stub` error because the previous shared revision has already been removed.
* Silence test suite warning message about cryptoPaul J. Davis2019-12-101-0/+4
|
* Normalize fabric2 test suitePaul J. Davis2019-12-0517-209/+214
| | | | | This change ensures that all test names are visible and follows a single unified pattern for each test module.
* Retry for failed indexes buildsGarren Smith2019-12-023-6/+103
| | | | | Retry building a failing index for a set number of retries. If it never completes, then return the error to the user.
* Optimize view read latency when the view readyGarren Smith2019-11-261-20/+26
| | | | | If the view is already built, read the view status and view in the same transaction. This will improve view read latency.
* Change end-point /_up to check fdb connectivityEric Avdey2019-11-222-6/+11
|
* Add operation names for all HTTP endpointsPaul J. Davis2019-11-227-27/+540
| | | | | | This adds operation names to all valid HTTP end points. This covers all of `make elixir` except for seven requests that are testing specific error conditions in URL and Methods for various endpoints.
* Implement node typesNick Vatamaniuc2019-11-216-18/+163
| | | | | | The implementation follows the RFC [1] [1]: https://github.com/apache/couchdb-documentation/blob/master/rfcs/013-node-types.md
* Trace fdb transactionsILYA Khlopotov2019-11-203-11/+39
|
* Trace http endpointsILYA Khlopotov2019-11-204-6/+146
|
* Add ctrace applicationILYA Khlopotov2019-11-2016-9/+1757
|
* Implement fabric2_server:fdb_cluster/0ILYA Khlopotov2019-11-201-12/+23
|
* Support regexp based blacklist in configILYA Khlopotov2019-11-201-16/+19
|
* Check membership when calling get_security/1 in fabric2_dbNick Vatamaniuc2019-11-182-73/+64
| | | | | | | | | | | | | | Previously, membership check was disabled when fetching the security doc. That was not correct, as membership should be checked before every db operation, including when fetching the security doc itself. Also, most of the security tests relied on patching the user context in the `Db` handle then calling `check_*` functions. Those functions however call `get_security/1` before doing the actual check, and in some cases, like when checking for admin, the failure was coming from the membership check in `get_security/1` instead. Also some tests were going through the regular request path of opening a new db by name. In order, make the tests more uniform, switch all the tests to apply the tested `UserCtx` in the open call.
* Assert Db handle field existence in `load_config/1` in fabric2_fdbNick Vatamaniuc2019-11-141-3/+5
| | | | | | Forgot to push this in the previous PR so made a new commit. https://github.com/apache/couchdb/pull/2300#discussion_r346592418
* Update fabric2_fdb's set_config to take un-encoding valuesNick Vatamaniuc2019-11-142-34/+26
| | | | | | | | | | | Previously `set_config/3` needed keys and values to be transalted to binaries, now that is done inside the function. It's a bit more consistent as binary config values and encodings are better encapsulated in the `fabric2_fdb` module. Since `set_config` does, it made sense to update get_config as well. There, it turns out it was used only to load configuration setting after a db open, so the function was renamed to `load_config` and was made private.
* Before starting a db transanction, refresh the db handle from the cacheNick Vatamaniuc2019-11-142-21/+50
| | | | | | | | | Previously, a stale db handle could be re-used across a few separate transactions. That would result in the database getting re-opened before every one of those operations. To prevent that from happening, check the cache before the transaction starts, and if there is a newer version of the db handle and use that.
* Check security properties in the main transactionNick Vatamaniuc2019-11-145-56/+71
| | | | | | | | | | | Previously we checked security properties in a separate transaction, after opening the db or fetching it from the cache. To avoid running an extra transaction move the check inside the main transaction right after the metadata check runs. That ensure it will be consistent and it won't be accidentally missed as all operations run the `ensure_current` metadata check. Also remove the special `get_config/2` function in `fabric2_fdb` for getting revs limit and security properties and just read them directly from the db map.
* Ensure we can create partitioned design docs with FDBNick Vatamaniuc2019-11-042-12/+20
| | | | Users should be able to replicate their partitioned dbs to the new environment.
* Merge pull request #2279 from cloudant/refactor-user-ctx-handlingiilyak2019-10-311-61/+42
|\ | | | | Pass contexts to fabric2_db functions
| * Pass contexts to fabric2_db functionsILYA Khlopotov2019-10-241-61/+42
| | | | | | | | | | | | | | | | | | | | Since the db structure returned from fabric2_db:open and fabric2_db:create includes `user_ctx` there is no need to pass it explicitly in every `fabric2_db` call. This means we can simplify few things: - Don't pass user_ctx in `chttpd_db:db_req/2` since we pass db already - Don't have to use `db_open_options` in `chttpd_changes` - Don't have to pass `user_ctx` to `fabric2_db:open_doc` and `fabric2_db:update_doc`
* | Abandon a view job if the db or ddoc is deletedPaul J. Davis2019-10-301-2/+21
| | | | | | | | | | If we don't explicitly bail out of running the job it will loop indefinitely in the couch_jobs retry logic.
* | Use "\xFF/metadataVersion" key for checking metadataNick Vatamaniuc2019-10-302-15/+3
| | | | | | | | | | | | | | | | Need to use FDB version 6.1+ and erlfdb version that has this commit: https://github.com/cloudant-labs/couchdb-erlfdb/commit/7718a3d7e1994e1384c56d39fae5cad3d8c6c4b3 Since fabric2.hrl is a public include now, use that in couch_jobs to avoid redefining a bunch of things.
* | add test to prove we can view swapGarren Smith2019-10-281-0/+67
| |
* | Add a special error for an invalid legacy local doc revsionNick Vatamaniuc2019-10-241-2/+7
| | | | | | | | | | | | Since we are dealing with upgrades and both versions start out as binaries, make sure we add extra belts and suspenders to surface any issues with encoding errors.
* | Chunkify local docsNick Vatamaniuc2019-10-246-21/+491
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously local docs were not chunkified and it was possible for replications which checkpointed a few dozen times to create local documents above the 100KB limit. Documents are chunkiefied according to the same scheme as the regular docs -- rev values are in a main `?DB_LOCAL_DOCS` subspace, and doc body chunks in a separate `?DB_LOCAL_DOC_BODIES` subspace that looks like: {?DB_LOCAL_DOC_BODIES, DocId, ChunkId} = BinaryChunk where `ChunkId` is an incrementing integer and BinaryChunk is a 100KB chunk of the term_to_binary of the body. We also go to some lengths to read and silently upgrade docs written with the old encoding. Upgrades happen on doc writes as a first step, to ensure stats update logic is not affected.
* | Merge pull request #2275 from cloudant/remove-ints-client-remainsiilyak2019-10-242-6/+0
|\ \ | | | | | | Remove old clause which is no longer used
| * | Remove old clause which is no longer usedILYA Khlopotov2019-10-242-6/+0
| |/ | | | | | | | | | | | | | | | | | | | | | | | | The history of `send_error(_Req, {already_sent, Resp, _Error})` clause is bellow: - it was added on [2009/04/18](https://svn.apache.org/viewvc/couchdb/trunk/src/couchdb/couch_httpd.erl?r1=762574&r2=765819&diff_format=h) - we triggered that clause [in couch_httpd:do](https://svn.apache.org/viewvc/couchdb/trunk/src/couchdb/couch_httpd.erl?revision=642432&view=markup#l88) - at that time we were using inets webserver [see use of `httpd_response/3`](https://svn.apache.org/viewvc/couchdb/trunk/src/couchdb/couch_httpd.erl?revision=642432&view=markup#l170) - The inets OTP codebase uses `already_sent` messages [here](https://github.com/erlang/otp/blob/50214f02501926fee6ec286efa68a57a47c2e531/lib/inets/src/http_server/httpd_response.erl#L220) It should be safe to remove this clause because we are not using inets anymore and search of `already_sent` term across all dependencies doesn't return any results.
* | Merge pull request #2274 from cloudant/fix-warningiilyak2019-10-241-2/+0
|\ \ | |/ |/| Remove compiler warning
| * Remove compiler warningILYA Khlopotov2019-10-241-2/+0
|/
* Take better advantage of metadata version key featureNick Vatamaniuc2019-10-183-31/+137
| | | | | | | | | | | | | | | | | | | | | FDB's metadata version key allows more efficient metadata invalidation (see https://github.com/apple/foundationdb/pull/1213). To take advantage of that feature update the caching logic to check the metadata version first, and if it is current, skip checking the db version altogether. When db version is bumped we now update the metadata version as well. There is a bit of a subtlety when the metadata version is stale. In that case we check the db version, and if that is current, we still don't reopen the database, instead we continue with the transaction. Then, after the transaction succeeds, we update the cached metadata version for that db handle. Next client would get the updated db metadata, it will be current, and they won't need to check the db version. If the db version is stale as well, then we throw a `reopen` exception and the handle gets removed from the cache and reopened. Note: this commit doesn't actually use the new metadata version key, it still uses the old plain key. That update will be a separate commit where we also start setting a new API version (610) and will only work on FDB version 6.1.x
* Enable FDB transaction tracingNick Vatamaniuc2019-10-171-0/+18
| | | | | | | | | | | | | | | | | | | | | To trace FDB transactions: 1. Enable tracing in erlfdb application environment: `network_options = [{trace_enable, ...}]` OR with an environment variable: `FDB_NETWORK_OPTION_TRACE_ENABLE = ""` 2. Set `[fabric] fdb_trace=true` configuration value 3. Add the `x-couchdb-fdb-trace:true` header to each request that should be traced. The transaction name is set to the nonce value, which is already used by CouchDB to track API requests. Only transactions started from the main request process will be traced. So if a process is spawned without inheriting the `erlfdb_trace` process dict key, that transaction will not be traced.
View Lorry Controller Status