summaryrefslogtreecommitdiff
path: root/doc
Commit message (Collapse)AuthorAgeFilesLines
...
| * sharder: support rows_per_shard in config fileAlistair Coles2021-07-071-7/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Make rows_per_shard an option that can be configured in the [container-sharder] section of a config file. For auto-sharding, this option was previously hard-coded to shard_container_threshold // 2. The swift-manage-shard-ranges command line tool already supported rows_per_shard on the command line and will now also load it from a config file if specified. Any value given on the command line takes precedence over any value found in a config file. Change-Id: I820e133a4e24400ed1e6a87ebf357f7dac463e38
* | reconstructor: make quarantine delay configurableAlistair Coles2021-07-061-0/+13
|/ | | | | | | | | | Previously the reconstructor would quarantine isolated durable fragments that were more than reclaim_age old. This patch adds a quarantine_age option for the reconstructor which defaults to reclaim_age but can be used to configure the age that a fragment must reach before quarantining. Change-Id: I867f3ea0cf60620c576da0c1f2c65cec2cf19aa0
* Switch IRC references from freenode to OFTCTim Burke2021-06-011-1/+1
| | | | | | | | See http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022718.html and http://lists.opendev.org/pipermail/service-discuss/2021-May/000249.html for more information. Change-Id: I6ac37c23153c5c9d42d96e448e0656dbf8257991
* Merge "Add absolute values for shard shrinking config options"Zuul2021-06-011-6/+37
|\
| * Add absolute values for shard shrinking config optionsAlistair Coles2021-05-201-6/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add two new sharder config options for configuring shrinking behaviour: - shrink_threshold: the size below which a shard may shrink - expansion_limit: the maximum size to which an acceptor shard may grow The new options match the 'swift-manage-shard-ranges' command line options and take absolute values. The new options provide alternatives to the current equivalent options 'shard_shrink_point' and 'shard_shrink_merge_point', which are expressed as percentages of 'shard_container_threshold'. 'shard_shrink_point' and 'shard_shrink_merge_point' are deprecated and will be overridden by the new options if the new options are explicitly set in a config file. The default values of the new options are the same as the values that would result from the default 'shard_container_threshold', 'shard_shrink_point' and 'shard_shrink_merge_point' i.e.: - shrink_threshold: 100000 - expansion_limit: 750000 Change-Id: I087eac961c1eab53540fe56be4881e01ded1f60e
* | Merge "s3api: Fix prefix/delimiter/marker quoting"Zuul2021-05-251-0/+1
|\ \
| * | s3api: Fix prefix/delimiter/marker quotingTim Burke2020-05-151-0/+1
| | | | | | | | | | | | | | | | | | | | | And stop sending WSGI strings on py3. Change-Id: I9b769e496aa7c8ed5862c2d7310f643838328084 Closes-Bug: #1853654
* | | Merge "s3api: Fix blank delimiter handling"Zuul2021-05-251-2/+0
|\ \ \ | |/ / | | / | |/ |/|
| * s3api: Fix blank delimiter handlingTim Burke2020-05-151-2/+0
| | | | | | | | | | | | | | | | Real AWS only includes an empty delimiter element when doing a version-aware listing. Change-Id: Id246a157c576eac93375be084ada3740f1e09793 Closes-Bug: #1853663
* | Use ContainerSharderConf class in sharder and manage-shard-rangesAlistair Coles2021-05-201-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Change the swift-manage-shard-ranges default expansion-limit to equal the sharder daemon default merge_size i.e 750000. The previous default of 500000 had erroneously differed from the sharder default value. Introduce a ContainerSharderConf class to encapsulate loading of sharder conf and the definition of defaults. ContainerSharder inherits this and swift-manage-shard-ranges instantiates it. Rename ContainerSharder member vars to match the equivalent vars and cli options in manage_shard_ranges: shrink_size -> shrink_threshold merge_size -> expansion_limit split_size -> rows_per_shard (This direction of renaming is chosen so that the manage_shard_ranges cli options are not changed.) Rename ContainerSharder member vars to match the conf file option name: scanner_batch_size -> shard_scanner_batch_size Remove some ContainerSharder member vars that were not used outside of the __init__ method: shrink_merge_point shard_shrink_point Change-Id: I8a58a82c08ac3abaddb43c11d26fda9fb45fe6c1
* | docs: Get rid of useless pageTim Burke2021-04-262-11/+0
| | | | | | | | | | | | | | I think this was mostly an artifact of the old admin-guide getting pushed in-tree. Change-Id: I61e63bd66ffd8207599b61721e7f0e4dc45d2e01
* | Merge "Create a separate doc for audit watcher"Zuul2021-04-023-20/+19
|\ \
| * | Create a separate doc for audit watcherMatthew Oliver2021-01-063-20/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We have a separate doc page for middlewares that pulls the docstring from each middleware's docstring[0]. This makes it easy to look up the docs in our documentation and easy to find the middleware doc by looking in the code of the middleware itself. This patch does the same with the audit watchers. There is now a page that generates a list of audit watchers, even though currently it's only one, and pulls the docs from their docstrings. Giving us an easy way to maintain each audit watcher doc along with it's code. [0] - https://docs.openstack.org/swift/latest/middleware.html Change-Id: I1456aba0158d29fa0a879dcc2dfb13245c45ad16
* | | Update docs to discourage policy names being numbersAlistair Coles2021-03-261-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are times when it is convenient to specify a policy by name or by index (see Related-Change), but policy names can unfortunately collide with indexes. Using a number as a policy name should at least be discouraged. Change-Id: I0cdd3b86b527d6656b7fb50c699e3c0cc566e732 Related-Change: Icf1517bd930c74e9552b88250a7b4019e0ab413e
* | | Merge "relinker: use abs path index in part power replace"Zuul2021-03-171-2/+2
|\ \ \
| * | | relinker: use abs path index in part power replaceClay Gerrard2021-03-171-2/+2
| | | | | | | | | | | | | | | | Change-Id: I799bdacb7ef5850096c2178bfb12959f4205920d
* | | | Merge "s3api: Allow CORS preflight requests"Zuul2021-03-161-0/+1
|\ \ \ \
| * | | | s3api: Allow CORS preflight requestsTim Burke2021-03-151-0/+1
| |/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Unfortunately, we can't identify the user, so we can't map to an account, so we can't respect whatever CORS metadata might be set on the container. As a result, the allowed origins must be configured cluster-wide. Add a new config option, cors_preflight_allow_origin, for that; default it to blank (ie, deny preflights from all origins, preserving existing behavior), but allow either a comma-separated list of origins or * (to allow all origins). Change-Id: I985143bf03125a05792e79bc5e5f83722d6431b3 Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
* | | | Merge "Replace dequeue_from_legacy_queue in documentation"Zuul2021-03-131-10/+10
|\ \ \ \
| * | | | Replace dequeue_from_legacy_queue in documentationPete Zaitcev2021-03-121-10/+10
| |/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When I tried to follow documentation, the expirer said: object-expirer[391675]: This node is not configured to dequeue tasks from the legacy queue. Our documentation was incorrect, the actual name of the option is "dequeue_from_legacy". Change-Id: I5ca7ac589a405d0b6250922aa9bcaabecb3c4fb0
* | | | Add a config file option to swift-manage-shard-rangesMatthew Oliver2021-03-121-2/+2
|/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | While working on the shrinking recon drops, we want to display numbers that directly relate to how tool should behave. But currently all options of the s-m-s-r tool is driven by cli options. This creates a disconnect, defining what should be used in the sharder and in the tool via options are bound for failure. It would be much better to be able to define the required default options for your environment in one place that both the sharder and tool could use. This patch does some refactoring and adding max_shrinking and max_expanding options to the sharding config. As well as adds a --config option to the tool. The --config option expects a config with at '[container-sharder]' section. It only supports the shard options: - max_shrinking - max_expanding - shard_container_threshold - shard_shrink_point - shard_merge_point The latter 2 are used to generate the s-m-s-r's: - shrink_threshold - expansion_limit - rows_per_shard Use of cli arguments take precedence over that of the config. Change-Id: I4d0147ce284a1a318b3cd88975e060956d186aec
* | | Add sharder section to container config docAlistair Coles2021-02-221-0/+224
| | | | | | | | | | | | Change-Id: I33c0168c1bb89f780be6fc317a4df89322cbc28d
* | | relinker: Rehash as we complete partitionsTim Burke2021-02-181-14/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, we would rely on replication/reconstruction to rehash once the part power increase was complete. This would lead to large I/O spikes if operators didn't proactively tune down replication. Now, do the rehashing in the relinker as it completes work. Operators should already be mindful of the relinker's I/O usage and are likely limiting it through some combination of cgroups, ionice, and/or --files-per-second. Also clean up empty partitions as we clean up. Operators with metrics on cluster-wide primary/handoff partition counts can use them to monitor relinking/cleanup progress: P 3N .----------. a / '. r / '. t 2N / .----------------- i / | t / | i N -----'---------' o n s 0 ---------------------------------- t0 t1 t2 t3 t4 t5 At t0, prior to relinking, there are N := <replica count> * 2 ** <old part power> primary partitions throughout the cluster and a negligible number of handoffs. At t1, relinking begins. In any non-trivial, low-replica-count cluster, the probability of a new-part-power partition being assigned to the same device as its old-part-power equivalent is low, so handoffs grow while primaries remain constant. At t2, relinking is complete. The total number of partitions is now 3N (N primaries + 2N handoffs). At t3, the ring with increased part power is distributed. The notion of what's a handoff and what's a primary inverts. At t4, cleanup begins. The "handoffs" are cleaned up as hard links and now-empty partitions are removed. At t5, cleanup is complete and there are now 2N total partitions. Change-Id: Ib5bf426cf38559091917f2d25f4f60183cd16354
* | | relinker: Allow conf files for configurationTim Burke2021-02-084-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | Swap out the standard logger stuff in place of --logfile. Keep --device as a CLI-only option. Everything else is pretty standard stuff that ought to be in [DEFAULT]. Co-Authored-By: Alistair Coles <alistairncoles@gmail.com> Change-Id: I32f979f068592eaac39dcc6807b3114caeaaa814
* | | Update Memcached Considerations docsClay Gerrard2021-01-281-2/+11
| | | | | | | | | | | | Change-Id: I561ca419104bfb4b64affc477aac53a2fb98137b
* | | Merge "Document memcache max size with shard ranges"Zuul2021-01-271-0/+23
|\ \ \
| * | | Document memcache max size with shard rangesMatthew Oliver2021-01-081-0/+23
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When you start getting more then 3k shards in a root container the cached shard range listing can get bigger then the default max size for memcache (1MB). So add a mention about it in the configuration guide. Fixes bug 1890643 Change-Id: If380410c17ed9ebc014b8198af0ea8d502deacc8
* | | docs: Clarify that encryption should not be in reconciler pipelineTim Burke2021-01-211-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | UpgradeImpact ============= Operators should verify that encryption is not enabled in their reconciler pipelines; having it enabled there may harm data durability. For more information, see https://launchpad.net/bugs/1910804 Change-Id: I1a1d78ed91d940ef0b4eba186dcafd714b4fb808 Closes-Bug: #1910804
* | | Remove vendor names from deployment guideAlistair Coles2021-01-071-33/+33
|/ / | | | | | | | | | | | | | | Also fix link to python paste. Closes-Bug: #1734740 Closes-Bug: #1719887 Change-Id: I27ff441ed746f2919bcf9ca9a77c26371b18540b
* | Let developers/operators add watchers to object auditSamuel Merritt2020-12-262-0/+127
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Swift operators may find it useful to operate on each object in their cluster in some way. This commit provides them a way to hook into the object auditor with a simple, clearly-defined boundary so that they can iterate over their objects without additional disk IO. For example, a cluster operator may want to ensure a semantic consistency with all SLO segments accounted in their manifests, or locate objects that aren't in container listings. Now that Swift has encryption support, this could be used to locate unencrypted objects. The list goes on. This commit makes the auditor locate, via entry points, the watchers named in its config file. A watcher is a class with at least these four methods: __init__(self, conf, logger, **kwargs) start(self, audit_type, **kwargs) see_object(self, object_metadata, data_file_path, **kwargs) end(self, **kwargs) The auditor will call watcher.start(audit_type) at the start of an audit pass, watcher.see_object(...) for each object audited, and watcher.end() at the end of an audit pass. All method arguments are passed as keyword args. This version of the API is implemented on the context of the auditor itself, without spawning any additional processes. If the plugins are not working well -- hang, crash, or leak -- it's easier to debug them when there's no additional complication of processes that run by themselves. In addition, we include a reference implementation of plugin for the watcher API, as a help to plugin writers. Change-Id: I1be1faec53b2cdfaabf927598f1460e23c206b0a
* | Merge "Add a new URL parameter to allow for async cleanup of SLO segments"Zuul2020-11-181-0/+1
|\ \
| * | Add a new URL parameter to allow for async cleanup of SLO segmentsTim Burke2020-11-101-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a new config option to SLO, allow_async_delete, to allow operators to opt-in to this new behavior. If their expirer queues get out of hand, they can always turn it back off. If the option is disabled, handle the delete inline; this matches the behavior of old Swift. Only allow an async delete if all segments are in the same container and none are nested SLOs, that way we only have two auth checks to make. Have s3api try to use this new mode if the data seems to have been uploaded via S3 (since it should be safe to assume that the above criteria are met). Drive-by: Allow the expirer queue and swift-container-deleter to use high-precision timestamps. Change-Id: I0bbe1ccd06776ef3e23438b40d8fb9a7c2de8921
* | | Move config option documentation to separate docsAlistair Coles2020-11-058-1749/+1803
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch moves the tables describing configuration options for each server type from the deployment_guide.rst doc to separate per-server documents. The new per-server documents are grouped under a config directory with a config index doc. The config index doc is listed in the top level index and provides a single starting point to navigate to the individual server docs. Change-Id: I6cedd98586febb5dc949c088ee44e160385ed324
* | | saio: Stop processes more forcefully in resetswiftTim Burke2020-11-041-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We sometimes see probe test failures in the gate where `resetswift` fails like umount: /mnt/sdb1: target is busy. It seems to be because `swift-init all kill` is just a stop with a status check after the fact. Add an orphan sweep afterwards to make sure worker subprocesses got cleaned up, too. Change-Id: I533b436a35e5c3deb7488e9961b55f2274c14751
* | | New proxy logging field for wire statusClay Gerrard2020-10-071-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | Capture the on the wire status code for logging because we change the logged status code sometimes. Closes-Bug: #1896518 Change-Id: I27feabe923a6520e983637a9c68a19ec7174a0df
* | | Clean up apache deployment docsRenich Bon Ćirić2020-10-011-45/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | docs: Removing the use of NameVirtualHost from the apache examples It's not used anymore. It's deprecated in fact: https://httpd.apache.org/docs/2.4/mod/core.html#namevirtualhost Change-Id: I76999cfacc10a244024ee0cca66dda95a0169a67 docs: Added more spacing to the apache2 examples They're easier to read and a bit less bloated. Change-Id: I5e21a66018b7ef309918fbbde93f2494286d291e docs: Switching to /srv/www to be more FHS 3.0 conformat It's more modern and well supported to use /srv/www now in place of /var/www. Change-Id: Icd09ed4d5fb4e2b9b84ddead21313ea1c0a87c91 ref: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s17.html docs: Added user, group and display name in WSGI examples This properly sets the user and group for the wsgi processes in the examples; as well as adding a display name for easier identification. Change-Id: Ie5783081e4054e5b2fbf3a856716101a1aaf61b8 docs: Replace apachectl for systemctl commands It's safe to asume that all modern distros; supported by OpenStack, will have systemd implemented. It's better to favor systemctl in those cases. Change-Id: Ic0d2e47c1ac53502ce638d6fc2424ab9df037262 docs: Emphasis to file paths and command options I've enclosed configuration options or parameters in interpreted text quotes. Also, I've enclosed fiel paths with inline literal quotes. Change-Id: Iec54b7758bce01fc8e8daff48498383cb70c62ce docs: Fixed wording used to indicate the restart of apache Just a little commit to make it clearer of what we're gonna do. Change-Id: Id5ab3e94519bcfe1832b92e456a1d1fa81dd54e3
* | | docs: Switching to /srv/www from /var/www to be more FHS 3.0 conformatRenich Bon Ćirić2020-10-011-52/+48
|/ / | | | | | | | | | | | | | | It's more modern and well supported to use /srv/www now in place of /var/www. Change-Id: Icd09ed4d5fb4e2b9b84ddead21313ea1c0a87c91 ref: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s17.html
* | docs: Clean up some formatting around using servers_per_portTim Burke2020-08-251-42/+42
| | | | | | | | Change-Id: Ic07b0bdd5f1c90e7e25991def58c587680d7bfab
* | Merge "Update SAIO & docker image to use 62xx ports"Zuul2020-08-0118-167/+167
|\ \
| * | Update SAIO & docker image to use 62xx portsTim Burke2020-07-2018-167/+167
| | | | | | | | | | | | | | | | | | | | | Note that existing SAIOs with 60xx ports should still work fine. Change-Id: If5dd79f926fa51a58b3a732b212b484a7e9f00db Related-Change: Ie1c778b159792c8e259e2a54cb86051686ac9d18
* | | docs: Clarify request_time in storage logsTim Burke2020-07-231-1/+2
|/ / | | | | | | | | | | | | It does *not* include transfer time; it's just the time until response is started. Change-Id: I67c5c37c8c669e1e2de2cbfca8bcec02dc98659f
* | Merge "docs: Improve replication-network remakerings"Zuul2020-07-201-11/+30
|\ \
| * | docs: Improve replication-network remakeringsTim Burke2020-07-161-11/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Lower the part-power -- 18 is way higher than is needed for a dev environment. Add commands for reduced-redundancy and erasure-coded storage policies. Related-Change: Ibe46011d8e6a6482d39b3a20ac9c091d9fbc6ef7 Related-Change: I6f11f7a1bdaa6f3defb3baa56a820050e5f727f1 Related-Change: I0403016a4bb7dad9535891632753b0e5e9d402eb Change-Id: I13de27674c81977c2470d43bbb2126ecc4bdd85a
* | | Add a reminder to configure firewall at storage nodesPete Zaitcev2020-07-131-0/+11
|/ / | | | | | | | | | | | | | | | | | | | | | | My customers use the upstream documentation for manual installation of Swift nodes and they come upon the lack of any mention that host firewall ought to accomodate services running on the node. Perhaps we thought it self-evident in Swift. Related RH bz#: 1797814 Change-Id: I337f8d0f1fbeee7ae927a581eecbbbcc6dc69340
* | Merge "Switch to newer openstackdocstheme and reno versions"Zuul2020-06-032-22/+9
|\ \
| * | Switch to newer openstackdocstheme and reno versionsAndreas Jaeger2020-06-032-22/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using these versions will allow especially: * Linking from HTML to PDF document * Allow parallel building of documents * Fix some rendering problems Update Sphinx version as well. Disable openstackdocs_auto_name to use 'project' variable as name. Set openstackdocs_pdf_link to link to PDF file. Note that the link to the published document only works on docs.openstack.org where the PDF file is placed in the top-level html directory. The site-preview places the PDF in a pdf directory. Change pygments_style to 'native' since old theme version always used 'native' and the theme now respects the setting and using 'sphinx' can lead to some strange rendering. Remove docs requirements from lower-constraints, they are not needed during install or test but only for docs building. openstackdocstheme renames some variables, so follow the renames before the next release removes them. A couple of variables are also not needed anymore, remove them. See also http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html Change-Id: I131850d2a5c6164dfd48c9c95885d4754b5236c6
* | | Merge "Remove <py3.5 dependencies from requirements.txt"Zuul2020-06-031-2/+1
|\ \ \ | |/ /
| * | Remove <py3.5 dependencies from requirements.txtAndreas Jaeger2020-06-031-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The requirements repo is support python 3.5 as oldest python version while swift still supports py27. thus, requirements-check will fail on a couple of lines in swift. The check is only run when these files are touched. The py2.7 packagers we know about aren't depending on upstream requirements.txt for correctness and aside from all the production deployments running on py2.7 we only realistically support >=py3.7 There's no good reason for our requirements.txt to be "unspported" by the openstack requirements check job. Since they only support >=py3.5 we can change our requirements.txt inline with that. This should be fine for everything we could hope to get out of both our requirements.txt and the check! Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Change-Id: Ibf8000498528c401707be8b0b91b8355cd993786
* | | docs: s3api doesn't support taggingTim Burke2020-06-021-2/+2
| | | | | | | | | | | | Change-Id: I227e8c2defe96fb5f33054167677ae551830d9e8
* | | Enable s3api and staticweb tests across all func testsThiago da Silva2020-06-011-0/+3
|/ / | | | | | | | | | | | | This patch removed the separate s3api, staticweb functional tests gate jobs and added them across all other functional test jobs. Change-Id: Ie1c606132a054defc2b3cc14a66031090e7b8449
View Lorry Controller Status