summaryrefslogtreecommitdiff
path: root/bin
Commit message (Collapse)AuthorAgeFilesLines
* swift-drive-audit: reload systemd after editing fstabMatthew Vernon2022-09-301-0/+4
| | | | | | | | | | | | | | | | Systemd does not monitor /etc/fstab for changes; so a filesystem unmounted and commented-out in fstab will be re-mounted by systemd after some time. This change means that swift-drive-audit will call systemcl daemon-reload (which causes systemd to reload its configuration including /etc/fstab) after editing /etc/fstab on systems where systemd is the running init. Check for that case by looking for the existence of the directory /run/systemd/system, as documented in sd_booted(3). Signed-off-by: Matthew Vernon <mvernon@wikimedia.org> Change-Id: I8830e3da9b6b085224511ac351f2d2860119c432
* swift-ring-builder: exit ERROR (2) on uncaught exceptionsMatthew Vernon2022-02-151-1/+14
| | | | | | | | | | | | | | swift-ring-builder has three exit statuses: 0 (OK), 1 (WARNING), 2 (ERROR). Uncaught exceptions in python result in an exit code of 1, so for example problems writing a builder file to disk will result in an exit of 1 (warning) rather than 2 (error). This addresses that by overriding sys.excepthook to produce the usual backtrace and then exit 2 (error); excepthook is called when an exception is unhandled, unless that is SystemExit. Closes-Bug: 1960657 Change-Id: I7cfeff4f436ade319cf21d0d29853931aef6d20f
* recon: refactor common recon names into a common locationMatthew Oliver2021-06-291-2/+3
| | | | Change-Id: I0a0766cfb6672377de0f152ce179c874c327ec54
* Add async_pending_last time to object.reconTim Burke2021-06-151-1/+5
| | | | | | | The async_pending count isn't near as useful when we don't know how out of date it is. Change-Id: I3e5e904ffc0eba7a7e141e1c2d9f9840e4952041
* swift-account-audit: Log the bad statusTim Burke2021-04-081-4/+6
| | | | Change-Id: Ib28d1948a571acf31926df82dd8c24910c227053
* relinker: Pull arg parsing into moduleTim Burke2021-02-081-21/+1
| | | | | | This allows us to do testing that's more end-to-end. Change-Id: Ifc47b00c597217efb4d705bd84dc8f7df117ae9d
* Run flake8 on bin/ filesTim Burke2021-02-015-1/+6
| | | | Change-Id: I58d4b5a00e97785584c6d3bd8b06243f481c1934
* Adjust initial month value from int to strits-not-a-bug-its-a-feature2021-01-231-1/+1
| | | | | | | | | | swift-drive-audit checks to see if a new year has recently ticked over by checking to see if the current month is January and the logs we are checking are in December. The logs use abbreviated month names, so we need to extract that from "now" to make valid comparisons. Closes-Bug: 1912508 Change-Id: Iabb53f5e4081d580d016bbf75d86e1d75e1f20bb
* Merge "Add option to swift-oldies to only print pids"Zuul2021-01-091-11/+18
|\
| * Add option to swift-oldies to only print pidsTim Burke2020-12-141-11/+18
| | | | | | | | Change-Id: I8877cf482616404eb7023b2975a24ad827efe2b6
* | Populate shrinking shards with shard ranges learnt from rootClay Gerrard2020-12-181-0/+4
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Shard shrinking can be instigated by a third party modifying shard ranges, moving one shard to shrinking state and expanding the namespace of one or more other shard(s) to act as acceptors. These state and namespace changes must propagate to the shrinking and acceptor shards. The shrinking shard must also discover the acceptor shard(s) into which it will shard itself. The sharder audit function already updates shards with their own state and namespace changes from the root. However, there is currently no mechanism for the shrinking shard to learn about the acceptor(s) other than by a PUT request being made to the shrinking shard container. This patch modifies the shard container audit function so that other overlapping shards discovered from the root are merged into the audited shard's db. In this way, the audited shard will have acceptor shards to cleave to if shrinking. This new behavior is restricted to when the shard is shrinking. In general, a shard is responsible for processing its own sub-shard ranges (if any) and reporting them to root. Replicas of a shard container synchronise their sub-shard ranges via replication, and do not rely on the root to propagate sub-shard ranges between shard replicas. The exception to this is when a third party (or auto-sharding) wishes to instigate shrinking by modifying the shard and other acceptor shards in the root container. In other circumstances, merging overlapping shard ranges discovered from the root is undesirable because it risks shards inheriting other unrelated shard ranges. For example, if the root has become polluted by split-brain shard range management, a sharding shard may have its sub-shards polluted by an undesired shard from the root. During the shrinking process a shard range's own shard range state may be either shrinking or, prior to this patch, sharded. The sharded state could occur when one replica of a shrinking shard completed shrinking and moved the own shard range state to sharded before other replica(s) had completed shrinking. This makes it impossible to distinguish a shrinking shard (with sharded state), which we do want to inherit shard ranges, from a sharding shard (with sharded state), which we do not want to inherit shard ranges. This patch therefore introduces a new shard range state, 'SHRUNK', and applies this state to shard ranges that have completed shrinking. Shards are now restricted to inherit shard ranges from the root only when their own shard range state is either SHRINKING or SHRUNK. This patch also: - Stops overlapping shrinking shards from generating audit warnings: overlaps are cured by shrinking and we therefore expect shrinking shards to sometimes overlap. - Extends an existing probe test to verify that overlapping shard ranges may be resolved by shrinking a subset of the shard ranges. - Adds a --no-auto-shard option to swift-container-sharder to enable the probe tests to disable auto-sharding. - Improves sharder logging, in particular by decrementing ranges_todo when a shrinking shard is skipped during cleaving. - Adds a ShardRange.sort_key class method to provide a single definition of ShardRange sort ordering. - Improves unit test coverage for sharder shard auditing. Co-Authored-By: Tim Burke <tim.burke@gmail.com> Co-Authored-By: Alistair Coles <alistairncoles@gmail.com> Change-Id: I9034a5715406b310c7282f1bec9625fe7acd57b6
* py3: Fix swift-dispersion-populateTim Burke2020-09-111-4/+5
| | | | | | | | | | We fixed swift-dispersion-report already; -populate needed the same fix or else it'd hit a "maximum recursion depth exceeded" error. Change-Id: I2d22e087a88c9e8003621feb26460ab6e5ce2a57 Related-Change: I24f4bcc3d62dc37fd9559032bfd25f5b15f98745 Closes-Bug: #1895346 Related-Bug: #1863680
* swift-container-info: Show shard ranges summaryTim Burke2020-07-221-0/+4
| | | | | | | | The current behavior is really painful when you've got hundreds of shard ranges in a DB. The new summary with the states is default. Users can add a -v/--verbose flag to see the old full detail view. Change-Id: I0a7d65f64540f99514c52a70f9157ef060a8a892
* Merge "swift-get-nodes: Allow users to specify either quoted or unquoted paths"Zuul2020-06-091-0/+2
|\
| * swift-get-nodes: Allow users to specify either quoted or unquoted pathsTim Burke2020-06-081-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Now that we can have null bytes in Swift paths, we need a way for operators to be able to locate such containers and objects. Our usual trick of making sure the name is properly quoted for the shell won't suffice; running something like swift-get-nodes /etc/swift/container.ring.gz $'AUTH_test/\0versions\0container' has the path get cut off after "AUTH_test/" because of how argv works. So, add a new option, --quoted, to let operators indicate that they already quoted the path. Drive-bys: * If account, container, or object are explicitly blank, treat them as though they were not provided. This provides better errors when account is explicitly blank, for example. * If account, container, or object are not provided or explicitly blank, skip printing them. This resolves abiguities about things like objects whose name is actually "None". * When displaying account, container, and object, quote them (since they may contain newlines or other control characters). Change-Id: I3d10e121b403de7533cc3671604bcbdecb02c795 Related-Change: If912f71d8b0d03369680374e8233da85d8d38f85 Closes-Bug: #1875734 Closes-Bug: #1875735 Closes-Bug: #1875736 Related-Bug: #1791302
* | relinker: Improve performance by limiting I/ORomain LE DISEZ2020-03-311-0/+2
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit reduce the number of I/O done by the swift-object-relinker. First, it saves a progress state of relinking and cleanup in case the process is interrupted during the operation. This allow to resume operation without rescanning all partitions. Secondly, it prevents from being scanned by relink and cleanup all partitions that are bigger than 2^part_power (or (2^next_part_power)/2). These partitions were not existing before the beginning of the part_power increase, so there is nothing to relink or cleanup. Thirdly, it reverse-orders the partitions to scan so that some useless work is avoided. If a device contains partitions 1 and 3, relinking partition 1 will create "new" objects in partition 3, that will need to be scanned when the relinker will work on partition 3. It is useless. If partition 3 is done first, it will only contain the objects that need to be relinked. Fourthly, it allows to specify a unique device to work on. To do that, some hooks were added in audit_location_generator to allow to execute some custom code before/after iterating a device/partition/suffix/hash. Change-Id: If1bf8ed9036fb0ec619b0d4f16061a81a1af2082
* Merge "swift-account-audit: work with encryption"Zuul2020-01-221-1/+6
|\
| * swift-account-audit: work with encryptionTim Burke2019-10-141-1/+6
| | | | | | | | Change-Id: I26c5fe9d45a9765da0d30138ea2df16fd4e73d57
* | Merge "swift-account-audit: clean up some error formatting"Zuul2019-10-161-2/+2
|\ \
| * | swift-account-audit: clean up some error formattingTim Burke2019-10-141-2/+2
| |/ | | | | | | | | | | "127.0.0.1s:sdas" is confusing at best. Change-Id: I37f78d5993082ac29b001e9563aa4b24fd009a27
* | Merge "py3: fix swift-account-audit"Zuul2019-10-151-2/+4
|\ \
| * | py3: fix swift-account-auditTim Burke2019-10-141-2/+4
| |/ | | | | | | | | | | Previously, we'd get a KeyError trying to read headers. Change-Id: I5d9f86784a3e39577ab010d29d8d03b26ffda357
* | Merge "py3: fix swift-dispersion-populate"Zuul2019-10-151-2/+4
|\ \
| * | py3: fix swift-dispersion-populateTim Burke2019-10-141-2/+4
| |/ | | | | | | Change-Id: I1f140ae00cbd25b23c9a40ee91dccee8c7c15d81
* | py3: Fix swift-drive-auditTim Burke2019-10-131-4/+13
|/ | | | | | | | | | Walking through the kernel logs backwards requires that we open them in binary mode. Add a new option to allow users to specify which encoding should be used to interpret those logs; default to the same encoding that open() uses for its default. Change-Id: Iae332bb58388b5521445e75beba6ee2e9f06bfa6 Closes-Bug: #1847955
* Find .d pid files with swift-orphansTim Burke2019-08-301-1/+1
| | | | Change-Id: I7a2f19862817abf15e51463bd124293730451602
* py3: fix up swift-orphansTim Burke2019-08-151-12/+16
| | | | Change-Id: Id1280abd92e8bb02fcaa4701a0e9d211d9d6e33e
* Fix locking in swift-recon-cronThiago da Silva2018-08-151-16/+10
| | | | | | | | The previous locking method would leave the lock dir lying around if the process died unexpectedly, preventing others swift-recon-cron process from running sucessfuly and requiring a manual clean. Change-Id: Icb328b2766057a2a4d126f63e2d6dfa5163dd223
* Merge "Experimental swift-ring-composer CLI to build composite rings"Zuul2018-06-151-0/+22
|\
| * Experimental swift-ring-composer CLI to build composite ringsAlistair Coles2018-06-141-0/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Provides a simple, experimental, CLI tool to generate a composite ring from a list of component builder files. For example: swift-ring-composer <composite-file> compose \ <builder-file> <builder-file> --output <ring-file> Commands available: - compose: compose a list of builder file to a composite ring - show: show the metadata for a composite ring Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp> Co-Authored-By: Matthew Oliver <matt@oliver.net.au> Change-Id: I25a79e71c13af352e19e4358f60545265b51584f
* | Add sharder daemon, manage_shard_ranges tool and probe testsMatthew Oliver2018-05-181-0/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The sharder daemon visits container dbs and when necessary executes the sharding workflow on the db. The workflow is, in overview: - perform an audit of the container for sharding purposes. - move any misplaced objects that do not belong in the container to their correct shard. - move shard ranges from FOUND state to CREATED state by creating shard containers. - move shard ranges from CREATED to CLEAVED state by cleaving objects to shard dbs and replicating those dbs. By default this is done in batches of 2 shard ranges per visit. Additionally, when the auto_shard option is True (NOT yet recommeneded in production), the sharder will identify shard ranges for containers that have exceeded the threshold for sharding, and will also manage the sharding and shrinking of shard containers. The manage_shard_ranges tool provides a means to manually identify shard ranges and merge them to a container in order to trigger sharding. This is currently the recommended way to shard a container. Co-Authored-By: Alistair Coles <alistairncoles@gmail.com> Co-Authored-By: Tim Burke <tim.burke@gmail.com> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Change-Id: I7f192209d4d5580f5a0aa6838f9f04e436cf6b1f
* | Merge "swift-recon-cron: do not get confused by files in /srv/node"Zuul2018-04-131-0/+2
|\ \
| * | swift-recon-cron: do not get confused by files in /srv/nodeStefan Majewsky2016-10-191-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | swift-recon-cron looks at the drives mounted in directories below /srv/node, but before this commit, it tried to call listdir() on everything in this directory, even if it is not a directory. Change-Id: Id281352f7ab6ecb520eb00f3649873d8c8678608 Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
* | | Merge "swift-(account|container)-info: tolerate LockTimeouts"Zuul2018-04-112-2/+4
|\ \ \
| * | | swift-(account|container)-info: tolerate LockTimeoutsTim Burke2018-04-102-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I'm not really clear on why a sqlite3.OperationalError should cause us to retry with stale_reads_ok=True, but swift.common.exceptions.LockTimeout *definitely* should. Change-Id: I707dec1d11b8db80bc8fbee30662b319bf10d6a5
* | | | Merge "Add manpage for swift-object-relinker"Zuul2018-04-111-2/+4
|\ \ \ \ | |/ / / |/| | |
| * | | Add manpage for swift-object-relinkerOndřej Nový2017-12-091-2/+4
| | | | | | | | | | | | | | | | Change-Id: I56dd9c646faba91e9f124f343ea0e08f8c3c4249
* | | | Merge "Support -d <devs> and -p <partitions> in DB replicators."Zuul2018-03-202-2/+24
|\ \ \ \
| * | | | Support -d <devs> and -p <partitions> in DB replicators.Samuel Merritt2018-03-052-2/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Similar to the object replicator and reconstructor, these arguments are comma-separated lists of device names and partitions, respectively, on which the account or container replicator will operate. Other devices and partitions are ignored. Change-Id: Ic108f5c38f700ac4c7bcf8315bf4c55306951361
* | | | | Merge "Optionally drop common prefixes in swift-*-info output"Zuul2018-03-133-0/+9
|\ \ \ \ \ | |_|_|_|/ |/| | | |
| * | | | Optionally drop common prefixes in swift-*-info outputTim Burke2017-09-133-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a --drop-prefixes flag to swift-account-info, swift-container-info, and swift-object-info. This makes the output between the three more consistent. Change-Id: I98252ff74c4983eaad0a93d9a9fc527c74ffce68
* | | | | py3: Port more CLI toolsTim Burke2018-02-281-0/+8
| |/ / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Bring under test - test/unit/cli/test_dispersion_report.py - test/unit/cli/test_info.py and - test/unit/cli/test_relinker.py I've verified that swift-*-info (at least) behave reasonably under py3, even swift-object-info when there's non-utf8 metadata on the data/meta file. Change-Id: Ifed4b8059337c395e56f5e9f8d939c34fe4ff8dd
* | | | Let recon-cron work with conf.dClay Gerrard2018-01-111-7/+2
| |/ / |/| | | | | | | | Change-Id: I862b74e0d9b20ba149581c1add6473dc1e5b2859
* | | Remove swift-temp-url scriptTim Burke2017-10-131-31/+0
| | | | | | | | | | | | | | | | | | | | | | | | This has been deprecated since Swift 2.10.0 (Newton) including a message that it would go away. Let's actually remove it. Change-Id: I7d3659761c71119363ff2c0c750e37b4c6374a39 Related-Change: Ifa8bf636f20f82db4845b02d1b58699edaa39356
* | | Replace SOSO auth prefix in examples with more-standard AUTHTim Burke2017-09-271-3/+3
| | | | | | | | | | | | Change-Id: I98643d6acf248840a8360f31e446bc8ecb834898
* | | swift-account-audit: compare each etag to the hash from containerTim Burke2017-09-271-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ...rather than only comparing the ETag from the last response over and over again. NB: This tool *does not* like EC data :-( Change-Id: Idd37f94b07f607ab8a404dd986760361c39af029 Closes-Bug: 1266636
* | | Add bin/swift-dispersion-reportChristian Schwede2017-09-131-0/+24
| | | | | | | | | | | | Change-Id: I81736080fc478c2b69d5b71edd0cada39aad9400
* | | Make swift-dispersion-report importableChristian Schwede2017-09-131-411/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch allows to import the dispersion report tool, and thus making it more easily usable within other Python tools. This can be also used in a follow up patch to add some tests for the report tool. It also fixes a bug when using the "--dump-json" option - until now it returned the policy name and made the JSON invalid. Change-Id: Ie0d52a1a54fc152bb72cbb3f84dcc36a8dad972a
* | | Clarify usage of replicator and reconstructor override optionsAlistair Coles2017-07-262-5/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Clarify in usage statement and man pages that CLI override options for swift-object-reconstructor and swift-object-replicator only have effect when --once is used. Also add a link to object reconstructor source code docs to the doc index page for consistency with the other object services. Change-Id: If348b340d59a672d3a19d4df231ebdb74f4aed51
* | | Fix swiftdir option and usage of storage policy aliasesChristian Schwede2017-07-122-0/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If swift-recon/swift-get-nodes/swift-object-info is used with the swiftdir option they will read rings from the given directory; however they are still using /etc/swift/swift.conf to find the policies on the current node. This makes it impossible to maintain a local swift.conf copy (if you don't have write access to /etc/swift) or check multiple clusters from the same node. Until now swift-recon was also not usable with storage policy aliases, this patch fixes this as well. Closes-Bug: 1577582 Closes-Bug: 1604707 Closes-Bug: 1617951 Co-Authored-By: Alistair Coles <alistairncoles@gmail.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Change-Id: I13188d42ec19e32e4420739eacd1e5b454af2ae3
View Lorry Controller Status