summaryrefslogtreecommitdiff
path: root/swift/common/middleware/staticweb.py
Commit message (Collapse)AuthorAgeFilesLines
* Merge "staticweb: Allow empty listings at the root of a container"Zuul2022-05-271-1/+1
|\
| * staticweb: Allow empty listings at the root of a containerTim Burke2021-05-111-1/+1
| | | | | | | | | | | | | | This is useful for testing that staticweb metadata was properly set before putting data into the container. Change-Id: I433bad1cb3f52ad60a81dcbf3209681f7068ed55
* | Move *_swift_info functions into a new registry moduleMatthew Oliver2022-02-031-1/+2
|/ | | | | | | | | | | | | | | | The *_swift_info functions use in module global dicts to provide a registry mechanism for registering and getting swift info. This is an abnormal pattern and doesn't quite fit into utils. Further we looking at following this pattern for sensitive info to trim in the future. So this patch does some house cleaning and moves this registry to a new module swift.common.registry. And updates all the references to it. For backwards compat we still import the *_swift_info methods into utils for any 3rd party tools or middleware. Change-Id: I71fd7f50d1aafc001d6905438f42de4e58af8421
* py3: Fix non-ascii chars in staticweb listingsTim Burke2020-09-061-6/+6
| | | | | Change-Id: Ifcf810f009a8036f250a09eec0d9a65a77342129 Closes-Bug: #1894357
* py38: cgi lost some namesTim Burke2020-01-291-9/+9
| | | | | | Drive-by: always use quote=True; it basically never hurts. Change-Id: Id91fdd19f226e9ec0d9c702d40d041c385c52b88
* Make staticweb return URL-encoded Location headersTim Burke2019-04-121-1/+2
| | | | | | | Also, enhance tests to verify proper encoding in generated responses. Change-Id: Ib696b1c3a34722f3a6de010973b90ef8b1917012 Partial-Bug: 1774238
* Merge "py3: port staticweb middleware"Zuul2019-04-111-12/+24
|\
| * py3: port staticweb middlewareMatthew Oliver2019-04-101-12/+24
| | | | | | | | | | | | This patch is porting the staticweb middleware to py3. Change-Id: I5d9a13baecedd13d2b7a8ae3dd639eaff0894441
* | Remove our urlparse wrapperTim Burke2019-04-101-1/+3
|/ | | | | | | | | | | It has not been necessary since we dropped support for Python 2.6. See https://github.com/python/cpython/commit/8c6d9d7 and https://bugs.python.org/issue2987. Be sure to keep a `urlparse` name in utils, though; swauth (at least) still expects there to be a swift.common.utils.urlparse. Change-Id: If2502868f251b8a83aa929ee22b10046e708d111
* Move listing formatting out to proxy middlewareTim Burke2017-09-151-3/+3
| | | | | | | | | | | | | | | | Make some json -> (text, xml) stuff in a common module, reference that in account/container servers so we don't break existing clients (including out-of-date proxies), but have the proxy controllers always force a json listing. This simplifies operations on listings (such as the ones already happening in decrypter, or the ones planned for symlink and sharding) by only needing to consider a single response type. There is a downside of larger backend requests for text/plain listings, but it seems like a net win? Change-Id: Id3ce37aa0402e2d8dd5784ce329d7cb4fbaf700d
* Fix some reST field lists in docstringsXieYingYun2017-03-291-1/+1
| | | | | | | | | | Probably the most common format for documenting arguments is reST field lists [1]. This change updates some docstrings to comply with the field lists syntax. [1] http://sphinx-doc.org/domains.html#info-field-lists Change-Id: I87e77a9bbd5bcb834b35460ce0adff5bc59d9168
* Add a configurable URL base to staticwebPete Zaitcev2016-10-031-10/+27
| | | | | | | | | | | | | | This came to light because someone ran Tempest against a standard installation of RDO, which helpfuly terminates SSL for Swift in a pre-configured load-balancer. In such a case, staticweb has no way to know what scheme to use and guesses wrong, causing Tempest to fail. Related upstream bug: https://bugs.launchpad.net/mos/+bug/1537071 Change-Id: Ie15cf2aff4f7e6bcf68b67ae733c77bb9353587a Closes-Bug: 1572011
* Stop staticweb revealing container existence to unauth'd requestsAlistair Coles2016-02-251-3/+15
| | | | | | | | | | | | | When a container has `X-Container-Meta-Web-Listings: false` then staticweb will return a 404 in response to a GET or HEAD on the container, regardless of whether the request is auth'd. That provides a way to probe for container existence. It should return a 401 if the request is not auth'd. This patch adds a call to swift.authorize before returning the 404. Closes-Bug: 1506116 Change-Id: I382323b49dc8f6d67bf4494db7084a860a10db59
* Add option for label to static web listingsRichard Hawkins2016-02-041-7/+18
| | | | | | | | | | | | By default listings are rendered with a label of: "Listing of /v1/account/container/path" This change adds a new custom metadata key/value pair "X-Container-Meta-Web-Listings-Label: <label>" that when set, will cause the following: "Listing of <label>/path" to be rendered instead. Change-Id: I47c5d21834f47e566b380920196926882ff83abf
* Replace string slicing with proper string methodsFerenc Horváth2015-11-271-4/+4
| | | | | | | | | | Updated string prefix and suffix checker slicing to startswith() and endswith() methods. Using startswith() and endswith() improves readability, error-proneness and enhances maintainability. Change-Id: I1d5fbf116a61763346c6f92fd8023dbbe9bb37cf
* Do not use pre-authenticated requests in staticwebChristian Schwede2015-10-131-5/+12
| | | | | | | | | | | | | | | | | | | | | | staticweb middleware uses make_pre_authed_env, this makes it possible to anonymously list containers without any read acl set if the metadata "web-listings: true" is set on a container. Using make_env enforces correct read_acl validation; however it is now also required to add ".rlistings" to the read acl. Also, if the staticweb middleware is put in the proxy pipeline before an authentication middleware, it broke authenticated GET and HEAD requests. This has some side effects in clients, because a html response is sent that might be parsed wrongly by the client. In case of python-swiftclient this was shown as an empty container without any ACL or web-listings:true meta set. This might lead to information leaks, because a user trusts the output from python-swiftclient and assumes an empty, private container even if the container contains public readable data. staticweb now checks if "swift.authorize" is included in the environ and skips itself if not. Closes-Bug: 1489749 Change-Id: Icf159d7e567ac5481e710c5910db686bdcba6336 Depends-On: Ie24bb995023c377e49796910ad80a256b00daa03
* Remove simplejson from staticwebSamuel Merritt2015-05-291-8/+10
| | | | | | | | | | | | | | | | | | | | | | | | | Since we're dropping Python 2.6 support, we can rely on stdlib's json and get rid of our dependency on simplejson. This lets us get rid of some redundant Unicode encoding. Before, we would take the container-listing response off the wire, JSON-deserialize it (str -> unicode), then pass each of several fields from each entry to get_valid_utf8_str(), which would encode it, (unicode -> str), decode it (str -> unicode), and then encode it again (unicode -> str) for good measure. The net effect was that each object's name would, in the proxy server, go str -> unicode -> str -> unicode -> str. By replacing simplejson with stdlib json, we get a guarantee that each container-listing entry's name, hash, content_type, and last_modified are unicodes, so we can stop worrying about them being valid UTF-8 or not. This takes an encode and decode out of the path, so we just have str -> unicode -> str. While it'd be ideal to avoid this, the first transform (str -> unicode) happens when we decode the container-listing response body (json.loads()), so there's no way out. Change-Id: I00aedf952d691a809c23025b89131ea0f02b6431
* Improve StaticWeb 404 on web-listings/indexDonagh McCabe2014-02-181-1/+25
| | | | | | | | | | | | | | | | | | | | A common scenario is for users to make a container public via an ACL. They can then use a browser to display the objects in the container. Next, they enable StaticWeb with something such as "X-Container-Meta-Web-Index:index.html" -- but then get confused because they get 404 Not Found (when index.html does not exist). For someone who understands what they are doing, this makes sense. However, we've had several customer escalations because of this. Usually, they are just playing with our GUI and have no intention of using StaticWeb for real -- the 404 looks like something is broken; not the correct response. The solution is to provide a better error message. This message is only shown when StaticWeb is refusing to give a listing -- other 404 situations are not affected. Also, a custom 404 error page is not affected. Change-Id: I3ba8c48e0ce148d8e91a1e0dc16a0d37a692a24e
* Add documentation for GateKeeperClay Gerrard2014-01-231-1/+1
| | | | | | | | | | | | | The purpose of GateKeeper mostly relates to the development of new swift code, so I threw together a guide for development_middleware that covers some basics with a eye towards metadata handling in-particular. I also fixed up some missing autodoc's, split out middleware autodoc and added some ref's here and about so I could link to them from the development_middleware guide. DocImpact Change-Id: I20dd942ea8df9e33c3e794cb49669ffa1332c63e
* Added discoverable capabilities.Richard (Rick) Hawkins2013-11-221-1/+2
| | | | | | | | | | | | | | Swift can now optionally be configured to allow requests to '/info', providing information about the swift cluster. Additionally a HMAC signed requests to '/info?swiftinfo_sig=<sign>&swiftinfo_expires=<expires>' can be configured allowing privileged access to more sensitive information not meant to be public. DocImpact Change-Id: I2379360fbfe3d9e9e8b25f1dc34517d199574495 Implements: blueprint capabilities Closes-Bug: #1245694
* Change OpenStack LLC to FoundationZhiQiang Fan2013-09-201-1/+1
| | | | | Change-Id: I7c3df47c31759dbeb3105f8883e2688ada848d58 Closes-bug: #1214176
* Clean up utf8ness quoteYuan Zhou2013-08-291-20/+6
| | | | | | Move common codes on utf8ness and quote to common/utils.py Change-Id: I91d98a06fa94ac608119a0d70adedc6d73337c64
* Merge "Staticweb shortcut object 404s"Jenkins2013-08-161-1/+3
|\
| * Staticweb shortcut object 404sDavid Goetz2013-07-311-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | Right now static web checks to see if it should handle a request and if it decides that it shouldn't, in all cases, it lets the request pass through the pipeline normally. After this change on "x-web-mode: t" obj requests where staticweb is not setup and an object 404s it will just return the 404 instead of letting the app send a duplicate request to the object servers. Change-Id: I824f4b03f7546d26084fd51b97ea6e290169d22a
* | Clarify staticweb configuration with keystone.Sascha Peilicke2013-08-051-2/+5
|/ | | | | | | | Since staticweb works with anonymous requests by default, explictly mention delay_auth_decision for the authtoken middleware in /etc/swift/proxy-server.conf. Change-Id: I4879884b22224843cf9d4a50ba3359fac36957dc
* Encode unicode from JSON before using it as a string.Alex Gaynor2013-07-291-15/+15
| | | | | | | | | | Right now this code fails when used with a JSON decoder that always produces unicode. This isn't usually the case with CPython, where simplejson is used most of the time, however with the stdlib JSON library (as used on PyPy), this code fails. Change-Id: Ib2343243f40194d5b2784551a807c7f58970a6e9
* Fix incorrect status handling at staticwebKota Tsuyuzaki2013-07-241-1/+1
| | | | | | | | | | | | | | | | | | Staticweb middleware always searchs html for error response when response status code is not redirect. It causes swift to return error response even if the request succeeded. (e.g. When finding index.html with 200, staticweb returns 200error.html if it exist) This patch modifies the constraint on response status and fix bug 1204319. Fixes bug #1204319. Change-Id: Ib83c303917da7fb94999f2d4d35063b450d0e992
* Start using HackingDirk Mueller2013-07-151-1/+1
| | | | | | | | | | | | | | | Instead of blacklisting Hacking globally, only blacklist those that currently occur frequently (for a later followup patch), and fix the rest. In detail: H101 Use TODO(NAME) H201 no 'except:' at least use 'except Exception:' H231 octal number 022 should be written as 0o22 H401 docstring should not start with a space H701 Empty localization string Change-Id: Ib3b3d56b68d1cf15d3b67ac9749fcbdb876dc52a
* Remove dead code and tests for staticweb.Samuel Merritt2013-06-251-3/+0
| | | | | | Dead since e499b91. Change-Id: I3c571ae471eb4a0b95772b19eac90183cfc7fe22
* Clean up some remnants of StaticWeb's and TempURL's custom caching.Samuel Merritt2013-06-211-19/+0
| | | | | | | | As of e499b91, these config values and functions are unused in StaticWeb. As of 2e155e5, the comment in TempURL is false. Change-Id: I75b631ece8a9a6075c406765361629c549c449f6
* use get_container_info for staticwebMichael Barton2013-06-201-54/+12
| | | | | | Update staticweb to use get_container_info instead of its own memcache entry. Change-Id: I59bc0d94b9e45f3d5776a5462f53b82ab78101d4
* Treat directory objects as not foundDan Hersam2013-06-061-21/+80
| | | | | | | | | | | | | | | | | | | | Make StaticWeb search for an index file rather than returning a 0-byte object. Add new StaticWeb header to identify a directory marker object: X-Container-Meta-Web-Directory-Type (default to application/directory) so user can specify the content type for directory marker objects. This required requesting the container headers earlier in the code and clearing the value of _error for non-404 errors. Add unit tests for directory object scenarios. Fix end tags in an unrelated HTML block. Bug 1178817 DocImpact Change-Id: I561f00c099feaa82fd658f5050bd90c76717da24
* fixes some problems pyflakes complains aboutGreg Lange2013-03-281-1/+1
| | | | Change-Id: I1eb16fa595b0b1d97ae14279b8aa25bd2740f511
* Fix for unicode issues in staticwebEohyung Lee2013-03-211-2/+8
| | | | | | | When prefix or subdir is unicode, it fails. Fixes: bug #1158092 Change-Id: I966a7a198a5ca5ea152872c94240e8c2ebe57d36
* Cleanup of file permissionsChuck Thier2013-02-151-0/+0
| | | | | | Mostly removed execute on non-executable files Change-Id: Ibfbe7e0cf0fbeabef602d70b20f75e1dd3bdf9c9
* Swift MemcacheRing (set) interface is incompatible fixesTong Li2013-02-131-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch fixes the Swift MemcacheRing set and set_multi interface incompatible problem with python memcache. The fix added two extra named parameters to both set and set_multi method. When only time or timeout parameter is present, then one of the value will be used. When both time and timeout are present, the time parameter will be used. Named parameter min_compress_len is added for pure compatibility purposes. The current implementation ignores this parameter. To make swift memcached methods all consistent cross the board, method incr and decr have also been changed to include a new named parameter time. In future OpenStack releases, the named parameter timeout will be removed, keep the named parameter timeout around for now is to make sure that mismatched releases between client and server will still work. From now on, when a call is made to set, set_multi, decr, incr by using timeout parametner, a warning message will be logged to indicate the deprecation of the parameter. Fixes: bug #1095730 Change-Id: I07af784a54d7d79395fc3265e74145f92f38a893
* Fix spelling mistakesJoe Gordon2013-02-121-1/+1
| | | | | | | git ls-files | misspellings -f - Source: https://github.com/lyda/misspell-check Change-Id: I4132e6a276e44e2a8985238358533d315ee8d9c4
* Use a doubled proxy-logging instead of each middleware handling itDavid Goetz2013-02-111-97/+14
| | | | | | | | | | | | | | | | | differently (if at all) Adding a swift.source to wsgi pre_auth funcs and all middleware that makes subrequests to proxy server. NOTE: This change will result in a change in the number of proxy logs made for staticweb, formpost, tempurl, and any other middleware that performs sub requests (including swauth and SOS). Please see docs for details. DocImpact Change-Id: I80cf2806add1c3d34054147e2515944be340455b
* Cleanup based on pyflakes.Monty Taylor2013-02-011-6/+1
| | | | | | | | | | | pyflakes itself can't be used in any automated gating way, because there are two sets of false errors it raises. However, as an exercise, cleaning up the 'valid' ones uncovered three actual bugs. The other changes (mostly unused variables) are included here for fun. Command run: pyflakes swift | grep -v "undefined name '_'" Change-Id: I18696bf047dedad1a9fdbde3463e214fba95f7c6
* Allow optional, temporary healthcheck failure.Darrell Bishop2012-12-041-1/+2
| | | | | | | | | | | | | | | | | A deployer may want to remove a Swift node from a load balancer for maintenance or upgrade. This patch provides an optional mechanism for this. The healthcheck filter config can specify "disable_path" which is a filesystem path. If a file is present at that location, the healthcheck middleware returns a 503 with a body of "DISABLED BY FILE". So a deployer can configure "disable_path" and then touch that filesystem path, wait for the proxy to be removed from the load balancer pool, perform maintenance/upgrade, and then remove the "disable_path" file. Also cleaned up the conf file man pages a bit. Change-Id: I1759c78c74910a54c720f298d4d8e6fa57a4dab4
* changed TRUE_VALUES references to utils.config_true_value() callJohn Dickinson2012-10-291-28/+30
| | | | | | cleaned up pep8 (v1.3.3) in all files this patch touches Change-Id: I30e8314dfdc23fb70ab83741a548db9905dfccff
* local WSGI Request and Response classesMichael Barton2012-09-281-2/+1
| | | | | | | | | | | | | | | This change replaces WebOb with a mostly compatible local library, swift.common.swob. Subtle changes to WebOb's API over the years have been a huge headache. Swift doesn't even run on the current version. There are a few incompatibilities to simplify the implementation/interface: * It only implements the header properties we use. More can be easily added. * Casts header values to str on assignment. * Response classes ("HTTPNotFound") are no longer subclasses, but partials on Response, so things like isinstance no longer work on them. * Unlike newer webob versions, will never return unicode objects. Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
* Upating proxy-server StatsD logging.Darrell Bishop2012-08-291-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Removed many StatsD logging calls in proxy-server and added swift-informant-style catch-all logging in the proxy-logger middleware. Many errors previously rolled into the "proxy-server.<type>.errors" counter will now appear broken down by response code and with timing data at: "proxy-server.<type>.<verb>.<status>.timing". Also, bytes transferred (sum of in + out) will be at: "proxy-server.<type>.<verb>.<status>.xfer". The proxy-logging middleware can get its StatsD config from standard vars in [DEFAULT] or from access_log_statsd_* config vars in its config section. Similarly to Swift Informant, request methods ("verbs") are filtered using the new proxy-logging config var, "log_statsd_valid_http_methods" which defaults to GET, HEAD, POST, PUT, DELETE, and COPY. Requests with methods not in this list use "BAD_METHOD" for <verb> in the metric name. To avoid user error, access_log_statsd_valid_http_methods is also accepted. Previously, proxy-server metrics used "Account", "Container", and "Object" for the <type>, but these are now all lowercase. Updated the admin guide's StatsD docs to reflect the above changes and also include the "proxy-server.<type>.handoff_count" and "proxy-server.<type>.handoff_all_count" metrics. The proxy server now saves off the original req.method and proxy_logging will use this if it can (both for request logging and as the "<verb>" in the statsd timing metric). This fixes bug 1025433. Removed some stale access_log_* related code in proxy/server.py. Also removed the BaseApplication/Application distinction as it's no longer necessary. Fixed up the sample config files a bit (logging lines, mostly). Fixed typo in SAIO development guide. Got proxy_logging.py test coverage to 100%. Fixed proxy_logging.py for PEP8 v1.3.2. Enhanced test.unit.FakeLogger to track more calls to enable testing StatsD metric calls. Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576
* Fixed bug in staticweb with log_headersgholt2012-07-291-2/+6
| | | | Change-Id: I7e5e254660e2ce35bc7fa95435db0a987d5acd0a
* Reverted the pulling out of various middleware:gholt2012-05-161-0/+564
| | | | | | | | RateLimit StaticWeb TempURL/FormPOST Change-Id: I988e93e6f4aacb817a2e354d43a04e47516fdf88
* Pulled StaticWeb out to separate projectgholt2012-05-051-564/+0
| | | | | | | | | | | | | | StaticWeb is now at http://gholt.github.com/swift-staticweb/ For current users of StaticWeb, this will require installing the new package and changing the "use" line of the staticweb filter conf section to: use = egg:swiftstaticweb#middleware And then 'swift-init proxy reload'. Change-Id: Iab32adb5927698a667c5c6d6a572c44ca23414eb
* Merge "HTTP status value naming normalization"Jenkins2012-05-031-11/+13
|\
| * HTTP status value naming normalizationVictor Rodionov2012-04-251-11/+13
| | | | | | | | | | | | Replace HTTP status code with constants like HTTP_OK, HTTP_NOT_FOUND, etc. Change-Id: I76cd8bf880e7269454192a3bc4b027a297741247
* | Removed unused imports.Josh Kearney2012-04-051-1/+1
|/ | | | Change-Id: I5a837bea8699b78484265dc2434b190d0cf15de2
* Update middleware to use common pre_authed funcsgholt2012-03-291-34/+17
| | | | | | | Update StaticWeb, FormPost, and TempURL to use common make_pre_authed_env and make_pre_authed_req functions. Change-Id: I32cc3cddff0d2aaeb6314578872707c1a37b4bce
View Lorry Controller Status