summaryrefslogtreecommitdiff
path: root/tests/unit/test_web.py
Commit message (Collapse)AuthorAgeFilesLines
* Fix links for jobs with special charactersAlbin Vass2022-08-231-0/+5
| | | | Change-Id: I12e8a056a2e5cd1bb18c1f24ecd7db55405f0a8c
* Merge "Handle jwt decoding error, fix exception default messages"Zuul2022-07-141-0/+27
|\
| * Handle jwt decoding error, fix exception default messagesMatthieu Huin2022-05-121-0/+27
| | | | | | | | | | | | | | | | | | | | Using a badly formatted token resulted in an error 500 from zuul-web. Return a more precise error message and an error 401 in zuul-web when this occurs. Also fix a typo in default messages for some auth-related exceptions. Change-Id: I4abe013e76ac51c3dad7ccd969ffe79f5cb459e3
* | Merge "Fix read-only branches error in zuul-web"Zuul2022-07-061-0/+22
|\ \
| * | Fix read-only branches error in zuul-webSimon Westphahl2022-07-041-0/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When exclude-unprotected-branche is in effect and a project doesn't have any protected branches (e.g. a wrong branch protection rule in Github or none at all) the branch cache will contain an empty list. Since the empty list was so far used to indicate a fetch error, those projects showed up with a config error in zuul-web ("Will not fetch project branches as read-only is set"). For the branch cache we need to distinguish three different cases: 1. branche cache miss (no fetch yet) 2. branch cache hit (including empty list of branches) 3. fetch error Instead of storing an empty list in case of a fetch error we will store a sentinel value of `None` in those cases. However, with this change we can't use `None` as an indicator for a cache miss anymore, so we are now raising a `LookupError` instead. Change-Id: I5b51805f7d0a5d916a46fe89db7b32f14063d7aa
* | | Fix flaky zuul-web tests using cached tenant listSimon Westphahl2022-07-041-0/+10
| | | | | | | | | | | | | | | | | | | | | Disable caching for the remaining zuul-web tests that use the tenant list. Change-Id: I03c80a2171ee02fba41319b21f78003e4c8554b4
* | | Fix race is test_tenant_add_removeJames E. Blair2022-07-021-0/+2
|/ / | | | | | | | | | | | | | | | | | | | | Change I101fc42e29f4b376272c417ec536c17f05cb1a60 introduced a race in test_tenant_add_remove. If the test takes longer than 1 second, it will succeed, but less than the 1 second cache interval and it will fail. Correct that by disabling caching for the test. Change-Id: I313e519ab26cdbdad0cc17cd4d3489b49482f0b3
* | Add ssh_server option to Gerrit driverJames E. Blair2022-06-221-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In a multi-host Gerrit environment (HA or failover) it's plausible that admins may use one mechanism for managing ingress for HTTP requests and a different for SSH requests. Or admins may have different firewall rules for each. To accomodate these situations, Add an "ssh_server" configuration item for Gerrit. This makes the set of hostname-like items the following: * server: the HTTP hostname and default for all others * canonical_hostname: what to use for golang-style git paths * ssh_server: the hostname to use for SSH connections * baseurl: the base URL for HTTP connections The following are equivalent: server=review.example.com ssh_server=ssh-review.example.com and: server=ssh-review.example.com baseurl=https://review.example.com Change-Id: I6e9cd9f48c1a78d8d24bfe176efbb932a18ec83c
* | Zuul-web: return 404 when attempting autohold ops on an unknown tenantMatthieu Huin2022-06-031-0/+5
| | | | | | | | Change-Id: I03e78a1774e20af6f6895faa089017cdcf62bb48
* | Merge "Fix zuul-web layout update on full/tenant-reconfigure"Zuul2022-05-311-0/+89
|\ \
| * | Fix zuul-web layout update on full/tenant-reconfigureJames E. Blair2022-05-251-0/+89
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When an operator issues a full-reconfigure or tenant-reconfigure command to the scheduler, the branch cache is cleared. It will be fully populated again during the subsequent full-reconfiguration, or eventually populated more slowly over time after a tenant-reconfiguration. But during the time the branch cache is clear, zuul-web will be unable to update layouts since it is incapable of populating the branch cache itself. This produces a short time period of errors during a full-reconfig or a potentially long time period of errors after a tenant-reconfig. To correct this, we now detect whether a layout update in zuul-web is incomplete due to a branch cache error, and retry the update later. In the scheduler, we only clear the projects from the branch cache that are affected by the tenants we are reloading (still all the projects for a full-reconfigure). This limits the time during which zuul-web will be unable to update the layout to only the time that the scheduler spends actually performing a reconfiguration. (Note that in general, the error here is not because zuul-web is loading the layout for the tenant that zuul-scheduler is reconfiguring, but rather it is loading a layout for a tenant which has projects in common with the tenant that is being reconfigured.) Change-Id: I6794da4d2316f7df6ab302c74b3efb5df4ce461a
* | Merge "Add support for GHE repository cache"Zuul2022-05-301-0/+1
|\ \
| * | Add support for GHE repository cacheJames E. Blair2022-05-051-0/+1
| |/ | | | | | | Change-Id: Iec87857aa58f71875d780da3698047dae01120d7
* | Merge "Deduplicate jobs in dependency cycles"Zuul2022-05-261-0/+8
|\ \
| * | Deduplicate jobs in dependency cyclesJames E. Blair2022-05-241-0/+8
| |/ | | | | | | | | | | | | | | | | | | | | This adds support for deduplicating jobs within dependency cycles. By default, this will happen automatically if we can determine that the results of two builds would be expected to be identical. This uses a heuristic which should almost always be correct; the behavior can be overidden otherwise. Change-Id: I890407df822035d52ead3516942fd95e3633094b
* | Clarify zuul admin CLI scopeMatthieu Huin2022-05-191-5/+5
|/ | | | | | | | | | | | | | | | | | | We have two CLIs: zuul-client for REST-related operations, which cover tenant-scoped, workflow modifying actions such as enqueue, dequeue and promote; and zuul which supercedes zuul-client and covers also true admin operations like ZooKeeper maintenance, config checking and issueing auth tokens. This is a bit confusing for users and operators, and can induce code duplication. * Rename zuul CLI into zuul-admin. zuul is still a valid endpoint and will be removed after next release. * Print a deprecation warning when invoking the admin CLI as zuul instead of zuul-admin, and when running autohold-*, enqueue-*, dequeue and promote subcommands. These subcommands will need to be run with zuul-client after next release. * Clarify the scopes and deprecations in the documentation. Change-Id: I90cf6f2be4e4c8180ad0f5e2696b7eaa7380b411
* Fix background layout updates in zuul-webJames E. Blair2022-04-131-0/+17
| | | | | | | | | | | Change Ia0a524053a110e8e48f709d219651e2ad9c8513d updated the scheduler to more aggressively use the file cache, but did not make the corresponding updates to zuul-web. This means that configuration changes are not being reflected in the web UI until the service restarts. This corrects that and adds a test. Change-Id: Iebdf485105bd02e57fa0d3db5ba162308b640ca0
* Merge "Make promote work for any pipeline manager"Zuul2022-03-281-0/+166
|\
| * Make promote work for any pipeline managerJames E. Blair2022-03-101-0/+166
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This alters the behavior of the promote event handler so that it does something useful in independent pipelines as well as dependent. It not only re-orders changes within a pipeline's shared queue (the old behavior), but it also re-orders the shared queue within the pipeline. When used in an independent pipeline, this will give the item an advantage when requesting nodes or semaphores, or starting jobs. This behavior applies to dependent pipelines as well -- the behavior is the same for every pipeline. Restarting jobs for changes in independent pipelines would be counter-productive (and even in dependent pipelines we may have restarted more jobs than necessary if the change at the head wasn't being altered), so it has been altered to only dequeue/enqueue items when necessary to achieve the requested order. The event argument to addChange within the promote method has been changed from the promote event to the original item enqueue event. Methods within the pipeline manager assume that event type is a TriggerEvent rather than a ManagementEvent and could throw some (non-fatal) errors when reporting. Change-Id: Ib4ab855cff27bf8e96aa852333fb4ace656235b4
* | Merge "Report gross/total tenant resource usage stats"Zuul2022-03-231-0/+6
|\ \
| * | Report gross/total tenant resource usage statsBenjamin Schanzel2022-03-171-0/+6
| |/ | | | | | | | | | | | | | | | | | | | | | | Export a new statsd gauge with the total resources of a tenant. Currently, we only export resources of in-use nodes. With this, we additionally report the cummulative resources of all of a tenants nodes (i.e. ready, deleting, ...). This also renames the existing in-use resource stat to distinguish those clearly. Change-Id: I76a8c1212c7e9b476782403d52e4e22c030d1371
* | Merge "Support submitWholeTopic in Gerrit"Zuul2022-03-221-1/+3
|\ \
| * | Support submitWholeTopic in GerritJames E. Blair2022-03-211-1/+3
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds a query for changes which are set (by Gerrit) to be submitted together due to submitWholeTopic being enabled. In this case, changes which are of the same topic will be merged simultaneously by Gerrit, therefore Zuul should treat them as a set of circular dependencies. The behavior is automatic, since Gerrit will return a set of changes if the setting is enabled, or the empty list if it is not. Zuul still requires that circular dependencies be enabled in any queues where they appear. If users have submitWholeTopic enabled in Gerrit but do not have allow-circular-dependencies enabled, they may begin to see errors. However, the errors are self-explanatory, and installations such as these are already not testing what Gerrit will merge, so reporting the discrepancy is an improvement. Change-Id: Icf65913a049a9b9ddbedd20cc73bf44ffcc831b8
* | Merge "Add support for Microsoft login"Zuul2022-03-211-0/+2
|\ \ | |/ |/|
| * Add support for Microsoft loginJames E. Blair2022-02-111-0/+2
| | | | | | | | | | | | | | | | | | | | | | Microsoft login (or Azure AD) nearly supports the OpenID Connect protocol. With extensive configuration and some additional customization of the values expected in the tokens, it will work with Zuul. Add the necessary additional configuration and a HOWTO with the procedure. Change-Id: I445a0494796572762fbf78e580d7d6dd17be7239
* | Fix removal of tenant on layout updateSimon Westphahl2022-02-211-0/+25
|/ | | | | | | | | | | Removed tenants where not correctly handled by the layout update logic in the scheduler and zuul-web. The tenant was only removed from the scheduler that process the reconfiguration event but not from other scheduler and web instances that reacted on the layout update. In addition to that we now also clear the TPCs of the removed tenant. Change-Id: I0b3ff77388c27132a558207316bd42c8646212c6
* Add stats to web serverJames E. Blair2022-02-021-0/+7
| | | | | | | | | | | | This adds matrics which report the number of thread workers in use as well as the number of requests queued at the start of each request in cherrypy. It also reports the number of streamers currently running. These can help us detect and diagnose problems with the web server. Change-Id: Iadf9479ae84167892ab11ae122f275637c0c6c6f
* Merge "Remove rpcclient from connection test fixtures"Zuul2022-01-251-2/+2
|\
| * Remove rpcclient from connection test fixturesJames E. Blair2022-01-171-2/+2
| | | | | | | | | | | | | | There are no uses of the rpc client in tests now, remove the initialization and parameters. Change-Id: I2d22fa73138c8cc2db96f78ecbc2dc3792cdd8bb
* | Merge "Remove rpcclient from test_web"Zuul2022-01-251-53/+31
|\ \ | |/
| * Remove rpcclient from test_webJames E. Blair2022-01-171-53/+31
| | | | | | | | | | | | | | These tests incidentally use the rpcclient to create autoholds; do that directly instead. Change-Id: Ic3e63b424d2f01a168281f16e5f7615af3ceddda
* | autohold: use canonical project name in REST interfaceIan Wienand2022-01-181-1/+2
|/ | | | | | | | | | | | | | | | | | | | | | | | | | I noticed autoholds not matching as I expected when using the zuul-client like $ zuul-client autohold --tenant openstack \ --project opendev/system-config --job ... It turns out the autohold matching does a full match against the canonical project name when deciding if a hold matches. The gearman RPC client expands your --project argument to it's canonical name @ zuul/rpclistener.py:RPCListener.handle_autohold() (trusted, project) = tenant.getProject(args['project']) if project: params['project_name'] = project.canonical_name else: error = "Invalid project: %s" % args['project'] This does the same in the REST client side, so similarly formatted autohold commands from zuul-client work the same as for the gearman client. Change-Id: Id565d540cbd6a5a2108b49af5db2c2e308b26cc3
* Handle missing config files in zuul-webJames E. Blair2022-01-131-14/+35
| | | | | | | | | | | | | | | | | | | | | The following sequence is possible: 1) New project added to tenant config file 2) Scheduler begins smart reconfiguration 3) Scheduler encounters problem accessing the branch list for the new repo. This is treated as a ConfigurationError and added to the layout error list but proceeds. ?) If at some point a branch listing for the new project has succeeded, there will be an entry for the branch in the branch cache. 4) A zuul-web starts (or attempts a reconfiguration), sees the branch in the branch cache, attempts to load the files from the config cache, and fails to load the layout. The scheduler and web show different behaviors because web is unable to fetch files via mergers. To bring them closer to the same behavior, treat missing files from the config cache as layout errors. Change-Id: I76d659f558cc3ed95a6ba7259d11b457ca57976c
* Merge "REST API: add idx_min, idx_max params to getBuilds, getBuildsets"Zuul2021-12-161-0/+32
|\
| * REST API: add idx_min, idx_max params to getBuilds, getBuildsetsMatthieu Huin2021-12-021-0/+32
| | | | | | | | | | | | | | | | | | | | | | Allow filtering searches per primary index; ie return only builds or buildsets whose primary index key is greater than idx_min or lower than idx_max. This is expected to increase queries speed compared to using the offset argument when it is possible to do so, since "offset" requires the database to sift through all results until the offset is reached. Change-Id: I420d71d7c62dad6d118310525e97b4a546f05f99
* | Make TestWeb.test_web_components work with multiple schedulersFelix Edel2021-11-301-1/+1
|/ | | | Change-Id: I98fbda8b78a44d66efacef1a673a82cac7e2356d
* Load repo state from pipeline state on executorsSimon Westphahl2021-11-171-2/+4
| | | | | | | | | | | | | Instead of sending the repo state via the build request we only provide the path to the merge/extra repo state that we already as part of the pipeline state. The executor will then load the repo states from the given paths. This way we reduce the amount of duplicate data we store in Zookeeper which is especially important for the repo state, as it can become quite large. Change-Id: I7dc5d854dbee93af52d25b4462293c85eb7a1a8e
* Fix zuul-web startup config primingJames E. Blair2021-11-161-0/+65
| | | | | | | | | | | | | | | We didn't include the branch cache ltimes when loading the initial config in zuul-web, so if zuul-web starts before the branch cache is updated, it won't be told to refresh it when loading the layout. This matches what we do in zuul-scheduler, and adds a skipped test. The test does unusual things with the startup sequence and does not appear to be reliable when run upstream, so it is disabled for now. This also adds a debug line and adjust another to remove an apostraphe to make grepping easier. Change-Id: Ib184f4cadb21124fd3dedeea3797b01bb9c431cc
* Remove fake SQL classes and variables from testsFelix Edel2021-11-151-2/+1
| | | | | | | It looks like those are leftovers from early Scale-out-Scheduler development, but aren't used anymore. Change-Id: I6265a7fe159e4a395c09a77f58842f3e123e0b32
* Add testing for tenant authorizations overrideMatthieu Huin2021-11-121-0/+32
| | | | | | | | Add testing of the behavior of the {tenant}/authorizations endpoint in the REST API when authorizations override is allowed for a given issuer. Change-Id: I40df96cb956b0d8ef5647bdfb983ba474183703b
* Use abide for listing tenants in zuul-webFelix Edel2021-11-101-1/+0
| | | | | | | | | | This uses the abide in zuul-web to directly access the tenant list instead of asking the scheduler via RPC. Relevant RPC listener methods are removed as they are not used anywhere else. Change-Id: I918cc3b9b8a6e0ba95a24dc55bf06976c6e2cfb5
* Merge "zuul-web: add pipelines' manager, triggers data in status"Zuul2021-10-301-0/+2
|\
| * zuul-web: add pipelines' manager, triggers data in statusMatthieu Huin2021-10-271-0/+2
| | | | | | | | | | | | | | The /api/tenant/{tenant}/status includes information about pipelines' type (ie manager) and triggers drivers. Change-Id: I6abe4bdcf175426908c321a2a298b9743f5ea80b
* | Merge "Zuul-web: Add authentication-realm attribute to tenants"Zuul2021-10-301-0/+29
|\ \ | |/
| * Zuul-web: Add authentication-realm attribute to tenantsMatthieu Huin2021-10-271-0/+29
| | | | | | | | | | | | | | | | With this attribute, an operator can map a tenant to an authenticator. This is the authenticator that will be used by the Web UI on a whitelabeled setup, or when scoping the UI to the tenant. Change-Id: Ifd3d629ff3ed0c3c902433bdf808d457e755c528
* | Remove setBase from job freeze APIJames E. Blair2021-10-271-5/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As part of the work to freeze jobs into ZooKeeper, it was noted that the freeze job API called setBase on a frozen job. This is not what happens during actual job freezing and therefore the API produces output that differs from what would actually be run. The setBase method is already called at the start of the job freezing process and sets up frozen playbooks and the initial inheritance path for the first variant that is applied. Subsequent variants add their playbooks using applyVariant. It is not clear why setBase was being called a second time after all variants are applied. By removing this call (which should not be necessary on a frozen job) the only change in the test output is to remove a non-sensical item from the end of the inheritance path (job project-test1 on common-config/zuul.yaml line 53 -- that line is actually the 'base' job; it correctly appears as the first entry in the list). To resolve this, the apparently unnecessary setBase call is removed from the freeze job API. Change-Id: Ibb2bb8d17bf1a665467affe583f2b7780fb54322
* | Add test_freeze_noop_jobJames E. Blair2021-10-271-0/+58
|/ | | | | | | The noop job can be special; add a test to freeze it using the REST API to make sure that doesn't break. Change-Id: Ib91f123ec8bbcb5b3abeb8b2062f77efb21a8f06
* Store version information in component registryFelix Edel2021-10-201-1/+1
| | | | | | | This stores the zuul version of each component in the component registry and updates the API endpoint. Change-Id: I1855b2a6db2bd330343cad69d9d6cf21ea35a1f5
* Add /components API endpoint to zuul-webFelix Edel2021-10-201-0/+17
| | | | | | | | This endpoint returns the list of components coming from Zookeeper. We need this in order to create a "component overview" page in the UI. Change-Id: I3a478e3f37000adde23bf4fa8b8bc1f38c975f1b
* Make url an attribute of the base Ref classClark Boylan2021-10-081-0/+1
| | | | | | | | | | | | | | | | | | | | We expect that changeish.url is valid on Refs, Branches, Tags, and Changes but we only serialize and deserialize the url attribute in the Change class. This meant that when we shifted to relying on the zk change cache to load these objects back into memory we were getting them back without their url info. This is a minor issue when running with a single scheduler because it only causes problems for enqueing jobs after a restart and we rely on pulling newly merged changes from the cache. I suspect this would've caused bigger problems if we were running with multiple schedulers. To fix this we move the url attribute into the base Ref objects and serialize/deserialize it there. This shoudl ensure that we can load the complete data back again after a restart or when we switch to multiple schedulers. Change-Id: I10c3657924b6a5f075a439441fde6fe9e582030d
View Lorry Controller Status