summaryrefslogtreecommitdiff
path: root/zuul/model.py
Commit message (Collapse)AuthorAgeFilesLines
* Merge "Add Item.formatStatusUrl"Zuul2019-09-181-0/+21
|\
| * Add Item.formatStatusUrlJames E. Blair2019-09-171-0/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This method can be used by reporters to generate the best "status" url to report for an item. It will report the buildset page if that is available, or the per-change status page if it is not. Eventually we plan on making the database required, at which point we can insert buildsets in the db earlier. Then we can report the buildset page in all cases. By beginning to use this method now, we can seamlessly upgrade reporters in the future. Test coverage for this is added in change Ida0cdef682ca2ce117617eacfb67f371426a3131. Change-Id: Ib0c2ca84f6c4d30f233382048c8885fb73edfeec
* | Merge "Add report time to item model"Zuul2019-09-181-0/+2
|\ \ | |/
| * Add report time to item modelJames E. Blair2019-09-171-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This allows reporters to include the enqueue/dequeue time of an item. The item model already hase enqueue/dequeue times which we use when reporting status, however, a reporter runs right before the item is dequeued. So we need one more time value which corresponds to the start of the reporting phase -- thus report_time in this patch. Test coverage for this is added in change Ida0cdef682ca2ce117617eacfb67f371426a3131. Change-Id: I093626e098b7ce2deea2b0c25265cb48d38712ad
* | Merge "Add no-jobs reporter action"Zuul2019-09-181-0/+2
|\ \ | |/
| * Add no-jobs reporter actionJames E. Blair2019-09-171-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This facilitates integration with the gerrit checks API (and may prove useful for other similar APIs). It will allow us to report that a change has no jobs in a particular pipeline. A Zuul pipeline will correspond to a Gerrit check, which means we can update the status for that check from "SCHEDULED" to "NOT_RELEVANT" if we determine that no jobs should run for the change. This closes out the status of the check in Gerrit when a project is configured to participate in a check/pipeline but no jobs are actually configured. Test coverage for this will be added in change Ida0cdef682ca2ce117617eacfb67f371426a3131. Change-Id: Ide2a332b294d7efe23601d80eeb92b5af1d4c21b
* | Merge "Add enqueue reporter action"Zuul2019-09-171-0/+4
|\ \ | |/
| * Add enqueue reporter actionJames E. Blair2019-09-171-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This facilitates integration with the gerrit checks API (and may prove useful for other similar APIs). It will allow us to report that a change has been enqueued in a particular pipeline. A Zuul pipeline will correspond to a Gerrit check, which means we can update the status for that check from "NOT_STARTED" to "SCHEDULED" when it enters the pipeline. This is important for our check polling loop, and it will cause that check to stop appearing in the list of pending checks. Test coverage for this is added in change Ida0cdef682ca2ce117617eacfb67f371426a3131. Change-Id: I9ec329b446fa51e0911d4d9ff67eea7ddd55ab5d
* | Annotate QueueItem loggerTobias Henkel2019-08-231-1/+2
|/ | | | | | The queue item always has the event id so annotate its logger. Change-Id: I873462bf738dc24f5afa8d9c572a77972a680c61
* Add option to report build pageJames E. Blair2019-08-081-1/+13
| | | | | | | | | | | | | | | | | | This adds a tenant option to use the Zuul web build page as the URL reported to the code review system when a build completes. The setting is per-tenant (because it requires that the tenant have a working SQL reporter configured in all pipelines) and defaults to false, since we can't guarantee that. In the future, we expect to make SQL reporting implicit, then this can default to true and eventually be deprecated. A new zuul.conf option is added and marked required to supply the root web URL. As we perform further integration with the web app, we may be able to deprecate other similar settings, such as "status_url". Change-Id: Iaa3be10525994722d020d2aa5a7dcf141f2404d9
* Merge "Add Authorization Rules configuration"Zuul2019-07-311-1/+117
|\
| * Add Authorization Rules configurationmhuin2019-07-301-1/+117
| | | | | | | | | | | | | | | | | | | | Allow an operator to define authorization rules. Allow an operator to add authorization rules to a tenant. Add a rule parser and a rule registry. The authZ engine is not plugged in yet. Change-Id: I3a86c6c7d62ad2bce68a98dbd2fff18549b94fb9
* | Merge "Support squash merge in Github"Zuul2019-07-311-0/+2
|\ \ | |/ |/|
| * Support squash merge in GithubTobias Henkel2019-06-111-0/+2
| | | | | | | | | | | | | | | | | | Many projects running on Github want to use the squash merge strategy of github to keep their history clean. Now that the github reporter forwards the merge-mode that can be configured on a project it is easy to also support squash merge. Change-Id: I1f4131002dd380f2860557a9aebcc0bcf4dc2af8
* | Merge "Add "supercedes" pipeline option"Zuul2019-07-181-0/+12
|\ \
| * | Add "supercedes" pipeline optionJames E. Blair2019-07-151-0/+12
| | | | | | | | | | | | | | | | | | This lets us abort check jobs when a change goes straight to gate. Change-Id: I0e07a4be74ea708d0a0e186f523dd41174215b92
* | | Record and report time for trigger eventsSimon Westphahl2019-07-181-0/+1
|/ / | | | | | | | | | | | | | | | | | | Sometimes, e.g. during reconfiguration, it can take quite some time between the trigger event and when a change is enqueued. This change allows tracking the time it takes from receiving the event until it is processed by the scheduler. Change-Id: I347acf56bc8d7671d96f6be444c71902563684be
* | Handle existing broken config in job updatesJames E. Blair2019-07-121-6/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If Zuul was unable to freeze the job graph of the previous layout, then it would throw an exception when calculating whether jobs have changed. This is a problem if it does so on a change which fixes an error in the current tenant config, as it means the fix can not merge. To handle this case, we need to catch that exception, and then provide a default value indicating whether the job config has been updated or not. Because returning True could cause every job with a file matcher to run unecessarily, the safer default is False (so the job relies on the file matcher alone). Change-Id: I8a72e57e068b073e37274cdabeacffc9daee2e94
* | Fix typo in debug messageAndreas Jaeger2019-07-111-2/+2
| | | | | | | | | | | | This is a simple typo fix. Change-Id: Ibe5eef419c45886a0cc92a7988a01448c152a400
* | Run jobs when their own config changesJames E. Blair2019-07-081-15/+62
| | | | | | | | | | | | | | | | | | This causes file matchers to automatically match when the configuration of the job itself changes. This can be used instead of matching "^.zuul.yaml$" which may cause too many jobs to run in larger repos. Change-Id: Ieddaead91b597282c5674ba99b0c0f387843c722
* | Fix multi-tenant caching of extra config filesJames E. Blair2019-07-031-23/+86
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The cache of unparsed config data holds all of the config objects on a project-branch, for all of the tenants. That works as long as all of the tenants load the same config files. With the addition of the extra-config-paths tenant option, different tenants may now see different configuration files for the same project-branch. Because the cache did not account for that, the data for the last tenant to load a project-branch is what would be cached. To correct this, make the global unparsed config cache aware of which paths have been searched in constructing it, and cache each file individually. This allows the cache to continue to be efficient across systems with large tenants where the same repos appear multiple times, in that each configuration object will still be stored only once. However, by storing them along with their files, the unparsed config for each tenant can be created from the cache, even if the cache contains more files than that tenant should see. Because the cache remembers which files have been searched, we can determine whether the cache results from a prior tenant are valid for a later one, or if the later one is meant to search more files, we can run another cat job and fetch the larger set of files and expand the cache. A unit test is adjusted to add a tenant which loads fewer files before one which loads more to verify that the cache expands appropriately when loading the second tenant. Change-Id: I268d89dfaaaf65af2237ec852d706a2752535c4a
* | Merge "model: add cleanup-run to the job configuration"Zuul2019-07-031-4/+13
|\ \
| * | model: add cleanup-run to the job configurationTristan Cacqueray2019-05-291-4/+13
| | | | | | | | | | | | | | | | | | | | | This change adds a cleanup-run job attribute to specify a cleanup playbook to be executed after the job execution. Change-Id: Iebd93b2d6d13e1d4ccf2878e79e563cc9da2cce9
* | | Merge "Filter out unprotected branches from builds if excluded"Zuul2019-07-021-4/+7
|\ \ \
| * | | Filter out unprotected branches from builds if excludedTobias Henkel2019-06-251-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When working with GitHub Enterprise the recommended working model is branch&pull within the same repo. This is especially necessary for workflows that combine multiple repos in a single workspace. This has the side effect that those repos can contain a large number of branches that never will be part of a job. Having many branches in a repo can have a large impact on the executor performance so exclude them from the repo state if we exclude them in the tenant config. This change only affects branches, not tags or other references. Change-Id: Ic8e75fa8bf76d2e5a0b1779fa3538ee9a5c43411
* | | | Reload config on extra config files changesJames E. Blair2019-07-011-6/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The change to add extra-config-paths neclected to update the updatesConfig() method to account for the potential new paths; this could Zuul to fail to reload its configuration on changes which touched only the extra files. This corrects that and adds a test case. Change-Id: I79fa02a482a0bbebff0de7f3081bb7ed04f1214c
* | | | Add extra-config-paths tenant config optionJames E. Blair2019-06-281-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is intended to address the case of the zuul-jobs repo, where it would be convenient to include a project definition and job definitions which test the roles in the zuul-jobs repo. However, we want the repo te be consumable by anyone, which means we can not encode nodesets or required-projects in any jobs which may be loaded by another Zuul. To address this, this commit adds a feature which will allow us to put those job and project definitions in a separate file or directory which will not be loaded by default. But in the Zuul tenant of the OpenDev Zuul installation, we will configure the system to load this secondary configuration location, so the self-test jobs will be available. Change-Id: Ic205d1f93f583514757a100471c47688d6641c53
* | | | Merge "Differentiate between queued and waiting jobs in zuul web UI"Zuul2019-06-261-0/+3
|\ \ \ \
| * | | | Differentiate between queued and waiting jobs in zuul web UIFelix Schmidt2019-06-201-0/+3
| |/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When working with job dependencies, the zuul web UI does not differentiate between jobs that are able to run and those that are waiting for another job to finish before they can actually run. As both are marked as 'queued', this often leads to confusion and people are wondering why the queuing takes so long. To better differentiate between these states, this change introduces a new state 'waiting' in zuul web. Change-Id: Id0e85dc2db9660925209dc3cee365aa5c1241190
* | | | Allow config projects to override allowed-projectsJames E. Blair2019-06-241-1/+4
|/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To handle the case where an untrusted project defines a job with a secret which another project would like to run, allow a config project to attach that job to a project-pipeline and have it run regardless of the allowed-projects setting. Normally, untrusted jobs with secrets have an implicit and non-overridable allowed-projects setting of only that project, to avoid a situation where another project with a trusted post-review pipeline gains access to the secret by using a Depends-On to a change which lifts the allowed-projects restriction. This change allows a config project to bypass this, in effect saying that the projects involved trust each other sufficiently (or else, do not have access to a post-review pipeline which could be used to obtain secrets). Change-Id: I52ab193d0e39a37de64c8b3cb6953538e4073b43
* | | Merge "Show retry attempts in UI"Zuul2019-06-171-0/+1
|\ \ \ | |_|/ |/| |
| * | Show retry attempts in UIFelix Schmidt2019-05-231-0/+1
| |/ | | | | | | | | | | | | | | Currently, it's not visible in the Zuul UI if a job runs for the first time or had already a retry. This change adds the number of tries to the job name if the job had a least one retry. Change-Id: I75f400b7eec1e623045b600a7bd93846a6ac3090
* | Merge "Annotate logs around build completion and cancellation"Zuul2019-06-101-19/+20
|\ \
| * | Annotate logs around build completion and cancellationTobias Henkel2019-05-301-19/+20
| | | | | | | | | | | | | | | | | | | | | The logs around build completion and cancelations should be annotated with the event that triggered the action. Change-Id: Ifd039d0314a32647a0ec9ebe9bff064e503e0452
* | | Merge "Annotate builds with event id"Zuul2019-06-101-1/+2
|\ \ \ | |/ /
| * | Annotate builds with event idTobias Henkel2019-05-301-1/+2
| | | | | | | | | | | | | | | | | | | | | It's useful to be able to trace an event through the system including the builds. Change-Id: If852cbe8aecc4cf346dccc1b8fc34272c8ff483d
* | | model: add annotateLogger procedureTristan Cacqueray2019-06-071-4/+8
|/ / | | | | | | | | | | | | This change adds a new procedure to the ChangeQueue to limit direct access to the event variable member. Change-Id: Ic48b0a6c67c1ac888cf6d48f0164d4b2e260daec
* | Report tenant and project specific resource usage statsTobias Henkel2019-05-291-0/+1
|/ | | | | | | | | | | | | | | | | | | | | | | We currently lack means to support resource accounting of tenants or projects. Together with an addition to nodepool that adds resource metadata to nodes we can emit statsd statistics per tenant and per project. The following statistics are emitted: * zuul.nodepool.resources.tenant.{tenant}.{resource}.current Gauge with the currently used resources by tenant * zuul.nodepool.resources.project.{project}.{resource}.current Gauge with the currently used resources by project * zuul.nodepool.resources.tenant.{tenant}.{resource}.counter Counter with the summed usage by tenant. e.g. cpu seconds * zuul.nodepool.resources.project.{project}.{resource}.counter Counter with the summed usage by project. e.g. cpu seconds Depends-On: https://review.openstack.org/616262 Change-Id: I68ea68128287bf52d107959e1c343dfce98f1fc8
* Merge "Annotate node request processing with event id"Zuul2019-05-221-1/+7
|\
| * Annotate node request processing with event idTobias Henkel2019-05-201-1/+7
| | | | | | | | | | | | | | | | | | Adding the event ids to the ndoeset processing makes it much easier to trace events through the system. This also adds the event id as metadata to the node request so nodepool can continue annotating logs while working on the node request. Change-Id: I2fac44564552d1fd303b76b994f4c4c16c1370b1
* | Merge "pass-to-parent: Fix passing multiple secrets to parent"Zuul2019-05-221-2/+2
|\ \ | |/ |/|
| * pass-to-parent: Fix passing multiple secrets to parentChristian Fetzer2019-05-201-2/+2
| | | | | | | | | | | | | | | | | | When multiple secrets were marked as 'pass-to-parent', only the last one was actually past to the parent job. The issue was caused by an incorrect indentation. Change-Id: I74a672f313b042e3075d88e5d1f5a9f02c365868
* | Annotate job freezing logs with event idTobias Henkel2019-05-171-19/+23
| | | | | | | | | | | | | | When tracing an event through the system it's useful to have the job freezing process annotated with the event that caused the action. Change-Id: Iac64b437070da0f4ec570c9892c4447af0b7abf5
* | Attach event to queue itemTobias Henkel2019-05-171-3/+4
|/ | | | | | | As a preparation to further attach the event id to the logs we need to add the trigger event to the queue items so we can use it later. Change-Id: I9cc87e5be9a391a53a0dcfd92a990ca848616ca0
* Merge "Support fail-fast in project pipelines"Zuul2019-05-161-1/+10
|\
| * Support fail-fast in project pipelinesTobias Henkel2019-04-291-1/+10
| | | | | | | | | | | | | | | | | | | | | | In some cases like resource constrained environments it is beneficial to report on changes in a fail fast manner to immediately report if one job fails. This can be useful especially if a project has many expensive long-running jobs. This introduces a fail-fast flag in the project pipeline that let's the project choose the trade off between full information and quick feedback. Change-Id: Ie4a5ac8e025362dbaacd3ae82f2e8369f7447a62
* | Merge "github: add event filter debug"Zuul2019-05-161-0/+12
|\ \
| * | github: add event filter debugTristan Cacqueray2018-07-161-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | This change improves filter matches method to return a reason when the filter fails. This helps debug why an event doesn't enter a pipeline. Change-Id: Ic31a059813a54b7ec9fae66312923cdd7d1fd7de
* | | Merge "Centralize logging adapters"Zuul2019-05-161-0/+2
|\ \ \
| * | | Centralize logging adaptersTobias Henkel2019-05-121-0/+2
| | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Logging adapters are a great way to annotate log output in a common way. Those are alreay in use for adding build uuids in the executor and deliveries in parts of the github event handling. Before expanding the use of them centralize and refactor them so they can be used easily with low overhead. The goal will be to annotate most of the logs with an event id that initially caused this action. In the end tThis will help tracing errornous behavior through the entire system. Change-Id: Ifc19b906c4ff7410d632470388c2a571fce1fee8
View Lorry Controller Status