summaryrefslogtreecommitdiff
path: root/tests/fakegithub.py
Commit message (Collapse)AuthorAgeFilesLines
* Fix bug in getting changed filesDong Zhang2022-04-251-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | The fix including 2 parts: 1. For Gtihub, we use the base_sha instead of target branch to be passed as "tosha" parameter to get precise changed files 2. In method getFilesChanges(), use diff() result to filter out those files that changed and reverted between commits. The reason we do not direcly use diff() is that for those drivers other than github, the "base_sha" is not available yet, using diff() may include unexpected files when target branch has diverged from the feature branch. This solution works for 99.9% of the caseses, it may still get incorrect list of changed files in following corner case: 1. In non-github connection, whose base_sha is not implented, and 2. Files changed and reverted between commits in the change, and 3. The same file has also diverged in target branch. The above corner case can be fixed by making base_sha available in other drivers. Change-Id: Ifae7018a8078c16f2caf759ae675648d8b33c538
* Include original path of renamed file for a PRSimon Westphahl2022-03-091-3/+14
| | | | | | | | | When a file is moved/renamed Github will only return an entry for the file with the new name. However, the previous path also needs to be included in the list of files. This is especially important when a Zuul config file is renamed but also when `job.[irrelevant-]files` is used. Change-Id: Ieba250bed57c8a9c2e7811476c202d530f2b30f1
* github: more complete mocking for app setupIan Wienand2021-09-231-1/+8
| | | | | | | | | | | | This mocks the /app/installations and /installations/repositories GitHub API calls to better validate the GitHub project initalization in the driver. It implements enough that we can use GithubClientManager:_prime_installation_map() directly and better tests the token issuing, etc. Change-Id: I608f1540ef33b1a95595393e546afba308fef66a
* Save superfluous api requests in check run reportingTobias Henkel2020-11-041-0/+30
| | | | | | | | | | | | | | | | | | | | | When reporting the check run result zuul is currently doing four github api requests: 1. Get repository 2. Get head commit of pr 3. Get check runs of head commit 4. (optional) Create initial version of check run in case start reporting is disabled 5. Update check run that has been created in start reporting The first three requests are basically unneeded if we craft a direct PATCH request and remember the check run id that has been created in the start reporting. This is needed since those run in the critical path of the event processing and can cause large event processing delays in a very busy zuul system. Change-Id: I55df1cc28279bb6923e51686dde8809421486c6a
* Remove an unneeded api call when creating check_runsTobias Henkel2020-11-041-5/+48
| | | | | | | | | | | | | | | | During the start reporting the check run is being created in status in_progress. This currently does two api requests due to the github3.py object model. The first one is to get the repository object, the second one to actually create the check run. However we already have all information available that is required to directly craft a post request to create that check run. This is a problem because this start reporting runs within the main loop of zuul and needs to be as fast as possible. In a very busy zuul api requests are becoming a bottleneck so we need to remove any request that is unneeded. Change-Id: Idd166c61a4d0081c284dee5677b4cc2e48932307
* Restore correct reason reporting of merge failuresTobias Henkel2020-10-051-22/+12
| | | | | | | | | | | | The optimization of merges in GitHub [1] has a side effect that the merge failure reason always is 'Method not allowed'. This has been unnoticed because the test framework did not distinguish between status message and response body. GitHub reports such failures with 405 Method not allowed and a detailed reason in the response body. [1] I48dbb5fc1b7e7dec4cb09c779b9661d82935d9df Change-Id: Ic2e47ee14c7f97defba0905db68ef8706108c081
* Merge "Optimize GitHub requests on PR merge"Zuul2020-10-021-18/+25
|\
| * Optimize GitHub requests on PR mergeSimon Westphahl2020-09-301-18/+25
| | | | | | | | | | | | | | | | | | | | | | | | Using github3py for merging a pull request there is always one extra request to get the pull request before the merge can be done due to the github3py data model. Since merging is done synchronously in in the scheduler main loop, it makes sense to save the extra API request by building and executing the merge call directly. Change-Id: I48dbb5fc1b7e7dec4cb09c779b9661d82935d9df
* | Merge "Fix multiple prs found when commit is not head"Zuul2020-09-281-3/+18
|\ \
| * | Fix multiple prs found when commit is not headTobias Henkel2020-09-041-3/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When receiving a status or checks event we need to find the corresponding pull request to that event. Github stores the status on the head commit of a pr and not on the pr itself which leads to problems if multiple prs exist with the same head but different target branches. Therefore zuul errors out when more than one pr is found for that event. However the github search also returns all prs that contain the sha we search for but have a different head which leads to the same error as if we had two conflicting prs. This can be solved by delaying the error and filtering the pr bodies for the head sha first. Change-Id: Iadafd3cf68a742941e9189e84fca594bc3394084
* | | Merge "Fetch can-merge info when updating a pull-request"Zuul2020-09-251-4/+23
|\ \ \
| * | | Fetch can-merge info when updating a pull-requestSimon Westphahl2020-09-231-4/+23
| | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of synchronously checking via the GitHub API if a pull request is mergeable, we can already get the necessary information during Github event processing so the canMerge() check operates on cached data similar to how this is done for the Gerrit driver already. With this we can also remove the additional API requests for the commit status and check runs since this info can be retrieved via the same GraphQL request. Change-Id: I6b4848dcb5d27071deeb6a59642c0e3c03a799da
* | | Add commit id and owner to Change for mqtt reportervass2020-09-221-0/+3
|/ / | | | | | | Change-Id: I3d197996597c70165edbf2e26f621391c87873e6
* | Handle review requirements in canMergeTobias Henkel2020-09-031-2/+15
| | | | | | | | | | | | | | | | | | | | | | Since GitHub 2.21 we now can query the reviewDecision flag on a PR which tells us if a review is required and if it's approved or not [1]. This can be leveraged in the canMerge check so we now finally can accurately check if a change is allowed to enter the gate. [1] https://developer.github.com/enterprise/2.21/v4/object/pullrequest/ Change-Id: I792a28b9f3c7d40ac21e22438bd7c09d3174cbb2
* | Exercise github auth handling in testsTobias Henkel2020-09-031-4/+39
| | | | | | | | | | | | | | | | | | | | So far almost all of the github auth handling has been overridden in the fake test classes. We can increase the test coverage and gain more confidence when doing changes related to this by not replacing getGithubClient. Instead call the super method and enhance the resulting client with the relevant test data. Change-Id: Icaa9cfb9f28f8faec2dc86f7e6393f5b23765cac
* | Remove unneeded api requests when commenting in githubTobias Henkel2020-08-311-0/+12
| | | | | | | | | | | | | | | | | | | | Currently when commenting on a pr in github we first get the repo object and then the issue object. After that we call create_comment on this object. This issues two unnecessary api calls to get project and issue. Instead directly build the correct url and do the post manually to comment. Change-Id: I9afb19e4918109e98882aa9d5b716d34c65936f2
* | Move reports from FakeGithubConnection to github dataTobias Henkel2020-08-311-1/+3
|/ | | | | | | | | | | | This is needed to handle comment pull as direct post request in the followup. Further the reports are better located in the github data since this needs to be global data. Further this will be needed as preparation for the HA scheduler as well since there will be more than one connection and the test data needs to be centralized then. Change-Id: I3fa96d43f0ffe62d50630856b23972221bf2c69a
* GitHub Reporter: Fix User Email in Merge Commit MessageBenjamin Schanzel2020-08-261-2/+2
| | | | | | | | | | | | | | | | | | Re-introduce change https://review.opendev.org/#/c/738590/ There was a bug in fetching a GitHub user while loading pull requests. `getUser` was given a project object instead of a project name string, so no installation id was found for GitHub app authentication. That lead to unauthenticated requests to the user API which resulted in HTTP 401 errors. This change fixes this by passing the correct `project.name` to `getUser` and also refactors the parameter names of the involved methods from `project` to `project_name` to be more precise there. This reverts commit 56f355ab72aab2716e2f8b580a242a2a824ea5a2. Change-Id: Ib25f0ef5d27115626792890d0ec282ed0d562485
* Dequeue changes via github checks APIFelix Edel2020-08-061-6/+33
| | | | | | | | | | | | | | | | The Github checks API allows us to define custom actions on a check run. With that, we can define a custom "abort" action, that enables users to directly abort a running change by clicking on a respective button in Github's checks tab. A click on this button will emit a Github webhook event that is now handleded by Zuul to dequeue the respective change. To uniquely identify a change represented by a check run, each check run now comes with an external ID which is a combination of tenant, pipeline, queue and PR number. Change-Id: I574479cd06abd1ea4f3daa7850cf35ea6715a23d
* Fix github branch protection while already unprotectedGuillaume Chauvel2020-07-291-2/+3
| | | | | | It was causing the branch to be protected, tests were failing Change-Id: Idb7b3957159684787fe272181eee92af1abe0da1
* github: use change.message in squahsed commit messageGonéri Le Bouder2020-07-101-0/+1
| | | | | | | | So far, github use the subject of the first change in the commit message body. This is most of the time redundant with the PR title. Ths idea is to use instead the change body. Change-Id: I9a2bc8aa3ff60a28b2ed62a4ee5fa9f7a2c9dcda
* Simplify FakeGithubClient and FakeGithubSessionClark Boylan2020-04-281-6/+5
| | | | | | | | | | | | | These two fakes are related to each other. Before this change FakeGithubClient would create a FakeGithubSession and FakeGithubSession would create FakeGithubClients. THis recursion is confusing and unnecessary. Under normal operation the FakeGithubClient exists first so we pass it into its FakeGithubSessions and the fake session is able to use this client object rather than recursively creating a new client. Change-Id: I9ff52061d62e844ca3f2185ea752ca39962b2a9f
* Merge "Improve error reporting when pr merge fails"Zuul2020-03-061-10/+28
|\
| * Improve error reporting when pr merge failsTobias Henkel2020-02-231-10/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | When merging a pull request in GitHub fails we currently report the standard message when a merge confict happened. This is actually rarely the case because when using proper gating a merge should almost never fail because of a merge conflict. Instead report the error we get from GitHub so the user knows why the merge failed. In most cases this is because branch protection refuses to merge because of a condition zuul doesn't know about. This will then be reported to the user so he can do something about it without asking a zuul admin for the internal logs. Change-Id: I1c3e82a2ac20e2177352b1a4d078839d3e114467
* | Merge "Optimize canMerge using graphql"Zuul2020-03-061-0/+18
|\ \
| * | Optimize canMerge using graphqlTobias Henkel2020-02-281-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The canMerge check is executed whenever zuul tests if a change can enter a gate pipeline. This is part of the critical path in the event handling of the scheduler and therefore must be as fast as possible. Currently this takes five requests for doing its work and also transfers large amounts of data that is unneeded: * get pull request * get branch protection settings * get commits * get status of latest commit * get check runs of latest commit Especially when Github is busy this can slow down zuul's event processing considerably. This can be optimized using graphql to only query the data we need with a single request. This reduces requests and load on Github and speeds up event processing in the scheduler. Since this is the first usage of graphql this also sets up needed testing infrastructure using graphene to mock the github api with real test data. Change-Id: I77be4f16cf7eb5c8035ce0312f792f4e8d4c3e10
* | | Merge "Refactor branch protection test infrastructure"Zuul2020-03-021-13/+32
|\ \ \ | |/ /
| * | Refactor branch protection test infrastructureTobias Henkel2020-02-251-13/+32
| |/ | | | | | | | | | | | | Branch protection rules are not just about required status so refactor it in the test framework so we can extend them in the future. Change-Id: I1e32ebb57b27bce898b074e419299a1803853608
* | Support file comments via Github checks APIFelix Edel2020-02-191-1/+2
|/ | | | | | | | Zuul is already providing these file comments via the zuul_return value. So far, the Github reporter wasn't able to use those, but with the help of the checks API we can add so called "annotations" to each check run. Change-Id: Iff10172f95dc0430bec8e4dafb9a6c09bbe06077
* Implement basic github checks API workflowFelix Edel2020-02-191-0/+99
| | | | | | | | | | | | | | | | Utilizing the checks API to report the build state to Github provides some additional functionality that is not supported by the status API. Those are: - Defining custom actions to e.g. cancel a running build - Report line-based file annotations This change implements the basic checks API workflow. Once this is in place, the additional features could simply be added on top. Change-Id: I7e790783ee35971085863b5487ff094fa0b23d65 Story: 2007268 Task: 38672
* Handle draft pull requests in canMergeTobias Henkel2020-01-281-0/+3
| | | | | | | | | | Github now supports draft pull requests which are blocked from merging by github. Currently those can end up in a gate loop if all requirements for the gate pipeline are fulfilled except for un-drafting it. Zuul needs to handle those in the canMerge check in order to handle them correctly. Change-Id: Ie9164ad5127d19ca3d75660b9718979da7cc344c
* Fix occasionally wrong change url with githubTobias Henkel2020-01-141-1/+4
| | | | | | | | | | | | | | | | Github returns different urls for pull requests. One for api use and one for browser use. The change url we're interested in is the one for browser usage. However zuul takes the wrong one from the pull request data structure but most of the time overwrites the change url generated from event metadata. This leads to the issue that the change url is sometimes not correct. This can be fixed by taking the html_url from the pull request data structure. Further due to the fact that the url is always set on a pr and stays static throughout the whole lifecycle of the pr, be safe and just don't overwrite the url from event metadata. Change-Id: I2030b9dddd5bc618231b73d73ae64e2552231769
* Look for depends-on lines in dependency searchesClark Boylan2019-12-061-8/+16
| | | | | | | | | | | | | | Prior to this change we looked for the current change/PR's url in any other change/PR's message body. This meant any cross referencing of urls would create further lookups to determine if there was a real dependency there. Restrict this a bit more to require the Depends-On string too when searching to limit the number of spidering queries that must be done. This is particularly useful for the github driver because queries are expensive there and may be rate limited. Change-Id: Ie49fe1a72dc844b14003d942684fd3d2a9478d21
* Fix issue search in FakeGithubClientSimon Westphahl2019-11-151-20/+17
| | | | | | | | | | | | | | | | Mocked issue search was broken in multiple ways: - tokenize() was wrongly splitting search modifieres (e.g. type:pr) into tokens (e.g. [type, pr]) - code for removing search modifiers (type, is, in) from terms was never reached and also used set(...) in a wrong way - set intersection of search terms and body doesn't make any sense since this will almost always have ANY overlap Simply extracting the URLs from the query and checking for in PR body should make the mock work for most of the tests. Change-Id: I9f896af85e21770bba80857511aae8505f3e5b84
* Eliminate two github requests per _updateChangeTobias Henkel2019-06-171-0/+35
| | | | | | | | | | | | | When processing _updateChange we fetch the statuses of the commit. Due the object model of Github3.py we currently get the repository (request 1), then the commit (request 2) and finally the commit statuses (request 3). However we already have all information needed to directly perform the request 3. Further we're in any case only using the json structure of the statuses so we can easily do a direct request and avoid the first two. This essentially saves us two api requests per incoming events. Change-Id: I4185e15dcaf2b66bc1e6629e725dfd508368140a
* Switch getPullBySha to using the search apiTobias Henkel2019-06-171-1/+13
| | | | | | | | | | | | | | | | | | | We currently cache the sha to pull request mapping locally in order to be able to map status events to pull requests. This cache is not effective in some cases. For instance on merge a pull request changes its sha so it's not known by the cache. Some projects like ansible run post merge jobs on a pr which in turn emit status events for the now unknown sha's. This invalidates the cache and we need to sweep through all pull requests of the project to find the correct one. This can better be solved by using the search api. However the search api is more restricted and allows 30 requests per minute per installation. Based on local testing it's hard but not impossible to reach the limit so we need to handle it gracefully. Luckily we get all information we need to handle it when we reached the rate limit. Change-Id: Idb6c64caa9f948111a58135754174d898ca1528c
* Merge "Add support for submitting reviews on GitHub"Zuul2019-05-161-0/+23
|\
| * Add support for submitting reviews on GitHubClint Byrum2019-04-301-0/+23
| | | | | | | | | | | | | | | | | | GitHub has added a lot of controls around the review object, so it is useful to be able to run tests and then submit a review rather than simply merge. One use-case is also to be able to self-approve with a comment, such as is done in the test code added. Change-Id: I16872062e627b385f78023878bea348555ec5348
* | Use user.html_url for github reporter messagesPaul Belanger2019-04-231-0/+1
|/ | | | | | | | | There is no need to dynamically generate the github users url with zuul, we can rely on html_url from github. This fixes an issue where we constructed the wrong url for the merge commit when using a github app. Change-Id: I65028be07636aeaad48ccb6de80b3e2a0d29b436 Signed-off-by: Paul Belanger <pabelanger@redhat.com>
* Merge "Switch to LRU based sha to PR cache"Zuul2019-02-191-1/+3
|\
| * Switch to LRU based sha to PR cacheClark Boylan2019-02-191-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | This updates the github driver to cache PRs by sha using cachetool's LRUCache. We make this change because we need to cache closed PRs so can't rely on the action of closing a PR to remove the PR from the cache. Since we don't have a good method of evicting entries we fall back to LRU with a reasonable cache size (2k commits). Change-Id: I5fb6c8b33f9eed221a8b84e537f02e7dccf2d2df
* | Don't request PR issue dataClark Boylan2019-02-181-9/+1
|/ | | | | | | | | | The github api docs show that labels is now part of the PR dict returned in pulls API responses. github3 doesn't directly expose this as an attribute but we can use the as_dict representation to access any json field. Use this field on the PR dict repr to access the labels of a PR and stop fetching the issue of a PR for that info. Change-Id: Ib14bb387dbd233ff252cf57be7f0517770ade037
* Re-use the github PR object when fetching reviewsJames E. Blair2019-02-151-0/+3
| | | | | | | | | We inadvertently fetch the PR object twice because of the way we were fetching reviews. Keep it around so we can make one less API call. Co-Authored-By: Clark Boylan <cboylan@sapwetik.org> Change-Id: If5260278adb525566d99eedaecaf8b4f5077d43e
* Use combined status for Github status checksTobias Henkel2018-12-071-0/+19
| | | | | | | | | | | | When checking the required status checks we currently get all statuses and get the successful of them. However Github returns all historic status changes there. So a change that get a successful check, then a recheck and then a failed check still enters the gate but will be prohibited by Github to merge in the end. However github also offers us a combined status call that only returns the current state of the statuses. Using this fixes the issue. Change-Id: Iec3b2a3dfc8626870381604badd40de71e7257b9
* Retry queries for commitsTobias Henkel2018-11-221-0/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a pull_request event arrives we also get the commit statuses which are attached to the commit object. Sometimes the commit is not yet queryable and we get a 404 [1]. In this case we need to retry the query. In the future we probably want to use that as a feedback mechanism of a dynamic event delay. [1] Trace 2018-06-13 11:49:11,881 ERROR zuul.GithubEventConnector: Exception moving GitHub event: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/zuul/driver/github/githubconnection.py", line 391, in run self._handleEvent() File "/usr/lib/python3.6/site-packages/zuul/driver/github/githubconnection.py", line 226, in _handleEvent refresh=True) File "/usr/lib/python3.6/site-packages/zuul/driver/github/githubconnection.py", line 751, in _getChange self._updateChange(change) File "/usr/lib/python3.6/site-packages/zuul/driver/github/githubconnection.py", line 837, in _updateChange change.patchset) File "/usr/lib/python3.6/site-packages/zuul/driver/github/githubconnection.py", line 1226, in _get_statuses for status in self.getCommitStatuses(project.name, sha): File "/usr/lib/python3.6/site-packages/zuul/driver/github/githubconnection.py", line 1171, in getCommitStatuses commit = repository.commit(sha) File "/usr/lib/python3.6/site-packages/github3/repos/repo.py", line 342, in commit json = self._json(self._get(url), 200) File "/usr/lib/python3.6/site-packages/github3/models.py", line 156, in _json raise exceptions.error_for(response) github3.exceptions.NotFoundError: 404 Not Found Change-Id: I5cc6d95ed18148657d9e5a8d132a8e43f385771d
* Fix manual dequeue of github itemsTobias Henkel2018-11-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | Currently dequeuing a github item fails with [1]. This is caused because the DequeueEvent has no change_url attribute. In order to fix this we need to add a check and add the correct change_url if it doesn't exist. This by accident fixes a second issue where manually enqueued github items miss the change_url. This adds a fix and test case for both use cases. [1] Traceback: 2018-11-21 14:17:56,592 ERROR zuul.Scheduler: Exception in management event: Traceback (most recent call last): File "/opt/zuul/lib/python3.6/site-packages/zuul/scheduler.py", line 1096, in process_management_queue self._doDequeueEvent(event) File "/opt/zuul/lib/python3.6/site-packages/zuul/scheduler.py", line 917, in _doDequeueEvent change = project.source.getChange(event, project) File "/opt/zuul/lib/python3.6/site-packages/zuul/driver/github/githubsource.py", line 66, in getChange return self.connection.getChange(event, refresh) File "/opt/zuul/lib/python3.6/site-packages/zuul/driver/github/githubconnection.py", line 789, in getChange change.url = event.change_url AttributeError: 'DequeueEvent' object has no attribute 'change_url' Change-Id: Ifbaa67dc06c671f9235545a3215aa337ab27697e Story: 2003747 Task: 26432
* Use merger to get list of files for pull-requestSimon Westphahl2018-10-091-2/+5
| | | | | | | | | | | | | | The Github pull requests files API only returns at max the first 300 changed files of a PR in alphabetical order. In case the PR has more than 300 changed files, it might be that Zuul is not considering changes to the config and not using them speculatively due to the incomplete file list. Also jobs using file filters might not be considered in case the relevant files are not in the list of files for a change. Change-Id: I10a593e26ac85b8c12ca9c82051cad809382f50a
* Merge "Reduce number of reconfigurations on branch delete"Zuul2018-07-261-0/+3
|\
| * Reduce number of reconfigurations on branch deleteMarkus Hosch2018-07-161-0/+3
| | | | | | | | | | | | | | | | | | | | If a branch is deleted that didn't contain configuration files or was disregarded due to its protection state, we do not need to trigger a tenant reconfiguration. This patch checks whether there is a cached configuration and suppresses reconfiguration on branch deletion if there is no configuration to be removed for this branch. Change-Id: I34f5d3631d8112c792a72146268dad85813ceb4f
* | Merge "github: prevent AttributeError when missing branch protection"Zuul2018-07-111-0/+4
|\ \
View Lorry Controller Status