summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* morphsetondisk: removal of unset_defaults functionHEADmasterChristopher Phang2016-09-261-1/+0
| | | | | | | | unset_defaults were still used in save_morphology despite being deprecated in commit 7d5804051531147298085d4db1b81ffd4b3c10c6 in morph. This was found through use of baserock-import. Change-Id: Ic6a58a4515cebf1fe6c47bd43caf6cf1d18d0d2c
* Disable chunk morph generation in python importerRichard Ipsum2015-10-172-33/+1
| | | | | | These chunk morphs are redundant since the build tools have them already. Change-Id: Ibf8dbaf5c19bb77993ea2590f2ac5ff8e58557da
* Add build system detectionRichard Ipsum2015-10-172-0/+58
| | | | Change-Id: Iaa1572cf3d5fd7baf96d2084f7194953bede634c
* Fix morphsetondisk: read_morphology is now read_fileRichard Ipsum2015-10-161-1/+1
| | | | Change-Id: I55f6240c9af720103afb8e91629a2a46cf8d3796
* Fix no git dir err in cpan extRichard Ipsum2015-10-161-3/+7
| | | | | | | | | | If no definitions dir exists then we incorrectly use mkdir() alone to create a new directory, morphset later fails because the directory isn't a repository. This fixes it so that we git init the definitions dir after the mkdir Change-Id: I1dc665d2c51cdc7afac45db14e4d2e699d9cf1ec
* Add zip support to python.to_lorry extRichard Ipsum2015-08-262-18/+37
| | | | Change-Id: I04ed11272c7ddfb22829c71607772b86f216d457
* Use virtualenv and Pip to find runtime deps; remove searching for upstreamRichard Ipsum2015-08-266-211/+173
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Co-authored-by: Sam Thursfield <sam.thursfield@codethink.co.uk> Since upstream pip do not want to merge https://github.com/pypa/pip/pull/2371 we should avoid depending on this pull request. To find runtime dependencies we now run pip install inside a virtual env then run pip freeze to obtain the dependency set, this has the advantage that nearly all the work is being done by pip. Originally the python extensions were designed to look for upstream git repos, in practice this is unreliable and won't be compatible with obtaining dependencies using pip install, so the downside of this approach is that all lorries will be tarballs, the upshot is that we can now automatically import many packages that we couldn't import before. Another upshot of this approach is that we may be able to consider the removal of a lot of the spec processing and validation code if we're willing to worry less about build dependencies, we're not sure whether we should be willing to worry less about build dependencies though. We've had encouraging results using this patch so far, we are now able to import, without user intervention, packages that failed previously, such as boto, persistent-pineapple, jmespath, coverage, requests also almost imported successfully but appears to require a release of pytest that is uploaded as a zip. Change-Id: I705c6f6bd722df041d17630287382f851008e97a
* Add README.cpanRichard Ipsum2015-08-251-0/+42
| | | | Change-Id: I797bd50753742f05b5f78faef1f4facd48965d34
* Add cpan test scriptRichard Ipsum2015-08-251-0/+86
| | | | | | | | | This script is helpful for ensuring regressions haven't been introduced, naturally it's relying on CPAN which can change over time, but it's still useful to be able to check that the modules you could import before applying a patch can still be imported afterwards. Change-Id: I58ccadf08a3ad58a1c4aa8d3c730ced5e39426de
* Add cpan command to appRichard Ipsum2015-08-251-1/+161
| | | | Change-Id: I13ad34e8b1bb255f2d644088b32ffcf2ba9e0a27
* Add cpan.find_deps extensionRichard Ipsum2015-08-251-0/+696
| | | | Change-Id: I4719d7a15ef133d0dcd72061ac7e612723b828df
* Add cpan.to_lorry extensionRichard Ipsum2015-08-251-0/+194
| | | | Change-Id: I3e8077d1e91a28ac0ed30cb0e8102622c866a8e0
* Add perl importer baseRichard Ipsum2015-08-251-0/+70
| | | | Change-Id: Ieb1a0d2047746e68db0e494d9dc73fe5aae782a3
* Add pathname2distinfo utilRichard Ipsum2015-08-251-0/+46
| | | | Change-Id: I89ea4bbb26cff27a6a2dcb71e3101e52b8f91002
* Clone source repo into tempdirRichard Ipsum2015-08-251-30/+33
| | | | | | | | | | | Extensions will now be given a path to a clone of the checkout rather than a path to the checkout itself. This prevents untracked modifications and protects any data the user might have in the checkout dirs from the extensions themselves. It also helps prevent a possible bug where the extension corrupts a checkout so that on the second import the result differs from the first import. Change-Id: I95d551a02aa77d665ba07fb7ed5b01be330e24d4
* Add an ignore_version_field option to PackageRichard Ipsum2015-08-252-6/+14
| | | | | | | If we're using an external tool to resolve dependencies then we may be want to ignore the version field. Change-Id: Ic41f5b5ac080a5e9a0b4e1b0607464faabb1ec94
* Let parent packages pass metadata to their childrenRichard Ipsum2015-08-251-14/+47
| | | | | | | | | With this packages can pass metadata to their dependencies (children) by putting the path to the parent's foreign-dependencies file into the child's environment. This provides a flexible means for extensions to pass around extra data where necessary. Change-Id: I6446d02abd1a8d703c8ff88fe12b92345a0ee461
* Rework tag searchRichard Ipsum2015-08-251-12/+13
| | | | | | This should make it easier to search through a wider range of tag formats. Change-Id: Ie124a054d55b4a8800f11479e0389c16c97c53a8
* Rename lambda parametersRichard Ipsum2015-08-251-6/+8
| | | | Change-Id: I6ee07010fad5164c9a3a3d40f0b951a6c9d676f1
* Show errors encountered during import at the endRichard Ipsum2015-08-251-0/+5
| | | | Change-Id: I86d38e0116fc8b95dcd20c1d52b1b801cf16466c
* Simplify accessors in packageRichard Ipsum2015-08-242-18/+18
| | | | Change-Id: I38dc72c2348883abea56bd58a3195df4dc197c8a
* Make to_chunk stage optionalRichard Ipsum2015-08-242-48/+50
| | | | | | | | | Most CPAN distributions can be installed with a standard set of commands so we don't want to generate chunk morphs for each chunk. This also fixes stratum build dependency generation Change-Id: Ica51721b9b3809f91b8016c959202694d4bc6ea7
* python.find_deps: On failure, show stdout and stderr output from PipSam Thursfield2015-07-131-2/+5
| | | | | | | | | | | | | | | | | | | | | | It's nice to have some idea right away of what went wrong. A common error is running the import tool on a system without the patched version of Pip. This now produces the following sort of message: baserockimport/exts/python.find_deps: Failed to get runtime dependencies for numpy at checkouts/python_numpy-tarball/. Output from Pip: Usage: pip install [options] <requirement specifier> [package-index-options] ... pip install [options] -r <requirements file> [package-index-options] ... pip install [options] [-e] <vcs project url> ... pip install [options] [-e] <local project path> ... pip install [options] <archive url/path> ... no such option: --list-dependencies Change-Id: I902920b46b29b1e2b18736b2ff2b5f4f4cdb42df
* Clean up data so url is validZara Zaimeche2015-06-101-1/+6
| | | | | | | | | Some people now put urls in the registry with '+git' at the start. The tool doesn't recognise this, but these urls still end in '.git', so this change just checks for such urls and cuts out the '+git'. The tool finds the repos once more. Panic over. Change-Id: I4e807797ed914fa1dbe96cc7e05263228785c86d
* Fix: honour version flag in npm.to_lorry extensionZara Zaimeche2015-06-091-1/+8
| | | | | | | | | | | | | | | This fixes a version bug in npm.to_lorry that meant the most recent db entry was used for repo location, regardless of the version specified. The npm registry is a database of all versions of all packages, listing names, source locations, etc. There was a fun bug in npm.to_lorry that meant that the import tool would try access the repo listed in the database entry for the latest version of the package, rather than for the version specified. This meant that if version 1.0.1 of a package had the right url listed, but something was wrong with the url listed for version 1.4.6, the tool would error out, even if version 1.0.1 was requested. Change-Id: If40c8b4c85f5e27fee07ee88daa1e5d2d347944f
* Don't put version in error message if it's NoneRichard Ipsum2015-06-031-2/+2
| | | | Change-Id: Id7b0713263f3d68773f69bc4c2dadec414e1d902
* Allow import tool to run as non-rootRichard Ipsum2015-06-031-1/+2
| | | | | | | | | Morph runs extensions in a separate mount namespace by default, but constructing this namespace requires root privileges. The import tool has no need for a separate mount namespace, so this commit disables it. Change-Id: I952885860100453e4a531589b6acca7e5dc31435
* Import warn() and error() from utilsRichard Ipsum2015-06-031-0/+1
| | | | | | python.find_deps uses warn and error, which have moved into the utils module Change-Id: I7ce58c034cb83b7fc486487d6234282b650edfc1
* Don't put "build-depends: []" in strataRichard Ipsum2015-05-291-1/+4
| | | | Change-Id: I53d655e09726b57abb69e04264ef685f26519356
* Make the python ext use cachingRichard Ipsum2015-05-291-12/+14
| | | | | | | | This doesn't provide much performance benefit, since most of our queries go through xmlrpclib, but caching here does no harm. Change-Id: Id740a7ffab56defeddb3a6f3f481d81498a4411a
* Move WebServiceClient into ImporterBaseRichard Ipsum2015-05-292-34/+37
| | | | | | | | | | | | | | | | | | | To allow all extensions to use it, this also modifies the client so that the cache gets expired after 5 minutes by default. The error message returned by the rubygems extension on failure to get gem data will also be slightly more detailed, old message, ERROR: Request to http://rubygems.org/api/v1/gems/kittens.json failed: Not Found new message, ERROR: Request to http://rubygems.org/api/v1/gems/kittens.json failed: 404 Client Error: Not Found Change-Id: I63354fc7682bb01b1122007c1435bf35975db1aa
* Move common functions into a utils moduleRichard Ipsum2015-05-294-60/+88
| | | | | | These functions will also be useful for our forthcoming cpan extension Change-Id: I9df87dee09bbcf43dd0868f062fb873632f1f5ae
* Add check scriptRichard Ipsum2015-05-193-6/+37
| | | | | | | We switch to unittest.main() here so that the exit code gets set by unittest, which in turn sets check's exit code Change-Id: If9ace26ba5373cb78192b30922ee1bd0ea91f36f
* Add .gitreview fileRichard Ipsum2015-05-181-0/+5
| | | | Change-Id: I3a085b089b5f7415c0cea0e6b6acc9e28b4deab0
* Merge branch 'petefoth/readme'Sam Thursfield2015-03-161-0/+13
|\ | | | | | | | | Reviewed-By: Javier Jardón <javier.jardon@codethink.co.uk> Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
| * Add link to Import Tool wiki pagesPete Fotheringham2015-03-161-0/+13
| |
* | Fix str_repo_lorry for svn reposbaserock/richardipsum/import-lorry-fixRichard Ipsum2015-01-161-4/+7
| | | | | | | | | | | | Most svn repos use a standard layout, we assume svn repos we want to lorry are using a standard layout, so make str_repo_lorry add 'layout': 'standard' for svn repos
* | Add more possible namesbaserock/richardipsum/add-more-possible-namesRichard Ipsum2015-01-141-5/+8
| | | | | | | | | | | | This fixes the tool so that it can find refs such as foobar-1.4.0 when the version has been passed as 1.4 and refs such as foobar-1.4 when the version has been passed 1.4.0
* | Merge branch 'baserock/richardipsum/fix-lorry-bug-3'Richard Ipsum2015-01-1410-98/+149
|\ \ | |/ |/| | | | | Reviewed by: Sam Thursfield <sam.thursfield@codethink.co.uk>
| * Update tests for python.to_lorrybaserock/richardipsum/fix-lorry-bug-3Richard Ipsum2015-01-131-4/+7
| |
| * Make npm lorry ext work with new interfaceRichard Ipsum2015-01-131-5/+7
| |
| * Make rubygems lorry ext work with new interfaceRichard Ipsum2015-01-131-2/+3
| |
| * Move get_releases into common and make all extensions use itRichard Ipsum2015-01-132-2/+11
| | | | | | | | | | | | | | We also call package_releases with True, so that we also get versions of releases that have been hidden. pip is willing to install from hidden releases so we should too, the concept of hidden releases will eventually disappear from pypi as well.
| * Fix python.to_lorry to use version passed on cmdlineRichard Ipsum2015-01-131-33/+31
| |
| * Make import tool pass version to to_lorry extRichard Ipsum2015-01-121-5/+9
| |
| * Add debug msg stating requirementRichard Ipsum2015-01-121-0/+2
| |
| * Add option for debugging vcssRichard Ipsum2015-01-121-1/+4
| | | | | | | | | | It's sometime useful to see the output of the vcs, but having this enabled all the time clutters the log
| * Remove generic PythonExtension classRichard Ipsum2015-01-122-28/+25
| | | | | | | | The two extensions have diverged so this generic class is no longer useful.
| * Fix bug in name_or_closestRichard Ipsum2015-01-121-0/+1
| |
| * Don't convert package names to lower caseRichard Ipsum2015-01-121-4/+4
| | | | | | | | This is redundant.