summaryrefslogtreecommitdiff
path: root/baserockimport
Commit message (Collapse)AuthorAgeFilesLines
* Allow import tool to run as non-rootRichard Ipsum2015-06-031-1/+2
| | | | | | | | | Morph runs extensions in a separate mount namespace by default, but constructing this namespace requires root privileges. The import tool has no need for a separate mount namespace, so this commit disables it. Change-Id: I952885860100453e4a531589b6acca7e5dc31435
* Import warn() and error() from utilsRichard Ipsum2015-06-031-0/+1
| | | | | | python.find_deps uses warn and error, which have moved into the utils module Change-Id: I7ce58c034cb83b7fc486487d6234282b650edfc1
* Don't put "build-depends: []" in strataRichard Ipsum2015-05-291-1/+4
| | | | Change-Id: I53d655e09726b57abb69e04264ef685f26519356
* Make the python ext use cachingRichard Ipsum2015-05-291-12/+14
| | | | | | | | This doesn't provide much performance benefit, since most of our queries go through xmlrpclib, but caching here does no harm. Change-Id: Id740a7ffab56defeddb3a6f3f481d81498a4411a
* Move WebServiceClient into ImporterBaseRichard Ipsum2015-05-292-34/+37
| | | | | | | | | | | | | | | | | | | To allow all extensions to use it, this also modifies the client so that the cache gets expired after 5 minutes by default. The error message returned by the rubygems extension on failure to get gem data will also be slightly more detailed, old message, ERROR: Request to http://rubygems.org/api/v1/gems/kittens.json failed: Not Found new message, ERROR: Request to http://rubygems.org/api/v1/gems/kittens.json failed: 404 Client Error: Not Found Change-Id: I63354fc7682bb01b1122007c1435bf35975db1aa
* Move common functions into a utils moduleRichard Ipsum2015-05-294-60/+88
| | | | | | These functions will also be useful for our forthcoming cpan extension Change-Id: I9df87dee09bbcf43dd0868f062fb873632f1f5ae
* Add check scriptRichard Ipsum2015-05-192-6/+10
| | | | | | | We switch to unittest.main() here so that the exit code gets set by unittest, which in turn sets check's exit code Change-Id: If9ace26ba5373cb78192b30922ee1bd0ea91f36f
* Fix str_repo_lorry for svn reposbaserock/richardipsum/import-lorry-fixRichard Ipsum2015-01-161-4/+7
| | | | | | Most svn repos use a standard layout, we assume svn repos we want to lorry are using a standard layout, so make str_repo_lorry add 'layout': 'standard' for svn repos
* Add more possible namesbaserock/richardipsum/add-more-possible-namesRichard Ipsum2015-01-141-5/+8
| | | | | | This fixes the tool so that it can find refs such as foobar-1.4.0 when the version has been passed as 1.4 and refs such as foobar-1.4 when the version has been passed 1.4.0
* Update tests for python.to_lorrybaserock/richardipsum/fix-lorry-bug-3Richard Ipsum2015-01-131-4/+7
|
* Make npm lorry ext work with new interfaceRichard Ipsum2015-01-131-5/+7
|
* Make rubygems lorry ext work with new interfaceRichard Ipsum2015-01-131-2/+3
|
* Move get_releases into common and make all extensions use itRichard Ipsum2015-01-132-2/+11
| | | | | | | We also call package_releases with True, so that we also get versions of releases that have been hidden. pip is willing to install from hidden releases so we should too, the concept of hidden releases will eventually disappear from pypi as well.
* Fix python.to_lorry to use version passed on cmdlineRichard Ipsum2015-01-131-33/+31
|
* Make import tool pass version to to_lorry extRichard Ipsum2015-01-121-5/+9
|
* Add debug msg stating requirementRichard Ipsum2015-01-121-0/+2
|
* Add option for debugging vcssRichard Ipsum2015-01-121-1/+4
| | | | | It's sometime useful to see the output of the vcs, but having this enabled all the time clutters the log
* Remove generic PythonExtension classRichard Ipsum2015-01-122-28/+25
| | | | The two extensions have diverged so this generic class is no longer useful.
* Fix bug in name_or_closestRichard Ipsum2015-01-121-0/+1
|
* Don't convert package names to lower caseRichard Ipsum2015-01-121-4/+4
| | | | This is redundant.
* Use comparison callback in python importerRichard Ipsum2015-01-121-1/+6
| | | | | | | As well as being case insensitive project names on python treat '-' and '_' as equivalent characters [1] [1]: http://legacy.python.org/dev/peps/pep-0426/#name
* Add package name comparison callbackRichard Ipsum2015-01-122-3/+11
| | | | | | This lets the importer specify how package names will be compared, to do this the importer optionally supplies a callback, if no callback is supplied then the tool will default to == for string comparison.
* Make python.to_lorry output x-products-python fieldRichard Ipsum2015-01-121-2/+9
|
* Read lorry prefix from yaml fileRichard Ipsum2015-01-121-30/+39
| | | | | | | | | | | | This adds PythonLorryExtension class to python.to_lorry to run the extension in a more conventional way. Previously the PythonExtension class was used to execute any of the extensions (it would call the extensions main() function). We move away from this so the extension can access useful methods, such as local_data_path(), that are provided by the ImportExtension class. This also removes use of pkg_resources.parse_requirement which is redundant. This also removes the unused import of select.
* Add python.yamlRichard Ipsum2015-01-121-0/+2
| | | | This adds a config file for the python extensions
* Add npm to_lorryZara Zaimeche2015-01-091-0/+51
|
* Add npm to_chunkZara Zaimeche2015-01-091-0/+72
|
* Add npm find_depsZara Zaimeche2015-01-091-0/+91
|
* Add helper functions for npm extensionsZara Zaimeche2015-01-091-0/+24
|
* Add npm subcommand to import tool via app.pyZara Zaimeche2015-01-091-0/+18
|
* fix app.py so that ruby doesn't always import from masterZara Zaimeche2015-01-071-1/+1
|
* Update copyrightsbaserock/richardipsum/fix-hyphen-bugRichard Ipsum2015-01-053-3/+6
|
* Treat hyphens and underscores equivalentlyRichard Ipsum2015-01-052-7/+22
| | | | | | | | | | | | | | There isn't yet an official spec for distribution names in python, however there is a draft at http://legacy.python.org/dev/peps/pep-0426/#name In particular, "All comparisons of distribution names MUST be case insensitive, and MUST consider hyphens and underscores to be equivalent." pkg_resource.parse_requirements will replace any underscores in the package name as hyphens, so when we search pypi we need to look for the package name with underscores as well as with hyphens.
* Add python.to_chunkRichard Ipsum2014-12-051-0/+33
| | | | | | Morph doesn't need a chunk morph to build/install setuptools packages, but the import tool needs to grow the ability to have 'to_chunk' as an optional stage before we can remove this part of the python extension.
* Add tests for requirement conflict detectionRichard Ipsum2014-12-051-0/+362
| | | | | python.find_deps does some pretty basic validation of the requirement specs, this commit adds the tests for this validation.
* Add python.find_depsRichard Ipsum2014-12-051-0/+352
| | | | | This takes source_dir, name, version and returns the dependencies for the package as json on stdout.
* Add tests for python.to_lorryRichard Ipsum2014-12-052-1/+73
|
* Add python.to_lorryRichard Ipsum2014-12-051-0/+219
| | | | This takes source, name, version and produces a lorry
* Add common functions used by python extensionsRichard Ipsum2014-12-051-0/+87
|
* Log when we receive a msg on stderrRichard Ipsum2014-12-051-0/+1
| | | | | | This is helpful for discovering when messages are being put on stderr, we're collecting messages on stderr, but these could come from different subprocesses leading to a confusing mixture of error messages.
* Add python subcommandRichard Ipsum2014-12-051-0/+19
|
* Make rubygems importer depend on strata/ruby.morphRichard Ipsum2014-12-051-2/+2
|
* Make stratum build-depends be set by importerRichard Ipsum2014-12-051-6/+11
|
* rubygems: Detect more signed GemsSam Thursfield2014-12-041-1/+1
| | | | | | | | The multi_json Gem wasn't being detected as signed, because the 'signing_key' field is an expression that can evaluate to 'nil' in some cases. In this Gem the 'cert_chain' field was still a standard string. Hopefully checking for the presence of either will catch all cases (and false positives should be harmless anyway).
* rubygems: Improve heuristic for when homepage_uri points to GithubSam Thursfield2014-12-031-3/+35
| | | | | This was motivated by <https://github.com/mislav/will_paginate>, which links to <https://github.com/mislav/will_paginate/wiki> as its homepage.
* Fix % interpolation crashSam Thursfield2014-12-031-1/+1
| | | | | | | | | | | | | | | | | | | | | | Previously if we got a BaserockImportException which contained a '%' in the message, you'd see this... Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/src/import/baserockimport/app.py", line 102, in process_args super(BaserockImportApplication, self).process_args(args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/src/import/baserockimport/app.py", line 185, in import_rubygems loop.run() File "/src/import/baserockimport/mainloop.py", line 176, in run self.app.status(str(e), error=True) File "/src/import/baserockimport/app.py", line 105, in status text = msg % args TypeError: not enough arguments for format string
* rubygems: Fix broken build-commands when .gemspec is in a subdirectorySam Thursfield2014-12-033-10/+20
| | | | | | The rubygems.to_chunk tool was assuming the .gemspec file always lived at the top of the chunk repo, but this isn't the case for <https://github.com/rails/rails>. Now it is smarter.
* Fix potential issue where two packages of different kinds share a nameSam Thursfield2014-12-012-18/+19
| | | | | | | This isn't a perfect fix. If this situation occurs the tool will generate an invalid stratum and the user will need to rename one of the chunks. But this is a better than what would have happened before: one of the chunks would have been silently ignored.
* Don't re-enqueue packages that already failedSam Thursfield2014-12-011-5/+12
|
* Give better errors when one .lorry duplicates another oneSam Thursfield2014-12-011-3/+19
| | | | | | | | | | | | | | | | | Now the tool gives the filenames of both files involved in the conflict. The most likely reason for seeing this error is that, at least in the lorries.git on git.baserock.org[1], the toplevel directory contains some 'disabled' subdirectories, so the user needs to point --lorries-dir to the correct subdirectory so that duplicate disabled lorries aren't loaded. This isn't really the final word on that problem, because having the tool load disabled lorries is quite bad and if there are no conflicts the user won't notice that it's happening and other confusing stuff might occur. But hopefully it's good enough for now. 1. http://git.baserock.org/cgi-bin/cgit.cgi/baserock/local-config/lorries.git/tree/
View Lorry Controller Status