How to use the Baserock Import Tool =================================== The tool helps you generate Baserock build instructions by importing metadata from a foreign packaging system. The process it follows is this: 1. Pick a package from the processing queue. 2. Find its source code, and generate a suitable .lorry file. 3. Make it available as a local Git repo. 4. Check out the commit corresponding to the requested version of the package. 5. Analyse the source tree and generate a suitable chunk .morph to build the requested package. 6. Analyse the source tree and generate a list of dependencies for the package. 7. Enqueue any new dependencies, and repeat. Once the queue is empty: 8. Generate a stratum .morph for the package(s) the user requested. The tool is not magic. It can be taught the conventions for each packaging system, but these will not work in all cases. When an import fails it will continue to the next package, so that the first run does as many imports as possible. For imports that could not be done automatically, you will need to write an appropriate .lorry or .morph file manually and rerun the tool. It will resume processing where it left off. It's possible to teach the code about more conventions, but it is only worthwhile to do that for common patterns. Package-system specific code and data ------------------------------------- For each supported package system, there should be an xxx.to_lorry program, an xxx.to_chunk program and a xxx.find_deps program. These should output on stdout a .lorry file, and a .morph file, and dependency information respectively. Each packaging system can have static data saved in a .yaml file, for known metadata that the programs cannot discover automatically. The following field should be honoured by most packaging systems: `known-source-uris`. It maps package name to source URI. Help with .lorry generation --------------------------- The quickest fix is to write a .lorry file by hand. If you write a .lorry file by hand, be sure to fill in the `x-products-YYY` field. 'x' means this field is an extension to the .lorry format. YYY is the name of the packaging system, e.g. 'rubygems'. It should contain a list of which packages this repository contains the source code for. Some import extensions have a .yaml file containing 'static metadata', where you can hardcode URIs for packages in the 'known-source-uris' dict. This is useful if you need to repeatedly import something (which you shouldn't need to do, unless you're developing the Import Tool). Help with linking package version to Git tag -------------------------------------------- Some projects do not tag releases. Currently, you must create a tag in the local checkout for the tool to continue. In future, the Lorry tool should be extended to handle creation of missing tags, so that they are propagated to the project Trove. The .lorry file would need to contain a dict mapping product version number to commit SHA1. If you are in a hurry, you can use the `--use-master-if-no-tag` option. Instead of an error, the tool will use whatever is the `master` ref of the component repo. Help with chunk .morph generation --------------------------------- Copy an existing one :-) For package-system specific information, see the relevant README file, e.g README.rubygems for RubyGem imports. Help with writing a .foreign-dependencies file ---------------------------------------------- If you need to write a .foreign-dependencies file, it's simply a JSON document containing two nested dicts. You should be able to copy an existing one. You may not need to create one. For example, if you get an error from the 'rubygems.find_deps' program because a package has an invalid Ruby .gemspec file, you could try fix the .gemspec file in the appropriate repo in the 'checkouts/' subdirectory, and rerun the Import Tool. If you ignore errors from the .find_deps program, you will have two problems. First, some dependencies might be missed. Second, a stratum won't be generated unless you use the `--force-stratum-generation` option. Common errors ------------- If you see the program showing this prompt and waiting for input: Username for 'https://github.com': This means that 'Lorry' has tried to pull an invalid Github URL. The .lorry entry for the current package lists the wrong URL. This may be due to bad metadata, an error in the .to_lorry program (check the log file), or a wrong URL in a .lorry file because someone wrote it wrong. The TODO file notes that if the .to_lorry program tried fetching the URL first and detecting if it returns a 404 'not found' error then we wouldn't hit this problem. More information ---------------- More information about the Import Tool can be found in the 'Guides' section of wiki.baserock.org - http://wiki.baserock.org/guides/import-tool/quickstart/ - http://wiki.baserock.org/guides/import-tool/rubygems/ - http://wiki.baserock.org/misc-docs/draft-rubygems-build-rules/ - http://wiki.baserock.org/guides/import-tool/npm/ - http://wiki.baserock.org/guides/import-tool/pypi/