1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
|
How to use the Baserock Import Tool
===================================
The tool helps you generate Baserock build instructions by importing metadata
from a foreign packaging system.
The process it follows is this:
1. Pick a package from the processing queue.
2. Find its source code, and generate a suitable .lorry file.
3. Make it available as a local Git repo.
4. Check out the commit corresponding to the requested version of the package.
5. Analyse the source tree and generate a suitable chunk .morph to build the
requested package.
6. Analyse the source tree and generate a list of dependencies for the package.
7. Enqueue any new dependencies, and repeat.
Once the queue is empty:
8. Generate a stratum .morph for the package(s) the user requested.
The tool is not magic. It can be taught the conventions for each packaging
system, but these will not work in all cases. When an import fails it will
continue to the next package, so that the first run does as many imports as
possible.
For imports that could not be done automatically, you will need to write an
appropriate .lorry or .morph file manually and rerun the tool. It will resume
processing where it left off.
It's possible to teach the code about more conventions, but it is only
worthwhile to do that for common patterns.
Package-system specific code and data
-------------------------------------
For each supported package system, there should be an xxx.to_lorry program, an
xxx.to_chunk program and a xxx.find_deps program. These should output on stdout
a .lorry file, and a .morph file, and dependency information respectively.
Each packaging system can have static data saved in a .yaml file, for known
metadata that the programs cannot discover automatically.
The following field should be honoured by most packaging systems:
`known-source-uris`. It maps package name to source URI.
Help with .lorry generation
---------------------------
The quickest fix is to write a .lorry file by hand.
If you write a .lorry file by hand, be sure to fill in the `x-products-YYY`
field. 'x' means this field is an extension to the .lorry format. YYY is the
name of the packaging system, e.g. 'rubygems'. It should contain a list of
which packages this repository contains the source code for.
Some import extensions have a .yaml file containing 'static metadata', where
you can hardcode URIs for packages in the 'known-source-uris' dict. This is
useful if you need to repeatedly import something (which you shouldn't need
to do, unless you're developing the Import Tool).
Help with linking package version to Git tag
--------------------------------------------
Some projects do not tag releases.
Currently, you must create a tag in the local checkout for the tool to continue.
In future, the Lorry tool should be extended to handle creation of missing
tags, so that they are propagated to the project Trove. The .lorry file would
need to contain a dict mapping product version number to commit SHA1.
If you are in a hurry, you can use the `--use-master-if-no-tag` option. Instead
of an error, the tool will use whatever is the `master` ref of the component
repo.
Help with chunk .morph generation
---------------------------------
Copy an existing one :-)
For package-system specific information, see the relevant README file, e.g
README.rubygems for RubyGem imports.
Help with writing a .foreign-dependencies file
----------------------------------------------
If you need to write a .foreign-dependencies file, it's simply a JSON
document containing two nested dicts. You should be able to copy an
existing one.
You may not need to create one. For example, if you get an error from the
'rubygems.find_deps' program because a package has an invalid Ruby .gemspec
file, you could try fix the .gemspec file in the appropriate repo in the
'checkouts/' subdirectory, and rerun the Import Tool.
If you ignore errors from the .find_deps program, you will have two problems.
First, some dependencies might be missed. Second, a stratum won't be generated
unless you use the `--force-stratum-generation` option.
Common errors
-------------
If you see the program showing this prompt and waiting for input:
Username for 'https://github.com':
This means that 'Lorry' has tried to pull an invalid Github URL. The .lorry
entry for the current package lists the wrong URL. This may be due to bad
metadata, an error in the .to_lorry program (check the log file), or a wrong
URL in a .lorry file because someone wrote it wrong.
The TODO file notes that if the .to_lorry program tried fetching the URL first
and detecting if it returns a 404 'not found' error then we wouldn't hit this
problem.
More information
----------------
More information about the Import Tool can be found in the 'Guides' section of
wiki.baserock.org
- http://wiki.baserock.org/guides/import-tool/quickstart/
- http://wiki.baserock.org/guides/import-tool/rubygems/
- http://wiki.baserock.org/misc-docs/draft-rubygems-build-rules/
- http://wiki.baserock.org/guides/import-tool/npm/
- http://wiki.baserock.org/guides/import-tool/pypi/
|
