expand Porting/release_managers_guide.pod

and remove some similar or out-of-date stuff from Porting/pumpkin.pod

2 files changed, 396 insertions, 208 deletions

diff --git a/Porting/pumpkin.pod b/Porting/pumpkin.pod
index fa13db6a44..2c53c7712a 100644
--- a/Porting/pumpkin.pod
+++ b/Porting/pumpkin.pod
@@ -309,51 +309,12 @@ information on obtaining the metaconfig units.
 
 =head1 How to Make a Distribution
 
-There really ought to be a 'make dist' target, but there isn't.
-The 'dist' suite of tools also contains a number of tools that I haven't
-learned how to use yet.  Some of them may make this all a bit easier.
+This section has now been expanded and moved into its own file,
+F<Porting/release_managers_guide.pod>.
 
-Here are the steps I go through to prepare a patch & distribution.
-
-Lots of it could doubtless be automated but isn't.  The Porting/makerel
-(make release) perl script does now help automate some parts of it.
-
-=head2 Announce your intentions
-
-First, you should volunteer out loud to take the patch pumpkin.  It's
-generally counter-productive to have multiple people working in secret
-on the same thing.
-
-At the same time, announce what you plan to do with the patch pumpkin,
-to allow folks a chance to object or suggest alternatives, or do it for
-you.  Naturally, the patch pumpkin holder ought to incorporate various
-bug fixes and documentation improvements that are posted while he or
-she has the pumpkin, but there might also be larger issues at stake.
-
-One of the precepts of the subversion idea is that we shouldn't give
-the patch pumpkin to anyone unless we have some idea what he or she
-is going to do with it.
-
-=head2 refresh pod/perltoc.pod
-
-Presumably, you have done a full C<make> in your working source
-directory.  Before you C<make spotless> (if you do), and if you have
-changed any documentation in any module or pod file, change to the
-F<pod> directory and run C<make toc>.
-
-=head2 run installhtml to check the validity of the pod files
-
-=head2 update patchlevel.h
-
-Don't be shy about using the subversion number, even for a relatively
-modest patch.  We've never even come close to using all 99 subversions,
-and it's better to have a distinctive number for your patch.  If you
-need feedback on your patch, go ahead and issue it and promise to
-incorporate that feedback quickly (e.g. within 1 week) and send out a
-second patch.
-
-If you update the subversion number, you may need to change the version
-number near the top of the F<Changes> file.
+I've kept some of the subsections here for now, as they don't  direclty
+eleate to building a release any more, but still contain what might be
+useful information - DAPM 7/2009.
 
 =head2 run metaconfig
 
@@ -386,57 +347,12 @@ a better place for your changes.
 
 =head2 MANIFEST
 
-Make sure the MANIFEST is up-to-date.  You can use dist's B<manicheck>
-program for this.  You can also use
-
-    perl -w -MExtUtils::Manifest=fullcheck -e fullcheck
-
-Both commands will also list extra files in the directory that are not
-listed in MANIFEST.
-
-The MANIFEST is normally sorted.
-
 If you are using metaconfig to regenerate Configure, then you should note
 that metaconfig actually uses MANIFEST.new, so you want to be sure
 MANIFEST.new is up-to-date too.  I haven't found the MANIFEST/MANIFEST.new
 distinction particularly useful, but that's probably because I still haven't
 learned how to use the full suite of tools in the dist distribution.
 
-=head2 Check permissions
-
-All the tests in the t/ directory ought to be executable.  The
-main makefile used to do a 'chmod t/*/*.t', but that resulted in
-a self-modifying distribution--something some users would strongly
-prefer to avoid.  The F<t/TEST> script will check for this
-and do the chmod if needed, but the tests still ought to be
-executable.
-
-In all, the following files should probably be executable:
-
-    Configure
-    configpm
-    configure.gnu
-    embed.pl
-    installperl
-    installman
-    keywords.pl
-    myconfig
-    opcode.pl
-    t/TEST
-    t/*/*.t
-    *.SH
-    vms/ext/Stdio/test.pl
-    vms/ext/filespec.t
-    x2p/*.SH
-
-Other things ought to be readable, at least :-).
-
-Probably, the permissions for the files could be encoded in MANIFEST
-somehow, but I'm reluctant to change MANIFEST itself because that
-could break old scripts that use MANIFEST.
-
-I seem to recall that some SVR3 systems kept some sort of file that listed
-permissions for system files; something like that might be appropriate.
 
 =head2 Run Configure
 
@@ -565,32 +481,6 @@ The pumpking can delegate the synchronization responsibility to anybody
 else, but the release process is the only place where we can make sure
 that no new macros fell through the cracks.
 
-=head2 Changes
-
-Be sure to update the F<Changes> file.  Try to include both an overall
-summary as well as detailed descriptions of the changes.  Your
-audience will include other developers and users, so describe
-user-visible changes (if any) in terms they will understand, not in
-code like "initialize foo variable in bar function".
-
-There are differing opinions on whether the detailed descriptions
-ought to go in the Changes file or whether they ought to be available
-separately in the patch file (or both).  There is no disagreement that
-detailed descriptions ought to be easily available somewhere.
-
-If you update the subversion number in F<patchlevel.h>, you may need
-to change the version number near the top of the F<Changes> file.
-
-=head2 Bumping perl's version
-
-If you bump perl's version, you will need to update a few things:
-the L<perlhist> manpage for the date of release, the version number and
-perldelta reference in the top level F<README> (and maybe the copyright
-year too), the F<META.yml> file (generated via F<Porting/makemeta>, be
-sure to run it with the current bleadperl), and the meta-info about
-dual-lived modules in Module::Corelist (F<Porting/corelist.pl> does that).
-Make sure the numbered feature bundles in F<lib/feature.pm> are also
-correct.
 
 =head2 Todo
 
@@ -624,19 +514,6 @@ things that need to be fixed in Configure.
 The Perl revision number appears as "perl5" in configure.com.
 It is courteous to update that if necessary.
 
-=head2 Making the new distribution
-
-Suppose, for example, that you want to make version 5.004_08.  Then you can
-do something like the following
-
-	mkdir ../perl5.004_08
-	awk '{print $1}' MANIFEST | cpio -pdm ../perl5.004_08
-	cd ../
-	tar cf perl5.004_08.tar perl5.004_08
-	gzip --best perl5.004_08.tar
-
-These steps, with extra checks, are automated by the Porting/makerel
-script.
 
 =head2 Making a new patch
 
diff --git a/Porting/release_managers_guide.pod b/Porting/release_managers_guide.pod
index 30c28e3484..9518b30266 100644
--- a/Porting/release_managers_guide.pod
+++ b/Porting/release_managers_guide.pod
@@ -3,171 +3,482 @@
 
 release_managers_guide - Releasing a new version of perl 5.x
 
+XXX as of Jul 2009, this file is still a work-in-progress - DAPM
+
 =head1 SYNOPSIS
 
+This document describes the series of tasks required - some automatic, some
+manual - to produce a perl release of some description, be that a snaphot,
+release candidate, or final release.
+
 The release process is primarily executed by the current pumpking.
 
 This document both helps as a check-list for the pumpking and is
 a base for ideas on how the various tasks could be automated or 
 distributed.
 
-The process has two major parts. In the first part 
-the pumpking needs to determine if the current head revision in Git
-is ready for shipment. The second part is the actual release 
-and packaging process.
+The outline of a release cycle is as follows:
+
+    (5.10.1 is released, and post-release action have been done)
+
+    ...time passes...
+
+    an occasional snapshot is released, that still identifies itself as
+	5.10.1
+
+    ...time passes...
+
+    a few weeks before the release, a number of steps are performed,
+	including bumping the version to 5.10.2
+    
+    perl-5.10.2-RC1 is released
+
+    perl-5.10.2 is released
+
+    post-release actions are performed, including creating new
+	perl5103delta.pod
+
+    ... the cycle continues ...
 
 =head1 DETAILS
 
-=head2 Is it ready?
 
-In this step we need to make sure that:
+=head2 Building a snapshot
+
+A snapshot is intended to encourage in-depth testing from time-to-time,
+for example after a key point in the stabilisation of a branch. It
+requires less steps than a full release, and the version number of perl in
+the tarball will usually still be the old one.
+
+=over 4
+
+=item *
+
+As there are no regular smokes [ XXX yet - please fix?] find out about the
+state of VMS. If it's bad, think again.
+
+=item *
+
+Rebuild META.yml:
+
+    $ rm META.yml
+    $ make META.yml
+
+and commit it if it's changed.
+
+=item *
+
+Check that the manifest is sorted and correct:
+
+    $ make distclean 
+    $ perl Porting/manisort
+    $ perl Porting/manicheck
+
+=item *
+
+If this is a release candidate or final release, add an entry to
+F<pod/perlhist.pod> with the current date, e.g.
+
+          5.8.9-RC1     2008-Nov-10
+
+Make sure the correct pumpking is listed, and if this your first time,
+append your name to C<THE KEEPERS OF THE PUMPKIN>.
+
+=item *
+
+Build perl, then make sure it passes its own test suite, and installs.
+
+=item *
+
+Create a tarball. Use the C<-s> option to specify a suitable suffix for
+the tarball and directory name:
+
+    $ cd root/of/perl/tree
+    $ git clean -xdf		#  make sure perl and git agree on files
+
+    $ perl Porting/makerel -s `git describe` # for a snapshot
+    $ perl Porting/makerel -s RC1	     # for a release candidate
+    $ perl Porting/makerel 		     # for a final release
+
+This creates the  directory F<../perl-x.y.z-RC1> or similar, copies all
+the MANIFEST files into it, sets the correct permissions on them,
+adds DOS line endings to some, then tars it up as
+F<../perl-x.y.z-RC1.tar.gz>.
+
+XXX if we go for extra tags and branches stuff, then add the extra details
+here
+
+=item *
+
+Copy the tarball to a web server somewhere you have access to.
+
+=item *
+
+Download the tarball to some other machine (for a release candidate, to two or
+more servers: IRC is good for this).
+
+=item *
+
+Check that C<./Configure -des && make all test> works in one place.
+
+=item *
+
+Check that C<./Configure ... && make all test_harness install> works.
+
+=item *
+
+Check that the output of C<perl -v> and C<perl -V> are as expected,
+especially as regards version numbers, patch and/or RC levels, and @INC
+paths. Note that the results may be different without a F<.git/> directory,
+which is why you should test from the tarball.
+
+=item *
+
+Bootstrap the CPAN client on the clean install.
+
+=item *
+
+Install CPANPLUS.
+XXX pick something new; this is now bundled
+
+=item *
+
+Bootstrap the CPANPLUS client.
+
+=item *
+
+Install an XS module.
+
+=item *
+
+If all is well, announce the snaphot to p5p. (For a release candidate,
+instead follow the further steps described later.)
+
+=back
+
+=head2  Actions prior to the first release candidate
+
+A week or two before the first release candidate, there are some additional
+tasks you should perform (actually, some of these should be done regularly
+anyway, but they definitely need doing now):
 
 =over 4
 
-=item 1
+=item *
+
+Check F<Maintainers.pl> for consistency; both these commands should
+produce no output:
+
+    perl Porting/Maintainers --checkmani
+    perl Porting/Maintainers --checkmani lib/ ext/
+
+=item *
+
+Ensure that dual-life CPAN modules are synchronised with CPAN.  Basically,
+run the following:
+
+    ./perl -Ilib Porting/core-cpan-diff -a -o /tmp/corediffs
 
-perl passes its own test suite and 
+to see any inconsistencies between the core and CPAN versions of distros,
+then fix the core, or cajole CPAN authors as appropriate. See also the
+C<-d> and C<-v> options for more detail.  You'll probably want to use the
+C<-c cachedir> option to avoid repeated CPAN downloads.
 
-=item 2
+To see which core distro versions differ from the current CPAN versions:
 
-CPAN works
+    ./perl -Ilib Porting/core-cpan-diff -x -a
 
+=item *
 
-which comes down to:
+Ensure dual-life CPAN modules are stable, which comes down to:
 
     for each module that fails its regression tests on $current
-      did it fail identically on $previous?
-      if yes, "SEP" (Somebody Else's Problem)
-      else work out why it failed (a bisect is useful for this)
+        did it fail identically on $previous?
+        if yes, "SEP" (Somebody Else's Problem)
+        else work out why it failed (a bisect is useful for this)
 
     attempt to group failure causes
 
     for each failure cause
-      is that a regression?
-      if yes, figure out how to fix it
-        (more code? revert the code that broke it)
-      else
-        (presumably) it's relying on something un-or-under-documented
-        should the existing behaviour stay?
-          yes - goto "regression"
-          no - note it in perldelta as a significant bugfix
-          (also, try to inform the module's author)
-
-( TBD based on http://www.nntp.perl.org/group/perl.perl5.porters/2009/05/msg146680.html )
+        is that a regression?
+        if yes, figure out how to fix it
+            (more code? revert the code that broke it)
+        else
+            (presumably) it's relying on something un-or-under-documented
+            should the existing behaviour stay?
+                yes - goto "regression"
+                no - note it in perldelta as a significant bugfix
+                (also, try to inform the module's author)
 
-=back
+=item *
 
+Similarly, monitor the smoking of core tests, and try to fix.
 
-=head2 The Actual release process
+=item *
 
-The set of tasks that can be "scripted" for Perl 5
+Run F<Porting/cmpVERSION.pl> to compare the current source tree with the
+previous version to check for for modules that have identical version
+numbers but different contents, e.g.:
 
-=over 4
+     $ cd ~/some-perl-root
+     $ ./perl -Ilib Porting/cmpVERSION.pl -xd ~/my_perl-tarballs/perl-5.10.0 .
+
+then bump the version numbers of any non-dual-life modules that have
+changed since the previous release, but which still have the old version
+number. If there is more than one maintenance branch (e.g. 5.8.x, 5.10.x),
+then compare against both.
+
+Note that some of the files listed may be generated (e.g. copied from ext/
+to lib/, or a script like lib/lib_pm.PL is run to produce lib/lib.pm);
+make sure you edit the correct file!
+
+Once all version numbers have been bumped, re-run the checks.
+
+Then run again without the -x option, to check that dual-life modules are
+also sensible.
+
+
+=item *
+
+Get perldelta in a mostly finished state.
+XXX expand
+
+=item *
+
+Bump the perl version number (e.g. from 5.10.0 to 5.10.1).
+
+There is a tool to semi-automate this process. It works in two stages.
+First, it generates a list of suggested changes, which you review and
+edit; then you feed this list back and it applies the edits. So, first
+scan the source dir looking for likely  candidates:
+
+    $ Porting/bump-perl-version -s 5.10.0 5.10.1 > /tmp/scan
+
+This produces a file containing a list of suggested edits, eg:
+
+    NetWare/Makefile
+
+       89: -MODULE_DESC     = "Perl 5.10.0 for NetWare"
+	   +MODULE_DESC     = "Perl 5.10.1 for NetWare"
+
+i.e. in the file F<NetWare/Makefile>, line 89 would be changed as shown.
+Review the file carefully, and delete any -/+ line pairs that you don't
+want changing. Remember that this tool is largely just grepping for '5.10.0'
+or whatever, so it will generate false positives. Be careful not change
+text like "this was fixed in 5.10.0"! Then run:
+
+    $ Porting/bump-perl-version -u < /tmp/scan
+
+which will update all the files shown; then commit the changes.
+
+Be particularly careful with F<INSTALL>, which contains a mixture of
+C<5.10.0>-type strings, some of which need bumping on every release, and
+some of which need to be left. Also note that this tool currently only
+performs a single change per line, so in particular, this line in
+README.vms needs special handling:
+
+    rename perl-5^.10^.1.dir perl-5_10_1.dir
 
-=item 0
+=item *
 
-so you think you have a source control in a state that won't break CPAN,
-at least not in "surprising" ways.
+Update the F<Changes> file to contain the git log command which would show
+all the changes in this release. You will need assume the existence of a
+not-yet created tag for the forthcoming release; e.g.
 
-=item 1
+    git log ... perl-5.10.0..perl5.12.0
 
-As there are no regular smokes (yet - please fix?) find out about the state
-of VMS. If it's bad, think again.
+Due to warts in the perforce-to-git migration, some branches require extra
+exclusions to avoid other branches being pulled in. Make sure you have the
+correct incantation: replace the not-yet-created tag with C<HEAD> and see
+if git log produces roughly the right number of commits across roughly the
+right time period.
 
-=item 2
+=item *
 
-Re-read the perldelta to try to find any embarrassing typos
+Check some more build configurations, e.g.
 
-=item 3
+     -Duseshrplib -Dd_dosuid
+     make suidperl
 
-Run Porting/makemeta
+Check that setuid installs works (for < 5.11.0 only).
+XXX any other configs?
 
-=item 4
+=item *
 
-[used to be run autodoc.pl, but I eliminated that]
+Make sure you have a PAUSE account suitable for uploading a perl release.
+If you don't have a PAUSE account, then request one:
 
-=item 5
+    https://pause.perl.org/pause/query?ACTION=request_id
 
-[used to be run pod/buildtoc, but I eliminated that]
+Check that your account is allowed to upload perl distros: goto
+https://pause.perl.org/, login, then select 'upload file to CPAN'; there
+should be a "For pumpkings only: Send a CC" tickbox.  If not, ask Andreas
+to add your ID to the list of people allowed to upload something called
+perl.
 
-=item 6
+=item *
 
-update module corelist, but we need to fix that
+Update F<AUTHORS>, using the C<Porting/checkAUTHORS.pl> script, and if
+necessary, update the script to include new alias mappings.
 
-[it has been holding perforce revisions for releases, but we can't know
+XXX This script is currently broken (7/2009). It needs updating to work
+with git and a lack of Changes files.
+
+=back
+
+=head2  Building a release candidate
+
+(At this point you should already have performed the actions described in
+L</"Actions prior to the first release candidate">.)
+
+=over 4
+
+=item *
+
+Re-read the perldelta to try to find any embarrassing typos and thinkos;
+remove any C<TODO> or C<XXX> flags; and run through pod and spell
+checkers.   [XXX show how]
+
+=item *
+
+Update patchlevel.h to add a C<-RC1>-or-whatever string; or, if this is a
+final release, remove it.  [ XXX how now?? see 34813 for old way ]
+
+=item *
+
+Update C<Module::Corelist>.
+[XXX add how: presumably Porting/corelist.pl ]
+but we need to fix that: 
+it has been holding perforce revisions for releases, but we can't know
 hashes in advance for git. We need to agree a plan to move to git tags]
 
-=item 7
+=item *
+
+Follow the instructions in the section L</"Building a snapshot">, then
+carry on with the extra steps that follow here.
+
+=item *
 
-[update changes, but Dave has eliminated that]
+Wait for the smoke tests to catch up with the commit which this release is
+based on (or at least the last commit of any consequence).
 
-=item 8
+Then check that the smoke tests pass (particularly on Win32). If not, go
+back and fix things.
 
-update patchlevel.h to remove all local patches
 
-=item 9
+=item *
 
-make tarball with Porting/makerel
+Once smoking is okay, upload it to PAUSE. This is the point of no return.
+
+XXX check whether need to upload .bz2,.meta, .readme too?
+
+
+=item *
+
+Mail p5p to announce it, with a quote you prepared earlier.
+
+=item *
+
+Wait 24 hours or so, then post the announcement to use.perl.org.
+
+=back
+
+
+=head2  Making a final release
+
+At this point you should have a working release candidate with few or no
+changes since.
+
+It's essentially the same procedure as for making a release candidate, but
+with a whole bunch of extra post-release steps.
+
+=over 4
 
-=item 10
+=item *
 
-copy tarball to some other machine x 2 [or more - IRC is good for this]
+Follow the same steps as for making a release candidate, then
 
-=item 11
+=item *
 
-check that ./Configure -des && make all test works in one place
+Ask Jarkko to update http://www.cpan.org/src/README.html and
+Rafael to update http://dev.perl.org/perl5/
 
-=item 12
 
-check that ./Configure ... && make all test_harness install works
+=item *
 
+create a tag [XXX and branches and stuff?]
+[ XXX expand this ]
 
-=item 13
+=item *
 
-bootstrap the CPAN client on the clean install
+[ XXX disarm any patchlevel.h stuff ??? ]
 
-=item 14
+=item *
 
-install CPANPLUS
+Create a new empty perlNNNdelta.pod file for the current release + 1;
+see F<Porting/how_to_write_a_perldelta.pod>.
+[ XXX Perhaps we should have an empty template file we can copy in. ]
 
-=item 15
+In addition, add entries to the following files:
 
-bootstrap the CPANPLUS client
+    MANIFEST
+    pod.lst         (add new entry as 'D', and unmark previous entry as 'D')
+    pod/perl.pod
+    win32/pod.mak   (add entries for .pod, .man, .html, .tex)
+    vms/descrip_mms.template (also bump the perldelta.pod: entry)
 
-=item 16
+and change perlNNNdelta references to the new version in these files
 
-install an XS module
+    INSTALL
+    win32/Makefile.mk
+    win32/Makefile
+    Makefile.SH
+    README
 
-=item 17
+Also, edit the previous delta file to change the C<NAME> from C<perldelta>
+to C<perlNNNdelta>.
 
-if this is good, commit this.
-sit, and wait.
+These two lists of files probably aren't exhaustive; do a recursive grep
+on the previous filename to look for suitable candidates.
 
-=item 18
+(see 16410843ea for an example).
 
-do the smoke tests pass (particularly Win32)
+=item *
 
-=item 19
+If this was  major release, bump the version, e.g. 5.12.0 to 5.13.0
+[ XXX probably more complex stuff to do, including perldelta,
+C<lib/feature.pm> ]
 
-if yes, upload it to PAUSE. This is the point of no return
+=item *
 
-=item 20
+Copy the perlNNNdelta.pod for this release into the other branches, and
+remember to update these files on those branches too:
 
-mail p5p to announce it, with a quote I prepared earlier
+    MANIFEST
+    pod.lst
+    pod/perl.pod
+    vms/descrip_mms.template
+    win32/pod.mak
 
-=item 21
+(see fc5be80860 for an example).
 
-wait 24 hours or so
+=item *
 
-=item 22
+Make sure any recent F<pod/perlhist.pod> entries are copied to
+F<perlhist.pod> on other branches; typically the RC* and final entries,
+e.g.
 
-post the announcement to use.perl.org
+          5.8.9-RC1     2008-Nov-10
+          5.8.9-RC2     2008-Dec-06
+          5.8.9         2008-Dec-14
 
 =back
 
 =head1 SOURCE
 
-Based on http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2009-05/msg00608.html
+Based on
+http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2009-05/msg00608.html,
+plus a whole bunch of other sources, including private correspondence.
 
 =cut