From 3ecd967b2a9e23403935e2bc932597f7e03e7f24 Mon Sep 17 00:00:00 2001 From: Kai Engert Date: Thu, 28 Feb 2013 12:44:50 +0100 Subject: Bug 845556, reorganize NSS directory layout, moving files, very large changeset! r=wtc --- tests/doc/clean.gif | Bin 0 -> 5503 bytes tests/doc/nssqa.txt | 108 ++++++++++++++ tests/doc/platform_specific_problems | 110 ++++++++++++++ tests/doc/qa_wrapper.html | 269 +++++++++++++++++++++++++++++++++++ 4 files changed, 487 insertions(+) create mode 100644 tests/doc/clean.gif create mode 100755 tests/doc/nssqa.txt create mode 100644 tests/doc/platform_specific_problems create mode 100755 tests/doc/qa_wrapper.html (limited to 'tests/doc') diff --git a/tests/doc/clean.gif b/tests/doc/clean.gif new file mode 100644 index 000000000..08781cb2b Binary files /dev/null and b/tests/doc/clean.gif differ diff --git a/tests/doc/nssqa.txt b/tests/doc/nssqa.txt new file mode 100755 index 000000000..34fa0955b --- /dev/null +++ b/tests/doc/nssqa.txt @@ -0,0 +1,108 @@ +The new QA wrapper consistst mainly of 2 scripts, nssqa and qa_stat, both +include a common header (header) and a common environment (set_environment). +Also used is mksymlinks and path_uniq. + +The scripts that are used on a daily basis are located in /u/sonmi/bin. + +Parameters and Options are the same for both scripts. + +Parameters +---------- + nssversion (supported: 30b, 31, tip) + builddate (default - today, format mmdd) + +Options +------- + -y answer all questions with y - use at your own risk... ignores warnings + -s silent (only usefull with -y) + -h, -? -help you guessed right - displays the usage + -d debug + -f - write the (error)output to filename + -m - send filename to mailinglist (csl) only useful + with -f on nssqa + -l run on a local build - does not work at this time + -cron equivalient to -y -s -d -f $RESULTDIR/$HOST. + +nssqa and qa_stat are Beta at the most +-------------------------- +Please be aware that + +-) machinenames are still hardcoded --FIXED +-) other very iPlanet specific environments and features are being used. + +-d Debug option will be removed from cron in a few weeks - or maybe not +-l QA on local build is not fully implemented yet + +Please do not use on Windows 95 and 98, ME platforms yet. + +use -d if script behaves strange or exits unexpectedly + +How to use QA +------------- +To test a build, first run nssqa on the required QA platforms (some +buildplatforms require QA to be run on additional platforms - for +example Solaris 2.6 has to be tested on 2.8 32 and 64bit) If QA has +been run on multiple or all required platforms it makes sense to run +qa_stat on the output of nssqa as well. +Before used on a new system (even if the same platform has been +tested before) please use completely interactive, to see what the +variables are being initialized to, and read the warnings. Same is +true if being run from a different user account than svbld. + +In any case, if you are using it, please let me know the results. + +Pseudocode Description of nssqa: +-------------------------------- +not quite up to date + + header:init (global) + set flags and variables to default values + signal trap (for interupts and kills) + set HOST and DOMSUF variables if running from cron + parse parameters and options + determine os and set up the environment (espec. PATH) + set the directories to run in (influenced by parameters and -l option) + set and initialize the tmp / debugging / output files + + nssqa:init (local) + locking: if nssqa is already running on this systems (yes-exit, + no-lockfile) + set HOST and DOMSUF variables if running interavtively + set flag to kill remaining selfserv processes during cleanup + if QA platform different from build platform create neccessary + symbolic links + wait for the build to finish (max of 5h) + + main: + repeated per test (optimized, debug, 32, 64 bit) + set flags for this run of all.sh (optimized, debug, 32, 64 bit) + set the DIST directory (where the binaries reside) + kill running selfservers (sorry - just don't use the svbld + account if you need to do your own testing... I will fix + selfserv as soon as I can - but it hangs too often and + disturbs all following QA) + run all.sh + + header:exit (global) + remove temporary files + kill remaining selfservers + send email to the list + + + errorhandling + Option / Parameter errors: Exit with usage information + + Severe errors: Exit wit errormessage + example: directory in which all.sh resides does not exist + can't create files or directories + build not done after 5 hours + is already running + + Other errors: User is prompted with the "errormessage - continue (y/n)?" + example: local DIST dir does not exist (continues with next all.sh) + outputdirectory does not exist (user can specify other) + + Signals 2, 3, 15 are treated as severe errors + + + diff --git a/tests/doc/platform_specific_problems b/tests/doc/platform_specific_problems new file mode 100644 index 000000000..92a22ca03 --- /dev/null +++ b/tests/doc/platform_specific_problems @@ -0,0 +1,110 @@ +I will, eventually convert all files here to html - just right now I have no +time to do it. Anyone who'd like to - please feel free, mail me the file and +I will check it in +sonmi@netscape.com + + +The NSS 3.1 SSL Stress Tests fail for me on FreeBSD 3.5. The end of the output +of './ssl.sh stress' looks like this: + +********************* Stress Test **************************** +********************* Stress SSL2 RC4 128 with MD5 **************************** +selfserv -p 8443 -d +/local/llennox/NSS-PSM/mozilla/tests_results/security/conrail.20/server -n +conrail.cs.columbia.edu -w nss -i /tmp/tests_pid.5505 & strsclnt -p 8443 -d . -w nss -c 1000 -C A conrail.cs.columbia.edu +strsclnt: -- SSL: Server Certificate Validated. +strsclnt: PR_NewTCPSocket returned error -5974: +Insufficient system resources. +Terminated +********************* Stress SSL3 RC4 128 with MD5 **************************** +selfserv -p 8443 -d +/local/llennox/NSS-PSM/mozilla/tests_results/security/conrail.20/server -n +conrail.cs.columbia.edu -w nss -i /tmp/tests_pid.5505 & strsclnt -p 8443 -d . -w nss -c 1000 -C c conrail.cs.columbia.edu +strsclnt: -- SSL: Server Certificate Validated. +strsclnt: PR_NewTCPSocket returned error -5974: +Insufficient system resources. +Terminated + +Running ktrace on the process (ktrace is a system-call tracer, the equivalent of +Linux's strace) reveals that socket() failed with ENOBUFS after it was called +for the 953rd time for the first test, and it failed after the 27th time it was +called for the second test. + +The failure is consistent, both for debug and optimized builds; I haven't tested +to see whether the count of socket() failures is consistent. + +All the other NSS tests pass successfully. + + +------- Additional Comments From Nelson Bolyard 2000-11-01 23:08 ------- + +I see no indication of any error on NSS's part from this description. +It sounds like an OS kernel configuration problem on the +submittor's system. The stress test is just that. It stresses +the server by pounding it with SSL connections. Apparently this +test exhausts some kernel resource on the submittor's system. + +The only change to NSS that might be beneficial to this test +would be to respond to this error by waiting and trying again +for some limited number of times, rather than immediately +treating it as a fatal error. + +However, while such a change might make the test appear to pass, +it would merely be hiding a very serious problem, namely, +chronic system resource exhaustion. + +So, I suggest that, in this case, the failure serves the useful +purpose of revealing the system problem, which needs to be +cured apart from any changes to NSS. + +I'll leave this bug open for a few more days, to give others +a chance to persuade me that some NSS change would and should +solve this problem. + + +------- Additional Comments From Jonathan Lennox 2000-11-02 13:13 ------- + +Okay, some more investigation leads me to agree with you. What's happening is +that the TCP connections from the stress test stick around in TIME_WAIT for two +minutes; my kernel is only configured to support 1064 simultaneous open sockets, +which isn't enough for the 2K sockets opened by the stress test plus the 100 or +so normally in use on my system. + +So I'd just suggest adding a note to the NSS test webpage to the effect of "The +SSL stress test opens 2,048 TCP connections in quick succession. Kernel data +structures may remain allocated for these connections for up to two minutes. +Some systems may not be configured to allow this many simulatenous connections +by default; if the stress tests fail, try increasing the number of simultaneous +sockets supported." + +On FreeBSD, you can display the number of simultaneous sockets with the command + sysctl kern.ipc.maxsockets +which on my system returns 1064. + +It looks like this can be fixed with the kernel config option + options NMBCLUSTERS=[something-large] +or by increasing the 'maxusers' parameter. + +It looks like more recent FreeBSD implementations still have this limitation, +and the same solutions apply, plus you can alternatively specify the maxsockets +parameter in the boot loader. + + +--------------------------------- + +hpux HP-UX hp64 B.11.00 A 9000/800 2014971275 two-user license + +we had to change following kernelparameters to make our tests pass + +1. maxfiles. old value = 60. new value = 100. +2. nkthread. old value = 499. new value = 1328. +3. max_thread_proc. old value = 64. new value = 512. +4. maxusers. old value = 32. new value = 64. +5. maxuprc. old value = 75. new value = 512. +6. nproc. old formula = 20+8*MAXUSERS, which evaluated to 276. + new value (note: not a formula) = 750. + +A few other kernel parameters were also changed automatically +as a result of the above changes. + + diff --git a/tests/doc/qa_wrapper.html b/tests/doc/qa_wrapper.html new file mode 100755 index 000000000..755cca236 --- /dev/null +++ b/tests/doc/qa_wrapper.html @@ -0,0 +1,269 @@ + + + + + + + + +

+Author Sonja Mirtitsch

+ +

+Last updated: 4/4/2001

+ +

+NSS 3.2.QA Wrapper

+ +

The QA wrapper tests the nightly builds of NSS. The actual +tests are being run are called from the QA script all.sh. I will add documentation +for the actual QA soon. The main purpose of the wrapper is: find out which +build (NSS version, date, Build Platform) to test on which machine (OS, +OS version) and construct a summary report, which is then mailed to the +nss developers (aka mailing list nss-qa-report@netscape.com). Please see +also the feature section. +

nssqa - the script that calls the actual +qa script all.sh +
qa_stat - sends out status reports +
qaclean - if everything else fails +

Sample global result, +individual result and log +files +

The QA wrapper consistst mainly of scripts, most located in security/nss/tests +and subdirectories, but run from /u/sonmi/bin +

nssqa and qa_stat, the main scripts both include a common header (header) +and a common environment (set_environment). +
Also used is mksymlinks and path_uniq +and qaclean. +

The scripts that are used on a daily basis are located in /u/sonmi/bin +and checked into security/nss/tests +

Parameters and Options are the same for most scripts. +

Parameters +
nssversion (supported: 30b, 31, tip, default tip) +
builddate (default - today, format mmdd) +

Options +
    -y answer all questions with y - use at your own +risk... ignores warnings +
    -s silent (only usefull with -y) +
    -h, -? -help you guessed right - displays the usage +
    -d debug +
    -f <filename> - write the (error)output to filename +
    -fcron writes resultfile in the same location as +would the -cron +
    -m <mailinglist> - send filename to mailinglist +(csl) only useful +
        with -f on nssqa +
    -l <mozroot> run on a local build - does not +work at this time +
    -cron equivalient to -y -s -d -f $RESULTDIR/$HOST.<scriptname> +
+

Please be aware that some iPlanet specific environments and features +are being used. +

-d Debug option might be removed from cron in a few weeks - or maybe +not +
-l QA on local build is not fully implemented yet - will not be implemented, +all.sh can be called directly instead +

Please do not use on Windows 95 and 98, ME platforms yet. +

use -d if script behaves strange or exits unexpectedly +

How to use the QA-wrapper +
To test a build, first run nssqa on the required QA platforms (some +buildplatforms require QA to be run on additional platforms - for example +Solaris 2.6 has to be tested on 2.8 32 and 64bit) If QA has been run on +multiple or all required platforms it makes sense to run qa_stat on the +output of nssqa as well. +
Before used on a new system (even if the same platform has been tested +before) please use completely interactive, to see what the variables are +being initialized to, and read the warnings. Same is true if being run +from a different user account than svbld. +

In any case, if you are using it, please let me know the results. +

nssqa: +

the script that calls the actual qa script all.sh +

nssqa parameters and options +

view the script +

Pseudocode Description of nssqa +
not quite up to date +

    header:init (global) +
        set flags and variables +to default values +
        signal trap (for interupts +and kills) +
        set HOST and DOMSUF variables +if running from cron +
        parse parameters and options +
        determine os and set up +the environment (espec. PATH) +
        set the directories to run +in (influenced by parameters and -l option)
+        set the directories for backward +compatibility testing +
        set and initialize the tmp +/ debugging / output files +

    nssqa:init (local) +
        locking: if nssqa is already +running on this systems (yes-exit, +
            +no-lockfile) +
        set HOST and DOMSUF variables +if running interavtively +
        set flag to kill remaining +selfserv processes during cleanup +
        if QA platform different +from build platform create neccessary +
            +symbolic links +
        wait for the build to finish +(max of 5h) +

    main: +
        repeated per test (optimized, +debug, 32, 64 bit) +
            +set flags for this run of all.sh (optimized, debug, 32, 64 bit) +
            +set the DIST directory (where the binaries reside) +
            +kill running selfservers (sorry - just don't use the svbld +
                +account if you need to do your own testing... I will fix +
                +selfserv as soon as I can - but it hangs too often and +
                +disturbs all following QA) +
            +run all.sh +

header:exit (global) +
remove temporary files +

kill remaining selfservers +
send email to the list +
+

errorhandling +
Option / Parameter errors: +Exit with usage information +

        Severe errors: Exit wit errormessage +
            +example: directory in which all.sh resides does not exist +
                +can't create files or directories +
                +build not done after 5 hours +
                +is already running +

        Other errors: User is prompted +with the "errormessage - continue (y/n)?" +
            +example: local DIST dir does not exist (continues with next all.sh) +
                +outputdirectory does not exist (user can specify other) +

Signals 2, 3, 15 are treated +as severe errors +
+
+
+

qaclean:/u/sonmi/bin/qaclean +
+

Use qaclean as user "svbld" to get the propper permissions. It is supposed +to clean up after a "hanging" QA and will also brutally kill, interupt +and disturb any other nss related test or performance meassurement on the +named machine. NT and 2000 might require an additional reboot, since the +ps is not so good about telling us the actual programmname - so we can't +kill them... Please note that this is a brute force script, it should not +be used on a regular basis, file a bug whenever you have to use it, since +hanging QA is nothing that should occur frequently +

view the script +

What it does: +

+see if there is a lockfile (/tmp/nssqa.$$ or $TMP/nssqa.$$)

(future expansion +and if possible it's children )

+kill selfservers
+kill whatever other qa related processes might be hanging
+clean up tmp files

+QAClean Parameters: +
    machinename. +
    for example +
    qaclean kentuckyderby +
    started on any machine, will clean up on kentuckyderby +

qa_stat +

qa_stat is the script that is being started from the svbld cron on kentuckyderby +every morning at 10:00 and runs some (very primitive) analysis on the qa +results. +
I'd like to rewrite the whole thing in perl, and in a few weeks I might +just do this... +

view the script +

qa_stat parameters and options +

Why we need the QA wrapper +

We need the new QA wrapper, because we have to test on so many platforms, +that running the tests and evaluating the results for the nightly builds +took about an average workday. +

New Features: +

+runs from cron / rsh or interactive if desired
+generates summary (no need to look through 60-90 directories)
+sends email about results
+automatically recognizes common errors and problems and conflicts +and corrects them

+automatically determines which build to test (waits if build in +progress, exits if no build)
+runs on all required platforms (Windows 98 and before not functional +yet)
+Windows version runs on free Cygnus as well as on MKS
+debug mode, normal mode and silent mode
+locking mechanism so it won't run twice
+cleanup after being killed and most errors (no remaining selfservers, +tmpfiles, lock files)

+The 1st script is started via cron between 5:00 and 8:00 am on different +systems, and starts QA on the nightly build. At 10:00 the next script is +started, and sends a QA summary to the nss developers. +

Cygnus Advantages: +

+free
+better handling of processes (background, processIDs, Signals)
+Unix / Linux compatible sh / bash

+Disadvantages +

+MKS functionality needs to be preserved (makes 8 Windows platforms +instead of 4 for the QA suites - makes 32 testruns on Windows alone)

slow

+Porting the windows QA to Uwin as well is also being considered + + -- cgit v1.2.1