Difference between revisions of "Automatic Testing"

From Madagascar
Jump to navigation Jump to search
Line 38: Line 38:
  
 
The output at the end of this step will include a count of how many examples failed.  I usually capture the output of this step into a file and grep it for "error" to see which examples crashed.  Then I can look at the corresponding scons.log file to see what happened, or just cd to the directory and run scons manually to see it happen again.
 
The output at the end of this step will include a count of how many examples failed.  I usually capture the output of this step into a file and grep it for "error" to see which examples crashed.  Then I can look at the corresponding scons.log file to see what happened, or just cd to the directory and run scons manually to see it happen again.
 +
 +
===Compare the results===
 +
<bash>rm book/*/*/*/.rsftest
 +
sfbooklist size=1024 list=filter skipfile=admin/skiplist.txt command="sffiglist rsftest=y list=none > list.log" book</bash>
 +
 +
Step 3 is like Step 2 except for a different command string.  This command string runs the script '''sffiglist''' in each directory to compare the <tt>*.vpl</tt> files in the <tt>Fig</tt> directory with those in the lock directory.  A summary of the comparison is put into an <tt>.rsftest</tt> file in each directory.
  
 
==Useful Scripts==
 
==Useful Scripts==

Revision as of 07:13, 22 June 2010

Quick Test

To run a "quick" test run this command from $RSFSRC:

<bash>admin/quick_test.csh</bash>

This script currently (June 19, 2010) runs a set of 147 examples in $RSFSRC/book that make 4GB of total data. They run in about 2 minutes total on a 2.4 GHz Intel Core 2 Duo MacBook when nothing needs to be updated, and 15 minutes when everything is rebuilt from scratch. Your mileage will vary.

Seven of the 147 examples do not pass all the tests on my system, perhaps because of bugs not yet found :-( These examples are:

  1. book/gee/mda/levint
  2. book/jsg/lpf/lpf
  3. book/sep/bspl/plane3
  4. book/sep/pwd/alias
  5. book/sep/pwd/hole
  6. book/sep/pwd/signoi
  7. book/sep/stack/miginv

Your mileage will probably vary on this as well. Indeed, I will be curious to see how the tests work on your system (hint, hint :-) Post your testing report on rsf-devel.

Explanation of different steps

The following commands use sfbooklist and sffiglist. Run them from $RSFSRC.

Get an inventory

<bash>sfbooklist size=1024 public=y list=filter skipfile=admin/skiplist.txt book</bash>

This step is not really necessary, but I do it to get a quick list of all the examples and a list of the subset that will be tested. I usually capture the output in a file and do a diff with the previous run. That way I will notice when new examples have been added to book since the last time I ran the tests.

The filters size= public= skipfile= limit the collection of examples to be executed. The size= filter skips any example that makes more than 1 GB of data. public=y means it will run examples that download public data from the server (the default is public=n). The default options for local= and private= are both n, so examples that use that kind of external data will be skipped. skipfile= specifies a list of examples by name that will be skipped. An example skipfile is RSFSRC/admin/skiplist.txt, it specifies examples I can't run because of stuff I haven't installed and others that take a long time to run on my laptop. There is also a default filter to skip any example that does not yet have an .rsfproj file.

Running this step will tell you which examples pass these set of filters.

Run the examples

<bash>sfbooklist size=1024 list=filter skipfile=admin/skiplist.txt timer=file command="scons 2>&1 >scons.log" book</bash>

Step 2 is the same as Step 1 except I also specify a command string to run in each directory. The command string runs scons to update the examples and capture the console output in log files. There is also an option timer=file which appends a single line execution-time summary to a .rsftimer file in each directory. The other options for timer are none and log which sends the summary to the console instead of a file.

The output at the end of this step will include a count of how many examples failed. I usually capture the output of this step into a file and grep it for "error" to see which examples crashed. Then I can look at the corresponding scons.log file to see what happened, or just cd to the directory and run scons manually to see it happen again.

Compare the results

<bash>rm book/*/*/*/.rsftest sfbooklist size=1024 list=filter skipfile=admin/skiplist.txt command="sffiglist rsftest=y list=none > list.log" book</bash>

Step 3 is like Step 2 except for a different command string. This command string runs the script sffiglist in each directory to compare the *.vpl files in the Fig directory with those in the lock directory. A summary of the comparison is put into an .rsftest file in each directory.

Useful Scripts

  1. List all the examples in book that use <program>
    sfbooklist uses=<program> list=filter book
  2. rebuild all the examples in book that use <program>
    sfbooklist uses=<program> list=filter command=scons book
  3. Compare the newly generated figs with the corresponding reference figs stored in $RSFFIGS (defaulted to $RSFROOT/share/figs)
    sfbooklist uses=<program> list=filter command="sffiglist rsftest=y list=none" book
  4. Display any figs that are different from their $RSFFIGS counterparts
    sfbooklist uses=<program> list=filter command="sffiglist rsftest=y list=none show=diff" book