Difference between revisions of "Automatic Testing"

From Madagascar
Jump to navigation Jump to search
Line 18: Line 18:
  
 
Your mileage will probably vary on this as well.  Indeed, I will be curious to see how the tests work on your system (hint, hint :-)  Post your testing report on rsf-devel.
 
Your mileage will probably vary on this as well.  Indeed, I will be curious to see how the tests work on your system (hint, hint :-)  Post your testing report on rsf-devel.
 +
 +
==Explanation of different steps==
 +
 +
===Get an inventory===
 +
<bash>sfbooklist size=1024 public=y list=filter skipfile=admin/skiplist.txt book</bash>
 +
 +
This step is not really necessary, but I do it to get a quick list of all the examples and a list of the subset that will be tested.  I usually capture the output in a file and do a diff with the previous run.  That way I will notice when new examples have been added to book since the last time I ran the tests.
 +
 +
The filters <tt>size= public= skipfile=</tt> limit the collection of examples to be executed.  The '''size=''' filter skips any example that makes more than 1 GB of data.  '''public=y''' means it will run examples that download public data from the server (the default is <tt>public=n</tt>).  The default options for '''local=''' and '''private=''' are both <tt>n</tt>, so examples that use that kind of external data will be skipped.  '''skipfile=''' specifies a list of examples by name that will be skipped.  An example skipfile is [http://rsf.svn.sourceforge.net/viewvc/rsf/trunk/admin/skiplist.txt?revision=6224&view=markup RSFSRC/admin/skiplist.txt], it specifies examples I can't run because of stuff I haven't installed and others that take a long time to run on my laptop.  There is also a default filter to skip any example that does not yet have an .rsfproj file.
 +
 +
Running this step will tell you which examples pass these set of filters.
  
 
==Testing Scripts==
 
==Testing Scripts==

Revision as of 07:05, 22 June 2010

Quick Test

To run a "quick" test run this command from $RSFSRC:

<bash>admin/quick_test.csh</bash>

This script currently (June 19, 2010) runs a set of 147 examples in $RSFSRC/book that make 4GB of total data. They run in about 2 minutes total on a 2.4 GHz Intel Core 2 Duo MacBook when nothing needs to be updated, and 15 minutes when everything is rebuilt from scratch. Your mileage will vary.

Seven of the 147 examples do not pass all the tests on my system, perhaps because of bugs not yet found :-( These examples are:

  1. book/gee/mda/levint
  2. book/jsg/lpf/lpf
  3. book/sep/bspl/plane3
  4. book/sep/pwd/alias
  5. book/sep/pwd/hole
  6. book/sep/pwd/signoi
  7. book/sep/stack/miginv

Your mileage will probably vary on this as well. Indeed, I will be curious to see how the tests work on your system (hint, hint :-) Post your testing report on rsf-devel.

Explanation of different steps

Get an inventory

<bash>sfbooklist size=1024 public=y list=filter skipfile=admin/skiplist.txt book</bash>

This step is not really necessary, but I do it to get a quick list of all the examples and a list of the subset that will be tested. I usually capture the output in a file and do a diff with the previous run. That way I will notice when new examples have been added to book since the last time I ran the tests.

The filters size= public= skipfile= limit the collection of examples to be executed. The size= filter skips any example that makes more than 1 GB of data. public=y means it will run examples that download public data from the server (the default is public=n). The default options for local= and private= are both n, so examples that use that kind of external data will be skipped. skipfile= specifies a list of examples by name that will be skipped. An example skipfile is RSFSRC/admin/skiplist.txt, it specifies examples I can't run because of stuff I haven't installed and others that take a long time to run on my laptop. There is also a default filter to skip any example that does not yet have an .rsfproj file.

Running this step will tell you which examples pass these set of filters.

Testing Scripts

The following commands use sfbooklist and sffiglist. Run them from $RSFSRC.

  1. List all the examples in book that use <program>
    sfbooklist uses=<program> list=filter book
  2. rebuild all the examples in book that use <program>
    sfbooklist uses=<program> list=filter command=scons book
  3. Compare the newly generated figs with the corresponding reference figs stored in $RSFFIGS (defaulted to $RSFROOT/share/figs)
    sfbooklist uses=<program> list=filter command="sffiglist rsftest=y list=none" book
  4. Display any figs that are different from their $RSFFIGS counterparts
    sfbooklist uses=<program> list=filter command="sffiglist rsftest=y list=none show=diff" book