SCons - Revision history

Sfomel: /* Data processing flows with rsf.proj */

2018-11-21T22:39:39Z

Data processing flows with rsf.proj

← Older revision		Revision as of 22:39, 21 November 2018
Line 74:		Line 74:
	Obviously, one could also run similar commands with a shell script. What makes SCons convenient is the way it behaves when we make changes in the input files or in the script. Let us change, for example, the mute velocity parameter in the second <tt>Flow</tt> command. You can do that with an editor or on the command line as		Obviously, one could also run similar commands with a shell script. What makes SCons convenient is the way it behaves when we make changes in the input files or in the script. Let us change, for example, the mute velocity parameter in the second <tt>Flow</tt> command. You can do that with an editor or on the command line as
	<pre>		<pre>
	sed -i s/v0=0.31/v0=0.32/ SConstruct		bash$ sed -i s/v0=0.31/v0=0.32/ SConstruct
	</pre>		</pre>
	Now let us run <tt>scons</tt> again		Now let us run <tt>scons</tt> again

Sfomel at 22:35, 21 November 2018

2018-11-21T22:35:56Z

← Older revision		Revision as of 22:35, 21 November 2018
Line 41:		Line 41:
	== Data processing flows with <tt>rsf.proj</tt> ==		== Data processing flows with <tt>rsf.proj</tt> ==

	The [~~http~~://~~sourceforge~~.~~net~~/p/~~rsf~~/~~code~~/~~HEAD/tree/trunk~~/framework/rsf/proj.py rsf.proj] module provides SCons rules for Madagascar data processing workflows. An example <tt>SConstruct</tt> file is shown below and can be found in [http://ahay.org/RSF/book/bei/sg/denmark.html bei/sg/denmark]		The [https://github.com/ahay/src/blob/master/framework/rsf/proj.py rsf.proj] module provides SCons rules for Madagascar data processing workflows. An example <tt>SConstruct</tt> file is shown below and can be found in [http://ahay.org/RSF/book/bei/sg/denmark.html bei/sg/denmark]
	<syntaxhighlight lang="python">		<syntaxhighlight lang="python">
	from rsf.proj import *		from rsf.proj import *

Karl: Undo revision 3294 by Karl (talk)

2015-11-06T16:34:05Z

Undo revision 3294 by Karl (talk)

← Older revision		Revision as of 16:34, 6 November 2015
Line 43:		Line 43:
	The [http://sourceforge.net/p/rsf/code/HEAD/tree/trunk/framework/rsf/proj.py rsf.proj] module provides SCons rules for Madagascar data processing workflows. An example <tt>SConstruct</tt> file is shown below and can be found in [http://ahay.org/RSF/book/bei/sg/denmark.html bei/sg/denmark]		The [http://sourceforge.net/p/rsf/code/HEAD/tree/trunk/framework/rsf/proj.py rsf.proj] module provides SCons rules for Madagascar data processing workflows. An example <tt>SConstruct</tt> file is shown below and can be found in [http://ahay.org/RSF/book/bei/sg/denmark.html bei/sg/denmark]
	<syntaxhighlight lang="python">		<syntaxhighlight lang="python">
	~~<syntaxhighlight lanf="python">~~
	from rsf.proj import *		from rsf.proj import *

Karl: /* Data processing flows with rsf.proj */

2015-11-06T16:30:48Z

Data processing flows with rsf.proj

← Older revision		Revision as of 16:30, 6 November 2015
Line 43:		Line 43:
	The [http://sourceforge.net/p/rsf/code/HEAD/tree/trunk/framework/rsf/proj.py rsf.proj] module provides SCons rules for Madagascar data processing workflows. An example <tt>SConstruct</tt> file is shown below and can be found in [http://ahay.org/RSF/book/bei/sg/denmark.html bei/sg/denmark]		The [http://sourceforge.net/p/rsf/code/HEAD/tree/trunk/framework/rsf/proj.py rsf.proj] module provides SCons rules for Madagascar data processing workflows. An example <tt>SConstruct</tt> file is shown below and can be found in [http://ahay.org/RSF/book/bei/sg/denmark.html bei/sg/denmark]
	<syntaxhighlight lang="python">		<syntaxhighlight lang="python">
			<syntaxhighlight lanf="python">
	from rsf.proj import *		from rsf.proj import *

Karl: /* SConstruct commands */

2015-11-06T15:56:57Z

SConstruct commands

← Older revision		Revision as of 15:56, 6 November 2015
Line 88:		Line 88:

	====Fetch(<file[s]>,<directory>,[options])====		====Fetch(<file[s]>,<directory>,[options])====
	defines a rule for downloading data files from the specified directory on an external data server (by default) of from another directory on disk. The optional parameters that control its behavior are summarized below.		defines a rule for downloading data files from the specified directory on an external data server (by default) or from another directory on disk. The optional parameters that control its behavior are summarized below.

	{\|class="wikitable" align="center" cellspacing="0" border="1"		{\|class="wikitable" align="center" cellspacing="0" border="1"
Line 101:		Line 101:
	\|-		\|-
	\| top \|\| <tt>data</tt> \|\| name of the top data directory on the data server		\| top \|\| <tt>data</tt> \|\| name of the top data directory on the data server
			\|-
			\| dir \|\| None \|\| name of directory after top
			\|-
			\| usedatapath \|\| 1 \|\| usedatapath=1 - download to $DATAPATH with symbolic link.
			usedatapath=0 - download to pwd
	\|}		\|}

	In the example above, '''Fetch''' specifies the rule for getting the file <tt>wz.35.H</tt>: connect to the default data sever and download the file from the [http://www.reproducibility.org/data/wz data/wz] directory.		In the example above, '''Fetch''' specifies the rule for getting the file <tt>wz.35.H</tt>: connect to the default data sever and download the file from the [http://www.reproducibility.org/data/wz data/wz] directory.

			An example to Fetch with more parameters is:
			<syntaxhighlight lang="python">
			Fetch('KAHU-3D-PR3177-FM.3D.Final_Migration.sgy',
			dir='newzealand/Taranaiki_Basin/KAHU-3D',
			server='http://s3.amazonaws.com',
			top='open.source.geoscience/open_data',
			usedatapath=1)
			</syntaxhighlight>


	====Flow(<target[s]>,<source[s]>,<command>,[options])====		====Flow(<target[s]>,<source[s]>,<command>,[options])====

Sfomel: /* DBPageNotFoundError */

2014-04-08T21:42:18Z

DBPageNotFoundError

← Older revision		Revision as of 21:42, 8 April 2014
Line 272:		Line 272:
	=== DBPageNotFoundError===		=== DBPageNotFoundError===
	The scons database contains the information required so that scons executes only the parts of the data processing flow that were affected by changes in data, programs, and flows. Sometimes the database become corrupted. For example, when I ran out of disk space my scons stopped leaving a corrupted database. After deleting some files to create enough disk space running scons quickly fails with the message:		The scons database contains the information required so that scons executes only the parts of the data processing flow that were affected by changes in data, programs, and flows. Sometimes the database become corrupted. For example, when I ran out of disk space my scons stopped leaving a corrupted database. After deleting some files to create enough disk space running scons quickly fails with the message:
	<bash>		<syntaxhighlight lang="bash">
	scons: Reading SConscript files ...		scons: Reading SConscript files ...
	scons: done reading SConscript files.		scons: done reading SConscript files.
Line 278:		Line 278:
	scons: *** [31_81_IM.JPG] DBPageNotFoundError : (-30986, 'DB_PAGE_NOTFOUND: Requested page not found')		scons: *** [31_81_IM.JPG] DBPageNotFoundError : (-30986, 'DB_PAGE_NOTFOUND: Requested page not found')
	scons: building terminated because of errors.		scons: building terminated because of errors.
	</~~bash~~>		</syntaxhighlight>

	I was working in the directory $RSFSRC/book/data/alaska/line31-81. Scons keeps the database in the file $DATAPATH/data/alaska/line31-81/.sconsign.dbhash. It's a little tricky to find this file since it is hidden file in a directory below $DATAPATH~~. You can remove the $RSFSRC~~/~~book/data/alaska/line31-81 and restore from subversion, but the error will persist. You can test your SConstruct file in another directory and it will work just fine~~. Removing $DATAPATH/data/alaska/line31-81/.sconsign.dbhash fixed this problem.		I was working in the directory <tt>$RSFSRC/book/data/alaska/line31-81</tt>. Scons keeps the database in the file <tt>$DATAPATH/data/alaska/line31-81/.sconsign.dbhash</tt>. It's a little tricky to find this file since it is hidden file in a directory below <tt>$DATAPATH</tt>. Removing <tt>$DATAPATH/data/alaska/line31-81/.sconsign.dbhash</tt> fixed this problem.

Sfomel: /* Data processing flows with rsf.proj */

2014-04-08T21:40:18Z

Data processing flows with rsf.proj

← Older revision		Revision as of 21:40, 8 April 2014
Line 41:		Line 41:
	== Data processing flows with <tt>rsf.proj</tt> ==		== Data processing flows with <tt>rsf.proj</tt> ==

	The [http://~~rsf.svn.~~sourceforge.net/~~viewvc~~/rsf/trunk/framework/rsf/proj.py~~?&view=markup~~ rsf.proj] module provides SCons rules for Madagascar data processing workflows. An example <tt>SConstruct</tt> file is shown below and can be found in [http://~~rsf~~.~~svn.sourceforge.net~~/~~viewvc/rsf/trunk~~/book/bei/sg/denmark~~/SConstruct?&view=markup~~ bei/sg/denmark]		The [http://sourceforge.net/p/rsf/code/HEAD/tree/trunk/framework/rsf/proj.py rsf.proj] module provides SCons rules for Madagascar data processing workflows. An example <tt>SConstruct</tt> file is shown below and can be found in [http://ahay.org/RSF/book/bei/sg/denmark.html bei/sg/denmark]
	<python>		<syntaxhighlight lang="python">
	from rsf.proj import *		from rsf.proj import *

Line 56:		Line 56:

	End()		End()
	</~~python~~>		</syntaxhighlight>
	Note that <tt>SConstruct</tt> by itself does not do any job other than setting rules for building different targets. The targets get built when one executes <tt>scons</tt> on the command line. Running <tt>scons</tt> produces		Note that <tt>SConstruct</tt> by itself does not do any job other than setting rules for building different targets. The targets get built when one executes <tt>scons</tt> on the command line. Running <tt>scons</tt> produces
	<pre>		<pre>
Line 87:		Line 87:
	=== SConstruct commands ===		=== SConstruct commands ===

	====~~'''~~Fetch(<file[s]>,<directory>,[options])====		====Fetch(<file[s]>,<directory>,[options])====
	defines a rule for downloading data files from the specified directory on an external data server (by default) of from another directory on disk. The optional parameters that control its behavior are summarized below.		defines a rule for downloading data files from the specified directory on an external data server (by default) of from another directory on disk. The optional parameters that control its behavior are summarized below.

Line 206:		Line 206:

	Running the example above with <tt>TIMER=y</tt> produces		Running the example above with <tt>TIMER=y</tt> produces
	<bash>		<syntaxhighlight lang="bash">
	bash$ scons -Q TIMER=y		bash$ scons -Q TIMER=y
	/usr/bin/time < wind.rsf /RSF/bin/sfmutter v0=0.32 half=n > mute.rsf		/usr/bin/time < wind.rsf /RSF/bin/sfmutter v0=0.32 half=n > mute.rsf
Line 217:		Line 217:
	0.06user 0.03system 0:00.06elapsed 135%CPU (0avgtext+0avgdata 444416maxresident)k		0.06user 0.03system 0:00.06elapsed 135%CPU (0avgtext+0avgdata 444416maxresident)k
	0inputs+0outputs (1739major+0minor)pagefaults 0swaps		0inputs+0outputs (1739major+0minor)pagefaults 0swaps
	</~~bash~~>		</syntaxhighlight>
	In other words, every shell command is preceded by the Unix [http://en.wikipedia.org/wiki/Time_%28Unix%29 time] utility to measure the CPU time of the process.		In other words, every shell command is preceded by the Unix [http://en.wikipedia.org/wiki/Time_%28Unix%29 time] utility to measure the CPU time of the process.

	Running the example above with <tt>CHECKPAR=y</tt>, we will not see any difference. Suppose, however, that we made a typo in specifying one of the parameters, for example, by using <tt>v1=</tt> instead of <tt>v0=</tt> in the arguments to <tt>sfmutter</tt>.		Running the example above with <tt>CHECKPAR=y</tt>, we will not see any difference. Suppose, however, that we made a typo in specifying one of the parameters, for example, by using <tt>v1=</tt> instead of <tt>v0=</tt> in the arguments to <tt>sfmutter</tt>.
	<bash>		<syntaxhighlight lang="bash">
	bash$ sed -i s/v0=0.31/v1=0.31/ SConstruct		bash$ sed -i s/v0=0.31/v1=0.31/ SConstruct
	bash$ scons -Q CHECKPAR=y		bash$ scons -Q CHECKPAR=y
	No parameter "v1" in sfmutter		No parameter "v1" in sfmutter
	Failed on "mutter v1=0.31 half=n"		Failed on "mutter v1=0.31 half=n"
	</~~bash~~>		</syntaxhighlight>
	The parameter error gets detected by <tt>scons</tt> before anything is executed.		The parameter error gets detected by <tt>scons</tt> before anything is executed.

Sfomel: /* Useful SCons options */

2014-04-08T21:35:29Z

Useful SCons options

← Older revision		Revision as of 21:35, 8 April 2014
Line 13:		Line 13:

	* '''scons -n''' (no exec) outputs the commands required for building the specified target (or the default targets if no target is specified) without actually executing them. It can be used to generate a shell script out of <tt>SConstruct</tt> script, as follows:		* '''scons -n''' (no exec) outputs the commands required for building the specified target (or the default targets if no target is specified) without actually executing them. It can be used to generate a shell script out of <tt>SConstruct</tt> script, as follows:
	<bash>		<syntaxhighlight lang="bash">
	scons -nQ [target] > script.sh		scons -nQ [target] > script.sh
	</~~bash~~>		</syntaxhighlight>

	== Compilation ==		== Compilation ==

Sfomel: /* Compilation */

2014-04-08T21:35:00Z

Compilation

← Older revision		Revision as of 21:35, 8 April 2014
Line 20:		Line 20:

	SCons was designed primarily for compiling software code. An <tt>SConstruct</tt> file for compilation may look like		SCons was designed primarily for compiling software code. An <tt>SConstruct</tt> file for compilation may look like
	<python>		<syntaxhighlight lang="python">
	env = Environment()		env = Environment()
	env.Append(CPPFLAGS=['-Wall','-g'])		env.Append(CPPFLAGS=['-Wall','-g'])
	env.Program('hello',['hello.c', 'main.c'])		env.Program('hello',['hello.c', 'main.c'])
	</~~python~~>		</syntaxhighlight>

	and produce something like		and produce something like

Karl at 17:28, 24 August 2013

2013-08-24T17:28:24Z

← Older revision		Revision as of 17:28, 24 August 2013
Line 268:		Line 268:

	The default targret is set to be '''scons pdf'''.		The default targret is set to be '''scons pdf'''.

			== Errors and Debugging ==
			=== DBPageNotFoundError===
			The scons database contains the information required so that scons executes only the parts of the data processing flow that were affected by changes in data, programs, and flows. Sometimes the database become corrupted. For example, when I ran out of disk space my scons stopped leaving a corrupted database. After deleting some files to create enough disk space running scons quickly fails with the message:
			<bash>
			scons: Reading SConscript files ...
			scons: done reading SConscript files.
			scons: Building targets ...
			scons: *** [31_81_IM.JPG] DBPageNotFoundError : (-30986, 'DB_PAGE_NOTFOUND: Requested page not found')
			scons: building terminated because of errors.
			</bash>

			I was working in the directory $RSFSRC/book/data/alaska/line31-81. Scons keeps the database in the file $DATAPATH/data/alaska/line31-81/.sconsign.dbhash. It's a little tricky to find this file since it is hidden file in a directory below $DATAPATH. You can remove the $RSFSRC/book/data/alaska/line31-81 and restore from subversion, but the error will persist. You can test your SConstruct file in another directory and it will work just fine. Removing $DATAPATH/data/alaska/line31-81/.sconsign.dbhash fixed this problem.