2008 Implementation Workshop - Revision history

Nick: /* Summary */

2008-10-15T21:19:39Z

Summary

← Older revision		Revision as of 21:19, 15 October 2008
Line 20:		Line 20:

	==Summary==		==Summary==
	~~To be continued~~		The intended "deliverables" of this meeting were:

			* A stronger geophysical open-source community
			* A short-term Madagascar road map, created by discussing, extending, and prioritizing the feature request list
			* A real move towards version 1.0

			The community was indeed strengthened by participants getting to know each other.

			The second goal was not only fulfilled, but far exceeded. Not one road map was created, but as many road maps as participants.

			The real move to version 1.0 was done by Joe, who completed <tt>vplotdiff</tt>, allowing automatic testing. Joe also played a crucial role in clarifying and solving the IP issues created by Madagascar containing some code from 1994 versions of SEPlib and vplot. In the correspondence that followed, the copyright holders and Stanford University's Office of Technology Licensing have agreed to release the code under GPL (for the portions of SEPlib used) and LGPL (for vplot).

			There were other achievements as well. Bugs in vplot were fixed, documentation was created, and the source code tree was partly redesigned.

Nick: to be continued

2008-10-15T17:29:46Z

to be continued

@@ Line 1: / Line 1: @@
 [[Image:M8rsprint2.jpg|right|]]
 The workshop took place over the Memorial Day weekend, from Fri May 23 to Mon May 26, at the [http://www.mines.edu/academic/geophysics/prospective_students_maps.html Geophysics Department] of [http://www.mines.edu Colorado School of Mines], in [http://en.wikipedia.org/wiki/Golden,_Colorado Golden, Colorado].
-participants worked together on implementing features in Madagascar.
+==Participants==
-The "deliverables" of this meeting are:
+==Summary==
-* A stronger geophysical open-source community
+To be continued

Nick at 17:38, 14 October 2008

2008-10-14T17:38:03Z

← Older revision		Revision as of 17:38, 14 October 2008
Line 1:		Line 1:
	~~==Madagascar 2008 Implementation Workshop: Towards full automation and better robustness==~~

	[[Image:M8rsprint2.jpg\|right\|]]		[[Image:M8rsprint2.jpg\|right\|]]
	'''UNDER CONSTRUCTION. This page is being changed from an invitation into a description of the meeting.'''		'''UNDER CONSTRUCTION. This page is being changed from an invitation into a description of the meeting.'''

Nick: removed motivational illustration

2008-10-14T17:19:49Z

removed motivational illustration

← Older revision		Revision as of 17:19, 14 October 2008
Line 1:		Line 1:
	==Madagascar 2008 Implementation Workshop: Towards full automation and better robustness==		==Madagascar 2008 Implementation Workshop: Towards full automation and better robustness==
	~~[[Image:Fotolia_800161_XS.jpg\|right\|]]~~
	[[Image:M8rsprint2.jpg\|right\|]]		[[Image:M8rsprint2.jpg\|right\|]]
	'''UNDER CONSTRUCTION. This page is being changed from an invitation into a description of the meeting.'''		'''UNDER CONSTRUCTION. This page is being changed from an invitation into a description of the meeting.'''

Nick: /* Pool of suggested features (to be moved to Feature requests/another page) */ update features

2008-10-13T20:48:46Z

Pool of suggested features (to be moved to Feature requests/another page): update features

← Older revision		Revision as of 20:48, 13 October 2008
Line 20:		Line 20:
	## Vplot diffs: These would allow the m8r project to fulfill one of its main goals – having fully automatic regression tests. See [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]		## Vplot diffs: These would allow the m8r project to fulfill one of its main goals – having fully automatic regression tests. See [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
	## rsfbook completion, also from the [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]		## rsfbook completion, also from the [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
	~~## Moving the wiki to Dreamhost so that it actually functions as a wiki again~~
	## Automatically sync-ing the Wiki "Guide to programs" with the self-doc;		## Automatically sync-ing the Wiki "Guide to programs" with the self-doc;
	## Saving a static copy of the wiki so that consulting important parts of the documentation does not require a centralized, brittle software stack (local internet connectivity + remote web server + php + SQL + Mediawiki subject to constant spam attacks) to be running at that very moment. That would also allow splitting the "Guide to Programs" into one-page-per-program files, to be concatenated together into one searchable page in the static copy of the wiki, and for user-contributed parts to be included in the HTML self-doc;		## Saving a static copy of the wiki so that consulting important parts of the documentation does not require a centralized, brittle software stack (local internet connectivity + remote web server + php + SQL + Mediawiki subject to constant spam attacks) to be running at that very moment. That would also allow splitting the "Guide to Programs" into one-page-per-program files, to be concatenated together into one searchable page in the static copy of the wiki, and for user-contributed parts to be included in the HTML self-doc;
	# '''Features that make m8r more user-friendly'''		# '''Features that make m8r more user-friendly'''
			## OO-based or functional-style configuration process, in which each directory/program is queried about its requirements and testing is done as appropriate
	## A sane configuration script for the tex2pdf reproducible paper engine, that pulls a complete list of missing dependencies and has a way of extracting them from TeX Live (or instructing the user to do so). Right now the huge size of TeX Live (>1Gb) and its fast pace of advancement precludes its installation on legacy systems (i.e. RHEL 2,3,4). The goal is to make the pdf paper generating mechanism as easy to install as the rest of m8r, i.e. configuration either fails with a helpful error message or everything installs fine and it Just Works.		## A sane configuration script for the tex2pdf reproducible paper engine, that pulls a complete list of missing dependencies and has a way of extracting them from TeX Live (or instructing the user to do so). Right now the huge size of TeX Live (>1Gb) and its fast pace of advancement precludes its installation on legacy systems (i.e. RHEL 2,3,4). The goal is to make the pdf paper generating mechanism as easy to install as the rest of m8r, i.e. configuration either fails with a helpful error message or everything installs fine and it Just Works.
	## sfdatadoc – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]		## sfdatadoc – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]

Nick: New page: ==Madagascar 2008 Implementation Workshop: Towards full automation and better robustness== right| right| '''UNDER CONSTRUCTION. Thi...

2008-10-05T20:45:58Z

New page: ==Madagascar 2008 Implementation Workshop: Towards full automation and better robustness== right| right| '''UNDER CONSTRUCTION. Thi...

New page

==Madagascar 2008 Implementation Workshop: Towards full automation and better robustness==
[[Image:Fotolia_800161_XS.jpg|right|]]
[[Image:M8rsprint2.jpg|right|]]
'''UNDER CONSTRUCTION. This page is being changed from an invitation into a description of the meeting.'''

The workshop took place over the Memorial Day weekend, from Fri May 23 to Mon May 26, at the [http://www.mines.edu/academic/geophysics/prospective_students_maps.html Geophysics Department] of [http://www.mines.edu Colorado School of Mines], in [http://en.wikipedia.org/wiki/Golden,_Colorado Golden, Colorado].

participants worked together on implementing features in Madagascar.

The "deliverables" of this meeting are:
* A stronger geophysical open-source community
* A short-term Madagascar road map, created by discussing, extending, and prioritizing the feature request list
* A real move towards version 1.0

===Pool of suggested features (to be moved to Feature requests/another page)===
The m8r features under consideration can be grouped into several categories, which are listed below. The grouping attempts to describe the software equivalent of [http://en.wikipedia.org/wiki/Maslow's_hierarchy_of_needs Maslow's pyramid of needs]. First the basics (reproducibility, I/O, parallelization, graphics) should be consolidated, then efforts should move up to numerical tools like solvers, FFTs and transposes, then up to widely-used geophysical algorithms. Interested parties are invited to brainstorm below!

# '''Features that provide functionality that is needed in order to have a minimalistic fully-automated m8r project setup'''
## Vplot diffs: These would allow the m8r project to fulfill one of its main goals – having fully automatic regression tests. See [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
## rsfbook completion, also from the [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
## Moving the wiki to Dreamhost so that it actually functions as a wiki again
## Automatically sync-ing the Wiki "Guide to programs" with the self-doc;
## Saving a static copy of the wiki so that consulting important parts of the documentation does not require a centralized, brittle software stack (local internet connectivity + remote web server + php + SQL + Mediawiki subject to constant spam attacks) to be running at that very moment. That would also allow splitting the "Guide to Programs" into one-page-per-program files, to be concatenated together into one searchable page in the static copy of the wiki, and for user-contributed parts to be included in the HTML self-doc;
# '''Features that make m8r more user-friendly'''
## A sane configuration script for the tex2pdf reproducible paper engine, that pulls a complete list of missing dependencies and has a way of extracting them from TeX Live (or instructing the user to do so). Right now the huge size of TeX Live (>1Gb) and its fast pace of advancement precludes its installation on legacy systems (i.e. RHEL 2,3,4). The goal is to make the pdf paper generating mechanism as easy to install as the rest of m8r, i.e. configuration either fails with a helpful error message or everything installs fine and it Just Works.
## sfdatadoc – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
## Go through all the programs to make sure that there are no undocumented or unclearly specified parameters. Also look at inconsistencies (when parameters with similar meaning get different names in different programs). Establish and implement coding conventions.
## Binary packages – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker] (''Note 1: there has been [http://osdir.com/ml/programming.tools.scons.devel/2006-06/msg00021.html noise] since 2006 on SCons developer mail lists about providing a SCons command that would automatically create RPMs and maybe debs as well. Anyone know the implementation status of that? Note 2: Packages are typically split into a "package core" providing executables and help for them and a "package-devel" providing a [http://en.wikipedia.org/wiki/Software_development_kit Software development kit] consisting of development libraries, headers and other include files, API documentation, etc. Doing things this way would require deep changes of the current Madagascar build mechanisms'')
## man pages – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
## Decide how to ensure that the [[Task-centric program list]] is always sync-ed with the source code. (keywords picked up by rsfdoc?)
## Create "Migrant's dictionary" for newcomers from other packages, by using ''their'' task-centric pages such as [http://sepwww.stanford.edu/oldsep/cliner/files/suhelp/suhelp.html this one] to show which m8r programs correspond to which programs from their "home country"
## TKSU-based GUI – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker] (''Note: the Madagascar plugin for OpendTect is also [http://www2.opendtect.org/wiki/madagascar/ on track]'')
# '''Features that extend m8r's capabilities'''
## M8r-based programming
### Java API (more details on the [[GSOC2008]] page)
### Extending the Python interface (more details on the [[GSOC2008]] page)
### Finishing the Octave interface
### A tool to convert "normal" rsf to a "transfer-ready" rsf that has the binary converted to portable XDR, then gzipp-ed, the number of bytes of the compressed file and the MD5sum of the binary and of the compressed binary in the ASCII header, and this is shown by a keyword like form=xdr.gz . I checked with various types and sizes of data and even in the worst-case scenario, a factor of 2 in compression is attained. This would allow safe, fully-automatic data transfers.
### Introduce a sane way to control the optimization level for all languages in a m8r build, using a single flag. A researcher using Madagascar should not be an expert in compiler usage, he should just set the level. For example:
###* Level A, have all warnings and debugging info turned on. Link to rsflib version compiled similarly.
###* Level B, compile with -O2 and link against rsflib version compiled similarly.
###* Level C, same as level B, and also detect if compiler has interprocedural optimization abilities. If yes, compile everything with appropriate flags. If compiler cannot deal with optimizing across multiple files, have Python feed to compiler concatenated source code files that make a single program instead of "include" statements.
###* Level D, same as level C, and also have a special SCons "Optiflow" rule type, in which programs to be run are actually recompiled with parameters hardwired into them, so that optimizations such as dead branch elimination and loop unrolling are feasible. Use SCons to perform compilations in parallel. A fully-optimized compilation of most of m8r for a single flow may seem like overkill, but is necessary in the case of huge datasets that take weeks and months to process.
###* Level E, same as level D, but compile with -O3 and also, where available, use special alternative versions of the codes optimized by hand and with less error checking.
## Graphics
### Bezier curves in vplot – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
### xtpen antialiasing – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
### bargraph – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
### graph3 completion – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
## Geophysical/numerical tools
### Harlan's CG – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
### conjgrad extensions – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
### kirmod in layered media – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
### minidds – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]

[[Image:M8rmap.png]]