rough documentation outline

31 files changed, 714 insertions, 0 deletions

diff --git a/doc/0_00_welcome.adoc b/doc/0_00_welcome.adoc
new file mode 100644
index 00000000..70bdc90d
--- /dev/null
+++ b/doc/0_00_welcome.adoc
@@ -0,0 +1,46 @@
+Welcome
+=======
+
+== Colm = COmputer Language Machinery
+
+Colm is a programming language designed for the analysis and http://www.program-transformation.org/Transform/TransformationSystems[transformation of computer languages].
+Colm is influenced primarily by http://www.txl.ca/[TXL].
+
+=== What is a transformation language?
+
+A transformation language has a type system based on formal languages. 
+Rather than define classes or data structures, one defines grammars. 
+A parser is constructed automatically from the grammar, and the parser is used for two purposes: 
+* to parse the input language,
+* and to parse the structural patterns in the program that performs the analysis. 
+
+In this setting, grammar-based parsing is critical because it guarantees that both the input and the structural patterns are parsed into trees from the same set of types, allowing comparison.
+
+=== Colm's features
+
+Colm is not-your-typical-scripting-language (TM):
+* Colm's main contribution lies in the parsing method.
+  Colm's parsing engine is generalized, but it also allows for the construction of arbitrary global data structures that can be queried during parsing.
+  In other generalized methods, construction of global data requires some very careful consideration because of inherent concurrency in the parsing method.
+  It is such a tricky task that it is often avoided altogether and the problem is deferred to a post-parse disambiguation of the parse forest.
+* By default Colm will create an elf exectuable that be used standalone for that actual transformations.
+* Colm is a static and strong typed scripting language.
+* Colms' is very tiny and fast and can easily be embedded/linked with c/cpp programs.
+* Colm's runtime is a stackbased VM that starts with the bare minium of the language and bootstraps itself.
+* creates aVM is very tycan be embedded in C as it It runs in a embeddable vm, the language is bootstrapped.
+
+=== Where is colm used?
+
+Colm is developed and used intensively by http://www.colm.net/[Colm Networks] to develop fast network traffic automata and systems for traffic identification, decoding, pattern matching, and extraction of security events. 
+But colm is also the driving force in http://www.colm.net/open-source/ragel/[the Ragel State Machine Compiler]
+
+=== What is colm's history?
+
+Colm's development started by https://twitter.com/ehdtee[Adrian Thurston] during his http://www.colm.net/files/colm/thurston-phdthesis.pdf[Ph.D. thesis] period after intensive study of http://research.cs.queensu.ca/~cordy/Papers/TC_SCAM06_ETXL.pdf[TXL].
+
+
+=== When not to use Colm
+
+Colm is meant to create executables or object files that can be linked in other programs. 
+This make is ideal for tasks like high performance transformations, but not very convienient for throwaway-oneliners that are common with tools like 'sed' or 'awk'.
+
diff --git a/doc/0_01_installing.adoc b/doc/0_01_installing.adoc
new file mode 100644
index 00000000..a55bfa9c
--- /dev/null
+++ b/doc/0_01_installing.adoc
@@ -0,0 +1,37 @@
+Installing
+=========
+
+Installing on a linux system is a breeze
+
+[source,bash]
+----
+wget -q http://www.colm.net/files/colm/thurston-phdthesis.pdf
+wget -q http://www.colm.net/files/colm/colm-0.13.0.4.tar.gz
+tar -xaf colm-0.13.0.4.tar.gz
+cd colm-0.13.0.4
+./configure --prefix=/opt/colm
+make
+sudo make install
+----
+
+When we do:
+
+ /opt/colm/bin/colm
+
+We get:
+
+ error: colm: colm: no input file given
+
+It works!
+
+== First impression
+
+When we look a little bit closer we see that colm:
+
+* is able to be build as a static and/or shared libray.
+* is licenced under GPL 2
+* is equiped with a vim syntax highlighting file
+* is using the aapl (LGPL 2.1 licenced) library from Adrian Thurston (just like ragel does).
+* There is one file in the repository that stands out: 'colm.lm'
+  In the Ragel repository there are also serveral '.lm' files. And it's syntax looks like the colm language that is presented in the thesis.
+
diff --git a/doc/0_02_vim_syntax.adoc b/doc/0_02_vim_syntax.adoc
new file mode 100644
index 00000000..f29ca33c
--- /dev/null
+++ b/doc/0_02_vim_syntax.adoc
@@ -0,0 +1,17 @@
+Vim syntax
+==========
+
+To activatecd colm syntax highlighting in vi
+
+ cp /path_to_extracted_files/lcolm.vim  ~/.vim/syntax
+
+[source,vim]
+.~/.vimrc
+----
+" Work with colm
+au BufRead,BufNewFile *.lm set filetype=colm
+----
+
+When we open an '.lm' file, we will see some colors
+
+ vi src/colm.lm
diff --git a/doc/0_03_commandline.adoc b/doc/0_03_commandline.adoc
new file mode 100644
index 00000000..9613ab11
--- /dev/null
+++ b/doc/0_03_commandline.adoc
@@ -0,0 +1,25 @@
+Commandline
+===========
+
+Let's start colm with the '--help' command line argument.
+
+ colm --help
+
+NOTE: This reflects the development version 0.13.0.4;
+
+
+```usage: colm [options] file
+general:
+   -h, -H, -?, --help   print this usage and exit
+   -v --version         print version information and exit
+   -o <file>            write output to <file>
+   -c                   compile only (don't produce binary)
+   -e <file>            write C++ export header to <file>
+   -x <file>            write C++ export code to <file>
+   -m <file>            write C++ commit code to <file>
+   -a <file>            additional code file to include in output program
+```
+
+This reveals us some more insights: it reads a 'colm' file and creates a object file with eventually cpp/h/x code.
+
+
diff --git a/doc/0_04_hello_world.adoc b/doc/0_04_hello_world.adoc
new file mode 100644
index 00000000..4f9bea29
--- /dev/null
+++ b/doc/0_04_hello_world.adoc
@@ -0,0 +1,59 @@
+Hello world
+===========
+
+The obligatory 'hello world' program:
+
+[source,chapel]
+.hello_world.lm
+----
+print "hello world" "\n"
+----
+
+We run it with:
+
+[source,bash]
+----
+/opt/colm/bin/colm hello_world.lm
+----
+
+This creates a executable chmod+x file with the same name:
+
+
+[source,bash]
+----
+ls -l hello_world
+----
+
+ -rwxr-xr-x 1 peter peter 29848 Nov  2 10:06 /tmp/hello_world
+
+When we execute it:
+
+ ./hello_world
+
+We'll see:
+
+ hello world
+
+We can strip the file to check if we can reduce the executable.
+[source,bash]
+----
+strip ./hello_world
+ls -l hello_words
+----
+
+ -rwxr-xr-x 1 peter peter 10360 Nov  2 10:10 /tmp/hello_world
+
+== Deja-vu: python2-python3
+TIP: It turns out that print is also a function that can have multiple arguments.
+
+[source,chapel]
+.hello_world_ext.lm]
+----
+print( 'hello ', "world" "\r\n" )
+----
+
+We also notice that:
+* the quotes can be single and double
+* there is no need for a concat operator
+* the whitespace is not significant
+* the newlines '\n' appear to be '\r\n'
diff --git a/doc/0_05_fizzbuzz.adoc b/doc/0_05_fizzbuzz.adoc
new file mode 100644
index 00000000..c6f439d9
--- /dev/null
+++ b/doc/0_05_fizzbuzz.adoc
@@ -0,0 +1,111 @@
+FizzBuzz
+========
+
+== Slightly modified example of thesis: Figure 4.4, page 87
+
+The colm language has evolved since it has been described in the thesis.
+With some modifications we can reactivate this example.
+
+[source,chapel]
+.figure_44.lm
+----
+i: int = 0
+j: int = i
+
+while ( i < 10 ) {
+    if ( i * ( 10 - i ) < 20 ) {
+        print ( "hello ", i, ' ', j , '\n' )
+        j = j+ 1
+    }
+    i = i + 1
+}
+----
+
+Please Note:
+
+* the syntax is very c-ish
+* the variables are defined with their type
+* there is no postfix increment operator (i = i +1)
+
+[source,bash]
+----
+/opt/colm/bin/colm fizzbuzz.lm
+./fizzbuzz
+----
+
+That gives us:
+----
+hello 0 0
+hello 1 1
+hello 2 2
+hello 8 3
+hello 9 4
+----
+
+== Real FizzBuzz
+
+The fizzbuzz test is often used to check is someone has programming skils.
+It is the next logical step to 'hello world'.
+
+[source,chapel]
+.fizzbuzz.lm
+----
+int modulo( value:int, div:int) {
+    times:int = value / div
+    return value - ( times * div )
+}
+
+i:int = 0
+while( i < 20 ) {
+    mod5:int = modulo( i, 5 )
+    mod3:int = modulo( i, 3 )
+    if ( mod5 == 0 && mod3 == 0 ) {
+        print( "FizzBuzz\n" )
+    } elsif( mod5 == 0 ) {
+        print( "Buzz\n" )
+    } elsif( mod3 == 0 ) {
+        print( "Fizz\n" )
+    } else {
+        print( i, "\n" )
+    }
+    i = i + 1
+}
+----
+
+It appears that there is no modulo operator ('%').
+Therefor we'll resort to a function.
+Writing a function seems rather straight forward
+
+Please note:
+* that '&&' is working.
+* The return type is needed, but if 'nil' is retuned by default.
+
+[source,bash]
+----
+/opt/colm/bin/colm fizzbuzz.lm
+./fizzbuzz
+----
+
+That gives us
+----
+FizzBuzz
+1
+2
+Fizz
+4
+Buzz
+Fizz
+7
+8
+Fizz
+Buzz
+11
+Fizz
+13
+14
+FizzBuzz
+16
+17
+Fizz
+19
+----
diff --git a/doc/0_06_scope.adoc b/doc/0_06_scope.adoc
new file mode 100644
index 00000000..5025052c
--- /dev/null
+++ b/doc/0_06_scope.adoc
@@ -0,0 +1,121 @@
+Scope
+=====
+
+We saw in the previous paragraph that functions can be used, and that they can have parameter.
+This forces us to clarify 'scope'.
+
+[source,chapel]
+./scope.lm
+----
+str d (where:str) {
+    print( "in D ", where, "\n")
+    where = "d"
+    print( "in D ", where, "\n")
+}
+
+str c ( ) {
+    print( "in C ", where_g, "\n")
+    where_g = "c"
+    print( "in C ", where_g, "\n")
+}
+
+str b ( where:str ) {
+    print( "in B ", where, "\n")
+    where = "b" 
+    print( "in B ", where, "\n")
+}
+
+str a( where:str ) {
+    print( "in A ", where, "\n")
+    where = "a" 
+    b( where )
+    print( "in A ", where, "\n")
+}
+
+where: str =  "global"
+print( "in global ", where, "\n")
+a( where )
+print( "in global ", where, "\n")
+global where_g:str
+c( )
+print( "in global ", where_g, "\n")
+----
+
+We run it with
+[source,bash]
+----
+/opt/colm/bin/colm scope.lm
+./scope
+----
+
+That gives us:
+----
+in global global
+in A global
+in B a
+in B b
+in A a
+in global global
+in C NIL
+in C c
+in global c
+----
+
+The thesis also mentions that variables can be passed by reference instead of by value.
+
+[source,chapel]
+.nested_scope.lm
+----
+str a( where:str ) {
+    print( "before block1 ", where, "\n" )
+    while(true) {
+        where = "block1"
+        print( "in block1 ", where, "\n" )
+        i:int = 0
+        while( true ) {
+            where =  where + "a"
+            print( "in loop ", where, "\n" )
+            break
+        }
+        print( "in block1 ", where, "\n" )
+        break
+    }
+    print( "in A ", where, "\n" )
+    return where
+}
+
+where: str =  "global"
+print( "in global ", where, "\n" )
+a( where )
+print( "in global ", where, "\n" )
+----
+
+That gives us:
+----
+in global global
+in A global
+in B a
+in B b
+in A a
+in global global
+in C NIL
+in C c
+in global c
+----
+
+[source,bash]
+----
+/opt/colm/bin/colm nested_scope.lm
+./nested_scope
+----
+
+It seems that this is still the case.
+----
+in global global
+before block1 global
+in block1 block1
+in loop block1a
+in block1 block1a
+in A block1a
+in global global
+----
diff --git a/doc/1_00_language_overview.adoc b/doc/1_00_language_overview.adoc
new file mode 100644
index 00000000..8628a0ad
--- /dev/null
+++ b/doc/1_00_language_overview.adoc
@@ -0,0 +1,6 @@
+Language overview
+=================
+
+== TODO
+
+
diff --git a/doc/1_01_statements.adoc b/doc/1_01_statements.adoc
new file mode 100644
index 00000000..f5bf8880
--- /dev/null
+++ b/doc/1_01_statements.adoc
@@ -0,0 +1,6 @@
+Statements
+==========
+
+== TODO
+
+
diff --git a/doc/1_02_expressions.adoc b/doc/1_02_expressions.adoc
new file mode 100644
index 00000000..223c5124
--- /dev/null
+++ b/doc/1_02_expressions.adoc
@@ -0,0 +1,6 @@
+Expressions
+===========
+
+== TODO
+
+
diff --git a/doc/1_03_control_flow.adoc b/doc/1_03_control_flow.adoc
new file mode 100644
index 00000000..e8c045d3
--- /dev/null
+++ b/doc/1_03_control_flow.adoc
@@ -0,0 +1,6 @@
+Control flow
+============
+
+== TODO
+
+
diff --git a/doc/1_04_functions.adoc b/doc/1_04_functions.adoc
new file mode 100644
index 00000000..efd80131
--- /dev/null
+++ b/doc/1_04_functions.adoc
@@ -0,0 +1,36 @@
+Functions
+=========
+
+The 'FizzBuzz' example gave us an good overview.
+
+
+== Arguments 'by reference'
+
+We can also pass parameters by reference.
+This enables us to change the variable with in the function.
+
+
+[source,chapel]
+./reference.lm
+----
+str sa( where:ref < str > ) {
+    print( "in SA ", where, "\n" )
+    where = "sa"
+    print( "in SA ", where, "\n" )
+}
+
+where: str =  "global"
+print( "in global ", where, "\n" )
+sa( where )
+print( "in global ", where, "\n" )
+----
+
+Compiling and running would give us:
+
+----
+in global global
+in SA global
+in SA sa
+in global sa
+----
+
diff --git a/doc/1_05_variables.adoc b/doc/1_05_variables.adoc
new file mode 100644
index 00000000..d4a4e59a
--- /dev/null
+++ b/doc/1_05_variables.adoc
@@ -0,0 +1,6 @@
+Variables
+=========
+
+== TODO
+
+
diff --git a/doc/1_06_iterators.adoc b/doc/1_06_iterators.adoc
new file mode 100644
index 00000000..d20df9bc
--- /dev/null
+++ b/doc/1_06_iterators.adoc
@@ -0,0 +1,6 @@
+Iterators
+=========
+
+== TODO
+
+
diff --git a/doc/2_00_types.adoc b/doc/2_00_types.adoc
new file mode 100644
index 00000000..a1e54dd2
--- /dev/null
+++ b/doc/2_00_types.adoc
@@ -0,0 +1,6 @@
+Types
+=====
+
+== TODO
+
+
diff --git a/doc/2_01_basic.adoc b/doc/2_01_basic.adoc
new file mode 100644
index 00000000..d9a3f84f
--- /dev/null
+++ b/doc/2_01_basic.adoc
@@ -0,0 +1,4 @@
+Basic Types
+===========
+
+== TODO
diff --git a/doc/2_02_list_map_struct_alias.adoc b/doc/2_02_list_map_struct_alias.adoc
new file mode 100644
index 00000000..f10c1ec8
--- /dev/null
+++ b/doc/2_02_list_map_struct_alias.adoc
@@ -0,0 +1,77 @@
+List, Map, Struct and alias
+===========================
+
+Next to the basic types, there are the normal things that we can expect from scripting languages.
+As colm is static and strong typed, it is very convienient to use alias as well.
+
+An example is probably much clearer then 1000 loc.
+
+[source,chapel]
+.poker.md
+----
+alias Value_t map<int,str>
+values:Value_t = new Value_t()
+
+values->insert(0, "Ace")
+values->insert(1, "1")
+values->insert(2, "2")
+values->insert(3, "3")
+values->insert(4, "4")
+values->insert(5, "5")
+values->insert(6, "6")
+values->insert(7, "7")
+values->insert(8, "8")
+values->insert(9, "9")
+values->insert(10, "Ten")
+values->insert(11, "Jack")
+values->insert(12, "Queen")
+values->insert(13, "King")
+
+alias Suit_t map<int,str> 
+suit:Suit_t = new Suit_t()
+suit->insert(1, "hearts")
+suit->insert(2, "spades")
+suit->insert(3, "diamonds")
+suit->insert(4, "clubs")
+
+struct Card_t
+    s:int
+    v:int
+end
+
+alias Hand_t list<Card_t>
+
+struct Person_t
+    name:str
+    age:int
+    hand:Hand_t
+end
+
+john:Person_t
+
+john = new Person_t()
+john->name = "john"
+john->age = 18
+john->hand = new Hand_t()
+
+card:Card_t = new Card_t()
+card->s = 2
+card->v = 13
+john->hand->push(card)
+
+print("ok ", john->name, " ", john->age, "\n")
+for card:Card_t in john->hand {
+    print("\n\t", suit->find(card->s), " ", values->find(card->v), "\n")
+}
+----
+
+When we run this we get:
+
+----
+ok john 18
+
+	spades King
+----
+
+
+NOTE: this also illustrates how to iterate through a 'list' and access elements in a 'map'.
diff --git a/doc/2_03_def.adoc b/doc/2_03_def.adoc
new file mode 100644
index 00000000..bd3b87aa
--- /dev/null
+++ b/doc/2_03_def.adoc
@@ -0,0 +1,50 @@
+Def
+===
+
+The 'def' is where colm realy shines.
+A 'def' is somewhere between a struct and a regular expression.
+Again one example is much more clearer.
+
+[source,chapel]
+.assign.lm
+----
+lex start
+	token id / ('a' .. 'z' | 'A' .. 'Z' ) + /
+	token value / ( 'a' .. 'z' | 'A' .. 'Z' | '0' .. '9' )+ /
+	literal `= `;
+	ignore / [ \t\n] /
+end
+
+def assignment
+	[ id `=  value `;]
+
+	
+def assignment_list
+	[assignment assignment_list]
+|	[assignment]
+|	[]
+
+parse Simple: assignment_list[ stdin ]
+
+for I:assignment in Simple {
+	print( I.id, "->", I.value, "\n" )
+}
+----
+
+After the compilation we can pipe some input to it's stdin.
+
+[source,bash]
+----
+/opt/colm/bin/colm assign.lm
+echo -e 'b=3;a=1;\n c=2;' |./assign
+----
+
+This gives us:
+
+----
+b->3
+a->1
+c->2
+----
+
+NOTE: this also illustrates how to read from 'stdin'.
diff --git a/doc/3_00_language_transformation.adoc b/doc/3_00_language_transformation.adoc
new file mode 100644
index 00000000..72279c90
--- /dev/null
+++ b/doc/3_00_language_transformation.adoc
@@ -0,0 +1,6 @@
+Language transformation
+=======================
+
+== TODO
+
+
diff --git a/doc/3_01_lex.adoc b/doc/3_01_lex.adoc
new file mode 100644
index 00000000..bb5d1409
--- /dev/null
+++ b/doc/3_01_lex.adoc
@@ -0,0 +1,6 @@
+Lex
+===
+
+== TODO
+
+
diff --git a/doc/3_02_parse.adoc b/doc/3_02_parse.adoc
new file mode 100644
index 00000000..80f89e7f
--- /dev/null
+++ b/doc/3_02_parse.adoc
@@ -0,0 +1,6 @@
+Parse
+=====
+
+== TODO
+
+
diff --git a/doc/3_03_match.adoc b/doc/3_03_match.adoc
new file mode 100644
index 00000000..095e9e11
--- /dev/null
+++ b/doc/3_03_match.adoc
@@ -0,0 +1,6 @@
+Match
+=====
+
+== TODO
+
+
diff --git a/doc/3_04_undo.adoc b/doc/3_04_undo.adoc
new file mode 100644
index 00000000..68acfcf5
--- /dev/null
+++ b/doc/3_04_undo.adoc
@@ -0,0 +1,6 @@
+Undo
+====
+
+== TODO
+
+
diff --git a/doc/3_05_reduction.adoc b/doc/3_05_reduction.adoc
new file mode 100644
index 00000000..d1293f6c
--- /dev/null
+++ b/doc/3_05_reduction.adoc
@@ -0,0 +1,6 @@
+Reduction
+========
+
+== TODO
+
+
diff --git a/doc/8_00_advanced_topics.adoc b/doc/8_00_advanced_topics.adoc
new file mode 100644
index 00000000..896ba3a8
--- /dev/null
+++ b/doc/8_00_advanced_topics.adoc
@@ -0,0 +1,6 @@
+Advanced topics
+===============
+
+== TODO
+
+
diff --git a/doc/8_01_embedding.adoc b/doc/8_01_embedding.adoc
new file mode 100644
index 00000000..b05af1c0
--- /dev/null
+++ b/doc/8_01_embedding.adoc
@@ -0,0 +1,6 @@
+Embedding
+=========
+
+== TODO
+
+
diff --git a/doc/8_02_modularity.adoc b/doc/8_02_modularity.adoc
new file mode 100644
index 00000000..27683309
--- /dev/null
+++ b/doc/8_02_modularity.adoc
@@ -0,0 +1,6 @@
+Modularity
+==========
+
+== TODO
+
+
diff --git a/doc/8_03_performance.adoc b/doc/8_03_performance.adoc
new file mode 100644
index 00000000..77294519
--- /dev/null
+++ b/doc/8_03_performance.adoc
@@ -0,0 +1,6 @@
+Performance
+===========
+
+== TODO
+
+
diff --git a/doc/8_04_error_handling.adoc b/doc/8_04_error_handling.adoc
new file mode 100644
index 00000000..66255c30
--- /dev/null
+++ b/doc/8_04_error_handling.adoc
@@ -0,0 +1,6 @@
+Error handling
+==============
+
+== TODO
+
+
diff --git a/doc/8_05_contributing.adoc b/doc/8_05_contributing.adoc
new file mode 100644
index 00000000..7031a24c
--- /dev/null
+++ b/doc/8_05_contributing.adoc
@@ -0,0 +1,6 @@
+Contributing
+============
+
+== TODO
+
+
diff --git a/doc/9_00_q_and_a.adoc b/doc/9_00_q_and_a.adoc
new file mode 100644
index 00000000..a143ab02
--- /dev/null
+++ b/doc/9_00_q_and_a.adoc
@@ -0,0 +1,17 @@
+Q and A
+=======
+
+== FAQ
+
+.Q: I get this error
+ error while loading shared libraries: libcolm-0.13.0.4.so: cannot open shared object file: No such file or directory
+
+*A*: You probably configured and installed colm with the '--prexix' argument
+You can find the dynamic library and symlink it to a familiar place.
+
+[source,bash]
+----
+sudo updatedb
+ln -s `locate libcolm-0.13.0.4.so` /usr/lib/libcolm-0.13.0.4.so
+----
+