Tokenizer

Translates the ast into a Token stream

A Token is a object with a mode name, a name and a value dict. This is later used by the Translator to formate the text and create the coresoponding code.

example: [ IF(mode=javascript), LEFT_BRACKET(mode=common), VAR(mode=javascript,{name=foo}), EQUALS(mode=javascript), VAR(mode=python,{name=bar}), RIGHT_BRACKET(mode=common), BLOCK_START(mode=javascript), NEWLINE(mode=common), INDENT(mode=control), SET_ATTRIBUTE(mode=python,{var:bar,name=foo}), VAR(mode=java,{var:foo}) NEWLINE(mode=common), DEDENT(mode=control), BLOCK_END(mode=javascript) ]