com.codemesh.terp.eval
Class TokenStream

java.lang.Object
  com.codemesh.terp.eval.TokenStream

All Implemented Interfaces:

TokenTypes

Direct Known Subclasses:

PathTokenStream, TextTokenStream

public abstract class TokenStream
extends java.lang.Object
implements TokenTypes

The baseclass for token-producing streams. This class contains most of the functionality for terp token streams. There are different subtypes because we have slightly different tokenization rules depending on the text contents. Normal, general text should be tokenized using a TextTokenStream. Paths, i.e. sequences of file/directory names, should be tokenized using a PathTokenStream. While TextTokenStreams treat path separator characters as a regular, plain-text character, PathTokenStreams treat path separators as a different token type.

Field Summary

protected int	collectionDepth
protected int	index
protected java.util.Stack<java.lang.Integer>	indexStack
protected CharacterStream	input
protected TokenizerState	state
protected java.util.List<Token>	tokenList

Fields inherited from interface com.codemesh.terp.api.TokenTypes

AMPER, ASS, AT, BACKSLASH, BEGIN_EMBEDDED, BOOLEAN, CHAR_LITERAL, CIRCON, COLON, COMMA, DIV, DIV_ASS, DOT, DOT2, DQUOTE, END_EMBEDDED, EOF, EQUALS, ERROR, EXCLAM, FILESEP, GE, GT, HASH, HIDDEN, ID, LBRACE, LCURLY, LE, LOG_AND, LOG_OR, LPAREN, LT, MATCH, MINUS, MINUS_ASS, MINUS2, MOD, MOD_ASS, MULT, MULT_ASS, NORMAL, NOT_EQ, NOT_MATCH, NULL, NUMERIC_LITERAL, PATHSEP, PLUS, PLUS_ASS, PLUS2, QUEST, QUOTE, RBRACE, RCURLY, RPAREN, SEMI, STATE_EMBEDDED, STATE_START, STRING_LITERAL, TEXT, TILDE, VERT, WS

Constructor Summary

protected	TokenStream(CharacterStream input) Creates a stream that reads from the given CharacterStream instance and starts out in TokenizerState.TEXT state.
protected	TokenStream(CharacterStream input, TokenizerState state) Creates a stream that reads from the given CharacterStream instance.
protected	TokenStream(java.io.File source) Creates a stream that reads from the given file and starts out in TokenizerState.TEXT state.
protected	TokenStream(java.io.File source, java.lang.String encoding) Creates a stream that reads from the given file in the given encoding and starts out in TokenizerState.TEXT state.
protected	TokenStream(java.io.File source, java.lang.String encoding, TokenizerState state) Creates a stream that reads from the given file in the given encoding.
protected	TokenStream(java.io.File source, TokenizerState state) Creates a stream that reads from the given file.
protected	TokenStream(java.io.InputStream is) Creates a stream that reads from the given stream and starts out in TokenizerState.TEXT state.
protected	TokenStream(java.io.InputStream is, java.lang.String encoding) Creates a stream that reads from the given stream in the given encoding and starts out in TokenizerState.TEXT state.
protected	TokenStream(java.io.InputStream is, java.lang.String encoding, TokenizerState state) Creates a stream that reads from the given stream in the given encoding.
protected	TokenStream(java.io.InputStream is, TokenizerState state) Creates a stream that reads from the given stream.
protected	TokenStream(java.lang.Object theSource, java.lang.String encoding, boolean bBuiltin) Creates a stream that reads from the given reader and starts out in TokenizerState.TEXT state.
protected	TokenStream(java.io.Reader r) Creates a stream that reads from the given reader and starts out in TokenizerState.TEXT state.
protected	TokenStream(java.io.Reader r, TokenizerState state) Creates a stream that reads from the given reader.
protected	TokenStream(java.lang.String input) Creates a stream that reads from the given string and starts out in TokenizerState.TEXT state.
protected	TokenStream(java.lang.String input, TokenizerState state) Creates a stream that reads from the given string and starts out in the given tokenizer state.
protected	TokenStream(java.net.URI sourceUri) Creates a stream that reads from the given uri and starts out in TokenizerState.TEXT state.
protected	TokenStream(java.net.URI sourceUri, java.lang.String encoding) Creates a stream that reads from the given uri in the given encoding and starts out in TokenizerState.TEXT state.
protected	TokenStream(java.net.URI sourceUri, java.lang.String encoding, TokenizerState state) Creates a stream that reads from the given uri in the given encoding.
protected	TokenStream(java.net.URI sourceUri, TokenizerState state) Creates a stream that reads from the given uri.
protected	TokenStream(java.net.URL sourceUrl) Creates a stream that reads from the given url and starts out in TokenizerState.TEXT state.
protected	TokenStream(java.net.URL sourceUrl, java.lang.String encoding) Creates a stream that reads from the given url in the given encoding and starts out in TokenizerState.TEXT state.
protected	TokenStream(java.net.URL sourceUrl, java.lang.String encoding, TokenizerState state) Creates a stream that reads from the given url in the given encoding.
protected	TokenStream(java.net.URL sourceUrl, TokenizerState state) Creates a stream that reads from the given url.

Method Summary

protected int	character_value()
protected int	character(char terminator)
protected Token	createEmbedded() Returns a token for an embedded expression.
abstract Token	createToken() The token factory method.
protected int	dec_digit()
protected java.lang.Integer	exponent()
int	getColumn() The current column.
int	getLine() The current line.
InputSource	getSource()
TokenizerState	getStartingState() Returns the tokenizer state that this stream was started in.
protected int	hex_digit()
Token	LA(int offset) Returns the token at the specified offset or null if there is no such token.
void	mark() Marks the current position in the stream.
Token	match(int tokenType) Returns the next token if it matches the given toekn type.
Token	next() Returns the next non-hidden token in the stream or null if there are no more tokens.
protected java.lang.Number	numericLiteral(int firstDigit)
protected int	oct_digit()
void	reset() Returns the stream to its initial state.
void	rewind() Restores the current stream position to the last remembered value.
protected java.lang.String	string(char terminator, boolean bUseEscape) Returns the string value with all escape sequences resolved, if so desired.
void	unmark() Forget a remembered position without readjusting the current stream position.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

input

protected final CharacterStream input

tokenList

protected final java.util.List<Token> tokenList

indexStack

protected final java.util.Stack<java.lang.Integer> indexStack

state

protected TokenizerState state

index

protected int index

collectionDepth

protected int collectionDepth

Constructor Detail

TokenStream

protected TokenStream(java.lang.String input)

Creates a stream that reads from the given string and starts out in TokenizerState.TEXT state.

Parameters:

input - the string to read from.

TokenStream

protected TokenStream(java.lang.String input,
                      TokenizerState state)

Creates a stream that reads from the given string and starts out in the given tokenizer state.

Parameters:

input - the string to read from.

state - the tokenizer state to start in.

TokenStream

protected TokenStream(java.io.File source)

Creates a stream that reads from the given file and starts out in TokenizerState.TEXT state.

Parameters:

source - the file to read from.

TokenStream

protected TokenStream(java.io.File source,
                      TokenizerState state)

Creates a stream that reads from the given file.

Parameters:

source - the file to read from.

state - the tokenizer state to start in.

TokenStream

protected TokenStream(java.io.File source,
                      java.lang.String encoding)

Creates a stream that reads from the given file in the given encoding and starts out in TokenizerState.TEXT state.

Parameters:

source - the file to read from.

encoding - the file encoding.

TokenStream

protected TokenStream(java.io.File source,
                      java.lang.String encoding,
                      TokenizerState state)

Creates a stream that reads from the given file in the given encoding.

Parameters:

source - the file to read from.

encoding - the file encoding.

state - the tokenizer state to start in.

TokenStream

protected TokenStream(java.net.URI sourceUri)

Creates a stream that reads from the given uri and starts out in TokenizerState.TEXT state.

Parameters:

sourceUri - the URI to read from.

TokenStream

protected TokenStream(java.net.URI sourceUri,
                      TokenizerState state)

Creates a stream that reads from the given uri.

Parameters:

sourceUri - the URI to read from.

state - the tokenizer state to start in.

TokenStream

protected TokenStream(java.net.URI sourceUri,
                      java.lang.String encoding)

Creates a stream that reads from the given uri in the given encoding and starts out in TokenizerState.TEXT state.

Parameters:

sourceUri - the URI to read from.

encoding - the content encoding.

TokenStream

protected TokenStream(java.net.URI sourceUri,
                      java.lang.String encoding,
                      TokenizerState state)

Creates a stream that reads from the given uri in the given encoding.

Parameters:

sourceUri - the URI to read from.

encoding - the content encoding.

state - the tokenizer state to start in.

TokenStream

protected TokenStream(java.net.URL sourceUrl)

Creates a stream that reads from the given url and starts out in TokenizerState.TEXT state.

Parameters:

sourceUrl - the URL to read from.

TokenStream

protected TokenStream(java.net.URL sourceUrl,
                      TokenizerState state)

Creates a stream that reads from the given url.

Parameters:

sourceUrl - the URL to read from.

state - the tokenizer state to start in.

TokenStream

protected TokenStream(java.net.URL sourceUrl,
                      java.lang.String encoding)

Creates a stream that reads from the given url in the given encoding and starts out in TokenizerState.TEXT state.

Parameters:

sourceUrl - the URL to read from.

encoding - the content encoding.

TokenStream

protected TokenStream(java.net.URL sourceUrl,
                      java.lang.String encoding,
                      TokenizerState state)

Creates a stream that reads from the given url in the given encoding.

Parameters:

sourceUrl - the URL to read from.

encoding - the content encoding.

state - the tokenizer state to start in.

TokenStream

protected TokenStream(java.io.InputStream is)

Creates a stream that reads from the given stream and starts out in TokenizerState.TEXT state.

Parameters:

is - the stream to read from.

TokenStream

protected TokenStream(java.io.InputStream is,
                      TokenizerState state)

Creates a stream that reads from the given stream.

Parameters:

is - the stream to read from.

state - the tokenizer state to start in.

TokenStream

protected TokenStream(java.io.InputStream is,
                      java.lang.String encoding)

Creates a stream that reads from the given stream in the given encoding and starts out in TokenizerState.TEXT state.

Parameters:

is - the stream to read from.

encoding - the content encoding.

TokenStream

protected TokenStream(java.io.InputStream is,
                      java.lang.String encoding,
                      TokenizerState state)

Creates a stream that reads from the given stream in the given encoding.

Parameters:

is - the stream to read from.

encoding - the content encoding.

state - the tokenizer state to start in.

TokenStream

protected TokenStream(java.io.Reader r)

Creates a stream that reads from the given reader and starts out in TokenizerState.TEXT state.

Parameters:

r - the reader to read from.

TokenStream

protected TokenStream(java.io.Reader r,
                      TokenizerState state)

Creates a stream that reads from the given reader.

Parameters:

r - the reader to read from.

state - the tokenizer state to start in.

TokenStream

protected TokenStream(java.lang.Object theSource,
                      java.lang.String encoding,
                      boolean bBuiltin)

Creates a stream that reads from the given reader and starts out in TokenizerState.TEXT state.

Parameters:

theSource - the source to read from.

encoding - the content encoding.

bBuiltin - true if we;re being called from one of the builtin constructors.

TokenStream

protected TokenStream(CharacterStream input)

Creates a stream that reads from the given CharacterStream instance and starts out in TokenizerState.TEXT state.

Parameters:

input - the CharacterStream to read from.

TokenStream

protected TokenStream(CharacterStream input,
                      TokenizerState state)

Creates a stream that reads from the given CharacterStream instance.

Parameters:

input - the CharacterStream to read from.

state - the tokenizer state to start in.

Method Detail

getStartingState

public TokenizerState getStartingState()

Returns the tokenizer state that this stream was started in. This is the tokenizer state that the stream instance will be returned to when reset() is called.

Returns:

the starting state of the stream.

reset

public void reset()

Returns the stream to its initial state.

createToken

public abstract Token createToken()

The token factory method. Overridden by concrete subtypes to allow for different interpretation of static text characters.

Returns:

the next token created from characters read from the underlying CharacterStream.

getLine

public int getLine()

The current line.

Returns:

the current line number.

getColumn

public int getColumn()

The current column.

Returns:

the current column number.

next

public Token next()

Returns the next non-hidden token in the stream or null if there are no more tokens.

Returns:

the next token or null.

LA

public Token LA(int offset)

Returns the token at the specified offset or null if there is no such token.

Parameters:

offset - the offset from the current position.

Returns:

a token or null.

match

public Token match(int tokenType)

Returns the next token if it matches the given toekn type. Returns null otherwise.

Parameters:

tokenType - the expected token type.

Returns:

the next token or null.

mark

public void mark()

Marks the current position in the stream.

unmark

public void unmark()

Forget a remembered position without readjusting the current stream position.

rewind

public void rewind()

Restores the current stream position to the last remembered value.

createEmbedded

protected Token createEmbedded()

Returns a token for an embedded expression.

Returns:

an expression token.

string

protected java.lang.String string(char terminator,
                                  boolean bUseEscape)

Returns the string value with all escape sequences resolved, if so desired.

Parameters:

terminator - the temrination character.

bUseEscape - true if escape sequences should be processed.

Returns:

the string value.

character

protected int character(char terminator)

character_value

protected int character_value()

oct_digit

protected int oct_digit()

dec_digit

protected int dec_digit()

hex_digit

protected int hex_digit()

exponent

protected java.lang.Integer exponent()

numericLiteral

protected java.lang.Number numericLiteral(int firstDigit)

getSource

public InputSource getSource()