com.codemesh.terp.eval
Class TextTokenStream

java.lang.Object
  com.codemesh.terp.eval.TokenStream
      com.codemesh.terp.eval.TextTokenStream

All Implemented Interfaces:

TokenTypes

public class TextTokenStream
extends TokenStream

A token stream that distinguishes between text and embedded tokens.

Field Summary

Fields inherited from class com.codemesh.terp.eval.TokenStream

collectionDepth, index, indexStack, input, state, tokenList

Fields inherited from interface com.codemesh.terp.api.TokenTypes

AMPER, ASS, AT, BACKSLASH, BEGIN_EMBEDDED, BOOLEAN, CHAR_LITERAL, CIRCON, COLON, COMMA, DIV, DIV_ASS, DOT, DOT2, DQUOTE, END_EMBEDDED, EOF, EQUALS, ERROR, EXCLAM, FILESEP, GE, GT, HASH, HIDDEN, ID, LBRACE, LCURLY, LE, LOG_AND, LOG_OR, LPAREN, LT, MATCH, MINUS, MINUS_ASS, MINUS2, MOD, MOD_ASS, MULT, MULT_ASS, NORMAL, NOT_EQ, NOT_MATCH, NULL, NUMERIC_LITERAL, PATHSEP, PLUS, PLUS_ASS, PLUS2, QUEST, QUOTE, RBRACE, RCURLY, RPAREN, SEMI, STATE_EMBEDDED, STATE_START, STRING_LITERAL, TEXT, TILDE, VERT, WS

Constructor Summary

	TextTokenStream(CharacterStream input)
	TextTokenStream(CharacterStream input, TokenizerState state)
	TextTokenStream(java.io.File source)
	TextTokenStream(java.io.File source, java.lang.String encoding)
	TextTokenStream(java.io.File source, java.lang.String encoding, TokenizerState state)
	TextTokenStream(java.io.File source, TokenizerState state)
	TextTokenStream(java.io.InputStream is)
	TextTokenStream(java.io.InputStream is, java.lang.String encoding)
	TextTokenStream(java.io.InputStream is, java.lang.String encoding, TokenizerState state)
	TextTokenStream(java.io.InputStream is, TokenizerState state)
protected	TextTokenStream(java.lang.Object theSource, java.lang.String encoding, boolean bBuiltin)
	TextTokenStream(java.io.Reader r)
	TextTokenStream(java.io.Reader r, TokenizerState state)
	TextTokenStream(java.lang.String input)
	TextTokenStream(java.lang.String input, TokenizerState state)
	TextTokenStream(java.net.URI sourceUri)
	TextTokenStream(java.net.URI sourceUri, java.lang.String encoding)
	TextTokenStream(java.net.URI sourceUri, java.lang.String encoding, TokenizerState state)
	TextTokenStream(java.net.URI sourceUri, TokenizerState state)
	TextTokenStream(java.net.URL sourceUrl)
	TextTokenStream(java.net.URL sourceUrl, java.lang.String encoding)
	TextTokenStream(java.net.URL sourceUrl, java.lang.String encoding, TokenizerState state)
	TextTokenStream(java.net.URL sourceUrl, TokenizerState state)

Method Summary

Token

createToken() The token factory method.

Methods inherited from class com.codemesh.terp.eval.TokenStream

character_value, character, createEmbedded, dec_digit, exponent, getColumn, getLine, getSource, getStartingState, hex_digit, LA, mark, match, next, numericLiteral, oct_digit, reset, rewind, string, unmark

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TextTokenStream

public TextTokenStream(java.lang.String input)

TextTokenStream

public TextTokenStream(java.lang.String input,
                       TokenizerState state)

TextTokenStream

public TextTokenStream(java.io.InputStream is)

TextTokenStream

public TextTokenStream(java.io.InputStream is,
                       TokenizerState state)

TextTokenStream

public TextTokenStream(java.io.InputStream is,
                       java.lang.String encoding)

TextTokenStream

public TextTokenStream(java.io.InputStream is,
                       java.lang.String encoding,
                       TokenizerState state)

TextTokenStream

public TextTokenStream(java.io.Reader r)

TextTokenStream

public TextTokenStream(java.io.Reader r,
                       TokenizerState state)

TextTokenStream

public TextTokenStream(java.io.File source)

TextTokenStream

public TextTokenStream(java.io.File source,
                       TokenizerState state)

TextTokenStream

public TextTokenStream(java.io.File source,
                       java.lang.String encoding)

TextTokenStream

public TextTokenStream(java.io.File source,
                       java.lang.String encoding,
                       TokenizerState state)

TextTokenStream

public TextTokenStream(java.net.URI sourceUri)

TextTokenStream

public TextTokenStream(java.net.URI sourceUri,
                       TokenizerState state)

TextTokenStream

public TextTokenStream(java.net.URI sourceUri,
                       java.lang.String encoding)

TextTokenStream

public TextTokenStream(java.net.URI sourceUri,
                       java.lang.String encoding,
                       TokenizerState state)

TextTokenStream

public TextTokenStream(java.net.URL sourceUrl)

TextTokenStream

public TextTokenStream(java.net.URL sourceUrl,
                       TokenizerState state)

TextTokenStream

public TextTokenStream(java.net.URL sourceUrl,
                       java.lang.String encoding)

TextTokenStream

public TextTokenStream(java.net.URL sourceUrl,
                       java.lang.String encoding,
                       TokenizerState state)

TextTokenStream

public TextTokenStream(CharacterStream input)

TextTokenStream

public TextTokenStream(CharacterStream input,
                       TokenizerState state)

TextTokenStream

protected TextTokenStream(java.lang.Object theSource,
                          java.lang.String encoding,
                          boolean bBuiltin)

Method Detail

createToken

public Token createToken()

Description copied from class: TokenStream

The token factory method. Overridden by concrete subtypes to allow for different interpretation of static text characters.

Specified by:

createToken in class TokenStream

Returns:

the next token created from characters read from the underlying CharacterStream.