TokenStreamRewriteEngine Class Reference

#include <TokenStreamRewriteEngine.hpp>

Inheritance diagram for TokenStreamRewriteEngine:

Collaboration diagram for TokenStreamRewriteEngine:


Public Types
typedef operation_list	program_map
Public Member Functions
typedef	ANTLR_USE_NAMESPACE (std) list< RewriteOperation * > operation_list
	list of rewrite operations
typedef	ANTLR_USE_NAMESPACE (std) map< ANTLR_USE_NAMESPACE(std) string
	map program name to <program counter,program> tuple
	TokenStreamRewriteEngine (TokenStream &upstream)
	TokenStreamRewriteEngine (TokenStream &upstream, size_t initialSize)
RefToken	nextToken (void)
void	rollback (size_t instructionIndex)
void	rollback (const ANTLR_USE_NAMESPACE(std) string &programName, size_t instructionIndex)
void	deleteProgram ()
void	deleteProgram (const ANTLR_USE_NAMESPACE(std) string &programName)
void	insertAfter (RefTokenWithIndex t, const ANTLR_USE_NAMESPACE(std) string &text)
void	insertAfter (size_t index, const ANTLR_USE_NAMESPACE(std) string &text)
void	insertAfter (const ANTLR_USE_NAMESPACE(std) string &programName, RefTokenWithIndex t, const ANTLR_USE_NAMESPACE(std) string &text)
void	insertAfter (const ANTLR_USE_NAMESPACE(std) string &programName, size_t index, const ANTLR_USE_NAMESPACE(std) string &text)
void	insertBefore (RefTokenWithIndex t, const ANTLR_USE_NAMESPACE(std) string &text)
void	insertBefore (size_t index, const ANTLR_USE_NAMESPACE(std) string &text)
void	insertBefore (const ANTLR_USE_NAMESPACE(std) string &programName, RefTokenWithIndex t, const ANTLR_USE_NAMESPACE(std) string &text)
void	insertBefore (const ANTLR_USE_NAMESPACE(std) string &programName, size_t index, const ANTLR_USE_NAMESPACE(std) string &text)
void	replace (size_t index, const ANTLR_USE_NAMESPACE(std) string &text)
void	replace (size_t from, size_t to, const ANTLR_USE_NAMESPACE(std) string &text)
void	replace (RefTokenWithIndex indexT, const ANTLR_USE_NAMESPACE(std) string &text)
void	replace (RefTokenWithIndex from, RefTokenWithIndex to, const ANTLR_USE_NAMESPACE(std) string &text)
void	replace (const ANTLR_USE_NAMESPACE(std) string &programName, size_t from, size_t to, const ANTLR_USE_NAMESPACE(std) string &text)
void	replace (const ANTLR_USE_NAMESPACE(std) string &programName, RefTokenWithIndex from, RefTokenWithIndex to, const ANTLR_USE_NAMESPACE(std) string &text)
void	remove (size_t index)
void	remove (size_t from, size_t to)
void	remove (RefTokenWithIndex indexT)
void	remove (RefTokenWithIndex from, RefTokenWithIndex to)
void	remove (const ANTLR_USE_NAMESPACE(std) string &programName, size_t from, size_t to)
void	remove (const ANTLR_USE_NAMESPACE(std) string &programName, RefTokenWithIndex from, RefTokenWithIndex to)
void	discard (int ttype)
RefToken	getToken (size_t i)
size_t	getTokenStreamSize () const
void	originalToStream (ANTLR_USE_NAMESPACE(std) ostream &out) const
void	originalToStream (ANTLR_USE_NAMESPACE(std) ostream &out, size_t start, size_t end) const
void	toStream (ANTLR_USE_NAMESPACE(std) ostream &out) const
void	toStream (ANTLR_USE_NAMESPACE(std) ostream &out, const ANTLR_USE_NAMESPACE(std) string &programName) const
void	toStream (ANTLR_USE_NAMESPACE(std) ostream &out, size_t start, size_t end) const
void	toStream (ANTLR_USE_NAMESPACE(std) ostream &out, const ANTLR_USE_NAMESPACE(std) string &programName, size_t firstToken, size_t lastToken) const
void	toDebugStream (ANTLR_USE_NAMESPACE(std) ostream &out) const
void	toDebugStream (ANTLR_USE_NAMESPACE(std) ostream &out, size_t start, size_t end) const
size_t	getLastRewriteTokenIndex () const
size_t	getLastRewriteTokenIndex (const ANTLR_USE_NAMESPACE(std) string &programName) const
Static Public Member Functions
typedef static ANTLR_USE_NAMESPACE(std) vector< antlr const char *	DEFAULT_PROGRAM_NAME
Static Public Attributes
static const size_t	MIN_TOKEN_INDEX
static const int	PROGRAM_INIT_SIZE
Protected Member Functions
void	addToSortedRewriteList (RewriteOperation *op)
void	addToSortedRewriteList (const ANTLR_USE_NAMESPACE(std) string &programName, RewriteOperation *op)
Protected Attributes
TokenStream &	stream
size_t	index
token_list	tokens
program_map	programs
BitSet	discardMask
Data Structures
class	DeleteOp
struct	executeOperation
class	InsertBeforeOp
class	ReplaceOp
class	RewriteOperation
struct	tokenToStream

Detailed Description

This token stream tracks the *entire* token stream coming from a lexer, but does not pass on the whitespace (or whatever else you want to discard) to the parser.

This class can then be asked for the ith token in the input stream. Useful for dumping out the input stream exactly after doing some augmentation or other manipulations. Tokens are index from 0..n-1

You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :)

Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1.

Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example,

TokenStreamRewriteEngine rewriteEngine = new TokenStreamRewriteEngine(lexer); JavaRecognizer parser = new JavaRecognizer(rewriteEngine); ... rewriteEngine.insertAfter("pass1", t, "foobar");} rewriteEngine.insertAfter("pass2", u, "start");} System.out.println(rewriteEngine.toString("pass1")); System.out.println(rewriteEngine.toString("pass2"));

You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer.

If you don't use named rewrite streams, a "default" stream is used.

Terence Parr, parrt@cs.usfca.edu University of San Francisco February 2004

Definition at line 77 of file TokenStreamRewriteEngine.hpp.

Member Typedef Documentation

typedef static ANTLR_USE_NAMESPACE (std)vector<antlr const char* TokenStreamRewriteEngine::DEFAULT_PROGRAM_NAME [static]

Definition at line 80 of file TokenStreamRewriteEngine.hpp.

typedef operation_list TokenStreamRewriteEngine::program_map

Definition at line 138 of file TokenStreamRewriteEngine.hpp.

Constructor & Destructor Documentation

TokenStreamRewriteEngine::TokenStreamRewriteEngine ( TokenStream & upstream )

TokenStreamRewriteEngine::TokenStreamRewriteEngine	(	TokenStream &	upstream,
		size_t	initialSize
	)

Member Function Documentation

typedef TokenStreamRewriteEngine::ANTLR_USE_NAMESPACE ( std )

list of rewrite operations

typedef TokenStreamRewriteEngine::ANTLR_USE_NAMESPACE ( std )

map program name to <program counter,program> tuple

RefToken TokenStreamRewriteEngine::nextToken ( void ) [virtual]

Implements TokenStream.

void TokenStreamRewriteEngine::rollback ( size_t instructionIndex ) [inline]

Definition at line 195 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::rollback	(	const ANTLR_USE_NAMESPACE(std) string &	programName,
		size_t	instructionIndex
	)

Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!

void TokenStreamRewriteEngine::deleteProgram ( ) [inline]

Definition at line 206 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::deleteProgram ( const ANTLR_USE_NAMESPACE(std) string & programName ) [inline]

Reset the program so that no instructions exist

Definition at line 211 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::insertAfter	(	RefTokenWithIndex	t,
		const ANTLR_USE_NAMESPACE(std) string &	text
	)			`[inline]`

Definition at line 215 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::insertAfter	(	size_t	index,
		const ANTLR_USE_NAMESPACE(std) string &	text
	)			`[inline]`

Definition at line 221 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::insertAfter	(	const ANTLR_USE_NAMESPACE(std) string &	programName,
		RefTokenWithIndex	t,
		const ANTLR_USE_NAMESPACE(std) string &	text
	)			`[inline]`

Definition at line 225 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::insertAfter	(	const ANTLR_USE_NAMESPACE(std) string &	programName,
		size_t	index,
		const ANTLR_USE_NAMESPACE(std) string &	text
	)			`[inline]`

Definition at line 232 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::insertBefore	(	RefTokenWithIndex	t,
		const ANTLR_USE_NAMESPACE(std) string &	text
	)			`[inline]`

Definition at line 240 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::insertBefore	(	size_t	index,
		const ANTLR_USE_NAMESPACE(std) string &	text
	)			`[inline]`

Definition at line 247 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::insertBefore	(	const ANTLR_USE_NAMESPACE(std) string &	programName,
		RefTokenWithIndex	t,
		const ANTLR_USE_NAMESPACE(std) string &	text
	)			`[inline]`

Definition at line 251 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::insertBefore	(	const ANTLR_USE_NAMESPACE(std) string &	programName,
		size_t	index,
		const ANTLR_USE_NAMESPACE(std) string &	text
	)			`[inline]`

Definition at line 258 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::replace	(	size_t	index,
		const ANTLR_USE_NAMESPACE(std) string &	text
	)			`[inline]`

Definition at line 265 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::replace	(	size_t	from,
		size_t	to,
		const ANTLR_USE_NAMESPACE(std) string &	text
	)			`[inline]`

Definition at line 270 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::replace	(	RefTokenWithIndex	indexT,
		const ANTLR_USE_NAMESPACE(std) string &	text
	)			`[inline]`

Definition at line 276 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::replace	(	RefTokenWithIndex	from,
		RefTokenWithIndex	to,
		const ANTLR_USE_NAMESPACE(std) string &	text
	)			`[inline]`

Definition at line 282 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::replace	(	const ANTLR_USE_NAMESPACE(std) string &	programName,
		size_t	from,
		size_t	to,
		const ANTLR_USE_NAMESPACE(std) string &	text
	)			`[inline]`

Definition at line 289 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::replace	(	const ANTLR_USE_NAMESPACE(std) string &	programName,
		RefTokenWithIndex	from,
		RefTokenWithIndex	to,
		const ANTLR_USE_NAMESPACE(std) string &	text
	)			`[inline]`

Definition at line 296 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::remove ( size_t index ) [inline]

Definition at line 307 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::remove	(	size_t	from,
		size_t	to
	)			`[inline]`

Definition at line 311 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::remove ( RefTokenWithIndex indexT ) [inline]

Definition at line 315 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::remove	(	RefTokenWithIndex	from,
		RefTokenWithIndex	to
	)			`[inline]`

Definition at line 319 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::remove	(	const ANTLR_USE_NAMESPACE(std) string &	programName,
		size_t	from,
		size_t	to
	)			`[inline]`

Definition at line 323 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::remove	(	const ANTLR_USE_NAMESPACE(std) string &	programName,
		RefTokenWithIndex	from,
		RefTokenWithIndex	to
	)			`[inline]`

Definition at line 329 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::discard ( int ttype ) [inline]

Definition at line 335 of file TokenStreamRewriteEngine.hpp.

RefToken TokenStreamRewriteEngine::getToken ( size_t i ) [inline]

Definition at line 339 of file TokenStreamRewriteEngine.hpp.

size_t TokenStreamRewriteEngine::getTokenStreamSize ( ) const [inline]

Definition at line 344 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::originalToStream ( ANTLR_USE_NAMESPACE(std) ostream & out ) const [inline]

Definition at line 348 of file TokenStreamRewriteEngine.hpp.

References ANTLR_USE_NAMESPACE.

void TokenStreamRewriteEngine::originalToStream	(	ANTLR_USE_NAMESPACE(std) ostream &	out,
		size_t	start,
		size_t	end
	)			const

void TokenStreamRewriteEngine::toStream ( ANTLR_USE_NAMESPACE(std) ostream & out ) const [inline]

Definition at line 355 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::toStream	(	ANTLR_USE_NAMESPACE(std) ostream &	out,
		const ANTLR_USE_NAMESPACE(std) string &	programName
	)			const `[inline]`

Definition at line 359 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::toStream	(	ANTLR_USE_NAMESPACE(std) ostream &	out,
		size_t	start,
		size_t	end
	)			const `[inline]`

Definition at line 365 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::toStream	(	ANTLR_USE_NAMESPACE(std) ostream &	out,
		const ANTLR_USE_NAMESPACE(std) string &	programName,
		size_t	firstToken,
		size_t	lastToken
	)			const

void TokenStreamRewriteEngine::toDebugStream ( ANTLR_USE_NAMESPACE(std) ostream & out ) const [inline]

Definition at line 375 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::toDebugStream	(	ANTLR_USE_NAMESPACE(std) ostream &	out,
		size_t	start,
		size_t	end
	)			const

size_t TokenStreamRewriteEngine::getLastRewriteTokenIndex ( ) const [inline]

Definition at line 382 of file TokenStreamRewriteEngine.hpp.

size_t TokenStreamRewriteEngine::getLastRewriteTokenIndex ( const ANTLR_USE_NAMESPACE(std) string & programName ) const [inline]

Return the last index for the program named programName return 0 if the program does not exist or the program is empty. (Note this is different from the java implementation that returns -1)

Definition at line 390 of file TokenStreamRewriteEngine.hpp.

void TokenStreamRewriteEngine::addToSortedRewriteList ( RewriteOperation * op ) [inline, protected]

If op.index > lastRewriteTokenIndexes, just add to the end. Otherwise, do linear

Definition at line 409 of file TokenStreamRewriteEngine.hpp.