TokenBuffer Class Reference
#include <TokenBuffer.hpp>
Public Member Functions | |
| TokenBuffer (TokenStream &input_) | |
| virtual | ~TokenBuffer () |
| void | reset (void) |
| Reset the input buffer to empty state. | |
| int | LA (unsigned int i) |
| RefToken | LT (unsigned int i) |
| unsigned int | mark () |
| void | rewind (unsigned int mark) |
| void | consume () |
| virtual unsigned int | entries () const |
| Return the number of entries in the TokenBuffer. | |
Protected Attributes | |
| TokenStream & | input |
| Token source. | |
| unsigned int | nMarkers |
| Number of active markers. | |
| unsigned int | markerOffset |
| Additional offset used when markers are active. | |
| unsigned int | numToConsume |
| Number of calls to consume() since last LA() or LT() call. | |
| CircularQueue< RefToken > | queue |
| Circular queue with Tokens. | |
Private Member Functions | |
| void | fill (unsigned int amount) |
| void | syncConsume () |
| TokenBuffer (const TokenBuffer &other) | |
| const TokenBuffer & | operator= (const TokenBuffer &other) |
Detailed Description
A Stream of Token objects fed to the parser from a TokenStream that can be rewound via mark()/rewind() methods.A dynamic array is used to buffer up all the input tokens. Normally, "k" tokens are stored in the buffer. More tokens may be stored during guess mode (testing syntactic predicate), or when LT(i>k) is referenced. Consumption of tokens is deferred. In other words, reading the next token is not done by conume(), but deferred until needed by LA or LT.
- Todo:
- : see if we can integrate this one with InputBuffer into one template or so.
- See also:
- antlr.Token
antlr.TokenStream
antlr.TokenQueue
Definition at line 36 of file TokenBuffer.hpp.
Constructor & Destructor Documentation
| TokenBuffer::TokenBuffer | ( | TokenStream & | input_ | ) |
Create a token buffer
| virtual TokenBuffer::~TokenBuffer | ( | ) | [virtual] |
| TokenBuffer::TokenBuffer | ( | const TokenBuffer & | other | ) | [private] |
Member Function Documentation
| void TokenBuffer::reset | ( | void | ) | [inline] |
| int TokenBuffer::LA | ( | unsigned int | i | ) |
Get a lookahead token value
| RefToken TokenBuffer::LT | ( | unsigned int | i | ) |
Get a lookahead token
| unsigned int TokenBuffer::mark | ( | ) |
Return an integer marker that can be used to rewind the buffer to its current state.
| void TokenBuffer::rewind | ( | unsigned int | mark | ) |
Rewind the token buffer to a marker.
- Parameters:
-
mark Marker returned previously from mark()
| void TokenBuffer::consume | ( | ) | [inline] |
Mark another token for deferred consumption
Definition at line 68 of file TokenBuffer.hpp.
| virtual unsigned int TokenBuffer::entries | ( | ) | const [virtual] |
Return the number of entries in the TokenBuffer.
| void TokenBuffer::fill | ( | unsigned int | amount | ) | [private] |
Ensure that the token buffer is sufficiently full
| void TokenBuffer::syncConsume | ( | ) | [inline, private] |
Sync up deferred consumption
Definition at line 104 of file TokenBuffer.hpp.
References markerOffset, nMarkers, numToConsume, and queue.
| const TokenBuffer& TokenBuffer::operator= | ( | const TokenBuffer & | other | ) | [private] |
Field Documentation
TokenStream& TokenBuffer::input [protected] |
unsigned int TokenBuffer::nMarkers [protected] |
Number of active markers.
Definition at line 87 of file TokenBuffer.hpp.
Referenced by syncConsume().
unsigned int TokenBuffer::markerOffset [protected] |
Additional offset used when markers are active.
Definition at line 90 of file TokenBuffer.hpp.
Referenced by syncConsume().
unsigned int TokenBuffer::numToConsume [protected] |
Number of calls to consume() since last LA() or LT() call.
Definition at line 93 of file TokenBuffer.hpp.
Referenced by syncConsume().
CircularQueue<RefToken> TokenBuffer::queue [protected] |
Circular queue with Tokens.
Definition at line 96 of file TokenBuffer.hpp.
Referenced by syncConsume().
The documentation for this class was generated from the following file:
- antlr/include/antlr/TokenBuffer.hpp
