clang-tools 20.0.0git
|
A complete sequence of Tokens representing a source file. More...
#include <Token.h>
Public Member Functions | |
TokenStream (std::shared_ptr< void > Payload=nullptr) | |
Create an empty stream. | |
void | push (Token T) |
Append a token to the stream, which must not be finalized. | |
void | finalize () |
Finalize the token stream, allowing tokens to be accessed. | |
bool | isFinalized () const |
Token::Index | index (const Token &T) const |
Returns the index of T within the stream. | |
ArrayRef< Token > | tokens () const |
ArrayRef< Token > | tokens (Token::Range R) const |
MutableArrayRef< Token > | tokens () |
const Token & | front () const |
May return the end sentinel if the stream is empty. | |
std::shared_ptr< void > | getPayload () const |
Returns the shared payload. | |
void | addPayload (std::shared_ptr< void > P) |
Adds the given payload to the stream. | |
void | print (llvm::raw_ostream &) const |
Print the tokens in this stream to the output stream. | |
A complete sequence of Tokens representing a source file.
This may match a raw file from disk, or be derived from a previous stream. For example, stripping comments from a TokenStream results in a new stream.
A stream has sentinel 'eof' tokens at each end, e.g int main();
becomes: int main ( ) ; eof kw_int ident l_paren r_paren semi eof front() back() 0 1 2 3 4 5
Definition at line 130 of file support/Token.h.
|
explicit |
Create an empty stream.
Initially, the stream is appendable and not finalized. The token sequence may only be accessed after finalize() is called.
Payload is an opaque object which will be owned by the stream. e.g. an allocator to hold backing storage for synthesized token text.
|
inline |
Adds the given payload to the stream.
Definition at line 184 of file support/Token.h.
void clang::clangd::TokenStream::finalize | ( | ) |
Finalize the token stream, allowing tokens to be accessed.
Tokens may no longer be appended.
Definition at line 55 of file Token.cpp.
References isFinalized(), and LastLine.
|
inline |
May return the end sentinel if the stream is empty.
Definition at line 176 of file support/Token.h.
References isFinalized().
|
inline |
Returns the shared payload.
Definition at line 182 of file support/Token.h.
Referenced by clang::clangd::cook().
|
inline |
Returns the index of T within the stream.
T must be within the stream or the end sentinel (not the start sentinel).
Definition at line 155 of file support/Token.h.
References isFinalized().
Referenced by clang::clangd::operator<<().
bool clang::clangd::TokenStream::isFinalized | ( | ) | const |
void clang::clangd::TokenStream::print | ( | llvm::raw_ostream & | OS | ) | const |
|
inline |
Append a token to the stream, which must not be finalized.
Definition at line 142 of file support/Token.h.
References isFinalized().
|
inline |
Definition at line 170 of file support/Token.h.
References isFinalized().
|
inline |
Definition at line 162 of file support/Token.h.
References isFinalized().
Referenced by clang::clangd::cook(), clang::clangd::operator<<(), print(), and tokens().
|
inline |
Definition at line 166 of file support/Token.h.
References clang::clangd::Token::Range::Begin, clang::clangd::Token::Range::End, and tokens().