clang-tools 20.0.0git
Public Member Functions | List of all members
clang::clangd::TokenStream Class Reference

A complete sequence of Tokens representing a source file. More...

#include <Token.h>

Public Member Functions

 TokenStream (std::shared_ptr< void > Payload=nullptr)
 Create an empty stream.
 
void push (Token T)
 Append a token to the stream, which must not be finalized.
 
void finalize ()
 Finalize the token stream, allowing tokens to be accessed.
 
bool isFinalized () const
 
Token::Index index (const Token &T) const
 Returns the index of T within the stream.
 
ArrayRef< Tokentokens () const
 
ArrayRef< Tokentokens (Token::Range R) const
 
MutableArrayRef< Tokentokens ()
 
const Tokenfront () const
 May return the end sentinel if the stream is empty.
 
std::shared_ptr< void > getPayload () const
 Returns the shared payload.
 
void addPayload (std::shared_ptr< void > P)
 Adds the given payload to the stream.
 
void print (llvm::raw_ostream &) const
 Print the tokens in this stream to the output stream.
 

Detailed Description

A complete sequence of Tokens representing a source file.

This may match a raw file from disk, or be derived from a previous stream. For example, stripping comments from a TokenStream results in a new stream.

A stream has sentinel 'eof' tokens at each end, e.g int main(); becomes: int main ( ) ; eof kw_int ident l_paren r_paren semi eof front() back() 0 1 2 3 4 5

Definition at line 130 of file support/Token.h.

Constructor & Destructor Documentation

◆ TokenStream()

clang::clangd::TokenStream::TokenStream ( std::shared_ptr< void >  Payload = nullptr)
explicit

Create an empty stream.

Initially, the stream is appendable and not finalized. The token sequence may only be accessed after finalize() is called.

Payload is an opaque object which will be owned by the stream. e.g. an allocator to hold backing storage for synthesized token text.

Definition at line 49 of file Token.cpp.

Member Function Documentation

◆ addPayload()

void clang::clangd::TokenStream::addPayload ( std::shared_ptr< void >  P)
inline

Adds the given payload to the stream.

Definition at line 184 of file support/Token.h.

◆ finalize()

void clang::clangd::TokenStream::finalize ( )

Finalize the token stream, allowing tokens to be accessed.

Tokens may no longer be appended.

Definition at line 55 of file Token.cpp.

References isFinalized(), and LastLine.

◆ front()

const Token & clang::clangd::TokenStream::front ( ) const
inline

May return the end sentinel if the stream is empty.

Definition at line 176 of file support/Token.h.

References isFinalized().

◆ getPayload()

std::shared_ptr< void > clang::clangd::TokenStream::getPayload ( ) const
inline

Returns the shared payload.

Definition at line 182 of file support/Token.h.

Referenced by clang::clangd::cook().

◆ index()

Token::Index clang::clangd::TokenStream::index ( const Token T) const
inline

Returns the index of T within the stream.

T must be within the stream or the end sentinel (not the start sentinel).

Definition at line 155 of file support/Token.h.

References isFinalized().

Referenced by clang::clangd::operator<<().

◆ isFinalized()

bool clang::clangd::TokenStream::isFinalized ( ) const

Definition at line 66 of file Token.cpp.

Referenced by finalize(), front(), index(), push(), and tokens().

◆ print()

void clang::clangd::TokenStream::print ( llvm::raw_ostream &  OS) const

Print the tokens in this stream to the output stream.

The presence of newlines/spaces is preserved, but not the quantity.

Definition at line 73 of file Token.cpp.

References LastLine, OS, clang::clangd::Text, and tokens().

◆ push()

void clang::clangd::TokenStream::push ( Token  T)
inline

Append a token to the stream, which must not be finalized.

Definition at line 142 of file support/Token.h.

References isFinalized().

◆ tokens() [1/3]

MutableArrayRef< Token > clang::clangd::TokenStream::tokens ( )
inline

Definition at line 170 of file support/Token.h.

References isFinalized().

◆ tokens() [2/3]

ArrayRef< Token > clang::clangd::TokenStream::tokens ( ) const
inline

Definition at line 162 of file support/Token.h.

References isFinalized().

Referenced by clang::clangd::cook(), clang::clangd::operator<<(), print(), and tokens().

◆ tokens() [3/3]

ArrayRef< Token > clang::clangd::TokenStream::tokens ( Token::Range  R) const
inline

The documentation for this class was generated from the following files: