clang 22.0.0git
clang::syntax::TokenCollector Class Reference

Collects tokens for the main file while running the frontend action. More...

#include "clang/Tooling/Syntax/Tokens.h"

Public Member Functions

 TokenCollector (Preprocessor &P)
 Adds the hooks to collect the tokens.
TokenBuffer consume () &&
 Finalizes token collection.

Detailed Description

Collects tokens for the main file while running the frontend action.

An instance of this object should be created on FrontendAction::BeginSourceFile() and the results should be consumed after FrontendAction::Execute() finishes.

Definition at line 418 of file Tokens.h.

Constructor & Destructor Documentation

◆ TokenCollector()

TokenCollector::TokenCollector ( Preprocessor & PP)

Adds the hooks to collect the tokens.

Fills in the TokenBuffer by tracing the run of a preprocessor.

Should be called before the preprocessing starts, i.e. as a part of BeginSourceFile() or CreateASTConsumer().

The implementation tracks the tokens, macro expansions and directives coming from the preprocessor and:

  • for each token, figures out if it is a part of an expanded token stream, spelled token stream or both. Stores the tokens appropriately.
  • records mappings from the spelled to expanded token ranges, e.g. for macro expansions. FIXME: also properly record:
    • #include directives,
    • #pragma, #line and other PP directives,
    • skipped pp regions,
    • ...

Definition at line 681 of file Tokens.cpp.

References clang::Preprocessor::getSourceManager(), and clang::T.

Member Function Documentation

◆ consume()

TokenBuffer TokenCollector::consume ( ) &&
nodiscard

Finalizes token collection.

Should be called after preprocessing is finished, i.e. after running Execute().

Definition at line 893 of file Tokens.cpp.


The documentation for this class was generated from the following files: