clang 18.0.0git
Public Member Functions | List of all members
clang::syntax::TokenCollector Class Reference

Collects tokens for the main file while running the frontend action. More...

#include "clang/Tooling/Syntax/Tokens.h"

Public Member Functions

 TokenCollector (Preprocessor &P)
 Adds the hooks to collect the tokens.
TokenBuffer consume () &&
 Finalizes token collection.

Detailed Description

Collects tokens for the main file while running the frontend action.

An instance of this object should be created on FrontendAction::BeginSourceFile() and the results should be consumed after FrontendAction::Execute() finishes.

Definition at line 418 of file Tokens.h.

Constructor & Destructor Documentation

◆ TokenCollector()

TokenCollector::TokenCollector ( Preprocessor PP)

Adds the hooks to collect the tokens.

Fills in the TokenBuffer by tracing the run of a preprocessor.

Should be called before the preprocessing starts, i.e. as a part of BeginSourceFile() or CreateASTConsumer().

The implementation tracks the tokens, macro expansions and directives coming from the preprocessor and:

  • for each token, figures out if it is a part of an expanded token stream, spelled token stream or both. Stores the tokens appropriately.
  • records mappings from the spelled to expanded token ranges, e.g. for macro expansions. FIXME: also properly record:
    • #include directives,
    • #pragma, #line and other PP directives,
    • skipped pp regions,
    • ...

Definition at line 676 of file Tokens.cpp.

References clang::Preprocessor::addPPCallbacks(), clang::Preprocessor::getSourceManager(), clang::Token::isAnnotation(), and clang::Preprocessor::setTokenWatcher().

Member Function Documentation

◆ consume()

TokenBuffer TokenCollector::consume ( ) &&

Finalizes token collection.

Should be called after preprocessing is finished, i.e. after running Execute().

Definition at line 888 of file Tokens.cpp.

The documentation for this class was generated from the following files: