clang 20.0.0git
Public Member Functions | List of all members
clang::TokenConcatenation Class Reference

TokenConcatenation class, which answers the question of "Is it safe to emit two tokens without a whitespace between them, or would that cause implicit concatenation of the tokens?". More...

#include "clang/Lex/TokenConcatenation.h"

Public Member Functions

 TokenConcatenation (const Preprocessor &PP)
 
bool AvoidConcat (const Token &PrevPrevTok, const Token &PrevTok, const Token &Tok) const
 AvoidConcat - If printing PrevTok immediately followed by Tok would cause the two individual tokens to be lexed as a single token, return true (which causes a space to be printed between them).
 

Detailed Description

TokenConcatenation class, which answers the question of "Is it safe to emit two tokens without a whitespace between them, or would that cause implicit concatenation of the tokens?".

For example, it emitting two identifiers "foo" and "bar" next to each other would cause the lexer to produce one "foobar" token. Emitting "1" and ")" next to each other is safe.

Definition at line 30 of file TokenConcatenation.h.

Constructor & Destructor Documentation

◆ TokenConcatenation()

TokenConcatenation::TokenConcatenation ( const Preprocessor PP)

Definition at line 69 of file TokenConcatenation.cpp.

References clang::Preprocessor::getLangOpts(), and memset().

Member Function Documentation

◆ AvoidConcat()

bool TokenConcatenation::AvoidConcat ( const Token PrevPrevTok,
const Token PrevTok,
const Token Tok 
) const

AvoidConcat - If printing PrevTok immediately followed by Tok would cause the two individual tokens to be lexed as a single token, return true (which causes a space to be printed between them).

This allows the output of -E mode to be lexed to the same token stream as lexing the input directly would.

This code must conservatively return true if it doesn't want to be 100% accurate. This will cause the output to include extra space characters, but the resulting output won't have incorrect concatenations going on. Examples include "..", which we print with a space between, because we don't want to track enough to tell "x.." from "...".

Definition at line 160 of file TokenConcatenation.cpp.

References GetFirstChar(), clang::Token::getIdentifierInfo(), clang::Token::getKind(), clang::Preprocessor::getLangOpts(), clang::Token::getLength(), clang::Token::getLocation(), clang::SourceLocation::getLocWithOffset(), clang::Preprocessor::getSourceManager(), clang::Token::hasUDSuffix(), clang::Token::is(), clang::Token::isAnnotation(), clang::isDigit(), clang::Token::isNot(), clang::Token::isOneOf(), clang::isPreprocessingNumberBody(), and SM.


The documentation for this class was generated from the following files: