clang API Documentation

Public Types | Public Member Functions
clang::Token Class Reference

#include <Token.h>

List of all members.

Public Types

enum  TokenFlags {
  StartOfLine = 0x01, LeadingSpace = 0x02, DisableExpand = 0x04, NeedsCleaning = 0x08,
  LeadingEmptyMacro = 0x10, HasUDSuffix = 0x20, HasUCN = 0x40, IgnoredComma = 0x80
}

Public Member Functions

tok::TokenKind getKind () const
void setKind (tok::TokenKind K)
bool is (tok::TokenKind K) const
bool isNot (tok::TokenKind K) const
bool isAnyIdentifier () const
 Return true if this is a raw identifier (when lexing in raw mode) or a non-keyword identifier (when lexing in non-raw mode).
bool isLiteral () const
 Return true if this is a "literal", like a numeric constant, string, etc.
bool isAnnotation () const
 Return true if this is any of tok::annot_* kind tokens.
SourceLocation getLocation () const
 Return a source location identifier for the specified offset in the current file.
unsigned getLength () const
void setLocation (SourceLocation L)
void setLength (unsigned Len)
SourceLocation getAnnotationEndLoc () const
void setAnnotationEndLoc (SourceLocation L)
SourceLocation getLastLoc () const
SourceRange getAnnotationRange () const
 SourceRange of the group of tokens that this annotation token represents.
void setAnnotationRange (SourceRange R)
const char * getName () const
void startToken ()
 Reset all flags to cleared.
IdentifierInfogetIdentifierInfo () const
void setIdentifierInfo (IdentifierInfo *II)
StringRef getRawIdentifier () const
void setRawIdentifierData (const char *Ptr)
const char * getLiteralData () const
void setLiteralData (const char *Ptr)
void * getAnnotationValue () const
void setAnnotationValue (void *val)
void setFlag (TokenFlags Flag)
 Set the specified flag.
void clearFlag (TokenFlags Flag)
 Unset the specified flag.
unsigned getFlags () const
 Return the internal represtation of the flags.
void setFlagValue (TokenFlags Flag, bool Val)
 Set a flag to either true or false.
bool isAtStartOfLine () const
bool hasLeadingSpace () const
 Return true if this token has whitespace before it.
bool isExpandDisabled () const
 Return true if this identifier token should never be expanded in the future, due to C99 6.10.3.4p2.
bool isObjCAtKeyword (tok::ObjCKeywordKind objcKey) const
 Return true if we have an ObjC keyword identifier.
tok::ObjCKeywordKind getObjCKeywordID () const
 Return the ObjC keyword kind.
bool needsCleaning () const
 Return true if this token has trigraphs or escaped newlines in it.
bool hasLeadingEmptyMacro () const
 Return true if this token has an empty macro before it.
bool hasUDSuffix () const
 Return true if this token is a string or character literal which has a ud-suffix.
bool hasUCN () const
 Returns true if this token contains a universal character name.

Detailed Description

Token - This structure provides full information about a lexed token. It is not intended to be space efficient, it is intended to return as much information as possible about each returned token. This is expected to be compressed into a smaller form if memory footprint is important.

The parser can create a special "annotation token" representing a stream of tokens that were parsed and semantically resolved, e.g.: "foo::MyClass<int>" can be represented by a single typename annotation token that carries information about the SourceRange of the tokens and the type object.

Definition at line 37 of file Token.h.


Member Enumeration Documentation

Enumerator:
StartOfLine 
LeadingSpace 
DisableExpand 
NeedsCleaning 
LeadingEmptyMacro 
HasUDSuffix 
HasUCN 
IgnoredComma 

Definition at line 73 of file Token.h.


Member Function Documentation

void clang::Token::clearFlag ( TokenFlags  Flag) [inline]

Unset the specified flag.

Definition at line 210 of file Token.h.

Referenced by setFlagValue().

Definition at line 125 of file Token.h.

References clang::SourceLocation::getFromRawEncoding(), and isAnnotation().

Referenced by getAnnotationRange(), and getLastLoc().

SourceRange of the group of tokens that this annotation token represents.

Definition at line 140 of file Token.h.

References getAnnotationEndLoc(), and getLocation().

Referenced by clang::Parser::TryAnnotateTypeOrScopeToken().

void* clang::Token::getAnnotationValue ( ) const [inline]

Definition at line 195 of file Token.h.

References isAnnotation().

Referenced by clang::Parser::getTypeAnnotation(), and clang::Parser::ParseTopLevelDecl().

unsigned clang::Token::getFlags ( ) const [inline]

Return the internal represtation of the flags.

This is only intended for low-level operations such as writing tokens to disk.

Definition at line 218 of file Token.h.

Referenced by clang::ASTWriter::AddToken().

Definition at line 134 of file Token.h.

References getAnnotationEndLoc(), getLocation(), and isAnnotation().

Referenced by clang::Parser::TryAnnotateTypeOrScopeToken().

unsigned clang::Token::getLength ( ) const [inline]
const char* clang::Token::getLiteralData ( ) const [inline]

getLiteralData - For a literal token (numeric constant, string, etc), this returns a pointer to the start of it in the text buffer if known, null otherwise.

Definition at line 186 of file Token.h.

References isLiteral().

Referenced by GetFirstChar(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), PrintPreprocessedTokens(), and UseNSOptionsMacro().

Return a source location identifier for the specified offset in the current file.

Definition at line 113 of file Token.h.

Referenced by clang::Sema::ActOnCharacterConstant(), clang::Sema::ActOnNumericConstant(), clang::Sema::ActOnPragmaUnused(), clang::ASTWriter::AddToken(), clang::TokenConcatenation::AvoidConcat(), clang::Preprocessor::CheckEndOfDirective(), clang::Sema::ClassifyName(), ComputeDATE_TIME(), clang::Lexer::ComputePreamble(), clang::Preprocessor::ConcatenateIncludeName(), clang::Parser::ConsumeToken(), clang::Parser::Diag(), clang::Preprocessor::Diag(), clang::DoPrintPreprocessedInput(), clang::Preprocessor::DumpToken(), EvaluateBuildingModule(), EvaluateDefined(), EvaluateDirectiveSubExpr(), EvaluateHasIncludeCommon(), EvaluateValue(), clang::Parser::ExitScope(), clang::BalancedDelimiterTracker::expectAndConsume(), findDirectives(), FixDigraph(), getAnnotationRange(), getBeginningOfFileToken(), GetFirstChar(), getLastLoc(), GetLineValue(), clang::StringLiteralParser::getOffsetOfStringByte(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), clang::Preprocessor::HandleDirective(), clang::Preprocessor::HandleEndOfFile(), clang::Preprocessor::HandleIdentifier(), clang::Preprocessor::HandlePragmaDependency(), clang::Preprocessor::HandlePragmaIncludeAlias(), clang::Preprocessor::HandlePragmaPopMacro(), clang::Preprocessor::HandlePragmaSystemHeader(), clang::html::HighlightMacros(), clang::TokenLexer::Init(), clang::TokenLexer::Lex(), clang::Preprocessor::LexAfterModuleImport(), clang::PreprocessorLexer::LexIncludeFilename(), ParseAlignPragma(), ParseLoopHintValue(), clang::Parser::ParseMSAsmIdentifier(), clang::Preprocessor::ParsePragmaPushOrPopMacro(), clang::Preprocessor::parseSimpleIntegerLiteral(), clang::Parser::ParseTopLevelDecl(), PrintPreprocessedTokens(), ReadLineMarkerFlags(), clang::RewriteMacrosInInput(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), clang::html::SyntaxHighlight(), clang::Parser::TryAnnotateTypeOrScopeToken(), clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec(), clang::Parser::TryConsumeToken(), and updateConsecutiveMacroArgTokens().

const char* clang::Token::getName ( ) const [inline]

Definition at line 148 of file Token.h.

References clang::tok::getTokenName().

Return the ObjC keyword kind.

getObjCKeywordID - Return the ObjC keyword kind.

Definition at line 43 of file Lexer.cpp.

References getIdentifierInfo(), and clang::IdentifierInfo::getObjCKeywordID().

StringRef clang::Token::getRawIdentifier ( ) const [inline]

getRawIdentifier - For a raw identifier token (i.e., an identifier lexed in raw mode), returns a reference to the text substring in the buffer if known.

Definition at line 174 of file Token.h.

References getLength(), and is().

Referenced by clang::Lexer::ComputePreamble(), clang::Lexer::getSpelling(), and clang::Preprocessor::LookUpIdentifierInfo().

Return true if this token has an empty macro before it.

Definition at line 255 of file Token.h.

References LeadingEmptyMacro.

bool clang::Token::hasLeadingSpace ( ) const [inline]
bool clang::Token::hasUCN ( ) const [inline]

Returns true if this token contains a universal character name.

Definition at line 264 of file Token.h.

References HasUCN.

Referenced by clang::Lexer::getSpelling(), clang::Preprocessor::getSpelling(), and clang::Preprocessor::LookUpIdentifierInfo().

bool clang::Token::hasUDSuffix ( ) const [inline]

Return true if this token is a string or character literal which has a ud-suffix.

Definition at line 261 of file Token.h.

References HasUDSuffix.

Referenced by clang::TokenConcatenation::AvoidConcat(), EvaluateValue(), clang::Preprocessor::FinishLexStringLiteral(), and clang::Preprocessor::ParsePragmaPushOrPopMacro().

bool clang::Token::is ( tok::TokenKind  K) const [inline]

is/isNot - Predicates to check if this token is a specific kind, as in "if (Tok.is(tok::l_brace)) {...}".

Definition at line 91 of file Token.h.

Referenced by clang::TokenConcatenation::AvoidConcat(), buildMSAsmString(), clang::Preprocessor::CheckEndOfDirective(), clang::Preprocessor::CheckMacroName(), clang::Sema::ClassifyName(), clang::Preprocessor::ConcatenateIncludeName(), clang::BalancedDelimiterTracker::consumeClose(), clang::BalancedDelimiterTracker::consumeOpen(), clang::Preprocessor::CreateString(), clang::DoPrintPreprocessedInput(), EvaluateDefined(), EvaluateHasIncludeCommon(), EvaluateValue(), findDirectives(), clang::Preprocessor::FinishLexStringLiteral(), clang::PTHLexer::getEOF(), getRawIdentifier(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), getSpellingSlow(), clang::MacroArgs::getUnexpArgument(), clang::Preprocessor::HandleDirective(), clang::Preprocessor::HandleMicrosoftCommentPaste(), clang::Preprocessor::HandlePragmaDependency(), clang::Preprocessor::HandlePragmaIncludeAlias(), clang::Preprocessor::HandlePragmaPoison(), clang::html::HighlightMacros(), clang::format::FormatToken::is(), IsCommonTypo(), isEndOfTemplateArgument(), clang::TokenLexer::isNextTokenLParen(), clang::TokenLexer::isParsingPreprocessorDirective(), isResultTypeOrTemplate(), isValidAfterIdentifierInDeclarator(), clang::TokenLexer::Lex(), clang::Preprocessor::Lex(), clang::PreprocessorLexer::LexIncludeFilename(), LexRawTokensFromMainFile(), clang::Parser::ParseAssignmentExpression(), ParseLoopHintValue(), clang::Parser::ParseMSAsmIdentifier(), clang::Preprocessor::parseSimpleIntegerLiteral(), clang::Parser::ParseTopLevelDecl(), clang::Parser::ParseUnqualifiedId(), PrintPreprocessedTokens(), ReadLineMarkerFlags(), clang::Lexer::ReadToEndOfLine(), clang::RewriteMacrosInInput(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), setRawIdentifierData(), clang::Parser::SkipMalformedDecl(), clang::Parser::SkipUntil(), clang::MacroArgs::StringifyArgument(), clang::Parser::TryAnnotateCXXScopeToken(), clang::Parser::TryAnnotateTypeOrScopeToken(), and clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec().

bool clang::Token::isAnnotation ( ) const [inline]
bool clang::Token::isAnyIdentifier ( ) const [inline]

Return true if this is a raw identifier (when lexing in raw mode) or a non-keyword identifier (when lexing in non-raw mode).

Definition at line 96 of file Token.h.

References getKind().

bool clang::Token::isAtStartOfLine ( ) const [inline]

Return true if this identifier token should never be expanded in the future, due to C99 6.10.3.4p2.

Definition at line 240 of file Token.h.

References DisableExpand.

Referenced by clang::Preprocessor::DumpToken(), and clang::Preprocessor::HandleIdentifier().

bool clang::Token::isLiteral ( ) const [inline]

Return true if this is a "literal", like a numeric constant, string, etc.

Definition at line 102 of file Token.h.

References getKind().

Referenced by clang::Preprocessor::CreateString(), GetFirstChar(), getIdentifierInfo(), getLiteralData(), clang::Lexer::getSpelling(), clang::PTHLexer::Lex(), PrintPreprocessedTokens(), setLiteralData(), and UseNSOptionsMacro().

bool clang::Token::isNot ( tok::TokenKind  K) const [inline]

Definition at line 92 of file Token.h.

Referenced by clang::MacroArgs::ArgNeedsPreexpansion(), clang::TokenConcatenation::AvoidConcat(), clang::CacheTokens(), clang::Preprocessor::CheckEndOfDirective(), clang::Preprocessor::ConcatenateIncludeName(), clang::Preprocessor::DiscardUntilEndOfDirective(), DoPrintMacros(), EvaluateBuildingModule(), EvaluateDefined(), EvaluateDirectiveSubExpr(), EvaluateHasIncludeCommon(), EvaluateValue(), clang::DumpRawTokensAction::ExecuteAction(), clang::DumpTokensAction::ExecuteAction(), clang::PreprocessOnlyAction::ExecuteAction(), findDirectives(), clang::Preprocessor::FinishLexStringLiteral(), clang::MacroArgs::getArgLength(), getIdentifierInfo(), GetLineValue(), clang::Preprocessor::getSpelling(), clang::Preprocessor::HandleMicrosoftCommentPaste(), clang::Preprocessor::HandlePragmaDependency(), clang::Preprocessor::HandlePragmaIncludeAlias(), clang::Preprocessor::HandlePragmaPoison(), clang::html::HighlightMacros(), clang::format::FormatToken::isNot(), clang::Preprocessor::LexOnOffSwitch(), LexRawTokensFromMainFile(), ParseAlignPragma(), ParseLoopHintValue(), clang::Preprocessor::ParsePragmaPushOrPopMacro(), clang::Parser::ParseUnqualifiedId(), clang::RewriteIncludesInInput(), clang::RewriteMacrosInInput(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), clang::Parser::SkipUntil(), clang::MacroArgs::StringifyArgument(), clang::html::SyntaxHighlight(), clang::format::tokenCanStartNewLine(), clang::TokenRewriter::TokenRewriter(), and clang::Parser::TryConsumeToken().

Return true if we have an ObjC keyword identifier.

isObjCAtKeyword - Return true if we have an ObjC keyword identifier.

Definition at line 36 of file Lexer.cpp.

References getIdentifierInfo().

Referenced by clang::format::FormatToken::isObjCAtKeyword(), and clang::Parser::SkipMalformedDecl().

bool clang::Token::needsCleaning ( ) const [inline]
void clang::Token::setAnnotationValue ( void *  val) [inline]

Definition at line 199 of file Token.h.

References isAnnotation().

Referenced by EnterAnnotationToken(), clang::Preprocessor::HandleEndOfFile(), and ParseAlignPragma().

void clang::Token::setFlag ( TokenFlags  Flag) [inline]
void clang::Token::setFlagValue ( TokenFlags  Flag,
bool  Val 
) [inline]

Set a flag to either true or false.

Definition at line 223 of file Token.h.

References clearFlag(), and setFlag().

Referenced by clang::TokenLexer::Lex().

void clang::Token::setKind ( tok::TokenKind  K) [inline]
void clang::Token::setLength ( unsigned  Len) [inline]
void clang::Token::setLiteralData ( const char *  Ptr) [inline]

Definition at line 190 of file Token.h.

References isLiteral().

Referenced by clang::Preprocessor::CreateString(), and clang::PTHLexer::Lex().

void clang::Token::setRawIdentifierData ( const char *  Ptr) [inline]

Definition at line 178 of file Token.h.

References is().

Referenced by clang::Preprocessor::CreateString().

void clang::Token::startToken ( ) [inline]

The documentation for this class was generated from the following files: