clang API Documentation
#include <Token.h>
Public Types | |
| enum | TokenFlags { StartOfLine = 0x01, LeadingSpace = 0x02, DisableExpand = 0x04, NeedsCleaning = 0x08, LeadingEmptyMacro = 0x10 } |
Public Member Functions | |
| tok::TokenKind | getKind () const |
| void | setKind (tok::TokenKind K) |
| bool | is (tok::TokenKind K) const |
| bool | isNot (tok::TokenKind K) const |
| bool | isAnyIdentifier () const |
| bool | isLiteral () const |
| bool | isAnnotation () const |
| SourceLocation | getLocation () const |
| unsigned | getLength () const |
| void | setLocation (SourceLocation L) |
| void | setLength (unsigned Len) |
| SourceLocation | getAnnotationEndLoc () const |
| void | setAnnotationEndLoc (SourceLocation L) |
| SourceLocation | getLastLoc () const |
| SourceRange | getAnnotationRange () const |
| void | setAnnotationRange (SourceRange R) |
| const char * | getName () const |
| void | startToken () |
| IdentifierInfo * | getIdentifierInfo () const |
| void | setIdentifierInfo (IdentifierInfo *II) |
| const char * | getRawIdentifierData () const |
| void | setRawIdentifierData (const char *Ptr) |
| const char * | getLiteralData () const |
| void | setLiteralData (const char *Ptr) |
| void * | getAnnotationValue () const |
| void | setAnnotationValue (void *val) |
| void | setFlag (TokenFlags Flag) |
| setFlag - Set the specified flag. | |
| void | clearFlag (TokenFlags Flag) |
| clearFlag - Unset the specified flag. | |
| unsigned | getFlags () const |
| void | setFlagValue (TokenFlags Flag, bool Val) |
| setFlagValue - Set a flag to either true or false. | |
| bool | isAtStartOfLine () const |
| bool | hasLeadingSpace () const |
| bool | isExpandDisabled () const |
| bool | isObjCAtKeyword (tok::ObjCKeywordKind objcKey) const |
| isObjCAtKeyword - Return true if we have an ObjC keyword identifier. | |
| tok::ObjCKeywordKind | getObjCKeywordID () const |
| getObjCKeywordID - Return the ObjC keyword kind. | |
| bool | needsCleaning () const |
| bool | hasLeadingEmptyMacro () const |
| Return true if this token has an empty macro before it. | |
Token - This structure provides full information about a lexed token. It is not intended to be space efficient, it is intended to return as much information as possible about each returned token. This is expected to be compressed into a smaller form if memory footprint is important.
The parser can create a special "annotation token" representing a stream of tokens that were parsed and semantically resolved, e.g.: "foo::MyClass<int>" can be represented by a single typename annotation token that carries information about the SourceRange of the tokens and the type object.
| void clang::Token::clearFlag | ( | TokenFlags | Flag | ) | [inline] |
clearFlag - Unset the specified flag.
Definition at line 216 of file Token.h.
Referenced by setFlagValue().
| SourceLocation clang::Token::getAnnotationEndLoc | ( | ) | const [inline] |
Definition at line 128 of file Token.h.
References clang::SourceLocation::getFromRawEncoding(), and isAnnotation().
Referenced by getAnnotationRange(), and getLastLoc().
| SourceRange clang::Token::getAnnotationRange | ( | ) | const [inline] |
getAnnotationRange - SourceRange of the group of tokens that this annotation token represents.
Definition at line 143 of file Token.h.
References getAnnotationEndLoc(), and getLocation().
| void* clang::Token::getAnnotationValue | ( | ) | const [inline] |
Definition at line 201 of file Token.h.
References isAnnotation().
| unsigned clang::Token::getFlags | ( | ) | const [inline] |
| IdentifierInfo* clang::Token::getIdentifierInfo | ( | ) | const [inline] |
Definition at line 165 of file Token.h.
References isAnnotation(), isLiteral(), and isNot().
Referenced by clang::Sema::ActOnPragmaUnused(), clang::MacroArgs::ArgNeedsPreexpansion(), clang::TokenConcatenation::AvoidConcat(), EvaluateDefined(), EvaluateValue(), GetFirstChar(), getObjCKeywordID(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpelling(), clang::Preprocessor::HandleDirective(), clang::Preprocessor::HandleIdentifier(), clang::Preprocessor::HandlePoisonedIdentifier(), clang::PragmaGCCVisibilityHandler::HandlePragma(), clang::PragmaPackHandler::HandlePragma(), clang::PragmaMSStructHandler::HandlePragma(), clang::PragmaWeakHandler::HandlePragma(), clang::PragmaOpenCLExtensionHandler::HandlePragma(), clang::PragmaNamespace::HandlePragma(), clang::Preprocessor::HandlePragmaComment(), clang::PreprocessingRecord::InclusionDirective(), clang::TokenLexer::Init(), clang::MacroInfo::isIdenticalTo(), isObjCAtKeyword(), isSameToken(), isTrivialSingleTokenExpansion(), clang::TokenLexer::Lex(), clang::Preprocessor::LexAfterModuleImport(), clang::Preprocessor::LexOnOffSwitch(), clang::PreprocessingRecord::MacroDefined(), clang::PreprocessingRecord::MacroExpands(), clang::Preprocessor::MaybeHandlePoisonedIdentifier(), ParseAlignPragma(), and PrintPreprocessedTokens().
| tok::TokenKind clang::Token::getKind | ( | ) | const [inline] |
Definition at line 81 of file Token.h.
Referenced by clang::Sema::ActOnCharacterConstant(), clang::TokenConcatenation::AvoidConcat(), clang::Lexer::ComputePreamble(), clang::Preprocessor::DumpToken(), EvaluateDirectiveSubExpr(), EvaluateHasIncludeCommon(), EvaluateValue(), getBeginningOfFileToken(), clang::Preprocessor::HandleDirective(), clang::Preprocessor::HandlePragmaMessage(), clang::MacroInfo::isIdenticalTo(), isSameToken(), clang::Preprocessor::LexAfterModuleImport(), clang::Preprocessor::LexNonComment(), clang::Preprocessor::LexUnexpandedNonComment(), PrintPreprocessedTokens(), and clang::html::SyntaxHighlight().
| SourceLocation clang::Token::getLastLoc | ( | ) | const [inline] |
Definition at line 137 of file Token.h.
References getAnnotationEndLoc(), getLocation(), and isAnnotation().
| unsigned clang::Token::getLength | ( | ) | const [inline] |
Definition at line 117 of file Token.h.
References isAnnotation().
Referenced by clang::Sema::ActOnNumericConstant(), AreTokensAdjacent(), clang::TokenConcatenation::AvoidConcat(), clang::Lexer::ComputePreamble(), clang::Preprocessor::ConcatenateIncludeName(), clang::Preprocessor::DumpToken(), getBeginningOfFileToken(), GetFirstChar(), GetLineValue(), clang::StringLiteralParser::getOffsetOfStringByte(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), clang::TokenLexer::Lex(), clang::Preprocessor::LookUpIdentifierInfo(), clang::Lexer::MeasureTokenLength(), PrintPreprocessedTokens(), clang::RewriteMacrosInInput(), clang::MacroArgs::StringifyArgument(), clang::html::SyntaxHighlight(), and updateConsecutiveMacroArgTokens().
| const char* clang::Token::getLiteralData | ( | ) | const [inline] |
getLiteralData - For a literal token (numeric constant, string, etc), this returns a pointer to the start of it in the text buffer if known, null otherwise.
Definition at line 192 of file Token.h.
References isLiteral().
Referenced by GetFirstChar(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), and PrintPreprocessedTokens().
| SourceLocation clang::Token::getLocation | ( | ) | const [inline] |
getLocation - Return a source location identifier for the specified offset in the current file.
Definition at line 116 of file Token.h.
Referenced by clang::Sema::ActOnCharacterConstant(), clang::Sema::ActOnNumericConstant(), clang::Sema::ActOnPragmaUnused(), clang::Sema::ActOnStringLiteral(), AreTokensAdjacent(), clang::TokenConcatenation::AvoidConcat(), clang::Preprocessor::CheckEndOfDirective(), clang::Sema::ClassifyName(), ComputeDATE_TIME(), clang::Lexer::ComputePreamble(), clang::Preprocessor::ConcatenateIncludeName(), clang::Parser::Diag(), clang::Preprocessor::Diag(), clang::DoPrintPreprocessedInput(), clang::Preprocessor::DumpToken(), EvaluateDefined(), EvaluateDirectiveSubExpr(), EvaluateHasIncludeCommon(), EvaluateValue(), FindExpectedDiags(), FixDigraph(), getAnnotationRange(), getBeginningOfFileToken(), GetFirstChar(), getLastLoc(), GetLineValue(), clang::StringLiteralParser::getOffsetOfStringByte(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), clang::Preprocessor::HandleDirective(), clang::Preprocessor::HandleIdentifier(), clang::PragmaGCCVisibilityHandler::HandlePragma(), clang::PragmaPackHandler::HandlePragma(), clang::PragmaMSStructHandler::HandlePragma(), clang::PragmaUnusedHandler::HandlePragma(), clang::PragmaWeakHandler::HandlePragma(), clang::PragmaOpenCLExtensionHandler::HandlePragma(), clang::Preprocessor::HandlePragmaComment(), clang::Preprocessor::HandlePragmaDependency(), clang::Preprocessor::HandlePragmaMessage(), clang::Preprocessor::HandlePragmaPopMacro(), clang::Preprocessor::HandlePragmaSystemHeader(), clang::html::HighlightMacros(), clang::TokenLexer::Init(), clang::TokenLexer::Lex(), clang::Preprocessor::LexAfterModuleImport(), clang::PreprocessorLexer::LexIncludeFilename(), clang::PreprocessingRecord::MacroExpands(), ParseAlignPragma(), clang::Preprocessor::ParsePragmaPushOrPopMacro(), clang::PrettyStackTraceParserEntry::print(), ReadLineMarkerFlags(), clang::RewriteMacrosInInput(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), clang::html::SyntaxHighlight(), and updateConsecutiveMacroArgTokens().
| const char* clang::Token::getName | ( | ) | const [inline] |
Definition at line 151 of file Token.h.
References clang::tok::getTokenName().
| tok::ObjCKeywordKind Token::getObjCKeywordID | ( | ) | const |
getObjCKeywordID - Return the ObjC keyword kind.
Definition at line 53 of file Lexer.cpp.
References getIdentifierInfo(), and clang::IdentifierInfo::getObjCKeywordID().
| const char* clang::Token::getRawIdentifierData | ( | ) | const [inline] |
getRawIdentifierData - For a raw identifier token (i.e., an identifier lexed in raw mode), returns a pointer to the start of it in the text buffer if known, null otherwise.
Definition at line 180 of file Token.h.
References is().
Referenced by clang::Lexer::ComputePreamble(), clang::Lexer::getSpelling(), and clang::Preprocessor::LookUpIdentifierInfo().
| bool clang::Token::hasLeadingEmptyMacro | ( | ) | const [inline] |
Return true if this token has an empty macro before it.
Definition at line 262 of file Token.h.
References LeadingEmptyMacro.
| bool clang::Token::hasLeadingSpace | ( | ) | const [inline] |
hasLeadingSpace - Return true if this token has whitespace before it.
Definition at line 241 of file Token.h.
References LeadingSpace.
Referenced by clang::Preprocessor::ConcatenateIncludeName(), clang::Preprocessor::DumpToken(), clang::html::HighlightMacros(), clang::TokenLexer::Init(), clang::MacroInfo::isIdenticalTo(), PrintPreprocessedTokens(), clang::RewriteMacrosInInput(), and clang::MacroArgs::StringifyArgument().
| bool clang::Token::is | ( | tok::TokenKind | K | ) | const [inline] |
is/isNot - Predicates to check if this token is a specific kind, as in "if (Tok.is(tok::l_brace)) {...}".
Definition at line 86 of file Token.h.
Referenced by clang::TokenConcatenation::AvoidConcat(), clang::Preprocessor::CheckEndOfDirective(), clang::Sema::ClassifyName(), clang::Preprocessor::ConcatenateIncludeName(), clang::Preprocessor::CreateString(), clang::DoPrintPreprocessedInput(), EvaluateDefined(), EvaluateValue(), FindExpectedDiags(), clang::PTHLexer::getEOF(), getRawIdentifierData(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), clang::MacroArgs::getUnexpArgument(), clang::Preprocessor::HandleDirective(), clang::Preprocessor::HandleMicrosoftCommentPaste(), clang::PragmaPackHandler::HandlePragma(), clang::PragmaUnusedHandler::HandlePragma(), clang::PragmaWeakHandler::HandlePragma(), clang::Preprocessor::HandlePragmaComment(), clang::Preprocessor::HandlePragmaDependency(), clang::Preprocessor::HandlePragmaMessage(), clang::Preprocessor::HandlePragmaPoison(), clang::html::HighlightMacros(), clang::Parser::Initialize(), isAnyIdentifier(), IsCommonTypo(), isEndOfTemplateArgument(), isLiteral(), clang::TokenLexer::isNextTokenLParen(), clang::TokenLexer::isParsingPreprocessorDirective(), isResultTypeOrTemplate(), isValidAfterIdentifierInDeclarator(), clang::TokenLexer::Lex(), clang::PreprocessorLexer::LexIncludeFilename(), LexRawTokensFromMainFile(), MayBeDesignationStart(), clang::Parser::ParseTopLevelDecl(), clang::PrettyStackTraceParserEntry::print(), PrintPreprocessedTokens(), ReadLineMarkerFlags(), clang::Lexer::ReadToEndOfLine(), clang::RewriteMacrosInInput(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), setRawIdentifierData(), and clang::MacroArgs::StringifyArgument().
| bool clang::Token::isAnnotation | ( | ) | const [inline] |
Definition at line 106 of file Token.h.
Referenced by clang::Preprocessor::AnnotateCachedTokens(), getAnnotationEndLoc(), getAnnotationValue(), getIdentifierInfo(), getLastLoc(), getLength(), clang::TokenLexer::Lex(), clang::PrettyStackTraceParserEntry::print(), clang::Preprocessor::ReplaceLastTokenWithAnnotation(), setAnnotationEndLoc(), setAnnotationValue(), and setLength().
| bool clang::Token::isAnyIdentifier | ( | ) | const [inline] |
isAnyIdentifier - Return true if this is a raw identifier (when lexing in raw mode) or a non-keyword identifier (when lexing in non-raw mode).
Definition at line 91 of file Token.h.
References is().
| bool clang::Token::isAtStartOfLine | ( | ) | const [inline] |
isAtStartOfLine - Return true if this token is at the start of a line.
Definition at line 237 of file Token.h.
References StartOfLine.
Referenced by clang::Lexer::ComputePreamble(), clang::Preprocessor::DumpToken(), clang::html::HighlightMacros(), clang::TokenLexer::Init(), clang::MacroInfo::isIdenticalTo(), clang::PTHLexer::Lex(), PrintPreprocessedTokens(), clang::RewriteMacrosInInput(), clang::MacroArgs::StringifyArgument(), and clang::html::SyntaxHighlight().
| bool clang::Token::isExpandDisabled | ( | ) | const [inline] |
isExpandDisabled - Return true if this identifier token should never be expanded in the future, due to C99 6.10.3.4p2.
Definition at line 245 of file Token.h.
References DisableExpand.
Referenced by clang::Preprocessor::DumpToken(), and clang::Preprocessor::HandleIdentifier().
| bool clang::Token::isLiteral | ( | ) | const [inline] |
isLiteral - Return true if this is a "literal", like a numeric constant, string, etc.
Definition at line 97 of file Token.h.
References is().
Referenced by clang::Preprocessor::CreateString(), GetFirstChar(), getIdentifierInfo(), getLiteralData(), clang::Lexer::getSpelling(), clang::PTHLexer::Lex(), PrintPreprocessedTokens(), and setLiteralData().
| bool clang::Token::isNot | ( | tok::TokenKind | K | ) | const [inline] |
Definition at line 87 of file Token.h.
Referenced by clang::MacroArgs::ArgNeedsPreexpansion(), clang::TokenConcatenation::AvoidConcat(), clang::CacheTokens(), clang::Preprocessor::CheckEndOfDirective(), clang::Preprocessor::ConcatenateIncludeName(), clang::Preprocessor::DiscardUntilEndOfDirective(), DoPrintMacros(), EvaluateDefined(), EvaluateDirectiveSubExpr(), EvaluateHasIncludeCommon(), EvaluateValue(), clang::DumpRawTokensAction::ExecuteAction(), clang::DumpTokensAction::ExecuteAction(), clang::PreprocessOnlyAction::ExecuteAction(), FindExpectedDiags(), clang::MacroArgs::getArgLength(), getIdentifierInfo(), GetLineValue(), clang::Preprocessor::getSpelling(), clang::Preprocessor::HandleMicrosoftCommentPaste(), clang::PragmaGCCVisibilityHandler::HandlePragma(), clang::PragmaPackHandler::HandlePragma(), clang::PragmaMSStructHandler::HandlePragma(), clang::PragmaUnusedHandler::HandlePragma(), clang::PragmaWeakHandler::HandlePragma(), clang::PragmaOpenCLExtensionHandler::HandlePragma(), clang::Preprocessor::HandlePragmaComment(), clang::Preprocessor::HandlePragmaDependency(), clang::Preprocessor::HandlePragmaMessage(), clang::Preprocessor::HandlePragmaPoison(), clang::html::HighlightMacros(), clang::Preprocessor::LexOnOffSwitch(), LexRawTokensFromMainFile(), ParseAlignPragma(), clang::Preprocessor::ParsePragmaPushOrPopMacro(), clang::RewriteMacrosInInput(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), clang::MacroArgs::StringifyArgument(), clang::html::SyntaxHighlight(), and clang::TokenRewriter::TokenRewriter().
| bool Token::isObjCAtKeyword | ( | tok::ObjCKeywordKind | objcKey | ) | const |
isObjCAtKeyword - Return true if we have an ObjC keyword identifier.
Definition at line 46 of file Lexer.cpp.
References getIdentifierInfo().
| bool clang::Token::needsCleaning | ( | ) | const [inline] |
needsCleaning - Return true if this token has trigraphs or escaped newlines in it.
Definition at line 258 of file Token.h.
References NeedsCleaning.
Referenced by clang::Lexer::ComputePreamble(), clang::Preprocessor::DumpToken(), GetFirstChar(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), clang::Preprocessor::LookUpIdentifierInfo(), and PrintPreprocessedTokens().
| void clang::Token::setAnnotationEndLoc | ( | SourceLocation | L | ) | [inline] |
Definition at line 132 of file Token.h.
References clang::SourceLocation::getRawEncoding(), and isAnnotation().
Referenced by setAnnotationRange().
| void clang::Token::setAnnotationRange | ( | SourceRange | R | ) | [inline] |
Definition at line 146 of file Token.h.
References clang::SourceRange::getBegin(), clang::SourceRange::getEnd(), setAnnotationEndLoc(), and setLocation().
| void clang::Token::setAnnotationValue | ( | void * | val | ) | [inline] |
Definition at line 205 of file Token.h.
References isAnnotation().
Referenced by clang::PragmaGCCVisibilityHandler::HandlePragma().
| void clang::Token::setFlag | ( | TokenFlags | Flag | ) | [inline] |
setFlag - Set the specified flag.
Definition at line 211 of file Token.h.
Referenced by clang::Preprocessor::HandleIdentifier(), clang::PTHLexer::Lex(), clang::Lexer::Lex(), clang::ASTReader::ReadMacroRecord(), and setFlagValue().
| void clang::Token::setFlagValue | ( | TokenFlags | Flag, |
| bool | Val | ||
| ) | [inline] |
setFlagValue - Set a flag to either true or false.
Definition at line 228 of file Token.h.
References clearFlag(), and setFlag().
Referenced by clang::TokenLexer::Lex().
| void clang::Token::setIdentifierInfo | ( | IdentifierInfo * | II | ) | [inline] |
Definition at line 173 of file Token.h.
Referenced by clang::Preprocessor::HandleIdentifier(), clang::PTHLexer::Lex(), clang::Preprocessor::LookUpIdentifierInfo(), and clang::ASTReader::ReadMacroRecord().
| void clang::Token::setKind | ( | tok::TokenKind | K | ) | [inline] |
Definition at line 82 of file Token.h.
Referenced by clang::TokenRewriter::AddTokenBefore(), FindExpectedDiags(), FixDigraph(), clang::Preprocessor::HandleIdentifier(), clang::PragmaGCCVisibilityHandler::HandlePragma(), clang::PragmaUnusedHandler::HandlePragma(), clang::html::HighlightMacros(), clang::PTHLexer::Lex(), clang::TokenLexer::Lex(), clang::Preprocessor::LookUpIdentifierInfo(), clang::Parser::Parser(), clang::ASTReader::ReadMacroRecord(), and clang::MacroArgs::StringifyArgument().
| void clang::Token::setLength | ( | unsigned | Len | ) | [inline] |
Definition at line 123 of file Token.h.
References isAnnotation().
Referenced by clang::TokenRewriter::AddTokenBefore(), clang::Preprocessor::CreateString(), FixDigraph(), clang::PTHLexer::Lex(), and clang::ASTReader::ReadMacroRecord().
| void clang::Token::setLiteralData | ( | const char * | Ptr | ) | [inline] |
Definition at line 196 of file Token.h.
References isLiteral().
Referenced by clang::Preprocessor::CreateString(), and clang::PTHLexer::Lex().
| void clang::Token::setLocation | ( | SourceLocation | L | ) | [inline] |
Definition at line 122 of file Token.h.
Referenced by clang::TokenRewriter::AddTokenBefore(), clang::Preprocessor::CreateString(), FixDigraph(), clang::PragmaGCCVisibilityHandler::HandlePragma(), clang::PragmaUnusedHandler::HandlePragma(), clang::PTHLexer::Lex(), clang::TokenLexer::Lex(), clang::ASTReader::ReadMacroRecord(), setAnnotationRange(), and updateConsecutiveMacroArgTokens().
| void clang::Token::setRawIdentifierData | ( | const char * | Ptr | ) | [inline] |
Definition at line 184 of file Token.h.
References is().
Referenced by clang::Preprocessor::CreateString().
| void clang::Token::startToken | ( | ) | [inline] |
startToken - Reset all flags to cleared.
Definition at line 157 of file Token.h.
Referenced by clang::TokenRewriter::AddTokenBefore(), ComputeDATE_TIME(), clang::Preprocessor::HandleEndOfFile(), clang::PragmaGCCVisibilityHandler::HandlePragma(), clang::PragmaUnusedHandler::HandlePragma(), clang::PTHLexer::Lex(), clang::Lexer::Lex(), clang::Preprocessor::ParsePragmaPushOrPopMacro(), PrintPreprocessedTokens(), clang::ASTReader::ReadMacroRecord(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), and clang::MacroArgs::StringifyArgument().