clang::Lexer Class Reference

#include <Lexer.h>

Inheritance diagram for clang::Lexer:

Collaboration diagram for clang::Lexer:

List of all members.

Public Member Functions
	Lexer (FileID FID, const llvm::MemoryBuffer *InputBuffer, Preprocessor &PP)
	Lexer (SourceLocation FileLoc, const LangOptions &Features, const char *BufStart, const char *BufPtr, const char *BufEnd)
	Lexer (FileID FID, const llvm::MemoryBuffer *InputBuffer, const SourceManager &SM, const LangOptions &Features)
const LangOptions &	getFeatures () const
SourceLocation	getFileLoc () const
void	Lex (Token &Result)
bool	isPragmaLexer () const
	isPragmaLexer - Returns true if this Lexer is being used to lex a pragma.
void	IndirectLex (Token &Result)
bool	LexFromRawLexer (Token &Result)
bool	isKeepWhitespaceMode () const
void	SetKeepWhitespaceMode (bool Val)
bool	inKeepCommentMode () const
void	SetCommentRetentionState (bool Mode)
const char *	getBufferStart () const
std::string	ReadToEndOfLine ()
DiagnosticBuilder	Diag (const char *Loc, unsigned DiagID) const
SourceLocation	getSourceLocation (const char *Loc, unsigned TokLen=1) const
SourceLocation	getSourceLocation ()
const char *	getBufferLocation () const
	Return the current location in the buffer.
char	getAndAdvanceChar (const char *&Ptr, Token &Tok)
Static Public Member Functions
static Lexer *	Create_PragmaLexer (SourceLocation SpellingLoc, SourceLocation ExpansionLocStart, SourceLocation ExpansionLocEnd, unsigned TokLen, Preprocessor &PP)
static std::string	Stringify (const std::string &Str, bool Charify=false)
static void	Stringify (SmallVectorImpl< char > &Str)
static unsigned	getSpelling (const Token &Tok, const char *&Buffer, const SourceManager &SourceMgr, const LangOptions &Features, bool *Invalid=0)
static std::string	getSpelling (const Token &Tok, const SourceManager &SourceMgr, const LangOptions &Features, bool *Invalid=0)
static StringRef	getSpelling (SourceLocation loc, SmallVectorImpl< char > &buffer, const SourceManager &SourceMgr, const LangOptions &Features, bool *invalid=0)
static unsigned	MeasureTokenLength (SourceLocation Loc, const SourceManager &SM, const LangOptions &LangOpts)
static SourceLocation	GetBeginningOfToken (SourceLocation Loc, const SourceManager &SM, const LangOptions &LangOpts)
	Given a location any where in a source buffer, find the location that corresponds to the beginning of the token in which the original source location lands.
static SourceLocation	AdvanceToTokenCharacter (SourceLocation TokStart, unsigned Character, const SourceManager &SM, const LangOptions &Features)
static SourceLocation	getLocForEndOfToken (SourceLocation Loc, unsigned Offset, const SourceManager &SM, const LangOptions &Features)
	Computes the source location just past the end of the token at this source location.
static bool	isAtStartOfMacroExpansion (SourceLocation loc, const SourceManager &SM, const LangOptions &LangOpts, SourceLocation *MacroBegin=0)
	Returns true if the given MacroID location points at the first token of the macro expansion.
static bool	isAtEndOfMacroExpansion (SourceLocation loc, const SourceManager &SM, const LangOptions &LangOpts, SourceLocation *MacroEnd=0)
	Returns true if the given MacroID location points at the last token of the macro expansion.
static CharSourceRange	makeFileCharRange (CharSourceRange Range, const SourceManager &SM, const LangOptions &LangOpts)
	Accepts a range and returns a character range with file locations.
static StringRef	getSourceText (CharSourceRange Range, const SourceManager &SM, const LangOptions &LangOpts, bool *Invalid=0)
	Returns a string for the source that the range encompasses.
static StringRef	getImmediateMacroName (SourceLocation Loc, const SourceManager &SM, const LangOptions &LangOpts)
	Retrieve the name of the immediate macro expansion.
static std::pair< unsigned, bool >	ComputePreamble (const llvm::MemoryBuffer *Buffer, const LangOptions &Features, unsigned MaxLines=0)
	Compute the preamble of the given file.
static bool	isObviouslySimpleCharacter (char C)
static char	getCharAndSizeNoWarn (const char *Ptr, unsigned &Size, const LangOptions &Features)
static unsigned	getEscapedNewLineSize (const char *P)
static const char *	SkipEscapedNewLines (const char *P)
static SourceLocation	findLocationAfterToken (SourceLocation loc, tok::TokenKind TKind, const SourceManager &SM, const LangOptions &LangOpts, bool SkipTrailingWhitespaceAndNewLine)
	Checks that the given token is the first token that occurs after the given location (this excludes comments and whitespace). Returns the location immediately after the specified token. If the token is not found or the location is inside a macro, the returned source location will be invalid.
Friends
class	Preprocessor

---

Detailed Description

Lexer - This provides a simple interface that turns a text buffer into a stream of tokens. This provides no support for file reading or buffering, or buffering/seeking of tokens, only forward lexing is supported. It relies on the specified Preprocessor object to handle preprocessor directives, etc.

Definition at line 46 of file Lexer.h.

---

Constructor & Destructor Documentation

Lexer::Lexer	(	FileID	FID,
		const llvm::MemoryBuffer *	InputFile,
		Preprocessor &	PP
	)

Lexer constructor - Create a new lexer object for the specified buffer with the specified preprocessor managing the lexing process. This lexer assumes that the associated file buffer and Preprocessor objects will outlive it, so it doesn't take ownership of either of them.

Definition at line 117 of file Lexer.cpp.

References clang::Preprocessor::getCommentRetentionState(), and SetCommentRetentionState().

Lexer::Lexer	(	SourceLocation	fileloc,
		const LangOptions &	features,
		const char *	BufStart,
		const char *	BufPtr,
		const char *	BufEnd
	)

Lexer constructor - Create a new raw lexer object. This object is only suitable for calls to 'LexRawToken'. This lexer assumes that the text range will outlive it, so it doesn't take ownership of it.

Definition at line 132 of file Lexer.cpp.

References clang::PreprocessorLexer::LexingRawMode.

Lexer::Lexer	(	FileID	FID,
		const llvm::MemoryBuffer *	FromFile,
		const SourceManager &	SM,
		const LangOptions &	features
	)

Lexer constructor - Create a new raw lexer object. This object is only suitable for calls to 'LexRawToken'. This lexer assumes that the text range will outlive it, so it doesn't take ownership of it.

Definition at line 145 of file Lexer.cpp.

References clang::PreprocessorLexer::LexingRawMode.

---

Member Function Documentation

SourceLocation Lexer::AdvanceToTokenCharacter	(	SourceLocation	TokStart,
		unsigned	CharNo,
		const SourceManager &	SM,
		const LangOptions &	Features
	)		[static]

AdvanceToTokenCharacter - If the current SourceLocation specifies a location at the start of a token, return a new location that specifies a character within the token. This handles trigraphs and escaped newlines.

AdvanceToTokenCharacter - Given a location that specifies the start of a token, return a new location that specifies a character within the token.

Definition at line 658 of file Lexer.cpp.

References clang::SourceManager::getCharacterData(), getCharAndSizeNoWarn(), clang::SourceLocation::getLocWithOffset(), isObviouslySimpleCharacter(), and SkipEscapedNewLines().

Referenced by clang::StringLiteral::getLocationOfByte(), and ProcessUCNEscape().

std::pair< unsigned, bool > Lexer::ComputePreamble	(	const llvm::MemoryBuffer *	Buffer,
		const LangOptions &	Features,
		unsigned	MaxLines = 0
	)		[static]

Compute the preamble of the given file.

The preamble of a file contains the initial comments, include directives, and other preprocessor directives that occur before the code in this particular file actually begins. The preamble of the main source file is a potential prefix header.

Parameters:

Buffer	The memory buffer containing the file's contents.
MaxLines	If non-zero, restrict the length of the preamble to fewer than this number of lines.

Returns:

The offset into the file where the preamble ends and the rest of the file begins along with a boolean value indicating whether the preamble ends at the beginning of a new line.

Definition at line 510 of file Lexer.cpp.

References clang::SourceLocation::getFromRawEncoding(), clang::Token::getKind(), clang::Token::getLength(), clang::Token::getLocation(), clang::SourceLocation::getRawEncoding(), clang::Token::getRawIdentifierData(), clang::Token::isAtStartOfLine(), and clang::Token::needsCleaning().

Referenced by clang::PrintPreambleAction::ExecuteAction().

Lexer * Lexer::Create_PragmaLexer	(	SourceLocation	SpellingLoc,
		SourceLocation	ExpansionLocStart,
		SourceLocation	ExpansionLocEnd,
		unsigned	TokLen,
		Preprocessor &	PP
	)		[static]

Create_PragmaLexer: Lexer constructor - Create a new lexer object for _Pragma expansion. This has a variety of magic semantics that this method sets up. It returns a new'd Lexer that must be delete'd when done.

Create_PragmaLexer: Lexer constructor - Create a new lexer object for _Pragma expansion. This has a variety of magic semantics that this method sets up. It returns a new'd Lexer that must be delete'd when done.

On entrance to this routine, TokStartLoc is a macro location which has a spelling loc that indicates the bytes to be lexed for the token and an expansion location that indicates where all lexed tokens should be "expanded from".

FIXME: It would really be nice to make _Pragma just be a wrapper around a normal lexer that remaps tokens as they fly by. This would require making Preprocessor::Lex virtual. Given that, we could just dump in a magic lexer interface that could handle this stuff. This would pull GetMappedTokenLoc out of the critical path of the lexer!

Definition at line 171 of file Lexer.cpp.

References clang::SourceManager::createExpansionLoc(), clang::SourceManager::getBuffer(), clang::SourceManager::getCharacterData(), clang::SourceManager::getFileID(), clang::SourceManager::getLocForStartOfFile(), clang::Preprocessor::getSourceManager(), and clang::PreprocessorLexer::ParsingPreprocessorDirective.

DiagnosticBuilder Lexer::Diag	(	const char *	Loc,
		unsigned	DiagID
	)		const

Diag - Forwarding function for diagnostics. This translate a source position in the current buffer into a SourceLocation object for rendering.

Definition at line 1176 of file Lexer.cpp.

References clang::Preprocessor::Diag(), getSourceLocation(), and clang::PreprocessorLexer::PP.

Referenced by DecodeTrigraphChar(), and isEndOfBlockCommentWithEscapedNewLine().

SourceLocation Lexer::findLocationAfterToken	(	SourceLocation	loc,
		tok::TokenKind	TKind,
		const SourceManager &	SM,
		const LangOptions &	LangOpts,
		bool	SkipTrailingWhitespaceAndNewLine
	)		[static]

Checks that the given token is the first token that occurs after the given location (this excludes comments and whitespace). Returns the location immediately after the specified token. If the token is not found or the location is inside a macro, the returned source location will be invalid.

Definition at line 1270 of file Lexer.cpp.

References clang::SourceManager::getBufferData(), clang::SourceManager::getCharacterData(), clang::SourceManager::getDecomposedLoc(), getLocForEndOfToken(), clang::SourceManager::getLocForStartOfFile(), clang::SourceLocation::getLocWithOffset(), isAtEndOfMacroExpansion(), isHorizontalWhitespace(), clang::SourceLocation::isMacroID(), and isVerticalWhitespace().

Referenced by GenerateFixForUnusedDecl().

char clang::Lexer::getAndAdvanceChar	(	const char *&	Ptr,
		Token &	Tok
	)		[inline]

getAndAdvanceChar - Read a single 'character' from the specified buffer, advance over it, and return it. This is tricky in several cases. Here we just handle the trivial case and fall-back to the non-inlined getCharAndSizeSlow method to handle the hard case.

Definition at line 437 of file Lexer.h.

References isObviouslySimpleCharacter().

Referenced by ReadToEndOfLine().

SourceLocation Lexer::GetBeginningOfToken	(	SourceLocation	Loc,
		const SourceManager &	SM,
		const LangOptions &	LangOpts
	)		[static]

Given a location any where in a source buffer, find the location that corresponds to the beginning of the token in which the original source location lands.

Parameters:

Loc

Definition at line 481 of file Lexer.cpp.

References getBeginningOfFileToken(), clang::SourceManager::getDecomposedLoc(), clang::SourceLocation::getLocWithOffset(), clang::SourceManager::getSpellingLoc(), clang::SourceLocation::isFileID(), and clang::SourceManager::isMacroArgExpansion().

const char* clang::Lexer::getBufferLocation

(

)

const [inline]

Return the current location in the buffer.

Definition at line 219 of file Lexer.h.

const char* clang::Lexer::getBufferStart

(

)

const [inline]

Definition at line 199 of file Lexer.h.

Referenced by clang::html::SyntaxHighlight().

static char clang::Lexer::getCharAndSizeNoWarn	(	const char *	Ptr,
		unsigned &	Size,
		const LangOptions &	Features
	)		[inline, static]

getCharAndSizeNoWarn - Like the getCharAndSize method, but does not ever emit a warning.

Definition at line 488 of file Lexer.h.

References isObviouslySimpleCharacter().

Referenced by AdvanceToTokenCharacter(), getSpelling(), and isHexaLiteral().

unsigned Lexer::getEscapedNewLineSize

(

const char *

Ptr

)

[static]

getEscapedNewLineSize - Return the size of the specified escaped newline, or 0 if it is not an escaped newline. P[-1] is known to be a "\" on entry to this function.

getEscapedNewLineSize - Return the size of the specified escaped newline, or 0 if it is not an escaped newline. P[-1] is known to be a "\" or a trigraph equivalent on entry to this function.

Definition at line 1223 of file Lexer.cpp.

References isWhitespace().

Referenced by SkipEscapedNewLines().

const LangOptions& clang::Lexer::getFeatures

(

)

const [inline]

getFeatures - Return the language features currently enabled. NOTE: this lexer modifies features as a file is parsed!

Definition at line 122 of file Lexer.h.

Referenced by DecodeTrigraphChar(), and isEndOfBlockCommentWithEscapedNewLine().

SourceLocation clang::Lexer::getFileLoc

(

)

const [inline]

getFileLoc - Return the File Location for the file we are lexing out of. The physical location encodes the location where the characters come from, the virtual location encodes where we should *claim* the characters came from. Currently this is only used by _Pragma handling.

Definition at line 128 of file Lexer.h.

StringRef Lexer::getImmediateMacroName	(	SourceLocation	Loc,
		const SourceManager &	SM,
		const LangOptions &	LangOpts
	)		[static]

Retrieve the name of the immediate macro expansion.

This routine starts from a source location, and finds the name of the macro responsible for its immediate expansion. It looks through any intervening macro argument expansions to compute this. It returns a StringRef which refers to the SourceManager-owned buffer of the source where that macro name is spelled. Thus, the result shouldn't out-live that SourceManager.

Definition at line 927 of file Lexer.cpp.

References clang::PreprocessorLexer::FID, clang::SourceManager::getBufferData(), clang::SourceManager::getDecomposedLoc(), clang::SrcMgr::SLocEntry::getExpansion(), clang::SrcMgr::ExpansionInfo::getExpansionLocStart(), clang::SourceManager::getFileID(), clang::SourceManager::getImmediateExpansionRange(), clang::SourceManager::getSLocEntry(), clang::SrcMgr::ExpansionInfo::getSpellingLoc(), clang::SourceManager::getSpellingLoc(), clang::SourceLocation::isFileID(), clang::SourceManager::isInFileID(), clang::SrcMgr::ExpansionInfo::isMacroArgExpansion(), clang::SourceLocation::isMacroID(), and MeasureTokenLength().

Referenced by clang::ento::CheckerContext::getMacroNameOrSpelling().

SourceLocation Lexer::getLocForEndOfToken	(	SourceLocation	Loc,
		unsigned	Offset,
		const SourceManager &	SM,
		const LangOptions &	Features
	)		[static]

Computes the source location just past the end of the token at this source location.

This routine can be used to produce a source location that points just past the end of the token referenced by Loc, and is generally used when a diagnostic needs to point just after a token where it expected something different that it received. If the returned source location would not be meaningful (e.g., if it points into a macro), this routine returns an invalid source location.

Parameters:

Offset

an offset from the end of the token, where the source location should refer to. The default offset (0) produces a source location pointing just past the end of the token; an offset of 1 produces a source location pointing to the last character in the token, etc.

Definition at line 717 of file Lexer.cpp.

References clang::SourceLocation::getLocWithOffset(), isAtEndOfMacroExpansion(), clang::SourceLocation::isInvalid(), clang::SourceLocation::isMacroID(), and MeasureTokenLength().

Referenced by clang::Sema::CheckLogicalOperands(), findLocationAfterToken(), clang::arcmt::trans::findSemiAfterLocation(), and makeRangeFromFileLocs().

SourceLocation Lexer::getSourceLocation	(	const char *	Loc,
		unsigned	TokLen = 1
	)		const

getSourceLocation - Return a source location identifier for the specified offset in the current file.

Definition at line 1157 of file Lexer.cpp.

References clang::SourceLocation::getLocWithOffset(), GetMappedTokenLoc(), clang::SourceLocation::isFileID(), and clang::PreprocessorLexer::PP.

SourceLocation clang::Lexer::getSourceLocation

(

)

[inline, virtual]

getSourceLocation - Return a source location for the next character in the current file.

Implements clang::PreprocessorLexer.

Definition at line 216 of file Lexer.h.

References getSourceLocation().

Referenced by Diag(), and getSourceLocation().

StringRef Lexer::getSourceText	(	CharSourceRange	Range,
		const SourceManager &	SM,
		const LangOptions &	LangOpts,
		bool *	Invalid = 0
	)		[static]

Returns a string for the source that the range encompasses.

Definition at line 891 of file Lexer.cpp.

References clang::CharSourceRange::getBegin(), clang::SourceManager::getBufferData(), clang::SourceManager::getDecomposedLoc(), clang::CharSourceRange::getEnd(), clang::SourceManager::isInFileID(), clang::CharSourceRange::isInvalid(), and makeFileCharRange().

Referenced by GCRewriteFinalize().

unsigned Lexer::getSpelling	(	const Token &	Tok,
		const char *&	Buffer,
		const SourceManager &	SourceMgr,
		const LangOptions &	Features,
		bool *	Invalid = 0
	)		[static]

getSpelling - This method is used to get the spelling of a token into a preallocated buffer, instead of as an std::string. The caller is required to allocate enough space for the token, which is guaranteed to be at least Tok.getLength() bytes long. The length of the actual result is returned.

Note that this method may do two possible things: it may either fill in the buffer specified with characters, or it may *change the input pointer* to point to a constant buffer with the data already in it (avoiding a copy). The caller is not allowed to modify the returned buffer pointer if an internal buffer is returned.

getSpelling - This method is used to get the spelling of a token into a preallocated buffer, instead of as an std::string. The caller is required to allocate enough space for the token, which is guaranteed to be at least Tok.getLength() bytes long. The actual length of the token is returned.

Note that this method may do two possible things: it may either fill in the buffer specified with characters, or it may *change the input pointer* to point to a constant buffer with the data already in it (avoiding a copy). The caller is not allowed to modify the returned buffer pointer if an internal buffer is returned.

Definition at line 330 of file Lexer.cpp.

References clang::SourceManager::getCharacterData(), getCharAndSizeNoWarn(), clang::Token::getIdentifierInfo(), clang::Token::getLength(), clang::Token::getLiteralData(), clang::Token::getLocation(), clang::Token::getRawIdentifierData(), clang::Token::is(), clang::Token::isLiteral(), and clang::Token::needsCleaning().

Referenced by clang::ento::CheckerContext::getMacroNameOrSpelling(), and clang::StringLiteralParser::getOffsetOfStringByte().

std::string Lexer::getSpelling	(	const Token &	Tok,
		const SourceManager &	SourceMgr,
		const LangOptions &	Features,
		bool *	Invalid = 0
	)		[static]

getSpelling() - Return the 'spelling' of the Tok token. The spelling of a token is the characters used to represent the token in the source file after trigraph expansion and escaped-newline folding. In particular, this wants to get the true, uncanonicalized, spelling of things like digraphs UCNs, etc.

getSpelling() - Return the 'spelling' of this token. The spelling of a token are the characters used to represent the token in the source file after trigraph expansion and escaped-newline folding. In particular, this wants to get the true, uncanonicalized, spelling of things like digraphs UCNs, etc.

Definition at line 289 of file Lexer.cpp.

References clang::SourceManager::getCharacterData(), getCharAndSizeNoWarn(), clang::Token::getLength(), clang::Token::getLocation(), and clang::Token::needsCleaning().

StringRef Lexer::getSpelling	(	SourceLocation	loc,
		SmallVectorImpl< char > &	buffer,
		const SourceManager &	SM,
		const LangOptions &	options,
		bool *	invalid = 0
	)		[static]

getSpelling - This method is used to get the spelling of the token at the given source location. If, as is usually true, it is not necessary to copy any data, then the returned string may not point into the provided buffer.

This method lexes at the expansion depth of the given location and does not jump to the expansion or spelling location.

getSpelling() - Return the 'spelling' of this token. The spelling of a token are the characters used to represent the token in the source file after trigraph expansion and escaped-newline folding. In particular, this wants to get the true, uncanonicalized, spelling of things like digraphs UCNs, etc.

Definition at line 241 of file Lexer.cpp.

References clang::SourceManager::getBufferData(), getCharAndSizeNoWarn(), clang::SourceManager::getDecomposedLoc(), clang::Token::getLength(), and clang::SourceManager::getLocForStartOfFile().

void clang::Lexer::IndirectLex

(

Token &

Result

)

[inline, virtual]

IndirectLex - An indirect call to 'Lex' that can be invoked via the PreprocessorLexer interface.

Implements clang::PreprocessorLexer.

Definition at line 155 of file Lexer.h.

References Lex().

bool clang::Lexer::inKeepCommentMode

(

)

const [inline]

inKeepCommentMode - Return true if the lexer should return comments as tokens.

Definition at line 186 of file Lexer.h.

bool Lexer::isAtEndOfMacroExpansion	(	SourceLocation	loc,
		const SourceManager &	SM,
		const LangOptions &	LangOpts,
		SourceLocation *	MacroEnd = 0
	)		[static]

Returns true if the given MacroID location points at the last token of the macro expansion.

Parameters:

MacroBegin

If non-null and function returns true, it is set to end location of the macro.

Definition at line 765 of file Lexer.cpp.

References clang::PreprocessorLexer::FID, clang::SrcMgr::SLocEntry::getExpansion(), clang::SourceManager::getFileID(), clang::SourceLocation::getLocWithOffset(), clang::SourceManager::getSLocEntry(), clang::SourceManager::getSpellingLoc(), clang::SourceLocation::isFileID(), clang::SourceManager::isInFileID(), clang::SourceLocation::isMacroID(), clang::SourceLocation::isValid(), and MeasureTokenLength().

Referenced by findLocationAfterToken(), clang::arcmt::trans::findSemiAfterLocation(), getLocForEndOfToken(), and makeFileCharRange().

bool Lexer::isAtStartOfMacroExpansion	(	SourceLocation	loc,
		const SourceManager &	SM,
		const LangOptions &	LangOpts,
		SourceLocation *	MacroBegin = 0
	)		[static]

Returns true if the given MacroID location points at the first token of the macro expansion.

Parameters:

MacroBegin

If non-null and function returns true, it is set to begin location of the macro.

Definition at line 739 of file Lexer.cpp.

References clang::SourceManager::getDecomposedLoc(), clang::SrcMgr::SLocEntry::getExpansion(), clang::SourceManager::getSLocEntry(), clang::SourceLocation::isFileID(), clang::SourceLocation::isMacroID(), and clang::SourceLocation::isValid().

Referenced by makeFileCharRange().

bool clang::Lexer::isKeepWhitespaceMode

(

)

const [inline]

isKeepWhitespaceMode - Return true if the lexer should return tokens for every character in the file, including whitespace and comments. This should only be used in raw mode, as the preprocessor is not prepared to deal with the excess tokens.

Definition at line 172 of file Lexer.h.

Referenced by SetCommentRetentionState().

static bool clang::Lexer::isObviouslySimpleCharacter

(

char

C

)

[inline, static]

isObviouslySimpleCharacter - Return true if the specified character is obviously the same in translation phase 1 and translation phase 3. This can return false for characters that end up being the same, but it will never return true for something that needs to be mapped.

Definition at line 429 of file Lexer.h.

Referenced by AdvanceToTokenCharacter(), getAndAdvanceChar(), and getCharAndSizeNoWarn().

bool clang::Lexer::isPragmaLexer

(

)

const [inline]

isPragmaLexer - Returns true if this Lexer is being used to lex a pragma.

Definition at line 151 of file Lexer.h.

void clang::Lexer::Lex

(

Token &

Result

)

[inline]

Lex - Return the next token in the file. If this is the end of file, it return the tok::eof token. Return true if an error occurred and compilation should terminate, false if normal. This implicitly involves the preprocessor.

Definition at line 134 of file Lexer.h.

References clang::Token::setFlag(), clang::Token::StartOfLine, and clang::Token::startToken().

Referenced by IndirectLex(), LexFromRawLexer(), and ReadToEndOfLine().

bool clang::Lexer::LexFromRawLexer

(

Token &

Result

)

[inline]

LexFromRawLexer - Lex a token from a designated raw lexer (one with no associated preprocessor object. Return true if the 'next character to read' pointer points at the end of the lexer buffer, false otherwise.

Definition at line 160 of file Lexer.h.

References Lex(), and clang::PreprocessorLexer::LexingRawMode.

Referenced by clang::TokenRewriter::TokenRewriter().

CharSourceRange Lexer::makeFileCharRange	(	CharSourceRange	Range,
		const SourceManager &	SM,
		const LangOptions &	LangOpts
	)		[static]

Accepts a range and returns a character range with file locations.

Returns a null range if a part of the range resides inside a macro expansion or the range does not reside on the same FileID.

Definition at line 827 of file Lexer.cpp.

References clang::PreprocessorLexer::FID, clang::CharSourceRange::getBegin(), clang::SourceManager::getDecomposedLoc(), clang::CharSourceRange::getEnd(), clang::SrcMgr::SLocEntry::getExpansion(), clang::SourceLocation::getLocWithOffset(), clang::SourceManager::getSLocEntry(), clang::SrcMgr::ExpansionInfo::getSpellingLoc(), isAtEndOfMacroExpansion(), isAtStartOfMacroExpansion(), clang::CharSourceRange::isCharRange(), clang::SourceLocation::isFileID(), clang::SourceManager::isInFileID(), clang::FileID::isInvalid(), clang::SourceLocation::isInvalid(), clang::SrcMgr::ExpansionInfo::isMacroArgExpansion(), clang::SourceLocation::isMacroID(), clang::CharSourceRange::isTokenRange(), makeRangeFromFileLocs(), clang::CharSourceRange::setBegin(), and clang::CharSourceRange::setEnd().

Referenced by getSourceText().

unsigned Lexer::MeasureTokenLength	(	SourceLocation	Loc,
		const SourceManager &	SM,
		const LangOptions &	LangOpts
	)		[static]

MeasureTokenLength - Relex the token at the specified location and return its length in bytes in the input file. If the token needs cleaning (e.g. includes a trigraph or an escaped newline) then this count includes bytes that are part of that.

Definition at line 389 of file Lexer.cpp.

References clang::SourceManager::getBufferData(), clang::SourceManager::getDecomposedLoc(), clang::SourceManager::getExpansionLoc(), clang::Token::getLength(), clang::SourceManager::getLocForStartOfFile(), isWhitespace(), and SetCommentRetentionState().

Referenced by clang::TextDiagnostic::emitDiagnosticLoc(), EmitLocation(), getImmediateMacroName(), getImmediateMacroName(), getLocForEndOfToken(), clang::Rewriter::getRangeSize(), clang::Rewriter::getRewrittenText(), clang::html::HighlightRange(), and isAtEndOfMacroExpansion().

std::string Lexer::ReadToEndOfLine

(

)

ReadToEndOfLine - Read the rest of the current preprocessor line as an uninterpreted string. This switches the lexer out of directive mode.

Definition at line 2231 of file Lexer.cpp.

References clang::Preprocessor::CodeCompleteNaturalLanguage(), getAndAdvanceChar(), clang::Token::is(), Lex(), clang::PreprocessorLexer::ParsingFilename, clang::PreprocessorLexer::ParsingPreprocessorDirective, and clang::PreprocessorLexer::PP.

void clang::Lexer::SetCommentRetentionState

(

bool

Mode

)

[inline]

SetCommentRetentionMode - Change the comment retention mode of the lexer to the specified mode. This is really only useful when lexing in raw mode, because otherwise the lexer needs to manage this.

Definition at line 193 of file Lexer.h.

References isKeepWhitespaceMode().

Referenced by FindExpectedDiags(), getBeginningOfFileToken(), Lexer(), LexRawTokensFromMainFile(), and MeasureTokenLength().

void clang::Lexer::SetKeepWhitespaceMode

(

bool

Val

)

[inline]

SetKeepWhitespaceMode - This method lets clients enable or disable whitespace retention mode.

Definition at line 178 of file Lexer.h.

References clang::PreprocessorLexer::LexingRawMode.

Referenced by clang::DumpRawTokensAction::ExecuteAction(), and clang::TokenRewriter::TokenRewriter().

const char * Lexer::SkipEscapedNewLines

(

const char *

P

)

[static]

SkipEscapedNewLines - If P points to an escaped newline (or a series of them), skip over them and return the first non-escaped-newline found, otherwise return P.

Definition at line 1246 of file Lexer.cpp.

References getEscapedNewLineSize().

Referenced by AdvanceToTokenCharacter().

std::string Lexer::Stringify	(	const std::string &	Str,
		bool	Charify = false
	)		[static]

Stringify - Convert the specified string into a C string by escaping '\' and " characters. This does not add surrounding ""'s to the string. If Charify is true, this escapes the ' character instead of ".

Stringify - Convert the specified string into a C string, with surrounding ""'s, and with escaped \ and " characters.

Definition at line 209 of file Lexer.cpp.

Referenced by clang::HeaderSearch::NormalizeDashIncludePath(), and clang::MacroArgs::StringifyArgument().

void Lexer::Stringify

(

SmallVectorImpl< char > &

Str

)

[static]

Stringify - Convert the specified string into a C string by escaping '\' and " characters. This does not add surrounding ""'s to the string.

Definition at line 223 of file Lexer.cpp.

---

Friends And Related Function Documentation

friend class Preprocessor [friend]

Reimplemented from clang::PreprocessorLexer.

Definition at line 88 of file Lexer.h.

---

The documentation for this class was generated from the following files:

• Lexer.h
• Lexer.cpp