clang 19.0.0git
Classes | Typedefs | Enumerations | Functions
Token extraction and manipulation

The routines in this group provide access to the tokens within a translation unit, along with a semantic mapping of those tokens to their corresponding cursors. More...

Collaboration diagram for Token extraction and manipulation:

Classes

struct  CXToken
 Describes a single preprocessing token. More...
 

Typedefs

typedef enum CXTokenKind CXTokenKind
 Describes a kind of token.
 

Enumerations

enum  CXTokenKind {
  CXToken_Punctuation , CXToken_Keyword , CXToken_Identifier , CXToken_Literal ,
  CXToken_Comment
}
 Describes a kind of token. More...
 

Functions

CINDEX_LINKAGE CXTokenclang_getToken (CXTranslationUnit TU, CXSourceLocation Location)
 Get the raw lexical token starting with the given location.
 
CINDEX_LINKAGE CXTokenKind clang_getTokenKind (CXToken)
 Determine the kind of the given token.
 
CINDEX_LINKAGE CXString clang_getTokenSpelling (CXTranslationUnit, CXToken)
 Determine the spelling of the given token.
 
CINDEX_LINKAGE CXSourceLocation clang_getTokenLocation (CXTranslationUnit, CXToken)
 Retrieve the source location of the given token.
 
CINDEX_LINKAGE CXSourceRange clang_getTokenExtent (CXTranslationUnit, CXToken)
 Retrieve a source range that covers the given token.
 
CINDEX_LINKAGE void clang_tokenize (CXTranslationUnit TU, CXSourceRange Range, CXToken **Tokens, unsigned *NumTokens)
 Tokenize the source code described by the given range into raw lexical tokens.
 
CINDEX_LINKAGE void clang_annotateTokens (CXTranslationUnit TU, CXToken *Tokens, unsigned NumTokens, CXCursor *Cursors)
 Annotate the given set of tokens by providing cursors for each token that can be mapped to a specific entity within the abstract syntax tree.
 
CINDEX_LINKAGE void clang_disposeTokens (CXTranslationUnit TU, CXToken *Tokens, unsigned NumTokens)
 Free the given set of tokens.
 

Detailed Description

The routines in this group provide access to the tokens within a translation unit, along with a semantic mapping of those tokens to their corresponding cursors.

Typedef Documentation

◆ CXTokenKind

typedef enum CXTokenKind CXTokenKind

Describes a kind of token.

Enumeration Type Documentation

◆ CXTokenKind

Describes a kind of token.

Enumerator
CXToken_Punctuation 

A token that contains some kind of punctuation.

CXToken_Keyword 

A language keyword.

CXToken_Identifier 

An identifier (that is not a keyword).

CXToken_Literal 

A numeric, string, or character literal.

CXToken_Comment 

A comment.

Definition at line 4699 of file Index.h.

Function Documentation

◆ clang_annotateTokens()

CINDEX_LINKAGE void clang_annotateTokens ( CXTranslationUnit  TU,
CXToken Tokens,
unsigned  NumTokens,
CXCursor Cursors 
)

Annotate the given set of tokens by providing cursors for each token that can be mapped to a specific entity within the abstract syntax tree.

This token-annotation routine is equivalent to invoking clang_getCursor() for the source locations of each of the tokens. The cursors provided are filtered, so that only those cursors that have a direct correspondence to the token are accepted. For example, given a function call f(x), clang_getCursor() would provide the following cursors:

  • when the cursor is over the 'f', a DeclRefExpr cursor referring to 'f'.
  • when the cursor is over the '(' or the ')', a CallExpr referring to 'f'.
  • when the cursor is over the 'x', a DeclRefExpr cursor referring to 'x'.

Only the first and last of these cursors will occur within the annotate, since the tokens "f" and "x' directly refer to a function and a variable, respectively, but the parentheses are just a small part of the full syntax of the function call expression, which is not provided as an annotation.

Parameters
TUthe translation unit that owns the given tokens.
Tokensthe set of tokens to annotate.
NumTokensthe number of tokens in Tokens.
Cursorsan array of NumTokens cursors, whose contents will be replaced with the cursors corresponding to each token.

◆ clang_disposeTokens()

CINDEX_LINKAGE void clang_disposeTokens ( CXTranslationUnit  TU,
CXToken Tokens,
unsigned  NumTokens 
)

Free the given set of tokens.

◆ clang_getToken()

CINDEX_LINKAGE CXToken * clang_getToken ( CXTranslationUnit  TU,
CXSourceLocation  Location 
)

Get the raw lexical token starting with the given location.

Parameters
TUthe translation unit whose text is being tokenized.
Locationthe source location with which the token starts.
Returns
The token starting with the given location or NULL if no such token exist. The returned pointer must be freed with clang_disposeTokens before the translation unit is destroyed.

◆ clang_getTokenExtent()

CINDEX_LINKAGE CXSourceRange clang_getTokenExtent ( CXTranslationUnit  ,
CXToken   
)

Retrieve a source range that covers the given token.

◆ clang_getTokenKind()

CINDEX_LINKAGE CXTokenKind clang_getTokenKind ( CXToken  )

Determine the kind of the given token.

◆ clang_getTokenLocation()

CINDEX_LINKAGE CXSourceLocation clang_getTokenLocation ( CXTranslationUnit  ,
CXToken   
)

Retrieve the source location of the given token.

◆ clang_getTokenSpelling()

CINDEX_LINKAGE CXString clang_getTokenSpelling ( CXTranslationUnit  ,
CXToken   
)

Determine the spelling of the given token.

The spelling of a token is the textual representation of that token, e.g., the text of an identifier or keyword.

◆ clang_tokenize()

CINDEX_LINKAGE void clang_tokenize ( CXTranslationUnit  TU,
CXSourceRange  Range,
CXToken **  Tokens,
unsigned NumTokens 
)

Tokenize the source code described by the given range into raw lexical tokens.

Parameters
TUthe translation unit whose text is being tokenized.
Rangethe source range in which text should be tokenized. All of the tokens produced by tokenization will fall within this source range,
Tokensthis pointer will be set to point to the array of tokens that occur within the given source range. The returned pointer must be freed with clang_disposeTokens() before the translation unit is destroyed.
NumTokenswill be set to the number of tokens in the *Tokens array.