Back to the room

Pass 3 of 3 · Deep dive of PHAROSPass 1 · OverviewPass 2 · How it works

Code Observatory / function

tokenize

codeintel.text.tokenize

PINNED SOURCE Source and edges from the index at this commit. An execution appears only where one was recorded.

PHAROS at 975a38dMIT license

Around this symbol

Search all symbols

Loading

Calls (1)
SymbolFileResolved by
split_identifiertext.pysame file
Called by (3)
SymbolFileResolved by
_featuresembeddings.pythrough an import
query_termstext.pysame file
search_texttext.pysame file
Inside (1)
SymbolFileResolved by
texttext.pysame file
codeintel.text.tokenizebackend/app/codeintel/text.py, lines 39–55Open source
def tokenize(text: str, *, drop_stopwords: bool = True) -> list[str]:    """Lower-cased search tokens for arbitrary text (code or prose).    Each identifier yields its parts plus the joined compound when it has    more than one part.    """    out: list[str] = []    for ident in _IDENT_RE.findall(text):        parts = split_identifier(ident)        if not parts:            continue        out.extend(parts)        if len(parts) > 1:            out.append("".join(parts))    if drop_stopwords:        out = [t for t in out if t not in STOPWORDS and len(t) > 1]    return out

What the index recorded

Signature
def tokenize(text: str, *, drop_stopwords: bool) -> list[str]:
File
backend/app/codeintel/text.py
Lines
39–55
Parser
python-ast
Git blob
74781c9c047d5953611fd7634b2cf29e7661b895
Call sites
6 named, 1 resolved, 5 left unlinked
Outside this subset
1 more edges reach code outside the graph shown here

Read it on GitHub at this commit