Back to the room

Pass 3 of 3 · Deep dive of PHAROSPass 1 · OverviewPass 2 · How it works

Code Observatory / module

text

codeintel.text

PINNED SOURCE Source and edges from the index at this commit. An execution appears only where one was recorded.

PHAROS at 975a38dMIT license

Around this symbol

Search all symbols

Loading

Contains (9)
SymbolFileResolved by
_after_signaturetext.pysame file
body_excerpttext.pysame file
query_termstext.pysame file
search_texttext.pysame file
semantic_summarytext.pysame file
split_identifiertext.pysame file
tokenizetext.pysame file
tsquery_from_termstext.pysame file
with_descriptiontext.pysame file
Imports (1)
SymbolFileResolved by
basebase.pythrough an import
Imported by (4)
SymbolFileResolved by
describedescribe.pythrough an import
embeddingsembeddings.pythrough an import
indexerindexer.pythrough an import
retrievalretrieval.pythrough an import
codeintel.textbackend/app/codeintel/text.py, lines 1–40Open source
"""Text derived from symbols: embedding summaries and full-text search tokens.The same tokenizer is applied to indexed chunks and to queries, so keywordsearch is symmetric: ``parseHTTPResponse``, ``parse_http_response`` and thequery "parse http response" all produce the tokens ``parse http response``(plus the compound ``parsehttpresponse``)."""from __future__ import annotationsimport refrom .parsers.base import Symbol_WORD_RE = re.compile(r"[A-Za-z][A-Za-z0-9]*|[0-9]+")_CAMEL_RE = re.compile(r"[A-Z]+(?=[A-Z][a-z])|[A-Z]?[a-z]+[0-9]*|[A-Z]+[0-9]*|[0-9]+")_IDENT_RE = re.compile(r"[A-Za-z_$][A-Za-z0-9_$]*")# Words that carry no retrieval signal in natural-language queries or# docstrings. Kept small on purpose: identifiers like "get" or "set" matter.STOPWORDS = frozenset(    ["a", "an", "and", "are", "as", "at", "be", "by", "does", "do", "for", "from", "has", "have", "how", "i", "in", "into", "is", "it", "its", "me", "my", "of", "on", "or", "that", "the", "this", "to", "was", "what", "when", "where", "which", "who", "why", "will", "with", "you", "your", "can", "should", "would", "could", "there", "their", "them", "then", "than", "these", "those", "code", "function", "method", "class", "file", "implementation", "implement", "implemented", "self", "cls", "this", "def"])MAX_SEARCH_TEXT_CHARS = 4000def split_identifier(token: str) -> list[str]:    """``getHTTPResponse_v2`` -> ``['get', 'http', 'response', 'v2']``."""    parts: list[str] = []    for piece in re.split(r"[_$\-]+", token):        if not piece:            continue        sub = _CAMEL_RE.findall(piece)        parts.extend(s.lower() for s in (sub or [piece]))    return partsdef tokenize(text: str, *, drop_stopwords: bool = True) -> list[str]:    """Lower-cased search tokens for arbitrary text (code or prose).

Shows the first 40 of 179 lines. The rest is on GitHub at this commit.

What the index recorded

Signature
module backend.app.codeintel.text
File
backend/app/codeintel/text.py
Lines
1–179
Parser
python-ast
Git blob
74781c9c047d5953611fd7634b2cf29e7661b895
Call sites
2 named, 0 resolved, 2 left unlinked
Outside this subset
5 more edges reach code outside the graph shown here

Read it on GitHub at this commit