Pass 3 of 3 · Deep dive of PHAROSPass 1 · OverviewPass 2 · How it works
Code Observatory / module
text
codeintel.text
PINNED SOURCE Source and edges from the index at this commit. An execution appears only where one was recorded.
PHAROS at 975a38dMIT license
Around this symbol
Search all symbolsLoading
| Symbol | File | Resolved by |
|---|---|---|
| _after_signature | text.py | same file |
| body_excerpt | text.py | same file |
| query_terms | text.py | same file |
| search_text | text.py | same file |
| semantic_summary | text.py | same file |
| split_identifier | text.py | same file |
| tokenize | text.py | same file |
| tsquery_from_terms | text.py | same file |
| with_description | text.py | same file |
| Symbol | File | Resolved by |
|---|---|---|
| base | base.py | through an import |
| Symbol | File | Resolved by |
|---|---|---|
| describe | describe.py | through an import |
| embeddings | embeddings.py | through an import |
| indexer | indexer.py | through an import |
| retrieval | retrieval.py | through an import |
codeintel.textbackend/app/codeintel/text.py, lines 1–40Open source
"""Text derived from symbols: embedding summaries and full-text search tokens.The same tokenizer is applied to indexed chunks and to queries, so keywordsearch is symmetric: ``parseHTTPResponse``, ``parse_http_response`` and thequery "parse http response" all produce the tokens ``parse http response``(plus the compound ``parsehttpresponse``)."""from __future__ import annotationsimport refrom .parsers.base import Symbol_WORD_RE = re.compile(r"[A-Za-z][A-Za-z0-9]*|[0-9]+")_CAMEL_RE = re.compile(r"[A-Z]+(?=[A-Z][a-z])|[A-Z]?[a-z]+[0-9]*|[A-Z]+[0-9]*|[0-9]+")_IDENT_RE = re.compile(r"[A-Za-z_$][A-Za-z0-9_$]*")# Words that carry no retrieval signal in natural-language queries or# docstrings. Kept small on purpose: identifiers like "get" or "set" matter.STOPWORDS = frozenset( ["a", "an", "and", "are", "as", "at", "be", "by", "does", "do", "for", "from", "has", "have", "how", "i", "in", "into", "is", "it", "its", "me", "my", "of", "on", "or", "that", "the", "this", "to", "was", "what", "when", "where", "which", "who", "why", "will", "with", "you", "your", "can", "should", "would", "could", "there", "their", "them", "then", "than", "these", "those", "code", "function", "method", "class", "file", "implementation", "implement", "implemented", "self", "cls", "this", "def"])MAX_SEARCH_TEXT_CHARS = 4000def split_identifier(token: str) -> list[str]: """``getHTTPResponse_v2`` -> ``['get', 'http', 'response', 'v2']``.""" parts: list[str] = [] for piece in re.split(r"[_$\-]+", token): if not piece: continue sub = _CAMEL_RE.findall(piece) parts.extend(s.lower() for s in (sub or [piece])) return partsdef tokenize(text: str, *, drop_stopwords: bool = True) -> list[str]: """Lower-cased search tokens for arbitrary text (code or prose).Shows the first 40 of 179 lines. The rest is on GitHub at this commit.
What the index recorded
- Signature
module backend.app.codeintel.text- File
backend/app/codeintel/text.py- Lines
- 1–179
- Parser
- python-ast
- Git blob
74781c9c047d5953611fd7634b2cf29e7661b895- Call sites
- 2 named, 0 resolved, 2 left unlinked
- Outside this subset
- 5 more edges reach code outside the graph shown here
