Pass 3 of 3 · Deep dive of PHAROSPass 1 · OverviewPass 2 · How it works
Code Observatory / function
tokenize
codeintel.text.tokenize
PINNED SOURCE Source and edges from the index at this commit. An execution appears only where one was recorded.
PHAROS at 975a38dMIT license
Around this symbol
Search all symbolsLoading
| Symbol | File | Resolved by |
|---|---|---|
| split_identifier | text.py | same file |
| Symbol | File | Resolved by |
|---|---|---|
| _features | embeddings.py | through an import |
| query_terms | text.py | same file |
| search_text | text.py | same file |
| Symbol | File | Resolved by |
|---|---|---|
| text | text.py | same file |
codeintel.text.tokenizebackend/app/codeintel/text.py, lines 39–55Open source
def tokenize(text: str, *, drop_stopwords: bool = True) -> list[str]: """Lower-cased search tokens for arbitrary text (code or prose). Each identifier yields its parts plus the joined compound when it has more than one part. """ out: list[str] = [] for ident in _IDENT_RE.findall(text): parts = split_identifier(ident) if not parts: continue out.extend(parts) if len(parts) > 1: out.append("".join(parts)) if drop_stopwords: out = [t for t in out if t not in STOPWORDS and len(t) > 1] return outWhat the index recorded
- Signature
def tokenize(text: str, *, drop_stopwords: bool) -> list[str]:- File
backend/app/codeintel/text.py- Lines
- 39–55
- Parser
- python-ast
- Git blob
74781c9c047d5953611fd7634b2cf29e7661b895- Call sites
- 6 named, 1 resolved, 5 left unlinked
- Outside this subset
- 1 more edges reach code outside the graph shown here
