Pass 3 of 3 · Deep dive of PHAROSPass 1 · OverviewPass 2 · How it works
Code Observatory / method
_features
codeintel.embeddings.HashEmbedder._features
PINNED SOURCE Source and edges from the index at this commit. An execution appears only where one was recorded.
PHAROS at 975a38dMIT license
Around this symbol
Search all symbolsLoading
| Symbol | File | Resolved by |
|---|---|---|
| tokenize | text.py | through an import |
| Symbol | File | Resolved by |
|---|---|---|
| embed_documents | embeddings.py | same file |
| Symbol | File | Resolved by |
|---|---|---|
| HashEmbedder | embeddings.py | same file |
codeintel.embeddings.HashEmbedder._featuresbackend/app/codeintel/embeddings.py, lines 113–128Open source
def _features(self, text: str) -> dict[int, float]: tokens = tokenize(text) counts: dict[str, int] = {} for tok in tokens: counts[tok] = counts.get(tok, 0) + 1 for a, b in zip(tokens, tokens[1:], strict=False): key = f"{a}_{b}" counts[key] = counts.get(key, 0) + 1 features: dict[int, float] = {} for tok, tf in counts.items(): digest = hashlib.blake2b(tok.encode(), digest_size=8).digest() bucket = int.from_bytes(digest[:4], "little") % self.dim sign = 1.0 if digest[4] & 1 else -1.0 weight = (1.0 + math.log(tf)) * (0.5 if "_" in tok else 1.0) features[bucket] = features.get(bucket, 0.0) + sign * weight return featuresWhat the index recorded
- Signature
def _features(self, text: str) -> dict[int, float]:- File
backend/app/codeintel/embeddings.py- Lines
- 113–128
- Parser
- python-ast
- Git blob
df7f8202f23ca6ad2c57aca5f09b435eb6486d79- Call sites
- 9 named, 2 resolved, 7 left unlinked
- Outside this subset
- 1 more edges reach code outside the graph shown here
