Pass 3 of 3 · Deep dive of PHAROSPass 1 · OverviewPass 2 · How it works
Code Observatory / class
HashEmbedder
codeintel.embeddings.HashEmbedder
PINNED SOURCE Source and edges from the index at this commit. An execution appears only where one was recorded.
PHAROS at 975a38dMIT license
Around this symbol
Search all symbolsLoading
| Symbol | File | Resolved by |
|---|---|---|
| get_embedder | embeddings.py | same file |
| Symbol | File | Resolved by |
|---|---|---|
| _features | embeddings.py | same file |
| embed_documents | embeddings.py | same file |
| Symbol | File | Resolved by |
|---|---|---|
| embeddings | embeddings.py | same file |
codeintel.embeddings.HashEmbedderbackend/app/codeintel/embeddings.py, lines 107–137Open source
class HashEmbedder(Embedder): """Signed feature hashing of unigrams and bigrams (lexical, offline).""" model_id = "hash-v1" semantic = False def _features(self, text: str) -> dict[int, float]: tokens = tokenize(text) counts: dict[str, int] = {} for tok in tokens: counts[tok] = counts.get(tok, 0) + 1 for a, b in zip(tokens, tokens[1:], strict=False): key = f"{a}_{b}" counts[key] = counts.get(key, 0) + 1 features: dict[int, float] = {} for tok, tf in counts.items(): digest = hashlib.blake2b(tok.encode(), digest_size=8).digest() bucket = int.from_bytes(digest[:4], "little") % self.dim sign = 1.0 if digest[4] & 1 else -1.0 weight = (1.0 + math.log(tf)) * (0.5 if "_" in tok else 1.0) features[bucket] = features.get(bucket, 0.0) + sign * weight return features def embed_documents(self, texts: list[str]) -> list[list[float]]: out = [] for text in texts: vec = [0.0] * self.dim for idx, val in self._features(text).items(): vec[idx] = val out.append(_normalize(vec)) return outWhat the index recorded
- Signature
class HashEmbedder(Embedder):- File
backend/app/codeintel/embeddings.py- Lines
- 107–137
- Parser
- python-ast
- Git blob
df7f8202f23ca6ad2c57aca5f09b435eb6486d79- Outside this subset
- 8 more edges reach code outside the graph shown here
