Back to the room

Pass 3 of 3 · Deep dive of PHAROSPass 1 · OverviewPass 2 · How it works

Code Observatory / class

HashEmbedder

codeintel.embeddings.HashEmbedder

PINNED SOURCE Source and edges from the index at this commit. An execution appears only where one was recorded.

PHAROS at 975a38dMIT license

Around this symbol

Search all symbols

Loading

Called by (1)
SymbolFileResolved by
get_embedderembeddings.pysame file
Contains (2)
SymbolFileResolved by
_featuresembeddings.pysame file
embed_documentsembeddings.pysame file
Inside (1)
SymbolFileResolved by
embeddingsembeddings.pysame file
codeintel.embeddings.HashEmbedderbackend/app/codeintel/embeddings.py, lines 107–137Open source
class HashEmbedder(Embedder):    """Signed feature hashing of unigrams and bigrams (lexical, offline)."""    model_id = "hash-v1"    semantic = False    def _features(self, text: str) -> dict[int, float]:        tokens = tokenize(text)        counts: dict[str, int] = {}        for tok in tokens:            counts[tok] = counts.get(tok, 0) + 1        for a, b in zip(tokens, tokens[1:], strict=False):            key = f"{a}_{b}"            counts[key] = counts.get(key, 0) + 1        features: dict[int, float] = {}        for tok, tf in counts.items():            digest = hashlib.blake2b(tok.encode(), digest_size=8).digest()            bucket = int.from_bytes(digest[:4], "little") % self.dim            sign = 1.0 if digest[4] & 1 else -1.0            weight = (1.0 + math.log(tf)) * (0.5 if "_" in tok else 1.0)            features[bucket] = features.get(bucket, 0.0) + sign * weight        return features    def embed_documents(self, texts: list[str]) -> list[list[float]]:        out = []        for text in texts:            vec = [0.0] * self.dim            for idx, val in self._features(text).items():                vec[idx] = val            out.append(_normalize(vec))        return out

What the index recorded

Signature
class HashEmbedder(Embedder):
File
backend/app/codeintel/embeddings.py
Lines
107–137
Parser
python-ast
Git blob
df7f8202f23ca6ad2c57aca5f09b435eb6486d79
Outside this subset
8 more edges reach code outside the graph shown here

Read it on GitHub at this commit