Back to the room

Pass 3 of 3 · Deep dive of PHAROSPass 1 · OverviewPass 2 · How it works

Code Observatory / method

_features

codeintel.embeddings.HashEmbedder._features

PINNED SOURCE Source and edges from the index at this commit. An execution appears only where one was recorded.

PHAROS at 975a38dMIT license

Around this symbol

Search all symbols

Loading

Calls (1)
SymbolFileResolved by
tokenizetext.pythrough an import
Called by (1)
SymbolFileResolved by
embed_documentsembeddings.pysame file
Inside (1)
SymbolFileResolved by
HashEmbedderembeddings.pysame file
codeintel.embeddings.HashEmbedder._featuresbackend/app/codeintel/embeddings.py, lines 113–128Open source
    def _features(self, text: str) -> dict[int, float]:        tokens = tokenize(text)        counts: dict[str, int] = {}        for tok in tokens:            counts[tok] = counts.get(tok, 0) + 1        for a, b in zip(tokens, tokens[1:], strict=False):            key = f"{a}_{b}"            counts[key] = counts.get(key, 0) + 1        features: dict[int, float] = {}        for tok, tf in counts.items():            digest = hashlib.blake2b(tok.encode(), digest_size=8).digest()            bucket = int.from_bytes(digest[:4], "little") % self.dim            sign = 1.0 if digest[4] & 1 else -1.0            weight = (1.0 + math.log(tf)) * (0.5 if "_" in tok else 1.0)            features[bucket] = features.get(bucket, 0.0) + sign * weight        return features

What the index recorded

Signature
def _features(self, text: str) -> dict[int, float]:
File
backend/app/codeintel/embeddings.py
Lines
113–128
Parser
python-ast
Git blob
df7f8202f23ca6ad2c57aca5f09b435eb6486d79
Call sites
9 named, 2 resolved, 7 left unlinked
Outside this subset
1 more edges reach code outside the graph shown here

Read it on GitHub at this commit