Back to the room

Pass 3 of 3 · Deep dive of PHAROSPass 1 · OverviewPass 2 · How it works

Code Observatory / class

SentenceTransformerEmbedder

codeintel.embeddings.SentenceTransformerEmbedder

PINNED SOURCE Source and edges from the index at this commit. An execution appears only where one was recorded.

PHAROS at 975a38dMIT license

Around this symbol

Search all symbols

Loading

Called by (1)
SymbolFileResolved by
get_embedderembeddings.pysame file
Contains (6)
SymbolFileResolved by
__init__embeddings.pysame file
_encodeembeddings.pysame file
_loadembeddings.pysame file
embed_documentsembeddings.pysame file
embed_queryembeddings.pysame file
encode_rawembeddings.pysame file
Inside (1)
SymbolFileResolved by
embeddingsembeddings.pysame file
codeintel.embeddings.SentenceTransformerEmbedderbackend/app/codeintel/embeddings.py, lines 167–226Open source
class SentenceTransformerEmbedder(_PrefixMixin, Embedder):    def __init__(self, model_name: str, device: str | None = None, batch_size: int = 32):        self.model_name = model_name        self.prefixes = _use_prefixes(model_name)        self.model_id = model_name + ("+prefix" if self.prefixes else "")        self.batch_size = batch_size        self._device = device        self._model = None        self._lock = threading.Lock()    def _load(self):        if self._model is not None:            return self._model        with self._lock:            if self._model is None:                try:                    from sentence_transformers import SentenceTransformer                except ImportError as exc:                    raise EmbeddingUnavailable(                        "sentence-transformers is not installed "                        "(pip install -r requirements-edge.txt)"                    ) from exc                device = self._device                if device is None:                    try:                        import torch                        device = "cuda" if torch.cuda.is_available() else "cpu"                    except ImportError:                        device = "cpu"                logger.info("Loading embedding model %s on %s", self.model_name, device)                try:                    self._model = SentenceTransformer(                        self.model_name,                        device=device,                        trust_remote_code=True,                        # Avoids a native crash seen with pytorch_model.bin on                        # Windows + Python 3.13 (see project history).                        model_kwargs={"use_safetensors": True},                    )                except Exception as exc:                    raise EmbeddingUnavailable(f"failed to load {self.model_name}: {exc}") from exc        return self._model    def _encode(self, texts: list[str]) -> list[list[float]]:        model = self._load()        vectors = model.encode(            texts,            batch_size=self.batch_size,            normalize_embeddings=True,            show_progress_bar=False,            convert_to_numpy=True,        )        return [_pad(v.tolist(), self.dim) for v in vectors]    def embed_documents(self, texts: list[str]) -> list[list[float]]:        return self._encode([self._doc(t) for t in texts])    def embed_query(self, text: str) -> list[float]:        return self._encode([self._query(text)])[0]

Shows the first 60 of 63 lines. The rest is on GitHub at this commit.

What the index recorded

Signature
class SentenceTransformerEmbedder(_PrefixMixin, Embedder):
File
backend/app/codeintel/embeddings.py
Lines
167–229
Parser
python-ast
Git blob
df7f8202f23ca6ad2c57aca5f09b435eb6486d79

Read it on GitHub at this commit