Back to the room

Pass 3 of 3 · Deep dive of PHAROSPass 1 · OverviewPass 2 · How it works

Code Observatory / module

indexer

codeintel.indexer

PINNED SOURCE Source and edges from the index at this commit. An execution appears only where one was recorded.

PHAROS at 975a38dMIT license

Around this symbol

Search all symbols

Loading

Contains (8)
SymbolFileResolved by
_decodeindexer.pysame file
_looks_minifiedindexer.pysame file
_mean_vectorindexer.pysame file
_PreparedChunkindexer.pysame file
_utcnowindexer.pysame file
CodeIndexerindexer.pysame file
IndexStatsindexer.pysame file
LanguageStatsindexer.pysame file
Imports (6)
SymbolFileResolved by
embeddingsembeddings.pythrough an import
git_sourcegit_source.pythrough an import
graphgraph.pythrough an import
languageslanguages.pythrough an import
parsers__init__.pythrough an import
texttext.pythrough an import
Imported by (1)
SymbolFileResolved by
jobsjobs.pythrough an import
codeintel.indexerbackend/app/codeintel/indexer.py, lines 1–40Open source
"""Incremental, idempotent repository indexing.One call to :meth:`CodeIndexer.index` brings the database in line with arepository at one commit:1. Resolve the commit (shallow bare clone, or a local repository).2. If the repository is already indexed at that commit with the same   embedding model, stop: re-ingestion is a no-op.3. Diff tracked files against the index by git blob SHA:   * **new / changed** files are parsed, embedded and (re)written;   * **deleted** files are removed with their chunks and edges;   * **unchanged** files only have their commit pointers moved.4. Rebuild the repository's call / import / contains graph.5. Record the commit and statistics on the repository row.The repository's ``last_commit_sha`` is written only after every stepsucceeds. A crash mid-run leaves a mix of old and new files; the retrydiffs again and converges, because files already rewritten now match theirblob SHA. Raw source is never stored (hybrid storage): chunks keep apointer (commit, path, line span) and are resolved on demand."""from __future__ import annotationsimport loggingimport mathimport timefrom collections import defaultdictfrom collections.abc import Callablefrom dataclasses import asdict, dataclass, field, replacefrom datetime import UTC, datetimefrom sqlalchemy import delete, func, select, updatefrom sqlalchemy.orm import Sessionfrom ..database.models import ChunkDescription, CodeEdge, DocumentChunk, Repository, Resourcefrom ..utils.path_exclusions import has_excluded_ancestor, is_excluded_filefrom .embeddings import Embedder, EmbeddingUnavailablefrom .git_source import Checkout, RepoRef, TreeEntry, open_checkoutfrom .graph import ChunkNode, build_edges

Shows the first 40 of 563 lines. The rest is on GitHub at this commit.

What the index recorded

Signature
module backend.app.codeintel.indexer
File
backend/app/codeintel/indexer.py
Lines
1–563
Parser
python-ast
Git blob
e8fdfa7b5545e821f7649d68cfadd2aae85b89d0
Call sites
1 named, 0 resolved, 1 left unlinked
Outside this subset
8 more edges reach code outside the graph shown here

Read it on GitHub at this commit