​

Token Intelligence (Atlas)

​

What is Atlas?

• Builds a local knowledge graph from your entire codebase
• Extracts code structure (functions, classes, variables, imports, calls, types)
• Models relationships between symbols (calls, references, extensions, implementations)
• Stores everything in an SQLite database at .atlas/atlas.db
• Runs 100% locally with no cloud processing or data transmission
• Exposes a Model Context Protocol (MCP) server so agents get surgical context

• Structured node/edge metadata (start/end line, visibility, type info)
• Semantic search results ranked by relevance
• Call graphs, type hierarchies, and data flow paths
• Per-symbol code snippets instead of whole files

​

How Indexing Works

​

Initial Indexing

1. Spawns node .../atlas/dist/mcp/server-entry.js --init --path <project>
2. Initializes the .atlas/ directory structure
3. Creates an empty SQLite database at .atlas/atlas.db
4. Begins scanning the project root

• Discovers all source files matching supported extensions
• Applies .gitignore rules and default ignore patterns
• Respects custom exclusions in atlas.json (project root)
• Skips vendor directories, build output, caches, test resources, Android resource directories, and other generated content by default

• All recognized source code files (.ts, .js, .py, .go, .rs, .java, .cpp, .cs, .php, .rb, .swift, .kt, .dart, .lua, .vue, .svelte, .astro, and 20+ more)
• Configuration files (.json, .yaml, .toml, .xml) that define routes or dependencies
• Framework files (routes, modules, decorators, resolvers)

• node_modules/, .venv/, venv/, target/, vendor/, dist/, build/, and ~30 other dependency/build directories
• Android resource directories (res/layout, res/values, res/drawable, etc.)
• Test/spec files (unless explicitly included in the query)
• Generated code (detected by heuristics: @generated, autogenerated, do not edit markers)
• Files larger than 1 MB (minified bundles, compiled assets)

​

Parsing & Extraction

• Worker threads pool parses multiple files concurrently
• Each parser extracts nodes (symbols) and edges (relationships)
• Extracted data includes:
  • Node kind (function, class, variable, import, route, component, etc.)
  • Name and fully qualified name (e.g., src/utils.ts::MathHelper.calculateTotal)
  • Location (file, start/end line, start/end column)
  • Metadata (visibility, type parameters, return type, docstring, signature)
  • Decorators and modifiers (async, static, abstract, exported)
• Edge kinds tracked:
  • contains (file contains class, class contains method)
  • calls (function calls another)
  • imports (file imports from another)
  • exports (symbol exported from file)
  • extends / implements (inheritance)
  • references (generic reference)
  • type_of (variable has type)
  • returns (function return type)
  • instantiates, overrides, decorates

• Typical scan: 100-1000 files/second (I/O bound)
• Typical parse: 1000-10000 files/second (CPU bound, parallelized)
• A 10k-file project typically indexes in 5-15 minutes on first run
• Database size: ~50-500 MB depending on complexity (heavily indexed repos on the high end)

• phase: ‘scanning’ | ‘parsing’ | ‘storing’ | ‘resolving’
• current: Files processed so far
• total: Total files to process
• currentFile: (optional) Current file being parsed

​

Incremental Sync

• atlas.sync() checks disk for added/modified/removed files
• Only re-parses changed files (fast path)
• Re-indexes only affected references
• Typical sync: < 1 second on small changes

​

Reference Resolution

1. Import-based resolution: Follows import X from './file' to map names to definitions
2. Framework-specific: React Routes, Express handlers, NestJS controllers, Laravel middleware, etc.
3. Name-based matching: Falls back to symbol-name lookup in the same package
4. Type hierarchy traversal: Finds inherited members through extends/implements chains
5. Chained calls via conformance: Resolves method calls on protocol/interface implementations

• “Who calls this function?”
• “What does this class extend?”
• “Which routes are handled by this controller?”
• “What symbols are exported from this module?”

​

The AtlasIndexToast

<AtlasIndexToast projectPath={...} projectName={...} onDismiss={...} />

• Polls every 2 seconds for the existence of .atlas/atlas.db in the project
• Shows a spinner and “Indexing project” message while indexing runs
• Once the database file appears (indexing complete), displays “Index ready” and auto-dismisses after 2.5 seconds
• Users can manually dismiss at any time

​

How Atlas Reduces Token Usage

• “Show me src/handlers.ts” → 500+ lines → 2000+ tokens per file
• “Show me 10 related files” → 20k tokens before any actual reasoning
• Agents must parse file structure themselves, extract only relevant pieces
• Duplicated context when multiple symbols from the same file are relevant

• atlas_explore "How does authentication work?" → Returns a focused subgraph with:
  • Only relevant files (5-8 instead of 20+)
  • Per-symbol code snippets (20-50 lines) instead of whole files
  • Symbol names, signatures, and locations
  • Relationships between symbols (who calls who, what implements what)
  • ~300-800 tokens instead of 2000+
• atlas_node "Symbol/QualifiedName" → Returns just that symbol’s definition + immediate context
• atlas_graph "find_callers UserService.authenticate" → Returns call chain as a traversable graph
• atlas_search "authentication" → Returns FTS-ranked search results, top N only

• Token savings: 60-80% reduction in context tokens for typical agent queries
• Faster responses: Smaller context means faster inference
• Better accuracy: Agents reason about structure, not raw text parsing
• Cross-file awareness: Agents see relationships without reading every file

​

Data Storage

.atlas/
  atlas.db          # Main SQLite database
  atlas.db-wal      # WAL (write-ahead log) for concurrent access
  atlas.db-shm      # Shared memory for WAL synchronization
  daemon.log        # (Optional) MCP daemon logs
  daemon.lock       # (Optional) Daemon process lock
  daemon.sock       # (Optional) Unix socket for daemon communication

​

Database Schema

• id: Primary key (hash of file path + qualified name)
• kind: Node type (function, class, import, route, etc.)
• name: Simple name (e.g., “calculateTotal”)
• qualified_name: Full path (e.g., “MathHelper.calculateTotal”)
• file_path: Relative to project root
• language: Detected language
• start_line, end_line, start_column, end_column: Location
• docstring, signature: Documentation and type info
• visibility, is_exported, is_async, is_static: Modifiers
• decorators, type_parameters, return_type: Extra metadata
• updated_at: Last modified timestamp

• source, target: Node IDs
• kind: Edge type (calls, imports, extends, etc.)
• metadata: JSON with context (line, column, parameter info)
• Unique index on (source, target, kind, line, col) to prevent duplicates

• path: File path (primary key)
• content_hash: SHA256 of file contents
• language, size: File metadata
• modified_at, indexed_at: Timestamps
• node_count: Count of extracted symbols
• errors: JSON array of parse errors

• Tracks references waiting for resolution
• Clears after successful resolution pass

• nodes_fts virtual table indexes name, qualified_name, docstring, signature
• Enables semantic search across the graph

• Indexes on kind, name, qualified_name, file_path, language
• Composite indexes on (file_path, start_line), (source, kind), (target, kind) for fast traversal
• UNIQUE index on edge identity to prevent duplicates

​

Database Configuration

• Readers never block on a concurrent writer
• Writers don’t block readers
• Multiple processes can connect simultaneously (MCP daemon + git hooks)

• journal_mode = WAL: Write-ahead logging
• synchronous = NORMAL: Safe with WAL
• busy_timeout = 5s: Wait up to 5 seconds if database is locked
• cache_size = 64 MB: Large page cache for fast queries
• mmap_size = 256 MB: Memory-mapped I/O for sequential scans

• Small project (< 1k files): 10-50 MB
• Medium project (1k-10k files): 50-200 MB
• Large project (10k-100k files): 200-500 MB
• Very large (100k+ files): 500 MB-2+ GB

​

Per-Project Indexing

• Switching to a new project workspace automatically points Atlas to its .atlas/
• Multiple projects can be indexed simultaneously in separate processes
• The MCP daemon (when running) maintains one connection per project
• No shared index across projects

• Each workspace root needs its own index
• Use atlas.json at the workspace root to configure exclusions/extensions

​

Re-indexing

​

Automatic (on first enable)

1. Tempest detects no .atlas/ directory
2. Spawns atlas init --path <project>
3. Wauri AtlasIndexToast polls for .atlas/atlas.db
4. Background indexing runs, user is notified via toast

​

Automatic (on file changes)

• Debounced every 500ms
• Only re-parses changed files
• Happens in background without blocking the UI

​

Manual (via CLI or API)

# Full re-index (discards old database, starts fresh)
atlas index --path <project>

# Sync only (incremental update)
atlas sync --path <project>

# Initialize if not already indexed
atlas init --path <project>

import Atlas from '@tempest/atlas';

const atlas = await Atlas.open('/path/to/project');

// Full re-index (clears database, reindexes everything)
await atlas.indexAll({ 
  onProgress: (progress) => console.log(progress),
  verbose: true 
});

// Incremental sync (only changed files)
await atlas.sync();

// Check if index is stale
if (atlas.isIndexStale()) {
  console.log('Index was built with an older extraction engine');
}

​

When to Re-index

• Enabling Token Intelligence for the first time
• After major framework/dependency updates (npm install, pip install)
• After local branch changes that rewrote history
• Atlas recommends it (run atlas status --json to check)
• You see “database is locked” errors (indicates corruption)

• Normal code edits (auto-sync handles these)
• Switching branches with similar structure
• Temporary file changes

​

Supported Languages

• TypeScript, JavaScript, TSX, JSX
• Vue.js, Svelte, Astro
• Python, Ruby, PHP, Lua, Luau

• Go, Rust, C, C++, C#
• Java, Kotlin, Scala
• Swift, Objective-C
• Dart

• YAML, XML, JSON, Properties files
• Liquid (Jekyll templates)
• Twig (Symfony templates)
• Pascal

{
  "extensions": {
    ".dota_lua": "lua",
    ".tpl": "php",
    ".h": "cpp"
  }
}

​

Configuration

{
  "extensions": {
    ".custom": "typescript"
  },
  "includeIgnored": [
    "vendor/important-vendored-lib/"
  ],
  "exclude": [
    "static/themes/**",
    "assets/vendor/**"
  ]
}

​

Graph Operations

atlas_search "authentication middleware"

atlas_explore "How does user login work?" --maxFiles 6

atlas_node "src/auth.ts::authenticate" --includeCode

atlas_graph "find_callers authenticate"
atlas_graph "find_usages UserService"
atlas_graph "type_hierarchy BaseController"

atlas_status

​

Performance Considerations

• Typical: 100-1000 files/second
• Parallelized across worker threads
• Bottleneck: tree-sitter parsing, not I/O or database writes

• Symbol lookup: < 1ms (indexed by name)
• Full-text search: 10-100ms depending on query selectivity
• Graph traversal: 10-500ms depending on depth
• File dependencies: < 100ms

• Parsing workers: ~100 MB each (WASM heap grows during parsing, shrinks after)
• Database: 50-200 MB resident (SQLite page cache)
• CLI tool: 200-500 MB during indexing

• Index database: 50 MB - 2 GB depending on project size
• WAL files: 0-500 MB (temporary during concurrent writes, cleaned after sync)

​

Troubleshooting

• Check if another process is indexing (look for atlas processes)
• Ensure journal mode is WAL: sqlite3 .atlas/atlas.db "PRAGMA journal_mode;"
• If corrupted, remove .atlas/ and re-index

• Check for very large files (> 1 MB) that slow parsing
• Verify .gitignore is working (should exclude vendor dirs)
• Look for symbolic links to huge directories
• Use verbose: true to see per-file progress

• Run atlas index --path <project> to rebuild with latest extraction engine

• Check file language detection (atlas status --json shows language distribution)
• Verify file is not in excluded patterns
• Ensure file is under project root

• Set ATLAS_NO_DAEMON=1 to run in direct mode
• Check .atlas/daemon.log for errors
• Ensure no stale daemon lock at .atlas/daemon.lock

​

What’s Next: Nexus

• Interactive visualization of the code graph
• Node/edge filtering and search
• Call graph exploration UI
• Data flow visualization
• Type hierarchy browser