Current Architecture

Data Catalog Lives in GitHub

Why?

• Humans and agents can both see and fix metadata
• Incremental changes with version history
• Issues and review workflow
• Controlled deployments: branch staging production

Arad’s Wish

“Users should be able to ingest new data while chatting with Axes.”

What follows?

• Private datasets
• GitHub is no longer viable for the data catalog
• Growler lives in Chat Plot

New Data Workflow

• Users can upload their own data or sick Growler on a dataset in the wild
• They can guide the agent to curate the metadata for their datasets
• Data can be marked “public”
• Admins can mark data “approved” and it becomes available by default

A Social Network for Data

• MOAR Data Better
• Axes becomes a way to share things with others: data, artifacts, insights, generated UI, reports
• Network effects: more public data more useful more public data

Data Model

• Version history is a stream of CatalogUpdate events. See spec
• Databases have branches/checkpoints. All our databases (PostgreSQL, Qdrant and ClickHouse) have every version of every database
• More Sources: Periodic Updates, Connections to External/Live Data (s3, psql, etc are filtered through ClickHouse), API fetch
• Linear history vs forking/branching

Agent Architecture

• Coding tasks start and stop in the same subagent loop so the state of the filesystem is disposable
• Agent architecture spec
• Agent Loops Spec

New UI

• Subagents
• Upload a dataset
• Review metadata changes
• Browse datasets, version management, etc

Eval In The New Paradigm

Again: Just build it inside Chat Plot

• Test data and code changes in isolation
• The same data model
• Sources: raw data files, reports, academic publications, blog posts
• Claims: essentially numbers extracted from those sources; these can become “verified” if the agents can reproduce them
• Retrospective Eval: read a user’s chat
• Generated Eval: create a new chat with a known answer
• A test suite on metadata and schema updates
• Evaluations of tweaks to our agents and search results

Roadmap

• Coding Agents (2 weeks)
• Manual Data Updates (4 weeks)
• Curation Agent (3 weeks)
• 9 weeks

Seriously: We could just fix Growler and iterate on the current model