Results
On a 48-question benchmark I built with gold SQL, the agent answered 87.5% correctly, up from 83% without examples. Table retrieval reached 95% top-k with hybrid search. Routing sub-tasks to smaller models cut the cost per query about 20× compared with o3, at similar accuracy.
Approximate figures.
The problem
Analysts needed answers from an internal Oracle financial database without writing SQL. The evaluation subset alone has 30 core tables; the full database is larger, so the agent first has to find the right tables.
What I did
- Built a LangGraph agent with 7 tools: o4-mini plans the steps, GPT-4.1 writes the SQL and judges answers, GPT-4.1-nano filters context.
- Built the 48-question benchmark with gold SQL from about 12 starter questions, and scored runs with an LLM judge.
- Built table retrieval with BM25 and embeddings, merged with reciprocal rank fusion.
- Added validated question and SQL examples to the model's context.
Results
Accuracy rose from 83% to 87.5% on the benchmark, table retrieval reached 95% top-k, and routing sub-tasks to smaller models cut the cost per query about 20 times compared with o3, at similar accuracy. I also built the Streamlit interface for analysts and part of a FastAPI backend.
The code and final architecture belong to CFM. The internship report, approved for sharing, has more detail on the method.