DCS — Data Collection & Storage
Smart RAG pipeline · v1 of DAC
The first milestone of the DAC line at AIBC. AI document processing pipeline — Chandra OCR extracts text from scanned PDFs and images, structured chunks land in PostgreSQL with pgvector for hybrid retrieval, and a LangGraph RAG agent answers queries with persistent long-term memory.
Problem
The raw material was scanned PDFs and images — documents a vector database can't ingest until something reliable turns pixels into structured text. Off-the-shelf RAG demos assume clean text; real corpora don't offer that.
Approach
Chandra OCR, deployed on AWS SageMaker, handles extraction from scans and images. Structured chunks land in PostgreSQL with pgvector + pgai, giving hybrid retrieval — semantic similarity and keyword search in the same store.
On top sits a LangGraph RAG agent with persistent long-term memory, so answers can build on prior conversations rather than starting cold. Flask serves the API; a Next.js front-end and Docker packaging round it out.
Outcomes
- Live demo deployed and public.
- Proved the ingestion-and-retrieval foundation that DAC's knowledge graph and DACv2's agents were built on.