Show HN: Self-hosted RAG for docs and code (FastAPI, Docling, ChromaDB)

By Riley Chen, AI Technology Analyst – AI2Work

In late November 2025 a small open‑source project called

Community Edition

went viral on Hacker News. The stack—FastAPI, Docling, ChromaDB, and Ollama—offers a zero‑config, Docker‑native RAG solution that can ingest both codebases and technical documentation, keep all data on premises, and deliver sub‑second answers with consumer GPU hardware. For software teams that must protect intellectual property or satisfy strict privacy regulations, the stack is an instant win.

This article dissects the technology, maps it to real‑world use cases, and shows how engineering leaders can evaluate, deploy, and scale the solution in 2025. It blends deep technical analysis with business strategy, ROI projections, and a forward‑looking roadmap—everything you need to decide whether a self‑hosted RAG is right for your organization.

• Zero‑config deployment : A single docker-compose.yml boots the entire stack on any VPS or workstation.

• Dual collection architecture : Separate ChromaDB collections for code and prose yield precision retrieval.

• Local embeddings & LLMs : Ollama hosts Gemma 7B‑I for embeddings and can swap in Claude 3.5 Sonnet or Gemini 1.5 via GGUF Modelfiles.

• Performance : On an RTX 4090 the stack processes ~120k tokens/s for inference and 45k tokens/s for embedding—enough for dozens of concurrent users in a small office.

• No outbound traffic : All data stays on premises, meeting GDPR, CCPA, and other compliance frameworks.

• Business impact: Eliminates per‑token cloud costs, reduces vendor lock‑in, and accelerates onboarding by providing instant contextual answers to code questions.

The 2025 AI landscape is dominated by large‑scale LLM APIs that charge per token. Enterprises are increasingly wary of vendor lock‑in, high operating costs, and data privacy concerns. A self‑hosted RAG stack directly addresses these pain points.

• Cost Control : Cloud API usage can reach $1–$2 per 1k tokens for premium models. With a single RTX 4090 the on‑prem stack consumes $10/month in electricity and hardware amortization, cutting costs by >95%.

• Compliance & Data Sovereignty : Regulations such as GDPR Article 20 (right to data portability) and CCPA Section 1794.21 mandate that personal data not leave the jurisdiction. The stack’s zero outbound traffic satisfies these mandates automatically.

• Competitive Advantage : Fast, context‑aware code search speeds up onboarding by 30–40% , reducing ramp‑up time for new hires. It also empowers dev‑ops to automate bug triage and dependency updates with higher confidence.

• Strategic Flexibility : Because the stack is modular, teams can experiment with newer GGUF models (Gemma 27B, Llama 4 400B) without re‑architecting their infrastructure. This future‑proofs the investment as model sizes grow.

The stack’s architecture is intentionally lightweight: a FastAPI backend exposes a single

/chat

endpoint, Docling handles ingestion, ChromaDB stores vectors, and Ollama runs inference. Each component has a clear responsibility, enabling teams to integrate the stack into existing pipelines with minimal friction.

• CI/CD Ingestion Hook : A GitHub Action can trigger Docling to re‑index new commits, updating the ChromaDB collections automatically.

• IDE Extension Compatibility : The FastAPI endpoint can be consumed by VS Code or JetBrains plugins, turning code comments into instant Q&A windows.

• Observability & Telemetry : All components expose Prometheus metrics. Teams can track latency, token rates, and error counts to enforce SLAs.

• Security Hardening : Docker images are signed by maintainers; no hardcoded credentials exist. Combined with network policies (e.g., docker‑compose.yml exposing only port 8000), the stack fits into a zero‑trust architecture.

A recent GitHub Actions run on an NVIDIA RTX 4090 (8 GB VRAM) measured:

Latency

:

~350 ms

per chat request under moderate load (10 concurrent users).

• Inference throughput : 120,000 tokens/s for a Gemma 7B‑I query.

• Embedding throughput : 45,000 tokens/s when ingesting new documents.

• Embedding throughput : 45,000 tokens/s when ingesting new documents.

Scaling horizontally is straightforward: spinning up additional GPUs behind a simple round‑robin FastAPI deployment or leveraging a distributed ChromaDB cluster can linearly increase capacity. For terabyte‑scale codebases, the recommendation is to partition by repository and deploy a dedicated GPU per shard.

Assumptions:

• Hardware : RTX 4090 ($1,500) amortized over 3 years → $42/month.

• Electricity : 250 W average load, 24/7 operation → ~$10/month.

• Maintenance : DevOps time for updates (~2 hours/month).

• Cloud API alternative : OpenAI GPT‑4o at $0.03 per 1k tokens (inference) and $0.02 per 1k tokens (embeddings). For 10,000 requests/day (~120M tokens), cost ≈ $3,600/month.

Net savings:

$3,548/month

. Over a 3‑year horizon, the stack saves roughly $127k. Even accounting for potential future hardware upgrades (e.g., RTX 4090 Ti at $2,200), ROI remains positive within 12 months.

• Proof of Concept (Week 1) : Spin up the docker-compose.yml , ingest a small repo, and test the UI. Validate latency and data locality.

• Security Hardening (Week 2–3) : Enable TLS on FastAPI, restrict Docker network to localhost, and audit image signatures.

• CI/CD Pipeline Integration (Month 1) : Add a GitHub Action that triggers Docling on push events. Store embeddings in ChromaDB with repository tags.

• IDE Plugin Development (Month 2–3) : Build a lightweight VS Code extension that queries the FastAPI endpoint and displays answers inline.

• Scaling Strategy (Month 4+) : Deploy multiple GPU nodes behind an NGINX load balancer. Migrate ChromaDB to a distributed mode using its built‑in sharding feature.

Throughout, monitor

prometheus.yml

metrics and set alerts for latency spikes or embedding failures. Use the stack’s modularity to swap in newer GGUF models (e.g., Claude 4.5 via Ollama) without code changes.

The market offers several managed options—Pinecone + OpenAI, Azure Cognitive Search + GPT‑4o, and AWS Bedrock. Key differentiators for the self‑hosted stack are summarized below:

Feature

Self‑Hosted Stack (2025)

Managed Alternatives (2025)

Data Residency

On‑prem, no egress

Cloud region, potential egress

Cost per Token

$0.00 (hardware amortized)

$0.02–$0.03 (inference), $0.01–$0.02 (embeddings)

Vendor Lock‑In

No lock‑in; modular components

High lock‑in to provider APIs

Latency

~350 ms (GPU) vs 1–2 s (cloud)

Depends on region, typically 1–3 s

Compliance

GDPR/CCPA compliant by design

Requires VPC endpoints and compliance attestations

Scalability

Horizontal scaling with GPUs; ChromaDB sharding

Elastic scaling, but at higher cost

Operational Overhead

Docker + basic monitoring

Managed services reduce ops but increase cost

For mid‑size teams (10–50 developers) that need to protect code and documentation, the self‑hosted stack offers a superior total cost of ownership while delivering comparable or better performance.

• Model Drift & Hallucinations : Since models run locally, there is no audit trail. Implement a post‑hoc review layer: capture user queries and model responses in a secure log, then periodically audit a sample for hallucination rates.

• Hardware Failure : Deploy GPU nodes behind a Kubernetes cluster with pod autoscaling to recover from node crashes automatically.

• Data Integrity : Use ChromaDB’s built‑in checkpointing and snapshot features to guard against corruption. Store backups in an isolated storage tier.

• Security Updates : Subscribe to the maintainers’ release notes for Ollama, Docling, and ChromaDB. Automate image pulls with docker‑compose pull --no-cache on a nightly schedule.

• Model Licensing : Verify that the GGUF model you deploy (e.g., Gemma 7B‑I) permits commercial use under its license. For proprietary models, ensure your internal compliance team signs off.

The community roadmap includes integrating FAISS GPU for similarity search and supporting larger GGUF models such as Gemma 27B and Llama 4 400B. As these models become available, the stack will need:

• More GPU Memory : A single RTX 4090 may no longer fit 27B embeddings; teams should consider RTX 6000 or A100 GPUs.

• Quantization Strategies : Leveraging 8‑bit quantization via Ollama can reduce memory footprint by ~75% while maintaining < 5% loss in accuracy.

• Hybrid Retrieval : Combine ChromaDB with a lightweight on‑disk vector store (e.g., Milvus) to handle petabyte‑scale codebases.

Early adopters who implement the current stack will be well positioned to upgrade smoothly when these larger models arrive, maintaining performance parity without re‑architecting ingestion pipelines.

Plan for Scale

: If you anticipate >100 concurrent users or terabyte‑scale codebases, architect a multi‑GPU cluster with distributed ChromaDB from the outset.

• Run a Cost Comparison : Estimate your monthly token volume and compare cloud API costs against the hardware amortization of an RTX 4090. If savings exceed 80%, consider self‑hosting.

• Pilot with a Small Repo : Deploy the Docker stack on a dev machine, ingest a single repository, and measure latency. A single click Q&A experience validates the concept before scaling.

• Integrate into Onboarding : Add the FastAPI endpoint to your internal knowledge base or Slack bot. New hires can ask questions about legacy code instantly.

• Secure and Monitor : Enable TLS, restrict network access, and set up Prometheus alerts for latency spikes. This ensures compliance and operational reliability.

• Secure and Monitor : Enable TLS, restrict network access, and set up Prometheus alerts for latency spikes. This ensures compliance and operational reliability.

By adopting this self‑hosted RAG stack in 2025, enterprises can slash AI operating costs, eliminate vendor lock‑in, and empower developers with instant contextual knowledge—all while keeping sensitive data under their own control. The technology is mature enough for production, but its modularity guarantees that it will evolve alongside the next generation of large models.