Regional Telco · Telecommunications · 2025

RAG Pipeline

Enterprise knowledge assistant

91%answer accuracy on held-out eval set — regional telco, 2025

The Problem

A decade of critical engineering knowledge was locked in documents no one could find.

The client had accumulated 12 years of engineering documentation — network architecture specs, incident post-mortems, configuration guides, vendor runbooks — across multiple SharePoint libraries, shared drives, and email threads. Search was effectively useless: keyword queries returned hundreds of irrelevant results, and the right document often required knowing it existed.

New engineer onboarding took 6–8 weeks, most of which was spent tracking down senior engineers for knowledge that should have been self-serve. Senior engineers were spending 30–40% of their time answering questions that were documented somewhere — a compounding drain on the team's most experienced people.

Our Approach

We deployed a RAG system over the full document corpus, giving engineers a single interface that answers questions with cited, retrievable sources.

01

Document ingestion and normalisation

We ingested the full SharePoint corpus — 40,000+ documents across 12 years — cleaning, deduplicating, and chunking content into semantically meaningful segments. Scanned documents were processed through OCR with layout preservation.

02

Hybrid retrieval architecture

Rather than pure vector search, we implemented a hybrid retrieval layer combining dense embeddings with BM25 sparse retrieval. This significantly improved performance on technical queries with specific terminology, part numbers, and version strings.

03

Citation-grounded generation

Every answer generated by the system includes document-level citations with clickable links back to the source material. Engineers can verify answers and navigate to full context — the system cannot hallucinate a source that doesn't exist.

04

Evaluation and calibration

We built a held-out evaluation set of 500 questions drawn from real onboarding queries and past incident tickets. The system was iteratively calibrated against this set before go-live, with retrieval and generation parameters tuned to maximise accuracy on the client's specific domain.

The Outcome

Engineers get accurate, cited answers in seconds. The knowledge bottleneck is gone.

On the held-out evaluation set of 500 domain-specific questions, the system achieved 91% answer accuracy — exceeding the target threshold set at project kickoff. The 9% gap consisted almost entirely of questions requiring real-time network state data outside the document corpus.

New engineer onboarding time dropped from 6–8 weeks to 3–4 weeks. Engineers reported being able to find answers independently from day one, rather than depending on colleagues for basic orientiation.

Senior engineers reclaimed an estimated 25% of their week previously spent answering repetitive questions. The documentation corpus itself also improved: gaps identified by the system's 'I don't know' responses surfaced areas where documentation had never existed.

91%
answer accuracy on held-out eval set
50%
reduction in onboarding time
25%
senior engineer time reclaimed
40K+
documents indexed across 12 years

Work With Us

Ready to see results like these?

Book a strategy call and we'll map out what's possible for your business — no boilerplate decks, no vague roadmaps.