Reference architecture
IllustrativePermission-aware knowledge copilot for a multi-site operations team
A reference architecture for answering procedure questions from thousands of documents while respecting site-level access rights.
Illustrative scenario. Not a completed client engagement. It describes how we would approach the problem; no client, results or quotes are implied.
/ 01Problem
The situation
Consider an organisation with several operating sites, each with its own procedures layered on top of group policy. Staff search a document management system and a wiki, often find superseded versions and fall back on asking supervisors.
A pilot chatbot built on a general model gave fluent but unreliable answers and ignored the fact that some procedures are site-confidential.
/ 02Baseline
Measuring before building
Before design, the team would sample around 200 real questions from supervisors and help-desk logs, record which document actually answers each one, and time how long staff take to find it today. That sample becomes both the baseline and the evaluation set.
/ 03AI architecture
Architecture snapshot
- Document management system with site-level permissions
- Wiki spaces
- Superseded-version metadata
- Structure-aware chunking by heading and table
- Permission and site metadata on every passage
- Version filter excluding superseded documents
- Hybrid keyword and vector search
- Cross-encoder re-ranking
- Pre-generation permission filter
- Answer only from retrieved passages
- Inline citations
- Abstain below evidence threshold
- Evaluation set scored on every change
- Feedback to content owners
- Query logs with 90-day retention
/ 04Delivery
How it would be delivered
- 1
Weeks 1–2: discovery
Question sampling, source inventory, permission model review and baseline measurement.
- 2
Weeks 3–6: build
Ingestion pipeline, retrieval tuning against the evaluation set, answer service and Teams front end.
- 3
Weeks 7–8: pilot
One site uses the copilot; failures are reviewed weekly and added to the test set.
/ 05Outcome measures
What would be measured
- Retrieval accuracy: share of test questions where the correct passage is in the top results
- Faithfulness: share of answer statements supported by cited passages
- Abstention quality: correct refusals versus missed answers
- Time to find information compared with the measured baseline
Limitations
- Outcomes depend heavily on document quality and are not predicted here.
- Scanned procedures with poor OCR would need remediation before inclusion.
No results are claimed for this scenario. Targets would be set from the measured baseline in a real engagement.
/ 06Client quote
Illustrative scenarios do not carry client quotes.
Related solutions
Related product concepts
Want this architecture for your workflow?
We will adapt it to your systems and constraints and give you a phased, costed plan.