Guide
Running a six-week knowledge assistant pilot
A week-by-week playbook for piloting a retrieval-augmented knowledge assistant with real users, clear success criteria and a go or no-go decision.
2 min readLast reviewed
A knowledge assistant pilot should answer one question: does this help a defined group of people find accurate answers faster, safely enough to scale? This playbook structures six weeks around that question. Adjust durations to your organisation, but keep the order.
Before week 1: agree the frame
- One user group of 20 to 50 people with a shared need.
- Two to five content sources with known owners.
- Three or four success metrics agreed with the sponsor.
- Data protection and security sign-off for the sources and hosting option.
Week 1: questions and baseline
Collect 150 to 250 real questions from the user group, from help-desk logs, chat channels and interviews. For each, record the document that answers it. Time how long it takes users to find answers today for a sample.
Weeks 2 and 3: ingestion and retrieval
Connect the sources with their permissions, design chunking around document structure and tune hybrid retrieval against the question set. Report retrieval accuracy weekly. Identify content problems such as duplicates and outdated versions and give them to content owners.
Week 4: answers and safeguards
Add answer generation with citations and abstention. Measure faithfulness on the question set. Test permission boundaries explicitly by asking questions as users who should not see certain content.
Weeks 5 and 6: live pilot
Release to the user group in the tools they already use. Collect feedback on every answer, review failures twice a week and fix content or retrieval. Repeat the time-to-answer measurement at the end.
Decision meeting
| Evidence | Question it answers |
|---|---|
| Retrieval accuracy and faithfulness | Is it accurate enough? |
| Time-to-answer versus baseline | Does it save time? |
| Usage and feedback | Will people use it? |
| Permission tests and incidents | Is it safe enough to scale? |
The outcome is go, go with conditions, or stop. Any of the three is a successful pilot if it was decided on evidence.