Building an AI Copilot That Employees Actually Use

Everyone wants an internal AI assistant. The pitch is always the same: employees ask questions in natural language, the AI searches across Confluence, Slack, SharePoint, and Google Drive, and it returns accurate answers with source links. Simple in theory. In practice, most internal copilots fail because of three problems: bad search quality, no access control, and poor adoption.

The Architecture

The system has four layers. At the bottom, connectors pull data from every knowledge source: Confluence pages, Slack threads, SharePoint documents, Google Drive files, Notion databases, and the company wiki. Each connector handles authentication, incremental syncing, and format normalization.

The ingestion layer processes documents into chunks, generates embeddings, and stores them in a vector database (we used Pinecone on this project, though Weaviate and Qdrant work well too). Crucially, every chunk retains metadata about its source, last update time, and access permissions.

The AI engine handles retrieval and generation. When a user asks a question, we retrieve the top 10 relevant chunks, filter by the user's access permissions, rerank using a cross-encoder, and pass the top 5 to the LLM with the question. The model generates an answer with inline citations linking back to source documents.

The interface layer is where users interact. We built three: a Slack bot (the primary interface, since that is where people already spend their time), a web app for longer research sessions, and a Chrome extension that provides context-aware answers while browsing internal tools.

Access Control Is Non-Negotiable

This is the part most demos skip. If an intern asks the copilot about executive compensation, the answer should be "I do not have information on that topic" rather than pulling from an HR document they should not see. We mapped every document to its permission set in the source system and enforced those permissions at retrieval time, not at display time.

Getting this right required close coordination with the client's IT team. We built permission sync jobs that run every 15 minutes, so when someone's access changes in SharePoint, the copilot reflects it within the hour.

Why Most Copilots Fail at Adoption

The first version had 12% weekly active users after launch. That is typical. People tried it once, got a mediocre answer, and went back to searching Confluence manually. Three changes got us to 68% adoption over three months:

First, we fixed search quality. The initial chunking strategy was too aggressive, splitting documents at arbitrary character limits. We switched to semantic chunking that respects document structure (headings, sections, paragraphs). Retrieval relevance improved immediately.

Second, we added proactive suggestions. Instead of waiting for users to ask questions, the Slack bot monitors channels and offers relevant links when it detects questions or confusion. "I noticed you are asking about the PTO policy. Here is the latest version." This felt genuinely helpful rather than intrusive because we tuned the confidence threshold high enough that it only chimed in when it was clearly relevant.

Third, we let users give feedback. A simple thumbs up / thumbs down on every answer, plus a "flag this answer" button for incorrect information. We reviewed flagged answers weekly and used them to improve the retrieval pipeline. This created a visible feedback loop that made employees feel like the tool was getting better because of their input, which it was.

Metrics That Matter

We track: weekly active users, questions per user per week, answer satisfaction rate (thumbs up ratio), time saved per query (estimated by comparing to the alternative of manual search), and hallucination rate (flagged answers divided by total answers). The last metric is the most important one. If employees cannot trust the answers, they stop using the tool entirely.

Lessons Learned

Data quality matters more than model quality. Switching from GPT-4 to GPT-4o made a marginal difference. Fixing the chunking strategy and cleaning up stale Confluence pages made a massive difference. The copilot is only as good as the documents it searches.

Launch with one team first, not the whole company. We started with the customer success team (150 people) who had clear, frequent questions about product features and policies. Their positive experience created internal advocates who pushed adoption in other departments.

Plan for ongoing maintenance. Knowledge bases change. New tools get adopted. People leave and join. The copilot needs someone responsible for keeping the connectors running, the index fresh, and the retrieval quality high. Budget for it from the start.