文章三：Agentic RAG: Company Knowledge Slack Agents

內容整理與總結

引言

• 章節重點原文保留:

  An AI knowledge agent can dig through internal documentation — websites, PDFs, random docs — and answer employees in Slack (or Teams/Discord) within a few seconds. So, these bots should significantly reduce time sifting through information for employees. I’ve seen a few of these in bigger tech companies, like AskHR from IBM, but they aren’t all that mainstream yet. If you’re keen to understand how they are built and how much resources it takes to build a simple one, this is an article for you. I’ll go through the tools, techniques, and architecture involved, while also looking at the economics of building something like this. I’ll also include a section on what you’ll end up focusing the most on.

• 章節總結: 作者點出 AI 知識代理程式能透過挖掘內部文件（網站、PDF、各種文件）並在 Slack 等平台快速回答員工問題，從而大幅減少員工搜尋資訊的時間。雖然在大型科技公司已出現類似應用（如 IBM 的 AskHR），但尚未普及。本文旨在探討建構這類代理程式的工具、技術、架構、經濟成本以及開發過程中需重點關注的環節。

What is RAG and Agentic RAG?

• 章節重點原文保留:

  Retrieval-Augmented Generation (RAG) is a way to fetch information that gets fed into the large language model (LLM) before it answers the user’s question. This allows us to provide relevant information from various documents to the bot in real time so it can answer the user correctly. This retrieval system is doing more than simple keyword search, as it finds similar matches rather than just exact ones. For example, if someone asks about fonts, a similarity search might return documents on typography. In agent systems, the LLM can decide where and how it should fetch information, rather than just having content dumped into its context before generating a response. It’s important to remember that just because more advanced tools exist doesn’t mean you should always use them. You want to keep the system intuitive and also keep API calls to a minimum.

• 章節總結: 本章節解釋了檢索增強生成（RAG）和代理式 RAG（Agentic RAG）。RAG 是一種在大型語言模型（LLM）回答問題前，先擷取相關資訊並提供給模型的技術，使其能即時利用各種文件內容，正確回答使用者。其檢索系統超越關鍵字搜尋，能進行相似性比對。而代理式系統則更進一步，LLM 能自主決定從何處及如何獲取資訊，而非被動接收。作者提醒，並非永遠要選擇最先進的工具，保持系統直觀性並減少 API 呼叫至關重要。

Technical Stack

• 章節重點原文保留:

  For the deployment option, since we’re working with Slack webhooks, we’re dealing with event-driven architecture where the code only runs when there’s a question in Slack. To keep costs to a minimum, we can use serverless functions. The choice is either going with AWS Lambda or picking a new vendor. Platforms like Modal are technically built to serve LLM models, but they work well for long-running ETL processes, and for LLM apps in general. As for how to pick the agent framework, this is completely optional. I did a comparison piece a few weeks ago on open-source agentic frameworks that you can find here, and the one I left out was LlamaIndex. The last thing you need to pick is a vector database, or a database that supports vector search. This is where we store the embeddings and other metadata, so we can perform similarity search when a user’s query comes in. Both Qdrant and Milvus have pretty generous free tiers for their cloud options.

• 章節總結: 建構公司知識 Slack 代理程式的技術棧選擇多元。部署方面，由於涉及 Slack webhooks，屬於事件驅動架構，可採用無伺服器函數（如 AWS Lambda 或 Modal）以降低成本。Modal 雖為 LLM 模型設計，但也適用於 ETL 和 LLM 應用。代理框架方面，作者選擇嘗試 LlamaIndex。向量資料庫用於儲存嵌入和元資料以進行相似性搜尋，Qdrant 和 Milvus 是具有免費雲端方案的熱門選項，本文作者選擇了 Qdrant。

Cost & time to build

• 章節重點原文保留:

  In terms of time and cost, you have to account for engineering hours, cloud, embedding, and large language model (LLM) costs. What takes time is connecting the content properly, prompting the system, parsing the outputs, and making sure it runs fast enough. cloud costs to run the agent system is minimal for just one bot for one company using serverless functions for the vector databases, it will get more expensive the more data you store. As for the embeddings, these are generally very cheap. The one thing that matters the most though is what large language model (LLM) you use. You need to think about API prices, since an agent system will typically call an LLM two to four times per run. For this system, I’m using GPT-4o-mini or Gemini Flash 2.0, which are the cheapest options.

• 章節總結: 建構這類系統的成本和時間需考量工程時數、雲端費用、嵌入成本及 LLM 成本。主要耗時部分在於正確連接內容、設計提示詞、解析輸出及確保系統速度。使用無伺服器函數的單一代理程式雲端成本極低。向量資料庫成本隨儲存資料量增加而提升，但如 Zilliz 和 Qdrant Cloud 提供可觀的免費額度。嵌入成本通常不高。最關鍵的成本因素是 LLM 的選擇，因代理系統每次運行可能呼叫 LLM 2-4 次。作者選用 GPT-4o-mini 或 Gemini Flash 2.0 等較經濟的 LLM 模型，預估每日少量使用下，每月成本約 10-50 美元。

The architecture (processing documents)

• 章節重點原文保留:

  The system has two parts. The first is how we split up documents — what we call chunking — and embed them. This first part is very important, as it will dictate how the agent answers later. You need to be smart about ensuring each chunk has enough context (but not too much). You also need to make sure the chunk is attached to metadata so it’s easy to trace back to where it was found. This is where you’ll spend the most time, and honestly, I think there should be better tools out there to do this intelligently. Remember, if the bot is supposed to cite sources, each chunk needs to be attached to URLs, anchor tags, page numbers, block IDs, permalinks so the system can locate the information correctly being used. if the quality of the source information is poor, it’s hard to make the system work well.

• 章節總結: 系統架構的第一部分是文件處理，包含文件分塊（chunking）和嵌入（embedding），這對代理程式後續的回答至關重要。確保每個區塊包含足夠且不過量的上下文，並附帶元資料以便追溯來源，是此階段的重點，也是最耗時的部分。作者使用了 Docling 處理 PDF，並自行建置了網頁爬蟲。為了讓機器人能正確引用來源，每個區塊需附加 URL、錨點標籤、頁碼等資訊。原始資訊品質差會嚴重影響系統效能。作者甚至透過 LLM 總結文本並賦予更高權重，或讓 AI 進行更深入研究以補充資訊。

The architecture (the agent)

• 章節重點原文保留:

  For the second part, where we connect to this data, we need to build a system where an agent can connect to different tools that contain different amounts of data from our vector database. We keep to one agent only to make it easy enough to control. This one agent can decide what information it needs based on the user’s question I did set up a first LLM function that decides if we need to run the agent at all. This was primarily for the user experience, as it takes a few extra seconds to boot up the agent As for how to build the agent itself, this is easy, as LlamaIndex does most of the work for us. For this, you can use the FunctionAgent, passing in different tools when setting it up. ‘’’python

  Only runs if the first LLM thinks it is necessary

  access_links_tool = get_access_links_tool() public_docs_tool = get_public_docs_tool() onboarding_tool = get_onboarding_information_tool() general_info_tool = get_general_info_tool() formatted_system_prompt = get_system_prompt(team_name) agent = FunctionAgent( tools=[onboarding_tool, public_docs_tool, access_links_tool, general_info_tool], llm=global_llm, system_prompt=formatted_system_prompt ) ‘’’ The tools have access to different data from the vector database, and they are wrappers around the CitationQueryEngine. This engine helps to cite the source nodes in the text. To make sure the user experience is good, you can tap into the event stream to send updates back to Slack.

• 章節總結: 系統架構的第二部分是代理程式本身，它連接到包含向量資料庫中不同資料量的各種工具。為易於控制，作者僅使用單一代理程式，由其根據使用者問題決定所需資訊。為提升使用者體驗（避免啟動代理的延遲），作者設定了一個初始 LLM 函數來判斷是否需要運行代理。代理程式的建構相對簡單，LlamaIndex 的 FunctionAgent 可整合多個工具。這些工具能存取向量資料庫中的資料，並包裝了 CitationQueryEngine 以便在文本中引用來源節點。透過事件流向 Slack 發送更新，可提升使用者體驗。

Techniques you can try

• 章節重點原文保留:

  You should use hybrid search along with some kind of re-ranking. With hybrid search, supported by both Qdrant and LlamaIndex, we use both dense and sparse vectors. ‘’’python

  when setting up the vector store (both for embedding and fetching)

  vector_store = QdrantVectorStore( client=client, aclient=async_client, collection_name=“knowledge_bases”, enable_hybrid=True, fastembed_sparse_model=“Qdrant/bm25” ) ‘’’ Sparse is perfect for exact keywords but blind to synonyms, whereas dense is great for “fuzzy” matches (“benefits policy” matches “employee perks”) but they can miss literal strings like CAT-00568. Once the results are fetched, it’s useful to apply deduplication and re-ranking to filter out irrelevant chunks before sending them to the LLM for citation and synthesis. ‘’’python reranker = LLMRerank(llm=OpenAI(model=“gpt-3.5-turbo”), top_n=5) dedup = SimilarityPostprocessor(similarity_cutoff=0.9) engine = CitationQueryEngine( retriever=retriever, node_postprocessors=[dedup, reranker], metadata_mode=MetadataMode.ALL, ) ‘’’ This part wouldn’t be necessary if your data were exceptionally clean, which is why it shouldn’t be your main focus. It adds overhead and another API call.

• 章節總結: 本章節介紹了 RAG 系統中可嘗試的技術，重點是混合搜尋（hybrid search）和重新排序（re-ranking）。混合搜尋結合了稠密向量（用於模糊匹配）和稀疏向量（用於精確關鍵字匹配），Qdrant 和 LlamaIndex 皆支援此功能。擷取結果後，可進行去重複和重新排序，以過濾掉不相關的區塊，再交給 LLM 進行引用和整合。雖然這些技術易於設定，但若資料本身非常乾淨，則非必要，因為會增加額外開銷和 API 呼叫。

What you’ll actually spend time on

• 章節重點原文保留:

  Most of the things you’ll spend time on aren’t so sexy. It’s prompting, reducing latency, and chunking documents correctly. Before you start, you should look into different prompt templates from various frameworks to see how they prompt the models. You’ll spend quite a bit of time making sure the system prompt is well-crafted for the LLM you choose. The second thing you’ll spend most of your time on is making it fast. I’ve looked into internal tools from tech companies building AI knowledge agents and found they usually respond in about 8 to 13 seconds. The last thing, which takes a huge amount of work and which I’ve mentioned before, is chunking documents. If you had exceptionally clean data with clear headers and separations, this part would be easy. But more often, you’ll be dealing with poorly structured HTML, PDFs, raw text files, Notion boards, and Confluence notes — often scattered and formatted inconsistently.

• 章節總結: 開發過程中，實際耗時最多的並非炫技的技術，而是提示詞工程、降低延遲和正確地分塊文件。精心設計適用於所選 LLM 的系統提示詞至關重要。其次是提升系統回應速度，目標應在 8-13 秒內。最後，也是最耗費精力的，是處理各種格式不一致、結構不良的文件（HTML、PDF、純文字、Notion、Confluence 等），並將其有效地分塊以供系統使用。

How to build it out further

• 章節重點原文保留:

  At this point, it works well for what it’s supposed to do, but there are a few pieces I should cover (or people will think I’m simplifying too much). You’ll want to implement caching, a way to update the data, and long-term memory. Caching isn’t essential, but you can at least cache the query’s embedding in larger systems to speed up retrieval, and store recent source results for follow-up questions. You’ll also want a way to continuously update information in the vector databases. This is the biggest headache — it’s hard to know when something has changed, so you need some kind of change-detection method along with an ID for each chunk. The last thing I want to mention is long-term memory for the agent, so it can understand conversations you’ve had in the past. For that, I’ve implemented some state by fetching history from the Slack API. This lets the agent see around 3–6 previous messages when responding.

• 章節總結: 為進一步完善系統，需考慮實現快取、資料更新機制和長期記憶。快取查詢的嵌入和最近的來源結果能加速檢索。持續更新向量資料庫中的資訊是個難題，需要變更偵測方法和區塊 ID。長期記憶則讓代理程式能理解過去的對話，作者透過擷取 Slack API 的歷史記錄來實現，讓代理程式能參考最近的 3-6 條訊息。

Learnings and so on

• 章節重點原文保留:

  You learn a lot from using a framework, especially how to prompt well and how to structure the code. But at some point, working around the framework adds overhead. Most developers recommend using frameworks for quick prototyping or bootstrapping, then rewriting the core logic with direct calls in production. The general recommendation is to keep things as simple as possible and minimize LLM calls (which I am not even fully doing myself here). But if all you need is RAG and not an agent, stick with that.

• 章節總結: 作者分享了使用框架的學習心得：框架有助於學習提示詞技巧和程式碼結構，但最終可能增加額外開銷。許多開發者建議使用框架進行快速原型設計，然後在生產環境中用直接呼叫重寫核心邏輯。總體建議是盡可能保持簡單並最小化 LLM 呼叫。如果僅需 RAG 而非完整的代理系統，則應堅持使用 RAG。一個簡單的 LLM 呼叫來設定向量資料庫的正確參數，從使用者角度看，依然像是系統在「查詢資料庫」。

整篇文章重點結論

本文深入探討了如何建構一個整合 Slack 的公司內部知識問答代理程式，該代理程式運用了檢索增強生成（RAG）及代理式 RAG（Agentic RAG）的技術。文章從概念、技術棧選擇、成本與時間考量、系統架構（文件處理與代理程式設計）、可嘗試的進階技術（如混合搜尋、重新排序）等方面進行了詳細闡述。

核心觀點與實踐建議包括：

1. RAG 與 Agentic RAG 的應用：透過 RAG 使 LLM 能存取並利用內部文件回答問題；Agentic RAG 則賦予 LLM 更大的自主性來決定如何獲取資訊。
2. 技術棧選擇：推薦使用無伺服器函數（如 Modal）進行事件驅動部署，LlamaIndex 作為代理框架，以及 Qdrant 或 Milvus 等向量資料庫。
3. 成本控制：雲端和嵌入成本相對較低，LLM 的 API 呼叫是主要成本來源，建議選擇經濟型模型並最小化呼叫次數。
4. 架構關鍵：
   • 文件處理：耗時最長的部分，包括仔細分塊（chunking）文件以保留足夠上下文，並附加詳細元資料以供追溯和引用。原始資料品質至關重要。
   • 代理程式設計：使用 LlamaIndex 的 FunctionAgent 整合多個工具，並可加入初步 LLM 判斷以優化使用者體驗。
5. 實際開發重點：真正的挑戰在於提示詞工程、降低系統延遲以及處理結構不良的異質文件。
6. 系統擴展：可考慮加入快取機制、資料持續更新方案以及長期記憶功能。
7. 框架使用與簡潔性：框架有助於快速上手，但最終可能需要重寫核心邏輯以優化效能和控制力。始終追求系統的簡潔性，若僅需 RAG 功能，則不必強求完整的代理系統。

文章最後強調，雖然建構這樣的系統涉及許多細節（如評估、防護機制、監控），但最終能實現一個高效的內部知識檢索工具。作者也提及正在開發一個更易用的工具，讓使用者無需自行搭建架構即可建立類似的 Slack 機器人，並具備操作 API 的能力。