From Slack Threads to Structured Knowledge: Implementing RAG at Snyk

Our mission at Snyk is to help organizations build fast and stay secure. Internally, this means ensuring our teams can quickly tap into the rich institutional knowledge generated daily. A key source? Slack.

Slack serves as our central hub for updates, technical discussions, and valuable Q&A. However, as frequent users know, its conversational nature means critical information quickly gets buried under threads, channels, and countless messages.

Enter Retrieval Augmented Generation (RAG). RAG enhances Large Language Models (LLMs) by providing them access to specific, current information not in their original training data. Our goal: leverage Slack's wealth of insights to empower a smarter internal Q&A tool.

Understanding the data challenge

The journey began not with complex algorithms but by diving into Slack’s messy, real-world data. Slack messages aren't neatly organized, they include casual chats, emojis, and cryptic identifiers like <@U123ABC> for users and <#C456DEF> for channels. The foundational principle remained clear throughout this process: "If you put garbage into a model, you get garbage out." Before building our solution, several key challenges emerged:

• Finding valuable information: Identifying relevant data amid conversational noise required careful curation.

• Chunking strategy: Determining how to segment data was tricky. Threads varied significantly. Some channels had rapid-fire activity, while others were sporadic. Conventional approaches like daily or rolling-window chunking were problematic, lacking consistency and focus.

• Decoding Slack IDs: We had to translate user and channel IDs into meaningful names to ensure readability.

• Markdown conversion: Initially, converting threads into markdown seemed straightforward. Yet, raw markdown proved cluttered with emojis, reactions, and nested replies, potentially confusing the LLM.

The “Aha!” moment: Q&A as core structure

The breakthrough came from stepping back and refocusing on the essential reason behind the RAG system: answering employee questions. Instead of forcing data into predefined chunks, we shifted to structuring information around clear Q&A interactions. Each Slack thread asking a question and receiving a response became a single, structured document chunk.

This intuitive shift streamlined our process:

• Question identification: We focused on Slack channels explicitly designed for questions, such as #ask-product.

• LLM for summarization: Using Google's Gemini API, we fed Slack threads with prompts like, “Extract a clear question and comprehensive answer.” This generated high-quality, structured data ready for our RAG system.

• Retrieval strategy: We embedded only the extracted questions, allowing for efficient retrieval by comparing the user’s query to the stored question. Crucially, the LLM received the answer (stored as metadata) for the most similar stored question, not the question itself. This ensured the most relevant context was passed to the model.

• Fallback approach: Not all valuable threads were straightforward Q&A. For these cases, we formatted threads into clean markdown summaries to preserve broader context.

• Continuous updates: The system dynamically regenerates summaries whenever new messages are added, ensuring our knowledge base remains current.

Implementing the solution

With a refined focus on Q&A structures, our technical workflow became clear and effective:

• Slack SDK: Fetch messages directly from Slack.

• Gemini API: Transform threads into structured Q&A or summarized markdown.

• Snowflake Cortex Search Service: Store structured outputs and create embeddings from questions.

• RAG Application: Employees ask queries, which are embedded and compared semantically to stored questions. Relevant answers are retrieved (from metadata) to provide contextually accurate responses.

View image description

Beyond APIs: Real engineering insights

Our final solution didn't rely on groundbreaking algorithms but on deeply understanding our data, problem-solving creatively, and refining our approach iteratively. The real engineering achievement wasn't just integrating APIs or implementing embeddings, but meticulously structuring data to solve our core challenge.

Our Q&A-centric strategy elegantly aligns with RAG mechanics, converting Slack's conversational chaos into clear, accessible knowledge. This demonstrates how innovative AI solutions often emerge from human-centered insights rather than complex technological feats alone.