Skip to main content

In this section

Building a Security News Summary with CrewAI AI Agent

Written by
Headshot of Liran Tal

Liran Tal

0 mins read

Sifting through many security news sources and vulnerability databases is hard work, but with the rise of agentic AI workflows, we can delegate this work to an LLM, and this article demonstrates how to use the open-source CrewAI framework to build an AI agent that curates security news.

AI agents to curate security news

For the application we build, we will specifically focus on curating new security vulnerabilities from the Snyk vulnerability database as a security news feed.

Our AI agent will be a command-line tool that opens the Snyk vulnerability database web page for a given ecosystem (we will use the npm JavaScript ecosystem) and picks the top 3 most relevant CVE reports that it deems most important.

Getting started with CrewAI

Our first phase is to scaffold a new project directory where you see fit in your local development environment, and there to initialize the CrewAI framework.

Our requirements are:

  • The uv package manager is installed and available locally

  • A Python runtime environment >= 3.x (can be installed with uv)

In your chosen directory, let’s begin with installing the crewai library and tooling:

uv tool install crewai

Then add it to your working shell environment so we can run the command crewai like a global executable:

uv tool update-shell

Now, we can begin scaffolding a new crewai project:

crewai create crew <your_project_name>

Then install dependencies in this project by running this command:

crewai install

Now the scaffolded project directory should look as follows:

CrewAI project directory structure for an AI agent

At a high-level perspective, CrewAI has a crew.py file that defines the agents and tasks:

   @agent
    def researcher(self) -> Agent:
        return Agent(
            config=self.agents_config['researcher'], # type: ignore[index]
            verbose=True
        )

    @agent
    def reporting_analyst(self) -> Agent:
        return Agent(
            config=self.agents_config['reporting_analyst'], # type: ignore[index]
            verbose=True
        )

The same file also includes the tasks that we want to achieve with those agents:

   @task
    def research_task(self) -> Task:
        return Task(
            config=self.tasks_config['research_task'], # type: ignore[index]
        )

    @task
    def reporting_task(self) -> Task:
        return Task(
            config=self.tasks_config['reporting_task'], # type: ignore[index]
            output_file='report.md'
        )

The main.py file has several functions exported such as run(), train() and others that invoke the agents and provide them with the input they need to do the job.

Here’s how the agents kickoff flow looks like in the main Python file:

def run():
    """
    Run the crew.
    """
    inputs = {
        'topic': 'AI LLMs',
        'current_year': str(datetime.now().year)
    }

    try:
        SecurityNewsSummary().crew().kickoff(inputs=inputs)
    except Exception as e:
        raise Exception(f"An error occurred while running the crew: {e}")

All of the above describes the stock example directory setup for the project that CrewAI creates but we’re going to make changes and define our own agents, tasks and build our own custom tool for the agents to call.

Building custom agentic workflows with CrewAI

We will be making changes to three main parts of the scaffolded CrewAI project directory:

  • Changes to agents in config/agents.yaml

  • Changes to tasks in config/tasks.yaml

  • Changes to crew.py main CrewAI Python program

Additionally, we will build a custom tool in tools/ directory that scrapes a web page HTML syntax using the popular Python library BeautifulSoup.

AI agents

A good rule of thumb for AI agents is to break down the tasks for them as small as possible. Just like with software engineering and design approach, breaking down big problems into smaller ones proves helpful.

To follow up on this advice, we will define two agents:

  • A CVE Curator, purposed to find links in the web page to what it deems the most important CVEs.

  • A CVE Researcher, purposed to browse those individual CVE links (web pages) and extract data from them.

Open up config/agents.yaml and replace the agents definition that CrewAI created with the following:

1cve_curator:
2  role: >
3    Senior Security Analyst
4  goal: >
5    Find the links to the top most interesting CVEs reported in {security_vulnerabilities_url}. 
6  backstory: >
7    You're a seasoned web scraper with a knack for parsing HTML pages and extracting the most relevant information.
8
9cve_researcher:
10  role: >
11    Senior Security Researcher
12  goal: >
13    Summarize and format the CVE information found by the CVE curator.
14  backstory: >
15    You're a meticulous security researcher with a keen eye for detail. You're known for
16    your ability to turn complex data into clear and concise reports, making
17    it easy for others to understand and act on the information you provide.

Tasks

Next, we will need to define tasks for the above:

  • Initial task is to extract links for the CVEs from the vulnerability database directory (the main web page with the search results / vulnerabilities listing).

  • The follow-up task that the CVE Researcher agent needs to perform - extracting the CVE vulnerability data from the information page for each CVE.

Open up the config/tasks.yaml file and replace the contents that the CrewAI library created with the following:

cve_curation_task:
    description: >
      Parse the HTML page to find the top most interesting CVEs reported in {security_vulnerabilities_url}. 

      You need to select 3 CVEs from the page and extract the link to their vulnerability page.
      For example, the link to the vulnerability for the package @trpc/server is: https://security.snyk.io/vuln/SNYK-JS-TRPCSERVER-10060256

      You should de-prioritize choosing CVEs for malicious packages. Instead, you should prioritize CVEs for popular packages and those that are likely
      to have a high impact on the open-source ecosystem.

    expected_output: >
      A total of 3 CVEs, each formatted as a bullet point in the following markdown format 
      ```
      - https://security.snyk.io/vuln/SNYK-JS-TRPCSERVER-10060256
      ```
    agent: cve_curator

cve_research_task:
    description: >  
      For each of the CVE you should extract information by visiting the link to the vulnerability and extracting the information from the page.
      For example, the link to the vulnerability for the package @trpc/server is: https://security.snyk.io/vuln/SNYK-JS-TRPCSERVER-10060256

      Once you have the link, visit the page and extract the following information:
      1) CVE ID (example: CVE-2025-12345)
      2) Type of Vulnerability (example: RCE, SQL Injection, etc.)
      3) CVE Publication Date (example: 2025-01-01)
      4) Link to the CVE vulnerability page (example: https://security.snyk.io/vuln/SNYK-JS-KIBANA-10339388). The link exists as the href attribute of the anchor tag in the HTML page, set on the name of the vulnerability.
      5) Package name that was found vulnerable (example: kibana)

    expected_output: >
      A total of 3 CVEs, each formatted as a bullet point in the following markdown format (strict markdown formatting for example links should not have space between the [] and () chars):
      ```
      **@trpc/server** found vulnerable to CVE-2025-43855 [Uncaught Exception](https://security.snyk.io/vuln/SNYK-JS-TRPCSERVER-10060256), 24 Apr 2025
      ```
    agent: cve_researcher

You’ll notice how the tasks are carefully worded for our AI agents to parse and uphold, including an example for expected output and rules to follow.

Custom tools

For the first task and our first agent, the CVE Curation, we need to build our own custom tool in Python code. We have to resort to building this because the built-in CrewAI tool ScrapeWebsiteTool only extracts text from a web page, and isn’t able to further parse the HTML source in order to extract anchor elements (<a href>) for links to each CVE data on the page.

To begin building a new custom tool that our agents can use, first open up the tools/__init__.py file and update it to export a new TableScraperTool as follows:

from .custom_tool import MyCustomTool
from .table_scraper import TableScraperTool

__all__ = ['MyCustomTool', 'TableScraperTool']

Note, the MyCustomTool was already there from the CrewAI example project scaffold.

Next, we need to create the TableScraperTool so create a new file in tools/table_scraper.py and add the following Python code to it:

from typing import List, Dict, Any
from bs4 import BeautifulSoup
import requests
from crewai.tools import BaseTool

class TableScraperTool(BaseTool):
    name: str = "table_scraper"
    description: str = "Scrapes table data from a webpage, extracting rows, links, and other relevant information"

    def _run(self, url: str) -> List[Dict[str, Any]]:
        """
        Scrapes table data from the given URL.

        Args:
            url: The URL to scrape

        Returns:
            List of dictionaries containing table row data
        """
        try:
            response = requests.get(url)
            response.raise_for_status()
            soup = BeautifulSoup(response.text, 'html.parser')

            # Find all table rows
            rows = soup.find_all('tr', class_='table__row')
            table_data = []

            for row in rows:
                row_data = {}

                # Extract severity
                severity_elem = row.find('li', class_='severity__item')
                if severity_elem:
                    row_data['severity'] = severity_elem.get('class', [''])[0].replace('severity__item--', '')

                # Extract vulnerability title and link
                title_link = row.find('a', class_='anchor--underline')
                if title_link:
                    row_data['title'] = title_link.text.strip()
                    row_data['vuln_link'] = title_link.get('href', '')

                # Extract package name and link
                package_link = row.find('a', attrs={'data-snyk-test-package-manager': True})
                if package_link:
                    row_data['package_name'] = package_link.text.strip()
                    row_data['package_link'] = package_link.get('href', '')

                # Extract version ranges
                version_spans = row.find_all('span', class_='vulns-table__semver')
                if version_spans:
                    row_data['version_ranges'] = [span.text.strip() for span in version_spans]

                # Extract package manager and date
                package_manager = row.find('span', attrs={'type': True})
                if package_manager:
                    row_data['package_manager'] = package_manager.get('type', '')
                    row_data['published_date'] = package_manager.get('published', '')

                if row_data:  # Only add if we found some data
                    table_data.append(row_data)

            return table_data

        except Exception as e:
            return [{"error": f"Failed to scrape table data: {str(e)}"}]

    async def _arun(self, url: str) -> List[Dict[str, Any]]:
        """Async implementation of the tool"""
        return self._run(url)

And lastly, we also need to include and update the BeautifulSoup Python library as part of the project so update the dependencies list of myproject.toml to include it as follows:

dependencies = [
    ...
    "beautifulsoup4>=4.12.0"

And then apply it with the following command:

uv pip install -e .

CrewAI primary execution file

Let’s now focus on the main crew.py file, which you should find in the src/security_news_summary or other directory you’ve scaffolded the CrewAI project into.

Edit the file and first, make the following updates to the import (we are including our own custom TableScraperTool):

from crewai import Agent, Crew, Process, Task
from crewai.project import CrewBase, agent, crew, task
from crewai.agents.agent_builder.base_agent import BaseAgent
from crewai_tools import ScrapeWebsiteTool
from typing import List
from .tools import TableScraperTool

Next, scroll down and locate the functions with annotations for agents (noted by the @agent annotation) and tasks (noted by the @task annotation), and update them to reflect our own newly defined agents and tasks above:

@agent
    def cve_curator(self) -> Agent:
        return Agent(
            config=self.agents_config['cve_curator'], # type: ignore[index]
            verbose=True,
            tools=[ScrapeWebsiteTool(), TableScraperTool()]
        )

    @agent
    def cve_researcher(self) -> Agent:
        return Agent(
            config=self.agents_config['cve_researcher'], # type: ignore[index]
            verbose=True,
            tools=[ScrapeWebsiteTool()]
        )

    # To learn more about structured task outputs,
    # task dependencies, and task callbacks, check out the documentation:
    # https://docs.crewai.com/concepts/tasks#overview-of-a-task
    @task
    def cve_curation_task(self) -> Task:
        return Task(
            config=self.tasks_config['cve_curation_task'], # type: ignore[index]
            output_file='cves.md'
        )

    @task
    def cve_research_task(self) -> Task:
        return Task(
            config=self.tasks_config['cve_research_task'], # type: ignore[index]
            output_file='cves.md'
        )

Executing AI agents with CrewAI

That’s it. We’re now ready to run our AI agents that curate new security vulnerabilities for us based on the Snyk vulnerability database as a data source.

All we have to do is kick off the CrewAI agents with this command from the terminal:

crewai run

And the AI agents start their work:

CrewAI executing AI agents

And tasks are reported as the progress continues:

CrewAI agents complete tasks and report on progress, and reasoning

About AI security and agentic workflows

Building AI agents feels magical. When it works, it is executing surprisingly well, and the CrewAI library for agent orchestration is a breeze to work with.

Given how fast AI and agentic coding workflows progress you should keep a sharp eye on the AI trust and establish AI security guardrails to avoid introducing risks and security pitfalls like insecure MCP server deployments, prompt injection attacks and other AI security vulnerable surface.

I recommend the following resources:

Build secure Al development from the start

Learn how to integrate security into every step of your Al coding workflow-with real implementation tips from the experts.

Get the full guide