> ## Documentation Index
> Fetch the complete documentation index at: https://phidatainc-agui.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl4AI

**Crawl4aiTools** enable an Agent to perform web crawling and scraping tasks using the Crawl4ai library.

## Prerequisites

The following example requires the `crawl4ai` library.

```shell theme={null}
uv pip install -U crawl4ai
```

## Example

The following agent will scrape the content from the [https://github.com/agno-agi/agno](https://github.com/agno-agi/agno) webpage:

```python cookbook/14_tools/crawl4ai_tools.py theme={null}
from agno.agent import Agent
from agno.tools.crawl4ai import Crawl4aiTools

agent = Agent(tools=[Crawl4aiTools(max_length=None)])
agent.print_response("Tell me about https://github.com/agno-agi/agno.")
```

## Toolkit Params

| Parameter           | Type    | Default              | Description                                                                               |
| ------------------- | ------- | -------------------- | ----------------------------------------------------------------------------------------- |
| `max_length`        | `int`   | `1000`               | Specifies the maximum length of the text from the webpage to be returned.                 |
| `timeout`           | `int`   | `60`                 | Timeout in seconds for web crawling operations.                                           |
| `use_pruning`       | `bool`  | `False`              | Enable content pruning to remove less relevant content.                                   |
| `pruning_threshold` | `float` | `0.48`               | Threshold for content pruning relevance scoring.                                          |
| `bm25_threshold`    | `float` | `1.0`                | BM25 scoring threshold for content relevance.                                             |
| `headless`          | `bool`  | `True`               | Run browser in headless mode.                                                             |
| `wait_until`        | `str`   | `"domcontentloaded"` | Browser wait condition before crawling (e.g., "domcontentloaded", "load", "networkidle"). |
| `enable_crawl`      | `bool`  | `True`               | Enable the web crawling functionality.                                                    |
| `all`               | `bool`  | `False`              | Enable all available functions. When True, all enable flags are ignored.                  |

## Toolkit Functions

| Function      | Description                                                                                                                                                                                                      |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `web_crawler` | Crawls a website using crawl4ai's WebCrawler. Parameters include 'url' for the URL to crawl and an optional 'max\_length' to limit the length of extracted content. The default value for 'max\_length' is 1000. |

## Developer Resources

* View [Tools](https://github.com/agno-agi/agno/blob/main/libs/agno/agno/tools/crawl4ai.py)