Prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI

AWS published a technical walkthrough and companion sample repository showing how to prepare website content for Amazon Bedrock Knowledge Bases using Crawl4AI, an open source Python crawler, as an alternative to the managed Web Crawler connector. The workflow runs on a schedule, writes cleaned Markdown and metadata sidecars to Amazon S3, and then triggers a knowledge base ingestion job. The sample is meant for batch ingestion of known public web properties, not for fetching pages at inference time, which the post distinguishes from real-time options such as Web Grounding with Amazon Nova.
The default path for most public site content remains the Amazon Bedrock Web Crawler data source connector for managed knowledge bases. It takes seed URLs or sitemaps the operator is authorized to crawl, operates within configured scope and rate limits, respects robots.txt, and supports scheduled syncs that capture only added, updated and deleted content after the first full crawl. It can also wait for content rendered by JavaScript. According to the post, this connector requires no crawler to build, host or operate.
The Crawl4AI route is for sources that need finer control before ingestion. Three capabilities drive the choice. First, Markdown preparation through DefaultMarkdownGenerator with a pruning content filter, which strips navigation and boilerplate while keeping headings, lists and tables; the post notes pruning is heuristic and output should be reviewed on representative pages. Second, bounded extraction that can isolate a page region matched by an optional CSS selector, with breadth-first traversal constrained by seed origin, page/path/host scope, depth and page caps, and URL patterns. Third, custom metadata such as source URL, crawl timestamps, content type and user-defined attributes, which can later be used to filter retrieval results. Because managed and customer-managed knowledge bases both accept Amazon S3 data sources, this path works with either type.
The sample is deployed with the AWS CDK, which packages the implementation into a container image. When the schedule fires, an EventBridge rule starts an ECS task on Fargate, which runs scripts/container_entrypoint.py and executes crawl, upload and ingestion in order. Non-secret configuration is read at startup from AWS Systems Manager Parameter Store. By default the Fargate task runs in a public subnet and reaches sites through an internet gateway; the stack does not create or modify the knowledge base or data source.
The crawl step uses AsyncWebCrawler with headless browsing and JavaScript enabled, plus a BFSDeepCrawlStrategy carrying depth and page limits and a filter chain seeded to enforce scope. Optional URL pattern include and exclude filters can be added. Requests honor robots.txt checks, use a configurable mean delay with a single concurrent request (semaphore_count=1), and re-check scope against the effective URL after redirects. Duplicate canonical URLs and failed or empty results are skipped. The implementation falls back from fit_markdown to raw_markdown if pruning produces nothing. Markdown is chosen because it preserves structure without passing all raw HTML into ingestion; the knowledge base also accepts plain text, HTML, PDF and DOCX from S3.
Uploading is where naming matters. Each document is written under a configurable s3Prefix (default web-content/), and its sidecar must be named by appending .metadata.json to the full source filename, so page.md pairs with page.md.metadata.json in the same S3 location. Amazon Bedrock caps a sidecar at 10 KB, and the run errors out before writing either file if a sidecar would exceed that. Attributes use a typed format accepted by both knowledge base types, with string and number values wrapped in a metadataAttributes object. Only files produced by the current crawl are uploaded, so stale local files are not pushed. On the data source, the metadata files prefix must be left empty because sidecars sit beside their documents. For customer-managed knowledge bases, the vector store must be able to store and filter these attributes: OpenSearch Serverless indexes need the faiss engine, Aurora needs a metadata column or a column per attribute, and S3 Vectors allows up to 1 KB of custom metadata per vector, so attributes should stay short.
After the upload, the pipeline calls StartIngestionJob through the bedrock-agent API and can poll with GetIngestionJob or watch progress in the console; the same calls work for both knowledge base types. Ingestion is where Crawl4AI's role ends. A managed knowledge base parses each file with its managed parser and applies the chunking strategy chosen when the data source was connected. Managed options are default chunking at fixed 300-token chunks with 20 percent overlap, fixed-size chunking with a user-set maximum token count and overlap, and no chunking for pre-split content. Customer-managed knowledge bases additionally offer semantic and hierarchical chunking and extra parsing options.
One operational constraint deserves emphasis: the chunking strategy cannot be changed after the data source is created, so it must be evaluated against representative pages and test questions before production. Changing it later means creating a new data source and re-ingesting.
Deployment prerequisites include an AWS account, a managed knowledge base in a supported Region or an existing customer-managed one, an S3 data source with status AVAILABLE pointing at the configured bucket, and the knowledge base, bucket and stack in one Region. The workstation needs Git, Python 3.10 or later, Node.js, AWS CLI v2, the CDK CLI v2 and a container engine. Deployment uses cdk bootstrap, then cdk deploy with context keys for knowledgeBaseId, dataSourceId, s3Bucket and seedUrls, plus maxDepth and maxPages; the walkthrough starts at one level and 25 pages. A --dry-run mode previews which pages would be prepared. The sample builds an ARM64 image by default, with a cpuArchitecture=X86_64 override for x86_64 workstations that cannot cross-build. If VPC Block Public Access is on in ingress-only mode, the task needs egressMode=nat, which adds a NAT gateway with hourly charges; bidirectional mode requires an exclusion.
The schedule defaults to rate(24 hours), and the first run starts as soon as deployment creates the EventBridge rule. cdk deploy prints TailLogsCommand and RunTaskCommand outputs; a successful run logs job_status: COMPLETE, and on-demand runs can be launched after the current one finishes. The stack name is fixed, so redeploying updates the same stack, and cleanup steps are documented to remove resources.
Scope limits are explicit. The workflow handles public pages only and does not implement authenticated sessions. For sites behind sign-in, the post points to the Bedrock Web Crawler's authentication options, which include basic, form-based and SAML authentication; adding login support to the sample would require session handling, credentials in AWS Secrets Manager and a security review. Hosting and egress costs for Fargate, S3 storage, container image builds and any NAT gateway, plus Bedrock ingestion and retrieval charges, fall on the adopter.
Why it matters: teams building RAG applications on Bedrock now have a documented, open source starting point for the cases where the managed crawler is too coarse, namely main-content extraction, tight crawl boundaries and metadata-driven filtering at retrieval. Developers should budget time for reviewing pruned Markdown output and locking in a chunking strategy, since the latter is effectively permanent per data source; my inference is that organizations with heterogeneous or noisy sites will get the most value, while those with clean public sitemaps can likely stay on the managed connector and skip the extra infrastructure.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
This post shows how to use Crawl4AI, an open source web crawler, as a content preparation layer for Amazon Bedrock Knowledge Bases, handling JavaScript rendering, selective extraction, and custom metadata enrichment before web content reaches an Amazon S3 data source.