WebCrawlerAPI
Developer ToolsCrawl and scrape websites at scale with WebCrawlerAPI, starting an asynchronous job from a seed URL and collecting every page. Agents gather source material that no API exposes.
What This Integration Enables
WebCrawlerAPI is a crawler first and a monitoring service second, and the second part is what makes it interesting inside a flow. A crawl starts asynchronously from a seed URL, follows links until the page or depth limit stops it, and produces one job item per page in whichever output formats were requested, up to a single combined markdown file covering an entire documentation site. That is the bulk ingestion path. Feeds are the standing path: a feed re-crawls a URL on a schedule and reports which pages are new, changed or no longer reachable, and the On Feed Change trigger turns each of those into an event a flow can act on without polling.
Both paths draw on the same balance, which is why this connector exposes Get Account Balance and Get Usage Statistics as ordinary actions rather than dashboard trivia. Crawls, scrapes, feed runs and agent runs all fail with a payment required error when the balance is exhausted. An agent can therefore check what it is about to spend before it spends it, and a person can be asked when the answer is uncomfortable. That combination, an agent that can act and a moment where it chooses not to, is what Orchestration as a Service is for.
Without FlowRunner
With FlowRunner
Use Case Scenarios
Keeping a support knowledge base honest
An operations team indexes a vendor's documentation site so support agents can answer questions from it. A feed re-crawls the site on a schedule. When On Feed Change fires, the agent reads the change entries, pulls the stored content for each changed page, replaces just those documents in Pinecone, and posts a summary to Slack. Pages that come back as unavailable are handled differently from pages that changed, because a page that vanished should be removed from the index rather than reindexed with an error body.
Bulk ingestion of a documentation site
Before any monitoring exists, the site has to be loaded once. The agent calls Start Crawl Job with the Markdown output format and explicit page and depth limits, then polls Get Crawl Job until the status is done. Get Crawl Markdown Content then returns every successfully crawled page as one text payload, each page preceded by its source URL. Before committing to the full run, the agent calls Get Crawl Job URLs on a small probe crawl to see the path based clusters, which is how whitelist and blacklist expressions get tuned before the expensive job rather than after it.
Competitor changelog and pricing watch
A feed monitors a competitor's pricing and changelog pages. Rather than a daily email nobody reads, each detected change becomes an event. The agent uses Run Extraction Agent against the changed pages with a natural language prompt and a JSON output schema to pull out the specific fields product marketing cares about, with a mandatory spending cap that stops the agent when it is reached. The shaped records land in Notion alongside the change type and the amount the run actually cost, taken from Get Agent Run.
Human-in-Loop Highlight
The risk on this connector is not a single expensive crawl. It is the feed that nobody re-approves. A feed is a standing instruction to re-crawl a site on a schedule, billed per page every run, drawing on a balance shared with every other job on the account. Sites grow. A feed sized for a forty page help center quietly becomes a four hundred page feed when the vendor migrates their docs, and the only signal is a larger number in the recent runs history. So the agent watches the per run page count and cost that Get Feed reports, and when a feed's run cost moves materially past what it was set up to be, it calls Pause Feed and asks: "The vendor docs feed crawled 412 pages this run against 44 last week, and the account balance is now at 38 dollars. Resume it at the new size, narrow the URL scope, or leave it paused?" It deliberately calls Pause Feed rather than Delete Feed, because a paused feed can be resumed and a deleted feed is permanently cancelled and cannot. The agent picks the reversible option and lets a person choose the irreversible one.
Agent Capabilities
22 actionsCrawling
6- Start Crawl Job Starts an asynchronous crawl from a seed URL and returns the job ID immediately. The crawler follows links until the page or depth limit is reached, producing one job item per page in the requested output formats. Nothing is crawled synchronously, so the agent polls Get Crawl Job or supplies a webhook URL.
- Get Crawl Job Retrieves the status and full results of a crawl job. Status is new, in_progress, done or error, and job items hold one entry per crawled page with its original URL, HTTP status code, title, cost and content URLs for each output format.
- Get Crawl Job URLs Returns every URL discovered during a completed crawl, both as a flat list and grouped into path based clusters showing how many URLs live under each section. Used to map a site's structure and tune whitelist and blacklist expressions before a larger crawl.
- Get Crawl Markdown Link Returns a stable download URL for a single combined markdown file holding every successfully crawled page, each preceded by its source URL. Requires a job created with the Markdown output format and a status of done.
- Get Crawl Markdown Content Downloads the combined markdown of every successfully crawled page and returns it as text, each page separated by a block containing its source URL. The single step route for feeding an entire documentation site into a vector store or a language model.
- Cancel Crawl Job Cancels a running crawl. Every item not already done or in progress is marked canceled and is not charged, which makes this the way to stop a crawl consuming more balance than intended. Items that already finished remain available.
Scraping
2- Scrape Page Scrapes a single page and returns the content in the requested formats in the same call. A prompt has an AI model extract structured data instead, optionally constrained by a JSON schema. Failures come back as an HTTP 200 carrying success false with an error code, so the agent always checks the success field.
- Get Scrape Result Retrieves the result of an asynchronous single page scrape by job ID. While processing, the response contains only a pending status, so the agent polls until the finished payload arrives. Applies only to scrapes started with Run Asynchronously enabled.
Feeds
9- Create Feed Creates a scheduled feed that re-crawls a URL automatically and reports pages that are new, changed or no longer reachable. Each run is billed per page like a normal crawl. An organization may hold up to 100 feeds, and a webhook URL delivers notification when changes are detected.
- Get Feed Retrieves a feed's configuration and recent run history, including status, next and last run timestamps and last webhook delivery status. The recent runs array summarises each execution with pages crawled, changed, new, unavailable and failed, plus the run cost in USD.
- List Feeds Lists every feed belonging to the organization, newest first, active or otherwise. Each entry carries the monitored URL, output format, page limit, schedule timestamps and webhook configuration, which makes this the audit of what is currently being monitored.
- Get Feed Changes Returns the changes a feed detected in JSON Feed 1.1 format, newest page first. Every item carries the page URL and title plus the change type, HTTP status code, page size and a content URL for the stored content. Paginated at up to 1000 items per page.
- Pause Feed Temporarily stops a feed running its scheduled crawls. Scheduled runs are skipped and no pages are charged, and the feed can be restarted with Resume Feed. Feeds also pause automatically after three consecutive errors.
- Resume Feed Restarts a paused feed so scheduled crawls continue. The next run is scheduled from the moment of resuming rather than from the original schedule. Cancelled feeds are permanently deleted and cannot be resumed.
- Run Feed Now Triggers an immediate extra run of an active feed without waiting for its schedule, returning the run and crawl job identifiers. Limited to one forced run per hour per feed, requires sufficient balance, and is billed per page. The regular schedule is unaffected.
- Resend Feed Webhook Re-delivers the webhook notification for a feed's most recent completed run and reports the HTTP status the receiving endpoint returned. Used to recover changes missed because the receiver was unavailable.
- Delete Feed Permanently cancels a feed so it never runs again. This cannot be undone and the feed cannot be resumed, though past runs and detected changes stay readable through Get Feed Changes until the 30 day retention period expires. Pause Feed is the reversible alternative.
AI Agent
3- Run Extraction Agent Starts an AI agent that browses the supplied seed URLs, decides which links to follow, and extracts data described by a natural language prompt, optionally shaped by a JSON output schema. A spending cap in USD is mandatory and the agent stops once it is reached. Runs asynchronously.
- Get Agent Run Retrieves the status and result of a single extraction agent run. Status is in_progress, done, error or canceled; the extracted payload appears once done, success reports whether data came back non-empty, and the balance used shows what the run actually cost.
- List Agent Runs Returns a paginated list of extraction agent runs for the organization with the total count. Each entry summarises status, prompt, model, amount spent and creation time, which makes it the review surface for recent extractions and agent spend.
Organization
2- Get Account Balance Returns the organization's spendable balance in US dollars, combining prepaid and subscription credits. Crawls, scrapes, feed runs and agent runs all draw on it and fail with a payment required error when it is exhausted, so the agent checks it before launching a large job.
- Get Usage Statistics Returns total cost in US dollars and request count over a date range, with an optional day by day breakdown including days with no usage. Defaults to the last 30 days, may not exceed one year, and requires an admin API key.
Triggers
1 triggersEvent Triggers
1- On Feed Change Fires when a monitored feed reports a page change, optionally narrowed to a single change type. Each cycle reads the most recent feed entries and emits one event per entry not seen before, carrying the page URL, title, change summary, change type, HTTP status code and the content URL of the stored page. The polling interval is configurable with a minimum of 30 seconds. The first cycle establishes a baseline and emits nothing, so existing feed history is not replayed when the trigger is switched on.
Frequently Asked Questions
What can FlowRunner do with WebCrawlerAPI?
FlowRunner agents can run Start Crawl Job, Get Crawl Job, and Get Crawl Job URLs in WebCrawlerAPI, plus 19 more actions.
Does connecting WebCrawlerAPI to FlowRunner require OAuth?
No. WebCrawlerAPI connects to FlowRunner with an API key, no OAuth flow required.
Can WebCrawlerAPI trigger a FlowRunner workflow automatically?
Yes. WebCrawlerAPI supports 1 trigger that can start a FlowRunner workflow automatically.
Start building with WebCrawlerAPI
$100 in credits. No card required. Connect in minutes.