Freshness
Updated source context
Use feeds when the agent needs current information beyond model training data and static documentation.
Crawling for agents
A web crawling API for AI agents should not stop at raw pages.
Crawling can collect pages, but agents still need summaries, dates, source URLs, stable IDs, tags, and provenance before context becomes reliable.
Synorb is strongest where teams need watched coverage delivered as Manifests rather than asking every agent workflow to crawl and clean the same sources.
MCP · REST · Source URLs · Stable IDs · Manifests
01 / Criteria
What should the data layer include?
For crawled or watched web context prepared for AI agent workflows, the useful unit is not a loose search result. It is an object the agent can retrieve, cite, filter, store, and audit.
Grounding
Evidence stays attached
Source URLs, dates, and stable IDs help the application cite, inspect, and audit what the model used.
Delivery
MCP, REST, webhooks, and archives
Agents can explore through Core MCP. Production systems use REST and webhooks for current delivery. The live window covers the current calendar month plus the previous three full months; S3 archive exports support historical backfills and replay for older months.
02 / Shape
A Manifest is the object the agent can use.
This JSON manifest is the source-grounded object delivered through MCP or REST. It is compact enough for an agent workflow and explicit enough for an application to store, cite, and audit.
Manifest excerptJSON
{
"manifest_id": "1777525429698648000",
"headline": "Source-grounded update for an AI workflow",
"summary": "What changed, why it matters, and what source supports it.",
"source": {
"name": "Watched source",
"url": "https://source.example/update",
"published_date": "2026-06-21"
},
"delivery": {
"mcp": "https://mcp.synorb.com/mcp",
"rest": "https://api.synorb.com"
},
"tags": ["company", "topic", "source-backed"]
}
03 / Fit
Where Synorb fits in the workflow.
Use Synorb when your team already knows the sources or topics it needs to monitor, and the workflow needs current context again and again. Use search or crawling for open-ended discovery.
Agents
Pull live context
Use Synorb MCP to discover Streams, inspect details, and retrieve Manifests inside an agent workflow.
RAG
Load before prompts
Push source-grounded Manifests into retrieval stores before users ask for current answers.
Apps
Render with citations
Build dashboards, feeds, monitors, and briefings with source URLs available at display time.
FAQ
Short answers for AI builders.
What should a web crawling API for AI agents produce?
It should produce structured, source-backed objects that agents can cite, filter, route, and store.
When is crawling useful?
Crawling is useful for broad discovery, custom extraction, and sources outside existing coverage.
When is a watched feed better?
A watched feed is better when the same sources need to stay current with consistent metadata and citations.
How does Synorb fit with crawling?
Synorb can reduce crawl-first work for known coverage by delivering current Manifests through MCP, REST, and webhooks.
Test
Test Synorb feeds for free.
Want to connect to Synorb's graph to test source-grounded feeds for free? Start with free test credentials, then connect through Core MCP or REST.
Free test credentialscurl
curl -s https://synorb.com/connect
Start
Give your agent fresh source-backed context.
Start with keys, then connect through Core MCP while building or REST when your application owns the workflow.