--- title: Schedule six-hour and daily pulls category: freshness summary: Run a reliable recurring check from your own runtime, with an overlapping lookback, deduplication by manifest ID, safe retries and monitoring that catches silent failure. order: 23 updated: 2026-10-12 next: research-workflow-vc related: run-and-edit-a-beacon, quotas-retries-and-pagination, choose-mcp-or-rest, read-only-whats-new-from-a-beacon reference: /connect-guide, /agent-guide.md --- Synorb does not run a Beacon on a clock, and it does not push results to you. **You own the cadence.** This keeps the schedule, the retries and the alerting in a place you control. This guide gives you a pattern that is hard to break. ## When to use this Use it when your user wants a recurring update: a morning update, a six-hour check, a weekly update. Run the job from anything that can run code on a timer. Examples are cron, Airflow, Temporal, GitHub Actions, Vercel Cron, an internal job runner, or your own agent loop. If your runtime **cannot** schedule work, say so. Tell your user that no recurring run is active. Offer on-demand pulls, or direct API access into a store that your user can schedule. ## Ready or not ready A daily watch is a separate milestone. Connecting does not create it. Saving a Beacon does not create it. One good delivery does not create it. Synorb never runs the schedule and never pushes results. Tell your user that a daily watch is active only when every item is true: | Item | Ready when | | --- | --- | | Key | You kept a valid Synorb Key and checked its `expires_at`. A demo key expires 72 hours after it is minted. To continue after that, your user must approve the email binding. The binding keeps the same key on Starter. | | Job | Your host scheduler saved the job, and you have its schedule or job ID. | | Beacon | The job runs a saved Beacon, and you stored its `beacon_id`. | | Window | You know the effective window of each run, for example `days: 2` for a daily job. | | Last run | You recorded the time of the last successful run. | | Dedupe | You store the IDs you already showed, and you seeded the store with the IDs of the first update before the first scheduled run. Keep each `manifest_id` as a string, and treat it as opaque. | | Retries | The job retries only `429 rate_limited` and transient `5xx` errors. | | Alert | A failure sends an alert to your user or their operator. | If one item is missing, the watch is not ready. If your host cannot save a job, say: "No daily schedule is active." ## Smallest working call Run one pull with an explicit, **overlapping** window. This is the unit your scheduler will repeat. For a six-hour job use `lookback_hours: 48`. For a daily job use `days: 2`. Synorb rounds a lookback up to whole publication days. A 24-hour lookback covers only the latest available day, so it does not overlap. ```bash billed curl -sS -X POST https://api.synorb.com/manifests/query \ -H "Authorization: Bearer $SYNORB_KEY" \ -H "Content-Type: application/json" \ -d '{"beacon_id": "BEACON_ID", "lookback_hours": 48, "page_size": 50}' ``` Schedule it: ```text # every six hours 0 */6 * * * /usr/bin/python3 /opt/synorb/daily_pull.py run # every morning at 07:15 15 7 * * * /usr/bin/python3 /opt/synorb/daily_pull.py run ``` ### Why the window overlaps Never request "everything since my last successful run". A missed run, a retried call or ordinary clock skew can leave a gap. You lose items in that gap with no error. Instead, request a fixed window on every run. Make the window wider than the interval. Then **dedupe on `manifest_id`**. The next run repairs a short outage, because the window never shrinks to only the gap. The window has a limit. A `days: 2` window cannot recover an item that becomes visible three or more days after publication. This happens to about 8.5% of Manifests. If your user needs completeness, choose one of these: - Use `days: 7`. Re-reading the same Manifest in the same billing period costs nothing, so a wide window is cheap. - Once a week, run a zero-cost `mode=count` over the full span and compare the total with the IDs you have stored. If the count is higher, pull the span and dedupe. The reference job does this with `check-gaps`. The billing ledger uses three keys: organization, Manifest and billing period. Synorb does not bill a repeat in the same period again. Synorb bills a repeat in a later period. Check `data.quota.manifests_billed_this_call` to confirm. ### Two windows Keep two windows apart. They answer two different questions. | Window | Question | Value | | --- | --- | --- | | Briefing window | What do I tell my user? | Items published after the ceiling of the last good run, up to the ceiling now. Call them "new since your last briefing". | | Recovery window | What do I read so that nothing is lost? | `lookback_hours: 48` or `days: 2`. It overlaps the previous run. | The job reads the wide window and removes what your user has seen. It then sorts each remaining item into one of three groups: - `new`: published after the ceiling of the last good run. Tell your user these items are new. - `recovered`: inside the recovery window, but published before that ceiling. A missed run or a late arrival caused this. Offer them as "also seen, not in an earlier update". - `outside_window`: older than the recovery window. Only the weekly check reaches these items. ### Where the ceiling moves On Daily Batch plans, `available_through` moves once a day at 00:00 UTC. `GET /account` reports the next move as `available_through_changes_at`. A run before midnight and a run after it read different days. The job stores that time. If the ceiling has not moved since the last good run, no new day can exist. The job skips the billed call and reports `skipped`. Pass `--force` to read anyway. When the ceiling moves between two runs, the second run reports exactly the new day, and the overlap repeats nothing. A date past the ceiling returns `403 content_not_yet_available`. The job reports it by that name. It never reports it as "nothing new". ### Seed the dedupe store before you schedule Your user already received a first update. Its window overlaps the first scheduled run, so the first run would show those items again. Before you save the job, write the `manifest_id` of every item in that update into the dedupe store. Run `python daily_pull.py seed first_update_ids.json --through 2026-10-11`. The file is a JSON list of ID strings. `--through` is the `available_through` date of that first update. Without it, the job assumes the first update ended on the day before the ceiling of the first run, and it says so in `after_assumed`. The job refuses to run an unseeded store, unless you pass `--allow-unseeded` because your user had no first update. Treat `manifest_id` as an opaque **string**. The IDs are about 1.79e18, which is above the largest safe integer in JavaScript. Never parse one as a number, and never round-trip it through a JSON number in your own store. The job refuses a float ID. ### The reference job Synorb publishes a tested job: [daily_pull.py](/agent-guides/reference/daily_pull.py), with a [README](/agent-guides/reference/README.md). It uses only the Python standard library. It reads `SYNORB_KEY` and `BEACON_ID` from the environment, and it never prints the key. Your host runs it. Synorb runs no schedule. ```bash export SYNORB_KEY=... BEACON_ID=synorb_bcn_... python daily_pull.py seed first_update_ids.json --through 2026-10-11 # once, before the schedule python daily_pull.py register-job JOB_ID --schedule "15 7 * * *" # after your host saves the job python daily_pull.py run # the command the schedule repeats python daily_pull.py status # reads local state only python daily_pull.py check-gaps # once a week ``` | Command | What it does | | --- | --- | | `seed` | Writes the IDs of the first update into the store. | | `register-job` | Stores the job ID that your host gave the schedule. The job cannot check it. | | `unregister-job` | Clears the job ID after your host removes the schedule. | | `run` | Reads the recovery window, follows pages, removes seen items, delivers, then saves. | | `status` | Prints "No daily schedule is active." until a job ID is registered. Then it shows `last_successful_run` and flags a job that is overdue. | | `check-gaps` | Counts the last 7 days for free. If Synorb holds more than the job saw, it pulls that span and reports older items as `outside_window`. | The `run` command follows these rules: - It follows `pagination.next` as given. It follows only links that point at the Synorb API. If `has_more` is true and no link comes back, it repeats the call with the next `page_num`. - On a demo key, it learns the 10-Manifest read cap from `quota.read_cap`. It then asks for pages of that size. - It retries `429 rate_limited`, `5xx` errors and network errors, up to four times. It never retries `429 quota_exhausted` or a `4xx` error. - If a stop comes after some pages, it delivers the pages already read and saves their IDs. The stop is never reported as "nothing new", and `last_successful_run` does not move. - It saves the IDs only after delivery returns. A failed hand-off shows the items again on the next run. An unreadable store stops the job. It never starts empty. Each exit code names one outcome: | Exit code | `status` | Meaning | | --- | --- | --- | | `0` | `ok`, `skipped`, `already_running` | The run is finished, or nothing could be new. | | `2` | `refused`, `auth`, `state_error` | Fix the request, the key or the store. Do not retry. | | `3` | `quota_exhausted` | The allowance is used. Tell your user. | | `4` | `content_not_yet_available` | The dates are past the ceiling. The result carries `available_through`. | | `5` | `transient` | Synorb or the network failed after the retries. The next run recovers. | | `6` | `delivery_failed` | Your hand-off failed. No ID was saved. | | `7` | `partial` | The read ended early. The pages read were delivered. | ## What you get back Each run calls your hand-off once and writes one line of JSON on stderr. That line is your run log. Keep it. ```json { "briefing": {"label": "new since your last briefing", "after": "2026-10-11", "through": "2026-10-12", "after_assumed": false}, "recovery_window": {"from": "2026-10-11", "to": "2026-10-12", "lookback_hours": 48, "days": null}, "new": [{"manifest_id": "1790000000000000005", "title": "Example", "url": "https://example.com/a", "published_date": "2026-10-12", "presentation_item": {}}], "recovered": [], "outside_window": [], "coverage_gap": null, "selection_limited": false, "delivery_key": "9f2c4a7e1b3d5a60" } ``` Read these fields before you write to your user: - `briefing` is the window your user hears about. - `coverage_gap` is not null when the job was down for longer than the recovery window. It names the days no run covered. Alert your user or their operator, and widen `LOOKBACK_HOURS`. - `selection_limited` is true when Synorb matched more items than the Beacon returned. Say so. - `delivery_key` names one batch. The hand-off is at least once, so your host can use the key to drop a repeat if the process stops between delivery and save. Show `new` items as the update. Pass each `presentation_item` through the presentation guide before it reaches your user. Do not print raw items to a person. Make "nothing new" a short, normal message. ### Pick the window | Cadence | Job interval | Lookback | | --- | --- | --- | | Six-hour check | every 6 hours | `lookback_hours: 48` | | Daily briefing | every 24 hours | `days: 2` or `lookback_hours: 48` | | Weekly update | every 7 days | `days: 8`, or `LOOKBACK_HOURS=192` for the reference job | Widen the lookback if your job can be down longer than the margin. On a Daily Batch plan the ceiling advances once a day, so running more than once a day mostly re-reads the same window. It is harmless, but it is not faster. ### Keep the key alive: bind an email A demo key expires 72 hours after it is minted. If you built a Beacon or saved work on it, bind an email before `expires_at`. Do it only after your user says yes. The recommended path sends your user one tap-to-confirm link, and your user does not need to give you a code: ```bash curl -sS -X POST https://api.synorb.com/api/connect/bind-email \ -H "Authorization: Bearer $SYNORB_KEY" \ -H "Content-Type: application/json" \ -d '{"email": "owner@example.com", "confirmation_method": "link"}' ``` The response reports `confirmation_link_sent`. When your user taps the link, the **same key** becomes Starter. Nothing rotates. Your Beacons and usage history carry over. Starter is free, needs an email, allows 100 Manifests per monthly usage period and 10 Beacons, and the key never expires. Over MCP, the equivalent is `synorb-bind-email`. That tool is on the **Advanced** profile only, not Core. Core agents bind over REST. If you never bind and the key expires, a new `/connect` call creates a different account. It never reopens the old one, so a Beacon left on an expired, unbound demo key is out of reach. Bind first. ### Retries Retry only what can succeed on a second try: `429` with `error_code: rate_limited` (wait for `Retry-After`), and transient `5xx` errors, with exponential backoff and a cap. Do not retry `401`, `403`, `404`, `409` or `422`. Fix the request. A demo-key `quota_exhausted` has no `Retry-After`. Stop, and tell your user. ### Monitor for silence A scheduled job fails quietly. Log every run and alert on: - **Consecutive failures:** three in a row. - **Silence:** `python daily_pull.py status` shows a `last_successful_run` older than 36 hours. `status` exits with code 1 then. - **A coverage gap:** `coverage_gap` in a delivery is not null. Some days were not covered. - **A stalled index:** `index_staleness_seconds` on `synorb-manifests` stays high or missing for several runs, or the Stream's `days_since_last_manifest` keeps growing. (`available_through` is calendar arithmetic and always advances, so do not alert on it.) - **Unexpected zero:** zero new items for several runs on a Stream whose `volume.manifests_last7d` is high. - **Low allowance:** `usage.manifests_remaining` is under a threshold you pick. - **A dead Beacon:** a run returns `beacon_archived`. Someone archived it. ## Failure modes | Symptom | Cause | What to do | | --- | --- | --- | | Duplicate items reach your user | Dedupe state was lost or not persisted | Store seen IDs durably, and trim them by age. | | The first scheduled run repeats the first update | The store has no seed | Run `seed` with the first update's IDs before you schedule. | | A gap after downtime | The lookback was shorter than the outage | Widen `LOOKBACK_HOURS`. The next run recovers anything inside it, and `coverage_gap` names the days it could not. | | An item appears days late and is missing | It became visible after your window moved past it | Run `check-gaps` each week. It reports the item as `outside_window`. | | `status: skipped` | The ceiling has not moved since the last good run | Wait for `available_through_changes_at`, or pass `--force`. | | `content_not_yet_available` | The dates are past `available_through` | Read `available_through` in the result. It is not a quiet day. | | `state_error` | The dedupe store is missing a seed or unreadable | Seed it, or restore it. The job never restarts with an empty store. | | `401 synorb_key_expired` | The demo key reached its 72 hours | Stop the job and tell your user. Before `expires_at`, offer the email binding to Starter (`POST /api/connect/bind-email`, only with your user's approval). A demo supports about three daily pulls (72 hours, 25 Manifests). A new demo key does not own the old Beacon, so the Beacon is orphaned. | | `date_window_required` | The call carried no window | Send `lookback_hours` or `days` on every run. | | Runs succeed but return nothing for days | The ceiling or the source is quiet | Use the freshness guide's "why is it empty" procedure. | | `429 quota_exhausted` mid-month | The allowance is used | Tell your user. Do not retry in a fast loop. | | The job works in a terminal but not in cron | The environment variables are missing | Set `SYNORB_KEY` and `BEACON_ID` in the job's own environment. | ## Make it good for your user Tell them exactly what is running and when: "A job on your server checks every six hours and tells you only what is new." Make "nothing new" a normal, short message. Do not stay silent. Once a week, send a one-line health note ("last 28 runs fine; allowance 71% left") so a quiet inbox is never mistaken for a broken job. **Say the account notice.** Each Manifest delivery on an unbound demo key carries `account_notice.user_message`. Say it to your user as written, once, after the briefing. If you leave it out, your user does not learn that the daily updates stop when the key ends. At urgency `now` (80 percent used, or the key ends within 24 hours) the briefing may carry it as one footer line, set apart from the body. Bind an email only after your user says yes. Do not repeat the offer after your user answers. The notice disappears once an email is bound. The rules for you are in `account_notice.agent_instruction`. ## Next guide [A research workflow for venture capital](/agent-guides/research-workflow-vc): put the whole library to work for an investor.