Guides

Sites and robots.txt

Put the entry from the where.sites column in where.sites. A site marked "needs a terms review before use" is refused with site_not_reviewed until someone at Snag has read its terms; see robots.txt. A site that needs a key on our server says so with site_unavailable when the key isn't set, and so does a site that walls this server off ("not available").

For dates, attributes.from and attributes.to are YYYY-MM-DD, inclusive. Readers that take a time zone read attributes.time_zone (an IANA name like America/New_York). These and the other request settings below (party size, ids, selector) tell the reader what to fetch; they never filter results the way size or color do.

Stores

If a store isn't listed, preview a product URL from it. Most stores run on one of these platforms or publish schema.org data, and those work without anything from us.

Marketplaces

Apartments

A page whose path is about units to rent (/floorplans, /available.html, /apartments/...) goes to the apartment reader. If it shows neither AppFolio nor SightMap, it's read by a learned recipe instead.

Booking and appointments

Booking readers give one slot per open time. Use { "field": "availability", "op": "=", "value": "open_slot" }.

Restaurants

Ask for attributes.party_size (default 2) and either date or from/to in the restaurant's local time, at most 14 days per check. time_from and time_to (HH:MM) narrow the hours. Slots carry party_size, seating, local_date, local_time and time_zone.

Tickets and events

Classes, courts and tee times

Jobs

Each open job is a listing; a closed one drops out. Use match: line with words from the title.

Software, games and status pages

Feeds and posts

Books and streams

Markets

Quotes are exact: a price of $0.000002808 is compared as that, not rounded to a cent.

Rates and economic data

Government

Bills and regulations

status is our reading of the action's wording, not an official field.

Open data

Attributes for both: where (a SoQL or SQL condition), q or search words (Socrata's full-text search), order, date_field, key_field, title_fields, place_field, location_field (Socrata), time_zone (Socrata: turns the portal's local times into exact ones), and limit (1 to 1000, default 200). location.zip with location.radius keeps rows within that distance.

Research

Names and domains

Any page

Two readers cover pages nothing above reads. Both are looser than a site's own reader, so check the preview.

A page with no reader of its own that you name with its path ("example.com/book/haircut") is read by a learned recipe. A new page is learned only at preview and spawn, and a spawn that can't learn the page is refused with recipe_not_learned. If the layout changes and the recipe stops finding its items 3 runs in a row, it's learned again, at most once every 6 hours.

For page changes, set what to read in target.attributes: selector (a CSS selector; default the page's main text), contains (a phrase or a list), ignore (a CSS selector to drop, like a ticker), strip_noise: false to keep times with seconds, and browser: true for a page that builds its text with scripts. Also:

• list: a CSS selector for repeated items. One result per item, keyed by its link, for { "field": "items", "op": "added" }. At most 500 items; more is flagged list_truncated.
• pdf: true: read the link as a PDF. Links ending in .pdf are read as PDFs anyway.
• seo: true: the page's title, meta description, canonical, robots meta, h1 and script hosts, for watching your own site's tags.
• summarize: true: a one- or two-sentence summary of the latest change, from the server's LLM.

A selector or list that matches nothing on the first read fails with selector_not_found, naming close selectors that do match. To wait for an element to appear, watch a larger region with contains. Each change also carries change_excerpt, the changed lines with a line of context, which judged conditions read. The last version is kept in the database, so a deploy doesn't lose what changed. PDFs are fetched directly (never through the proxy), at most one request a second per host, and robots.txt is checked on every redirect.

A url match with any of these, or a link ending in .pdf, is a page-change watch even without the change: prefix. Timestamps, relative times ("5 minutes ago") and long tokens are dropped before hashing, so they don't count as a change.

Steps and location

With browser: true, target.attributes.steps runs in the page before it's read, one step per string:

End a step with ? to make it optional. {{name}} is filled from target.attributes, and {{zip}} from location.zip. At most 10 steps, 5 seconds each, and 25 seconds in all (less when the page's own load is quicker). Steps never type into password, file or card fields. A step that can't be done fails the check with step_failed and says which step. A bad step is refused at spawn with bad_steps. A captcha that guards the form is solved when the server has a captcha service set; otherwise the run is blocked.

Each account gets 200 browser runs with steps and 100 captcha solves a day (previews count), reset at 00:00 UTC. Past that, a run with steps is skipped and its check says usage_limit with the reset time; a captcha past the cap is left unsolved and the check says usage_limit instead of blocked. Steps that would send a contact, sign-up, registration, subscription, checkout or application form, or post a form after filling more than one field, are refused at spawn with steps_submit_refused (or fail the step with that reason when only the page shows it). Search forms, a single field then Enter, and GET forms are allowed.

For location, the browser gets the position (the ZIP's center when there's no lat/lng), locale ("en-US") and timezone. cookies (up to 5, name=value) are set on the page's host before it loads, like "GuestLocation=48104|42.270|-83.730|MI|US" on Target. Point steps at searches and lookups; see the limits above for the forms they won't send.

Blocks

We fetch directly, and through a residential proxy only after a store blocks us. APIs and sites that wall bots off on purpose (booking, restaurant, ticket and job sites, every API above) never go through the proxy: a block there is the site saying no.

When a page we load in a browser puts up reCAPTCHA (v2 or v3) or Cloudflare Turnstile, we solve it through a captcha-solving service and carry on, once the server has CAPTCHA_PROVIDER and CAPTCHA_API_KEY set. hCaptcha is reported as blocked. A server can turn solving off for sites nobody has reviewed (WATCHER_CAPTCHA_UNREVIEWED_HOSTS=off); a captcha there stays unsolved and the check says captcha_not_allowed. A page whose address is on a private or local network is never fetched (private_address). A site that can't be read 3 runs in a row (blocked, unreachable, or the page or subreddit is gone) sends a blocked event saying why, and its conditions stay unknown until it answers again.

robots.txt

We read each host's robots.txt once a day and don't fetch paths it disallows. A check that hits one fails with the issue robots_disallowed. A host whose robots.txt can't be read (a timeout, a server error, or a bot check in its place) is treated as disallowing everything until the next read.

Some sites disallow the paths that hold the data. Once Snag has reviewed such a site's terms, watchers on it must set:

"fetch": { "accept_site_terms": true }

By setting it, you confirm you're allowed to fetch that site under its terms, and that you, not Snag, are responsible for doing so.

Server keys

Self-hosting? These keys turn sites on. Each is optional; without it the site says so at spawn or reads less.