How does crawling work?

We fetch pages server-side (via a relay) to avoid browser CORS. You can set depth, page limit, timeouts, and User-Agent.

Do you store my data?

No persistent storage. Optionally a short cache is used on the relay to reduce repeated fetches.

Yes. We fetch /robots.txt first and skip disallowed paths for your chosen User-Agent.

Build XML sitemaps. Paste URLs or crawl a site with depth/limits. We auto-fill lastmod and include priority automatically.

1 • Choose input

Crawl a site Paste URL list

Advanced: Depth, Limits, UA & Timeouts

Max depth

Max pages

Concurrency

Connect timeout (s)

Request timeout (s)

Cache TTL (s)

User-Agent

Respect robots.txt Enable cache Bypass cache

Exclude paths (one per line, supports simple * wildcard)

Discovered URLs

Total: 0

URL	Status	Last-Modified