How to download an entire website
Crawl and copy a whole site — the flags that matter for wget and HTTrack, why client-rendered sites need a browser, and how to stay polite while doing it.
Copying one page and copying a site are different problems. One is fidelity; the other is graph traversal, politeness and knowing when to stop.
Decide the boundary first
Nearly every runaway crawl comes from skipping this step. Write down, before you start:
- Which hosts. The domain plus its CDN, usually. Not every domain it links to.
- How deep. Three levels covers most marketing sites. Unbounded depth on a site with a calendar or faceted search is a trap that will fetch a million URLs.
- Which paths. Include
/blog/, exclude/cart/,/search,?sort=,/tag/. - How many pages, hard limit. Set one even if you think you do not need one.
- Whether pages need JavaScript. This decides the whole approach.
Server-rendered sites: wget or HTTrack
wget --mirror \
--page-requisites \
--convert-links \
--adjust-extension \
--restrict-file-names=windows \
--span-hosts --domains=example.com,cdn.example.com \
--level=3 --wait=1 --random-wait \
--reject-regex='(\?|/cart|/search)' \
--user-agent='Mozilla/5.0 (offline archive)' \
https://example.com/The flags that decide whether the result is usable:
--page-requisitesfetches CSS, JS and images; without it you get HTML only.--convert-linksrewrites references to your local copies. Without it your "offline" mirror is a set of links to the internet — the default failure of this whole category. Even with it, expect leftovers: across eight single pages in our benchmark,wgetstill left 554 requests aimed at the original hosts.--adjust-extensiongives.htmlnames so a local browser opens them.--wait=1 --random-waitis the difference between an archive and a small denial-of-service.--reject-regexis what stops the calendar trap.
HTTrack does the same job with a GUI and a rules dialog (WinHTTrack on Windows), and it is friendlier for choosing include/exclude patterns interactively. Both are free, mature, and both fetch what the server sends.
Client-rendered sites: crawl in a browser
If the site is a React, Vue, Svelte or Astro app that hydrates in the browser, a server fetch returns a near-empty shell — a <div id="root"> and a script tag. The pages you want do not exist until a browser runs the JavaScript.
The only approach that works is to crawl with a browser: load each page, wait for it to settle, capture the rendered DOM and computed styles, then embed the assets. That is what CopyAnySite's crawl mode does from the side panel — you give it a start URL, a depth, a page cap and an include/exclude pattern, and it captures each page the way it captures a single one, then exports the set as a ZIP with links rewritten between the captured pages.
The trade-off is honest: rendering every page is far slower than fetching every page, and self-contained pages that each embed their own assets duplicate those assets. For a 50-page marketing site that is fine. For a 50,000-page e-commerce catalogue, no browser-based crawler is the right tool.
Being a good citizen
- Read
/robots.txtand respect it. It is not legally binding everywhere, but ignoring it is how tools get blocked for everyone. - Rate-limit. One request per second is polite; parallel unlimited fetching is how you end up in a WAF rule. We had to add per-host pacing to our own capture after Wikimedia started returning 429s and leaving images unembedded.
- Identify yourself in the user agent if you are crawling anything sizeable.
- Cache and resume rather than re-crawling from scratch.
- Do not crawl behind a login on someone else's site.
Verify the mirror
- Disconnect from the network and browse the copy from its entry page. Follow ten links at random.
- Grep the saved HTML for the original domain:
grep -ro "https://example.com" . | wc -l. Every hit is a link that will break. - Check a few asset-heavy pages, not just the home page.
- Confirm the page count matches your expectation — a mirror with 4 pages when you expected 40 usually means a client-rendered site or a blocked crawl.
What a mirror is not
Not the CMS, not the database, not the search, not the cart, not the forms, not the redirects, not the analytics config. A mirror is the published front end at a moment in time — which for archives, migrations, offline reading and evidence is exactly what you want, and for "recreate this business" is not close.
FAQ
What is the best tool to download an entire website?
For server-rendered sites, wget --mirror or HTTrack, both free and mature. For sites built in the browser with JavaScript, you need a browser-based crawler that captures each rendered page — that is what CopyAnySite's crawl mode does.
Can I download a website that uses React or JavaScript?
Not with wget or HTTrack: they fetch the server's HTML, which on those sites is an empty shell. Crawl with a real browser and capture the rendered DOM instead.
How do I stop a crawl from downloading the whole internet?
Set a depth limit, restrict the allowed domains, reject query strings and calendar or search paths, and set a hard page cap. Do all four; any one of them alone can be defeated by a site's URL structure.
Is downloading a whole website legal?
Copying published pages for archival, offline reading, migration or evidence is normal. Republishing the content, ignoring terms of service you agreed to, or crawling aggressively enough to affect the site are the ways it becomes a problem.
How big will the download be?
Roughly the transfer size of every page you visit plus their assets, minus caching. Media-heavy sites reach gigabytes quickly, so cap the page count and consider excluding video.
Try it on the page you are looking at
CopyAnySite is a Chrome extension: open any page you can reach, press capture, and take away a standalone HTML file, an editable WordPress page, or a kit a coding agent can build from. The free key needs no card.
Download CopyAnySite