+1
日增长
5,970
stars
820
forks
126
watchers
44
prs
276
issues
8
releases
README
Wayback Machine Downloader
Download an entire website from the Internet Archive Wayback Machine.
Installation
You need to install Ruby on your system (= 1.9.2) - if you don't already have it.
Then run:
gem install wayback_machine_downloader
Tip: If you run into permission errors, you might have to add sudo in front of this command.
Basic Usage
Run wayback_machine_downloader with the base url of the website you want to retrieve as a parameter (e.g., http://example.com):
wayback_machine_downloader http://example.com
How it works
It will download the last version of every file present on Wayback Machine to ./websites/example.com/. It will also re-create a directory structure and auto-create index.html pages to work seamlessly with Apache and Nginx. All files downloaded are the original ones and not Wayback Machine rewritten versions. This way, URLs and links structure are the same as before.
Advanced Usage
Usage: wayback_machine_downloader http://example.com
Download an entire website from the Wayback Machine.
Optional options:
-d, --directory PATH Directory to save the downloaded files into
Default is ./websites/ plus the domain name
-s, --all-timestamps Download all snapshots/timestamps for a given website
-f, --from TIMESTAMP Only files on or after timestamp supplied (ie. 20060716231334)
-t, --to TIMESTAMP Only files on or before timestamp supplied (ie. 20100916231334)
-e, --exact-url Download only the url provided and not the full site
-o, --only ONLY_FILTER Restrict downloading to urls that match this filter
(use // notation for the filter to be treated as a regex)
-x, --exclude EXCLUDE_FILTER Skip downloading of urls that match this filter
(use // notation for the filter to be treated as a regex)
-a, --all Expand downloading to error files (40x and 50x) and redirections (30x)
-c, --concurrency NUMBER Number of multiple files to download at a time
Default is one file at a time (ie. 20)
-p, --maximum-snapshot NUMBER Maximum snapshot pages to consider (Default is 100)
Count an average of 150,000 snapshots per page
-l, --list Only list file urls in a JSON format with the archived timestamps, won't download anything
Specify directory to save files to
-d, --directory PATH
Optional. By default, Wayback Machine Downloader will download files to ./websites/ followed by the domain name of the website. You may want to save files in a specific directory using this option.
Example:
wayback_machine_downloader http://example.com --directory downloaded-backup/
All Timestamps
-s, --all-timestamps
Optional. This option will download all timestamps/snapshots for a given website. It will uses the timestamp of each snapshot as directory.
Example:
wayback_macTop issues
#307Need Help: Getting this Error: `open_http': 400 BAD REQUEST42 comments · 2024-10-03#291Help please! port 44328 comments · 2024-05-13#292INSTRUCTIONS! Use this working version instead.19 comments · 2024-05-25#273Permission denied - connect(2) for "web.archive.org" port 44317 comments · 2024-01-02#182400 Bad Request?17 comments · 2021-02-05#207only downloads an index.html file rather than entire site15 comments · 2021-09-30#312Failed to dl my old site :/14 comments · 2024-11-27#281DO NOT USE unless you have a means of rate limiting yourself12 comments · 2024-03-07
Latest release
2.3.1 2.3.1 2021-09-042.3.1 Version