hartator /wayback-machine-downloader

Download an entire website from the Wayback Machine.

★ 5,970⑂ 820👁 126⑃ 44● 276▤ 8
RubyNOASSERTIONCreated 2015-07-25
下载:2.3.12.3.1 ↓2.3.12021-09-04
+1
日增长
5,970
stars
820
forks
126
watchers
44
prs
276
issues
8
releases

README

Wayback Machine Downloader




Download an entire website from the Internet Archive Wayback Machine.

 Installation

You need to install Ruby on your system (= 1.9.2) - if you don't already have it.
Then run:

    gem install wayback_machine_downloader

Tip: If you run into permission errors, you might have to add sudo in front of this command.

 Basic Usage

Run wayback_machine_downloader with the base url of the website you want to retrieve as a parameter (e.g., http://example.com):

    wayback_machine_downloader http://example.com

 How it works

It will download the last version of every file present on Wayback Machine to ./websites/example.com/. It will also re-create a directory structure and auto-create index.html pages to work seamlessly with Apache and Nginx. All files downloaded are the original ones and not Wayback Machine rewritten versions. This way, URLs and links structure are the same as before.

 Advanced Usage

	Usage: wayback_machine_downloader http://example.com

	Download an entire website from the Wayback Machine.

	Optional options:
	    -d, --directory PATH             Directory to save the downloaded files into
					     Default is ./websites/ plus the domain name
	    -s, --all-timestamps             Download all snapshots/timestamps for a given website
	    -f, --from TIMESTAMP             Only files on or after timestamp supplied (ie. 20060716231334)
	    -t, --to TIMESTAMP               Only files on or before timestamp supplied (ie. 20100916231334)
	    -e, --exact-url                  Download only the url provided and not the full site
	    -o, --only ONLY_FILTER           Restrict downloading to urls that match this filter
					     (use // notation for the filter to be treated as a regex)
	    -x, --exclude EXCLUDE_FILTER     Skip downloading of urls that match this filter
					     (use // notation for the filter to be treated as a regex)
	    -a, --all                        Expand downloading to error files (40x and 50x) and redirections (30x)
	    -c, --concurrency NUMBER         Number of multiple files to download at a time
					     Default is one file at a time (ie. 20)
	    -p, --maximum-snapshot NUMBER    Maximum snapshot pages to consider (Default is 100)
					     Count an average of 150,000 snapshots per page
	    -l, --list                       Only list file urls in a JSON format with the archived timestamps, won't download anything
	    
 Specify directory to save files to

    -d, --directory PATH

Optional. By default, Wayback Machine Downloader will download files to ./websites/ followed by the domain name of the website. You may want to save files in a specific directory using this option.

Example:

    wayback_machine_downloader http://example.com --directory downloaded-backup/
    
 All Timestamps

    -s, --all-timestamps 

Optional. This option will download all timestamps/snapshots for a given website. It will uses the timestamp of each snapshot as directory.

Example:

    wayback_mac

Top issues

Latest release

2.3.1 2.3.1 2021-09-04
2.3.1 Version

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON