Squidwarc by N0taN3rd

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head

created at July 20, 2017, 6:57 a.m.

10 +0

164 +0

26 +0

browsertrix by webrecorder

Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!

created at June 28, 2021, 10:46 p.m.

10 +0

131 +1

27 +0

awesome-website-change-monitoring by edgi-govdata-archiving

A curated list of awesome tools for website diffing and change monitoring.

created at May 24, 2017, 5:33 a.m.

31 +0

482 +0

31 +0

wail by machawk1

Web Archiving Integration Layer: One-Click User Instigated Preservation

created at March 20, 2013, 2:42 p.m.

14 +0

345 +0

32 +0

waybackpy by akamhy

Wayback Machine API interface & a command-line tool

created at May 2, 2020, 9:19 a.m.

10 +0

441 +3

33 +0

aut by archivesunleashed

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

created at July 6, 2017, 10:13 a.m.

15 +0

133 +0

33 +0

ipwb by oduwsdl

InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

created at March 4, 2016, 3:01 p.m.

23 +0

591 +1

39 +0

archivenow by oduwsdl

A Tool To Push Web Resources Into Web Archives

created at Feb. 9, 2017, 12:29 p.m.

21 +0

392 +0

41 +0

auto-archiver by bellingcat

Automatically archive links to videos, images, and social media content from Google Sheets (and more).

created at Jan. 15, 2021, 10:30 a.m.

19 +0

478 +1

53 +0

warcprox by internetarchive

WARC writing MITM HTTP/S proxy

created at Oct. 25, 2013, 11:27 p.m.

33 +0

366 +1

54 -1

warcio by webrecorder

Streaming WARC/ARC library for fast web archive IO

created at March 6, 2017, 6:17 p.m.

22 +0

349 +0

55 +1

wayback by wabarc

An archiving tool with an IM-style interface that prioritizes privacy and accessibility, integrated with various archival services including Internet Archive, archive.today, IPFS, Telegraph, and file systems.

created at June 13, 2020, 10:08 a.m.

9 +0

1,667 +5

61 +0

browsertrix-crawler by webrecorder

Run a high-fidelity browser-based crawler in a single Docker container

created at Nov. 2, 2020, 4:37 a.m.

24 +0

557 +5

72 +0

wpull by ArchiveTeam

Wget-compatible web downloader and crawler.

created at Dec. 7, 2013, 1:03 p.m.

23 +0

538 +2

77 +0

brozzler by internetarchive

brozzler - distributed browser-based web crawler

created at July 13, 2015, 11:48 p.m.

36 +0

630 +0

93 +0

grab-site by ArchiveTeam

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

created at Feb. 5, 2015, 5:01 a.m.

40 +0

1,273 +1

125 +0

DownloadNet by dosyago

💾 DownloadNet - All content you browse online available offline. Search through the full-text of all pages in your browser history. ⭐️ Star to support our work!

created at Dec. 20, 2019, 9:47 a.m.

42 +0

3,662 +5

137 +0

wikiteam by WikiTeam

Tools for downloading and preserving wikis. We archive wikis, from Wikipedia to tiniest wikis. As of 2023, WikiTeam has preserved more than 350,000 wikis.

created at June 25, 2014, 10:18 a.m.

40 +0

696 +3

145 +1

pywb by webrecorder

Core Python Web Archiving Toolkit for replay and recording of web archives

created at Dec. 9, 2013, 3:30 a.m.

61 +0

1,317 +4

206 -1

internetarchive by jjjake

A Python and Command-Line Interface to Archive.org

created at Aug. 15, 2012, 7:18 p.m.

51 +0

1,539 +6

211 +1