Skip to content

Latest commit

 

History

History
141 lines (115 loc) · 5.62 KB

README.rst

File metadata and controls

141 lines (115 loc) · 5.62 KB
Raw

warcprox - WARC writing MITM HTTP/S proxy

https://travis-ci.org/internetarchive/warcprox.png?branch=master

Based on the excellent and simple pymiproxy by Nadeem Douba. https://github.com/allfro/pymiproxy

License: because pymiproxy is GPL and warcprox is a derivative work of pymiproxy, warcprox is also GPL.

Install

Warcprox runs on python 3.5.

To install latest release run:

pip install warcprox

You can also install the latest bleeding edge code:

pip install git+https://github.com/internetarchive/warcprox.git

Trusting the CA cert

For best results while browsing through warcprox, you need to add the CA cert as a trusted cert in your browser. If you don't do that, you will get the warning when you visit each new site. But worse, any embedded https content on a different server will simply fail to load, because the browser will reject the certificate without telling you.

Usage

usage: warcprox [-h] [-p PORT] [-b ADDRESS] [-c CACERT]
                [--certs-dir CERTS_DIR] [-d DIRECTORY] [-z] [-n PREFIX]
                [-s SIZE] [--rollover-idle-time ROLLOVER_IDLE_TIME]
                [-g DIGEST_ALGORITHM] [--base32] [-j DEDUP_DB_FILE]
                [-P PLAYBACK_PORT]
                [--playback-index-db-file PLAYBACK_INDEX_DB_FILE] [--version]
                [-v] [-q]

warcprox - WARC writing MITM HTTP/S proxy

optional arguments:
  -h, --help            show this help message and exit
  -p PORT, --port PORT  port to listen on (default: 8000)
  -b ADDRESS, --address ADDRESS
                        address to listen on (default: localhost)
  -c CACERT, --cacert CACERT
                        CA certificate file; if file does not exist, it will
                        be created (default: ./desktop-nlevitt-warcprox-
                        ca.pem)
  --certs-dir CERTS_DIR
                        where to store and load generated certificates
                        (default: ./desktop-nlevitt-warcprox-ca)
  -d DIRECTORY, --dir DIRECTORY
                        where to write warcs (default: ./warcs)
  -z, --gzip            write gzip-compressed warc records (default: False)
  -n PREFIX, --prefix PREFIX
                        WARC filename prefix (default: WARCPROX)
  -s SIZE, --size SIZE  WARC file rollover size threshold in bytes (default:
                        1000000000)
  --rollover-idle-time ROLLOVER_IDLE_TIME
                        WARC file rollover idle time threshold in seconds (so
                        that Friday's last open WARC doesn't sit there all
                        weekend waiting for more data) (default: None)
  -g DIGEST_ALGORITHM, --digest-algorithm DIGEST_ALGORITHM
                        digest algorithm, one of sha384, sha512, md5, sha224,
                        sha256, sha1 (default: sha1)
  --base32              write digests in Base32 instead of hex (default:
                        False)
  -j DEDUP_DB_FILE, --dedup-db-file DEDUP_DB_FILE
                        persistent deduplication database file; empty string
                        or /dev/null disables deduplication (default:
                        ./warcprox-dedup.db)
  -P PLAYBACK_PORT, --playback-port PLAYBACK_PORT
                        port to listen on for instant playback (default: None)
  --playback-index-db-file PLAYBACK_INDEX_DB_FILE
                        playback index database file (only used if --playback-
                        port is specified) (default: ./warcprox-playback-
                        index.db)
  --version             show program's version number and exit
  -v, --verbose
  -q, --quiet

To do

  • (partly done) integration tests, unit tests
  • (done) url-agnostic deduplication
  • unchunk and/or ungzip before storing payload, or alter request to discourage server from chunking/gzipping
  • check certs from proxied website, like browser does, and present browser-like warning if appropriate
  • keep statistics, produce reports
  • write cdx while crawling?
  • performance testing
  • (done) base32 sha1 like heritrix?
  • configurable timeouts and stuff
  • evaluate ipv6 support
  • (done) more explicit handling of connection closed exception during transfer
  • dns cache?? the system already does a fine job I'm thinking
  • keepalive with remote servers?
  • (done) python3
  • special handling for 304 not-modified (write nothing or write revisit record... and/or modify request so server never responds with 304)
  • (done) instant playback on a second proxy port
  • special url for downloading ca cert e.g. http(s)://warcprox./ca.pem
  • special url for other stuff, some status info or something?
  • browser plugin for warcprox mode
    • accept warcprox CA cert only when in warcprox mode
    • separate temporary cookie store, like incognito
    • "careful! your activity is being archived" banner
    • easy switch between archiving and instant playback proxy port

To not do

The features below could also be part of warcprox. But maybe they don't belong here, since this is a proxy, not a crawler/robot. It can be used by a human with a browser, or by something automated, i.e. a robot. My feeling is that it's more appropriate to implement these in the robot.