Code

Dashboard#238

Code

Open

drzo wants to merge 49 commits intoArchiveTeam:dashboardArchiveTeam/grab-site:dashboardfrom drzo:dashboarddrzo/grab-site:dashboardCopy head branch name to clipboard

Commits

Commits on Jul 18, 2015

• Explain how to stop a crawl

  ivan

  committed
  47940fd

  Copy full SHA for 47940fd

• My* -> Grabber*

  ivan

  committed
  e42e33d

  Copy full SHA for e42e33d

• Spaces -> tabs

  ivan

  committed
  2222aaf

  Copy full SHA for 2222aaf

• Dup -> Dupe

  ivan

  committed
  d3715fe

  Copy full SHA for d3715fe

• chmod +x

  ivan

  committed
  d5d2d49

  Copy full SHA for d5d2d49

• Report concurrency level

  ivan

  committed
  5da054a

  Copy full SHA for 5da054a

• Add dashboard screenshot

  ivan

  committed
  bec8615

  Copy full SHA for bec8615

• Link to raw.githubusercontent.com for screenshot

  ivan

  committed
  b1d5f67

  Copy full SHA for b1d5f67

• Tweak README

  ivan

  committed
  210baaa

  Copy full SHA for 210baaa

• Use --level inf by default; add --level option

  ivan

  committed
  f848b28

  Copy full SHA for f848b28

• Require 400MB disk free

  ivan

  committed
  5331f4c

  Copy full SHA for 5331f4c

• Add --page-requisites-level= and --concurrency= options; use default concurrency of 2

  ivan

  committed
  63e8f18

  Copy full SHA for 63e8f18

• Set --max-redirect 8 like ArchiveBot

  ivan

  committed
  318fb3c

  Copy full SHA for 318fb3c

• Move some code around

  ivan

  committed
  877b170

  Copy full SHA for 877b170

• Recommend starting gs-server first

  ivan

  committed
  9f872f4

  Copy full SHA for 9f872f4

• Tweak README

  ivan

  committed
  55e3507

  Copy full SHA for 55e3507

• Don't assume WebSocket clients are dashboards by default; announce user agents

  ivan

  committed
  6a866ad

  Copy full SHA for 6a866ad

• Don't allow setting mode more than once

  ivan

  committed
  fe659e2

  Copy full SHA for fe659e2

• Allow only grabbers to announce download/stdout/stderr/ignore

  ivan

  committed
  2270523

  Copy full SHA for 2270523

• Enable faulthandler

  ivan

  committed
  a66b970

  Copy full SHA for a66b970

Commits on Jul 19, 2015

• Tell people to use GitHub issues

  ivan

  committed
  3ffed7d

  Copy full SHA for 3ffed7d

• Camelcase

  ivan

  committed
  8e2e1c5

  Copy full SHA for 8e2e1c5

• Mark finished jobs as finished on dashboard

  ivan

  committed
  7566de0

  Copy full SHA for 7566de0

• Bump version

  ivan

  committed
  a5cc1d8

  Copy full SHA for a5cc1d8

• Tell user where the output files are

  ivan

  committed
  a95ee28

  Copy full SHA for a95ee28

• Accept more exit codes from wpull as clean exit

  ivan

  committed
  6c6c319

  Copy full SHA for 6c6c319

Commits on Jul 20, 2015

• Keeping your crawling problems in perspective

  Show description for cd893cb
  ivan

  committed
  cd893cb

  Copy full SHA for cd893cb

• Add OS X support

  ivan

  committed
  0e38441

  Copy full SHA for 0e38441

• Document how to fix your PATH for grab-site

  ivan

  committed
  c7a272d

  Copy full SHA for c7a272d

• README: include suggestions from @ethus3h (thanks!) and wrap long lines

  ivan

  committed
  210c3d0

  Copy full SHA for 210c3d0

• README: there are control files in DIR too

  ivan

  committed
  e833753

  Copy full SHA for e833753

• Add --1 option for turning off recursion; document options

  ivan

  committed
  1fce3af

  Copy full SHA for 1fce3af

• Unbreak README

  ivan

  committed
  9af02f1

  Copy full SHA for 9af02f1

• README: improve docs for options

  ivan

  committed
  58b5602

  Copy full SHA for 58b5602

• README: use an <h3>

  ivan

  committed
  3f78e5f

  Copy full SHA for 3f78e5f

• Bump version

  ivan

  committed
  35d6d78

  Copy full SHA for 35d6d78

• Clarify ?host= dashboard option

  ivan

  committed
  08933f6

  Copy full SHA for 08933f6

• Allow archiving archive.org content despite it being in the global ignore set

  ivan

  committed
  c35b388

  Copy full SHA for c35b388

• README: document --concurrency=

  ivan

  committed
  5e85e00

  Copy full SHA for 5e85e00

• Document webarchiveplayer for viewing your WARCs

  ivan

  committed
  a7f2ee7

  Copy full SHA for a7f2ee7

• README: minor tweaks

  ivan

  committed
  8d2acd6

  Copy full SHA for 8d2acd6

Commits on Jul 21, 2015

• Use cchardet for faster encoding detection (imported by wpull/thirdparty/dammit.py)

  ivan

  committed
  915ed0e

  Copy full SHA for 915ed0e

Commits on Jul 27, 2015

• First take on converting grab-site to a Python program

  ivan

  committed
  637929a

  Copy full SHA for 637929a

• Write proper --help text and use aliased inputs too

  ivan

  committed
  84b183e

  Copy full SHA for 84b183e

• Update README

  ivan

  committed
  2e7d928

  Copy full SHA for 2e7d928

• Add --sitemaps/--no-sitemaps

  ivan

  committed
  b7c2f1d

  Copy full SHA for b7c2f1d

• Make --igsets actually work

  ivan

  committed
  4c9a935

  Copy full SHA for 4c9a935

• Add --version

  ivan

  committed
  99dfbe2

  Copy full SHA for 99dfbe2

• Put all temporary files in DIR/temp; don't let ctrl-c exit grab-site before wpull

  ivan

  committed
  472edf5

  Copy full SHA for 472edf5