Locust

Distributed web data discovery and collection framework

Quick Start

npm install @achannarasappa/locust

Features

• Configuration driven jobs
• Distributed execution model to support serverless architectures
• Handle client-side JavaScript execution
• Data extraction using CSS selectors
• Depth-based stop condition along with support for custom stop conditions
• Robust dev tooling with locust-cli to build and test jobs

Use Cases

• Web indexing (i.e. web crawling)
• Web data extraction (i.e. web scraping)

Reference

• Documentation
  • Quick start guide
  • API
  • CLI
• Examples
• Related
  • locust-cli
  • locust-aws-terraform
  • locust-website