Skip to content

Version 1.0.1
Compare
Choose a tag to compare
@JanPetterMG JanPetterMG released this 05 Apr 14:43
· 39 commits to master since this release
v1.0.1
40ac4dd
  • Improved robots.txt parser
  • Improved parsing of line separated text strings
  • New configuration option: strict (bool), enabled by default.

Strict standards: parse valid standardized content only.
Ex. when expecting XML, but an plain text URL list is returned instead.

Assets 2
Loading