Skip to content

Latest commit

 

History

History
71 lines (43 loc) · 1.59 KB

README.md

File metadata and controls

71 lines (43 loc) · 1.59 KB

RobotsForce1

Define the crawl zones. Control the AO.

RobotsForce1 – The presidential package of robots.txt generators. A joint task force of elite directives, securing the perimeter against unauthorized bots. Precision, control, and absolute authority over your domain.


npm version

License

Features

  • Elegant API - Simple, direct, and fluent.

  • Full Compliance - Supports the entire standard.

  • Zero dependencies - atomic, minimal, unopinionated.



Installation

npm  install  robotsforce1



Usage

import { RobotsTxt } from 'robotsforce1';

const robots = new RobotsTxt();
robots.agent('*').allow('/').disallow('/private');
robots.agent('Googlebot').allow('/').disallow('/sensitive').delay(5);
robots.sitemap('https://example.com/sitemap.xml');
console.log(robots.toRobotsTxt());

/**
User-agent: *
Allow: /
Disallow: /private

User-agent: Googlebot
Allow: /
Disallow: /sensitive
Crawl-delay: 5

Sitemap: https://example.com/sitemap.xml
*/



Need sitemap.xml?

I provide a package for sitemap.xml as well. If you use these together your project will be backed by some of the most elite webstandard writers in the world—if not the most elite.

https://github.com/alexstevovich/sitemapteam6



License

Licensed under the Apache License 2.0.