Disable certain user agents

[cuvelierm]

I have been running prerender-node for a while and would like to keep using it for certain crawlers and not for others. Is an addition of non-checked crawler-agents an idea you are open to?

My use case is a React application which can be rendered perfectly fine by Google and their bots, yet most others of which facebook annoys me the most are not allowing javascript to be run and those still would use my prerender servers.

This is meant to reduce the load on those servers and allow people to 'not care' about certain search engines or enable it only for the remaining ones.

If there is some interest I am willing to create a pull request for this.

[thoop]

Sorry for the delay here! You can do that currently without any changes like this.

To override the entire list of user agents:

var prerender = require('prerender-node');
prerender.crawlerUserAgents = [ 'facebookexternalhit' ]; //add any other user agents you'd like to support
app.use(prerender);

To remove certain user agents:

var prerender = require('prerender-node');
prerender.crawlerUserAgents = prerender.crawlerUserAgents.filter(function(ua) { return ua !== 'googlebot'; });
app.use(prerender);

I don't think we need a function yet to remove a specific user agent from the list since you can modify that directly if needed.