Added support for detecting bots through client-hints

[sanchezzzhak]

for matomo, you will need to add a new HTTP_X_CLIENT key
https://github.com/matomo-org/matomo/blob/d1eaaca1b7abdddea15ff8d1d8e2075b6f92c672/core/Http.php#L996-L1012

[!] this PR is worth viewing about when we have more bots through the xClient header.

[liviuconcioiu]

@sanchezzzhak I have something to add:

1. Can you also include the HTTP_FROM header? According to https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/From this one is a must for any crawler. So, if this header is present it should be detected as Generic bot for example.
2. Here are some HTTP_FROM values that should be added, to be detected as specific bots:

googlebot(at)googlebot.com
bingbot(at)microsoft.com
support@search.yandex.ru
the.knowledge.ai@gmail.com
crawler@alexa.com

[sanchezzzhak]

@liviuconcioiu Perhaps the implementation of HTTP_FROM should be added separately, since this PR is in limbo.
It would be good to have approximate statistics before implementing new features.

[sanchezzzhak]

ChatGPT support header HTTP_FROM;

From: gptbot(at)openai.com
User-Agent" Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)

[liviuconcioiu]

Here is a complete list of what I have so far:

bingbot(at)microsoft.com
googlebot(at)googlebot.com
gptbot(at)openai.com
robot@seokicks.de
support@search.yandex.ru
tech@babbar.tech
TGVnaXRpbWF0ZSBsaW5rIHRyYWNrZXI=
the.knowledge.ai@gmail.com
wc@verisign.com
crawler@alexa.com
pigafetta-bot(at)visual-seo.com
oai-searchbot@openai.com
"<?=print(9347655345-4954366);?>"
root@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.oast.site