Hi, I am looking for a way to detect on our NGINX server which traffic is coming from the crawl.
I thought that there was a config to personalize the User-agent that the crawler used but have not found it or any doc regarding if it uses an specific one.
Is it using an specific one when crawling? Any other specific requested header I could use?
Thanks.
Guest