New
#1
best way to replace or configure robots.txt to prevent indexing.
System is running Windows 10 and acting as an Apache website host.
We have always had good results with using a robots.txt file consisting of only a few brief statements;
The two lines of > "User-agent: *" and "Disallow: /"
Lately we have noticed occasional reports of some documents turning up in various google cache searches with the contents attributed to the exact folder they are in inside the private website.
If I wanted to prevent ALL indexing or scanning by ALL robots, what is the best way to accomplish this? These sites are meant to be accessed only by the people who have the link to them. Most contain private content that should never be attributed to anyone. I would prefer a method of keeping all indexing or scanning bots out of the sites.
Anti-advertising if you will?
Robots.txt doesn't seem to even stop Google-bots anymore