best way to replace or configure robots.txt to prevent indexing.

questorfla · 15 Jul 2020

System is running Windows 10 and acting as an Apache website host.
We have always had good results with using a robots.txt file consisting of only a few brief statements;
The two lines of > "User-agent: *" and "Disallow: /"

Lately we have noticed occasional reports of some documents turning up in various google cache searches with the contents attributed to the exact folder they are in inside the private website.

If I wanted to prevent ALL indexing or scanning by ALL robots, what is the best way to accomplish this? These sites are meant to be accessed only by the people who have the link to them. Most contain private content that should never be attributed to anyone. I would prefer a method of keeping all indexing or scanning bots out of the sites.

Anti-advertising if you will?
Robots.txt doesn't seem to even stop Google-bots anymore

Ztruker · 15 Jul 2020

Good question. That's all I have in my robots.text since 2001. In my case I think running my web server on a non-standard port is the main reason it never gets scanned. I don't use or support port 80.

questorfla · 19 Sep 2020

Ztruker:
i have always valued your advice. I have never tried running it on anything OTHER than the normal 80.
What port range do you suggest? Is there any disadvantage to using an alternate port in that manner?

Ztruker · 19 Sep 2020

I moved mine up above 20000, primarily because my ISP at the time did not allow the use of port 80. They did not like people running Web Servers for some reason. I don't think that's the case any more but my family is used to the port I use and it would be confusing if I changed it. Besides, it gets me away from a lot of the scanners.

To acces s it you need to include the port address in the URL, i.e. http:\\192.168.0.92:20955 (using IP instead of a name).