best way to replace or configure robots.txt to prevent indexing.


  1. Posts : 228
    Win 10 Ver 1903
       #1

    best way to replace or configure robots.txt to prevent indexing.


    System is running Windows 10 and acting as an Apache website host.
    We have always had good results with using a robots.txt file consisting of only a few brief statements;
    The two lines of > "User-agent: *" and "Disallow: /"

    Lately we have noticed occasional reports of some documents turning up in various google cache searches with the contents attributed to the exact folder they are in inside the private website.

    If I wanted to prevent ALL indexing or scanning by ALL robots, what is the best way to accomplish this? These sites are meant to be accessed only by the people who have the link to them. Most contain private content that should never be attributed to anyone. I would prefer a method of keeping all indexing or scanning bots out of the sites.

    Anti-advertising if you will?
    Robots.txt doesn't seem to even stop Google-bots anymore
      My Computer


  2. Posts : 13,960
    Windows 10 Pro X64 21H1 19043.1503
       #2

    Good question. That's all I have in my robots.text since 2001. In my case I think running my web server on a non-standard port is the main reason it never gets scanned. I don't use or support port 80.
      My Computers


  3. Posts : 228
    Win 10 Ver 1903
    Thread Starter
       #3

    Ztruker:
    i have always valued your advice. I have never tried running it on anything OTHER than the normal 80.
    What port range do you suggest? Is there any disadvantage to using an alternate port in that manner?
      My Computer


  4. Posts : 13,960
    Windows 10 Pro X64 21H1 19043.1503
       #4

    I moved mine up above 20000, primarily because my ISP at the time did not allow the use of port 80. They did not like people running Web Servers for some reason. I don't think that's the case any more but my family is used to the port I use and it would be confusing if I changed it. Besides, it gets me away from a lot of the scanners.

    To acces s it you need to include the port address in the URL, i.e. http:\\192.168.0.92:20955 (using IP instead of a name).
      My Computers


 

  Related Discussions
Our Sites
Site Links
About Us
Windows 10 Forums is an independent web site and has not been authorized, sponsored, or otherwise approved by Microsoft Corporation. "Windows 10" and related materials are trademarks of Microsoft Corp.

Designer Media Ltd
All times are GMT -5. The time now is 17:45.
Find Us




Windows 10 Forums