There is still nothing on this page, and now if we look at the source, they’ve removed that image. Now there’s a taunt that not even Google will find it this time.
Well, Google is a good little robot puppy who always obeys his master, but we are evil attackers. What rules do robots have to follow that we don’t? robots.txt. Let’s look at what they don’t want Google to see. robots.txt always lives at the root of the domain, so open http://natas3.natas.labs.overthewire.org/robots.txt.
They’ve hidden /s3cr3t/ from crawlers, but we don’t have to abide by that. Let’s take a look. Another directory listing, which contains a file which contains our password.
Lessons learned
Remember that hiding things from search engines is very different from making them inaccessible. Even some web crawlers won’t respect your robots.txt file, and attackers certainly won’t. If you need to make something inaccessible, configure that in your webserver, not in robots.txt.
See you soon for level 4!