Draft Disclaimer: Please note that this article is currently in draft form and may undergo revisions before final publication. The content, including information, opinions, and recommendations, is subject to change and may not represent the final version. We appreciate your understanding and patience as we work to refine and improve the quality of this article. Your feedback is valuable in shaping the final release.
Language Mismatch Disclaimer: Please be aware that the language of this article may not match the language settings of your browser or device.
Do you want to read articles in English instead ?
Robots.txt
What is it ?
- tell the crawlers which website not index
User-agent: <agent> Disallow: <folder or file name>
Be aware that some bots can deliberately choose not to follow your instructions
As helper for hackers points of interest
Since the robots.txt contains by definition folders and files that you do not want to be indexed. That's exactly the reason why it is interesting for