Yes, robots can keep your website from being indexed in the top search engines like Google and Yahoo… no, this is not a movie idea! Don’t worry, you can fix it, there is an easy solution. …Just to clarify first though, we are talking about robots – as in a robotx.txt file – in the root of your website, i.e. www.your-website.com/robots.txt .
If you want your website to be indexed – all of your webpages – make sure you don’t have a robots.txt file uploaded to your server – the easiest solution is to just delete the file if it exists in the “httpdocs” or “www” folder on your server (i.e. the root). The robots.txt file tells search engine robots what pages not to spider and index. Respectable search engines like Google and Yahoo will honor your request and not index those webpages on their search engine. In fact, if you have this in your robots.txt file, the search engines will not list any of your webpages:
User-agent: *
Disallow: /
“User-agent” field is for the list of search engine names (separated by commas) – “*” denotes all search engines. “Disallow” is for the list of webpages not to index (separated by commas), “/” denoting your whole website.
Note that the robots.txt file doesn’t force search engines to comply with your request, and also that your website robots.txt file is viewable to the public in the internet browser. So, it is best to use other methods to restrict access to your website or certain webpages of your website – our favorite is the members app and page access tool in Ultimate Web Builder software! We use php to do the behind-the-scenes work for you.
Copyright ©