Secret Casino Club
Gambling Home
Webmaster pages
SEO resources
-Robots.txt
-Sitemaps
--Sitemap generator
--Add to Google
-Meta codes
--Meta generator
-Articles
Affiliate programs
Traffic exchanges
Privacy Policy

Set as home-page Add to favorites Open in new window Recommend this page
iPowerWeb
Webmaster resources - Robots.txt - Search Engine Optimization ( SEO )
  Page topics
  Why do I need a robots.txt file
What is a simple robots.txt file
How do i make a robots.txt file
What should i do with my robots.txt file
Why do I need a robots.txt file

Creating a robots.txt file will not always improve your search engine positioning. The robots.txt file provide robots with information concerning files you don't want the search engines to crawl and index in the search engines.

The robots.txt file also indicates how often you want the search engines to "re-index" your website. This is very important if you have dynamic content ( content that often changes on the same page via php,asp, database use etc ).

When a search engine robot visits your site it starts by checking for the robots.txt file. If it can't find one, it automatically assumes that it may crawl and index the entire site.

Not having a robots.txt file can create unnecessary 404 errors in your server logs, making it more difficult to track "real" 404 errors.

Not having a robots.txt file can seriously increase your bandwidth useage because the search engine robots will crawl and index everything, including images and link to these.

If you want your entire site indexed and only want to stop the unnecessary 404 errors from occurring you can simply...

What is a simple robots.txt file

Please note: Disallowing a specific file to be crawled will keep it from being indexed. The file disallowed will not show up in the search engines. HOWEVER, this is only effective for friendly robots. Robots can choose to ignore your instructions.

These lines in a robots.txt file allows all robots to crawl all files except the files in the /images/ folder.

User-agent: *
Disallow: /images/

These lines in a robots.txt file allows all robots to crawl all files except the files in the "images" folder and the "stats" folder

User-agent: *
Disallow: /images/
Disallow: /stats/


How do i make a robots.txt file
Simply create a text document and save the new document as robots.txt Do not use a html editor to create the file unless is has the ability to create a plain text document (ASCII).
Most computers will allow you to create a text document using notepad.

  • Right click on your desktop

  • Choose new

  • Choose text document

  • Open the document you just created

  • Insert instructions to robots ( copy and paste from our website if you want )

  • Click on save as

  • Save document as robots.txt


What should i do with my robots.txt file
The robots.txt file should be places in the root folder of your website.
Our robots.txt file is located here: http://www.SecretCasinoClub.dk/robots.txt
AddThis Social Bookmark Button
Copyright © 2006, SecretCasinoClub.dk, All rights reserved

- Sitemap -