What is Robots.txt and Why It’s Important from SEO Point of View?


Hello friends, if you have a website and wants search engine not crawl or index some file or directory from your website then with the help of robots.txt you can do it. As you understand from robots.txt file extension it’s a simple text document file which is used to instruct search engine how to crawl or index web pages of their website. You must upload this robots.txt file in main directory (root) of hosting server otherwise search engine not able to search and read this file.

Syntax of Robots.txt: Basic structure of robots.txt file is endless combination of “user-agent” & “disallow” file and directories



Here “User-agent” stands for search engine crawlers/bots or “Disallow:” used to include file name and path of directories which you want to exclude from indexing.

There different number of combination for robots.txt to give different command to search engines which are as follow:

Block all search engine crawlers/bots from whole website:

User-agent: *

Disallow: /

Block a particular search engine crawlers/bots from particular directory/folder:

User-agent: Googlebot

Disallow: /abc/

Block a particular search engine crawlers/bots from particular file or web page:

User-agent: Googlebot

Disallow: /abc/xyz.html

Sitemap parameter:

User-agent: *


Sitemap: http://www.example.com/none-standard-location/sitemap.xml

You May Also Like

About the Author: SEOTutorialsPoint

Hello friends, I am Parvinder Singh having more than 5+ years of experience in digital marketing field and at SEOTutorialsPoint (STP) I would like to express my whole experience to make digital marketing strategies understandable in an easy manner. To get latest digital marketing tips keep reading my upcoming blogs and also follow me on facebook, twitter & google plus.

1 Comment

  1. Spin Rewriter 8.0 completely changed my thoughts about article spinning software.
    I have tried virtually every one that’s come out there (badly) and wound
    up just rewriting articles myself since
    every item basically made me do this anyway. So when I attempted Spin Rewriter 8.0,
    I was somewhat hesitant as I’m used to each product always having a massive
    hype with a lot of testimonials from people I’ve never heard about.
    With SR 4, my head has been changed, and I had been stunned
    at how easy the software was to work with, but it really focused on appropriate language and grammar.
    You see online, it’s obvious that it’s spun as it reads like something automated and off the
    railings wrote it. With Spin Rewriter 8.0, it genuinely is able to sound real with quality writing.
    You need to put a little effort into it for the best results, of course,
    but unlike all the others it’s worth that little bit of effort.
    I can not wait to see what changes are made for 8.0!

    I Enjoy EVERYTHING!!!! You understand that content is a cornerstone even if you’re
    new to advertising. We use to spend tens of thousands of dollars
    having content developed. Utilizing Spin Rewriter 8.0 has cut our costs by
    90%. The best part is that we did not have to sacrifice quality in order
    to raise production.

    Spin Rewriter 8.0 simply keeps getting better. A lot of spinners churn out
    crappy unreadable rubbish that takes ages to sort out, but Spin Rewriter 8.0 has a quality factor it is possible to pick dependant on the purpose
    that you want the post for. I am impressed that very little work
    has to be done on the spun article that was final.

    Spin Rewriter 8.0 is different because its ENL Semantic Spinning technology enables it
    analyze the true meaning of your articles.

    The problem is spinners don’t understand.

    Since they WORK enjoy a 2-year-old using a thesaurus what
    they create reads like something you’d receive using a thesaurus from a 2-year-old.

    They do not know that a word like ‘book’ can mean either ‘novel’ and ‘make a reservation’.

    That is the reason why the posts they create do not make any sense.

    Spin Rewriter 8.0 Delivers An Entirely

    New Level of Language Understanding

    Spin Rewriter 8.0’s got a huge — and growing —
    table of synonyms.

    But unlike many spinners, it moves further.

    It looks at the whole article. It does not only look words up,
    but investigations the meaning to be understood by them.

    So Spin Rewriter 8.0 understands what you are trying to say, and
    certainly will rewrite it intelligently.

    It is Named Emulated Natural Language.

    And it’s how Spin Rewriter 8.0 can deliver countless
    human-quality articles time and time again.

    Just imagine being able to produce ten times the amount of posts you can.

    Imagine having the ability to conduct more blogs.

    Imagine just how much more traffic you can drive and how many terms
    you are rank for.

    Imagine being able to take off the brakes. Being able to grow your company to the powerhouse you’ve always
    wanted it to be.

    Spin Rewriter 8.0 has saved me hundreds, or even thousands of hours.
    I have had articles and articles that needed printed and
    to be pushed to thousands of places. With Spin Rewriter 8.0, with a couple clicks, I had been given tens of thousands
    of special articles in a couple of minutes. The time saved by using Spin Rewriter 8.0 was invaluable.

    It Is Really Safe To Use An Article Spinner Like Spin Rewriter 8.0

    Many internet marketers have a tendency to avoid using article spinners like Spin Rewriter 8.0.
    Search engine positions as they believe that doing so will somehow impede their sites.
    This simply is not.

    Previously article spinning tools use to just spin articles in a really basic fashion and the article wasn’t of great readable
    quality. Search engines was at weeding out junk much efficient although complex as
    though they are today.

    The search engine algorithms completely read the text, videos, even photographs contained
    on a website and are more intelligent. Google wants the
    best results with the content relevance to the search for by a user.

    Thus providing quality content is key to gaining trust as a power site on any subject with all
    the search engines. This is but one of the chief variables Google considers when wanting
    to rank site higher.

    Report Spinners Like Spin Rewriter 8.0 Have Moved With The Times

    Since the sport has changed with respect to how sites are ranked,
    the programmers of article spinners like people at Spin Rewriter 8.0.
    Have had to produce intelligent parts of software that produce articles which are unique and
    fully readable .

    Spin Rewriter 8.0 for example has a highly developed algorithms that spins articles at word, sentence and paragraph
    level. Using a huge thesaurus database various synonyms
    are used by that the software in an assortment of combinations.

    You can spin 1 base article of 500 hundred words
    into a distinctive and human readable article within two seconds.
    With this capacity allows you to get different versions of an article describing the subject.

    So you can use these articles to bulk up the content of your business website.

    You might apply these articles to give quality backlinks.

    Platforms using a link back to your site just distributing these articles to on social media.
    Will go a long way into help your site look in the search engines.

    Using Article Spinners Won’t Make Your Website Penalised

    Don’t listen to these myths when it comes to utilizing article spinners.

    Algorithms that were new have adapted and advanced to the search engines.

    So if you’re searching for a economical and time efficient way to
    make quality articles for your website. The I would strongly recommend you with a post spinning service like Spin Rewriter 8.0.

    With this extremely powerful tool inside your internet advertising
    arsenal. You’ll be able to produce high content in your niche.
    And in the long term become an authority site that has been seen in a favourable light by the
    search engines.

    Spin Rewriter is THE tool that I use to generate hundreds of posts to my niche websites
    and it helps me to get grade link building. It may produce
    500 unique, replicate scape passed and human readable content with in 45
    minutes or less. It is cheaper than other post rewriters and operates on Tablet,
    Telephone, Mac and PC.

    In this deep dive review, I will be showing you the benefits that you
    will get together with the program. In addition, I’ll show you the
    procedure to make high quality 500 posts in 45 minutes
    or less.

    Internet marketing might be conceivably when attempting to build
    an income to your products or business, among the paths to follow.
    Particularly when you therefore are up along with companies and goods that have
    a strong and recognised existence on the most important
    search engines and are unfamiliar to the internet side.

    How several of these recognised brands have accomplished such a online presence is through generating thousands upon thousands of article content that has been posted over quite a
    few years to sites using a keyword improved one way link back
    to their website.

    Sure articles will require a number of backlinks
    aiming to this content to make any impression on those search engines and do not inevitably mean greater search engine.
    Nonetheless without content the juice having masses of proper niche content is essential to keep
    your brand and company fresh and hence from these backlinks would be going to waste.

    Spin Rewriter 8.0 Will Help Fortify Internet Marketing Efforts

    Creating a number of articles that are constant is a true inconvenience particularly when you don’t have a line-up
    of authors or bottomless pockets so that you can employ decent content creators.
    Even the content writing services which will deliver articles will put you back about $5 per article.

    These articles will build up and end up costing
    tens of thousands if we are talking about needing to create hundreds on a constant basis.
    This choice is rather unmanageable for minor businesses offering their products and
    services. Therefore a alternate have to be discovered!

    1 such choice is an article spinning service such as Spin Rewriter 8.0.
    You can find quite a couple of quality post spinners out there, but the thing that actually
    makes Spin Rewriter 8.0 unique is the fact that it’s entirely an internet based service that doesn’t require one to download and install software
    to utilize.

    Meaning you can generate any other apparatus that you can edit and produce content on and articles using your tablet smart phone, Windows or Apple desktop PC.
    Once you’ve created your niche article you can then easily
    log in to the Spin Rewriter 8.0 service and make
    tens of thousands of exclusive articles for your online promotions.

    So in what way will Spin Rewriter 8.0 significantly advance my internet advertising promotions?
    Behind utilizing article spinning tools such as Spin Rewriter the advantage is via swiftness and the
    effortlessness you can generate tens of thousands
    of posts from only one article that is solitary.

    What’s even more incredible is that this one article can be spun in word, sentence
    or paragraph amount using the potent Spin Rewriter 8.0 algorithm which will construct loads of articles which have over 90%
    uniqueness. So there’s not any actual risk of these search engines signaling that up the content as copying and penalize the search engine rankings of their articles.

    When you have these articles you can start the process of issuing to blogging and
    article platforms aiming back with niche proper text hyperlinks to your
    site. With any chance if your content is full of concrete content, folks will commence to obviously bookmark
    and distribute your articles around media platforms.

    Spin Rewriter 8.0 Is Your Cost Effective Article Spinning Service

    Utilising a service such as Spin Rewriter 8.0 will not cost as significant as you may consider,
    in fact you are able to use the terrific article spinning service for
    below $75 per year. That is less compared to the tens of thousands
    you would be committing to make precisely the quantity of content.

    If you’re fascinated in considerably advancing your internet advertising campaigns subsequently equipping yourself with a mighty tool such as Spin Rewriter 8.0
    is the only wise thing to do. Additionally you can use
    the Spin Rewriter service for 5 days entirely free so you can test out the item with no obligation to get a yearly subscription.

Leave a Reply

Your email address will not be published. Required fields are marked *