04-19-2014, 04:55 AM
The more often your site gets crawled, the sooner your new content
can start appearing in the search results, and bringing in traffic and
sales. So here's how, in just nine steps, you can keep Google coming
back for more.
The more often your site gets crawled, the more quickly
your content will appear in the search results. And that’s the point,
right? You wouldn’t create a new product then keep it in the shed, would
you? But the things you need to do to get your site crawled more
frequently are also best practice SEO and could even improve your
position in the search engines - getting even more of that money-making
traffic. All this you can do in just nine steps.
1. Create more content
Creating new content is good for a whole range of reasons. Users love
it, and so do search engines. Every time Google crawls your site, it
looks for stuff it hasn’t seen before. If it can’t find any it might
wait a little longer before visiting you again next time. But if it does
find new stuff, it’ll probably come back sooner (newspaper websites for
example, who update content almost constantly, get crawled several
times a day). Try to update your content as often as it makes sense to
do so. (Remember though: quality is better than quantity. So don’t just
create content for creating content’s sake.)
Yes Google likes seeing lots of content, but it doesn’t like seeing
the same content over and over. That’s just boring. So don’t duplicate
your content.
If there are legitimate reasons for you to duplicate your content (in
your heart of hearts, you know if you’re being naughty or nice), you
might want to deploy a redirect or a canonical tag In fact, if you have duplicate content you should. Whether you want to or not.
Just like you wouldn’t want to drive your car on a road covered in
melted tar, Google doesn’t like sending its spider to crawl slow-loading
sites. And users don’t like visiting them. Site speed is also becoming an increasingly important ranking factor so keep yours up to scratch.
4. You can also adjust the crawl frequency in Google Webmaster Tools.
Go into ‘Configuration’ then ‘Settings’, and you can go from there:
![[Image: webmaster-tools-settings.jpg]](http://www.wordtracker.com/attachments/webmaster-tools-settings.jpg)
This feature was designed so that webmasters could tell Google that it is crawling their site too aggressively, ie, it's overloading their servers. So if that’s happening, then by all means turn it down a notch here.
Some people however, use this feature to turn their crawl rate up. Personally, I wouldn’t do this because:
Google is pretty clever, and will usually work out an appropriate
frequency for crawling your site. I wouldn’t mess with it. I figure that
it knows what’s it’s doing more than I do, but if you really want to,
now you can. Hooray.
5. Add a sitemap
Google loves them. SEOs have been talking about them for ages. I‘ve
written an (amazing) article on how to create one. You have no excuse. Create a sitemap
6. Make sure your server is working as it should
The easiest way of checking this is to log into the Wordtracker's Keywords tool. Once you've logged in, just click on the campaigns you're working on, and the first thing you'll see will be the dashboard.
![[Image: wordtracker-server-crawls.png]](http://www.wordtracker.com/attachments/wordtracker-server-crawls.png)
Here, where I've put the arrow, you can see how many of your site's
pages have been crawled by Google and Bing. The higher this number, the
better. If the number is very low, there might be a problem. (But treat
it as a rough guide. There are lots of reasons why Google might not be
indexing your site. If only there was an article on crawling and indexing that explained this more fully? I jest, of course there is. It’s the previous link).
7. Check your server responses
Google wants your pages to be where it expects them to be. It doesn’t
like sites whose server keeps throwing back problems. Is your site like
this? Check your server responses by running a site audit in the Keywords tool
8. Get links
As well as improving your rank, bringing in more traffic and
generally being awesome, links potentially get your website crawled more
quickly. Why? Because when Google crawls the web, it will find
(hopefully) lots of links to your site littered throughout it. The more
times it sees other websites pointing to you with links, the harder it
is to stay away.
9. Know your crawl rate
Finally you should monitor your crawl rate. After all, that’s what
we’re talking about. Google probably won’t crawl your site all at once
at regular intervals: it’ll do a few pages one day, a few more the next.
But these are just details. You want Google to crawl as much of your
site as often as possible. So to find out, log into Webmaster Tools. In
the 'Health' section, click ‘Crawl stats’.
![[Image: webmaster-tools-crawl-stats.jpg]](http://www.wordtracker.com/attachments/webmaster-tools-crawl-stats.jpg)
And you’ll see a not-very-interesting graph that looks like this:
![[Image: crawl-stats-graph.jpg]](http://www.wordtracker.com/attachments/crawl-stats-graph.jpg)
Most of the time, it will look like this. It will go up a bit and it
will go down a bit. Don’t worry about individual movements of the line,
but do worry about trends.
If there’s a long term upward trend, that’s all good. Keep doing what
you’re doing. If there’s a long term downward trend, re-read this
article. Start at one, go down to seven, then lather, rinse, repeat as
needed. You’ll not turn the ship around overnight, but you will
eventually. We hope.
It’s also worth looking at spikes in the graph. Was there anything
you did on these days? Was there a Google algorithm change? Think about
the method behind Google’s madness and you’ll soon start understanding
how it actually works and what you can do to get in its good books.
Bonus tip! Use Google+
OK, so really this is point number 10, but whatever. 10 is such a overused number.
Your final top tip to get your site crawled more often is quite
simply, to share it on Google+, Google's social network. Remember, one
of Google's motivations for creating a social network was to get access
to real-time data. So, if you share your new content on Google+, it's
likely to get crawled super-fast. Just try it, and you'll see what I
mean. (And, if you're not sure how to do this, check out our Google+ guide
http://googelino.com
![[Image: banner.png]](http://picload.org/image/llcddir/banner.png)
Rep ?
can start appearing in the search results, and bringing in traffic and
sales. So here's how, in just nine steps, you can keep Google coming
back for more.
The more often your site gets crawled, the more quickly
your content will appear in the search results. And that’s the point,
right? You wouldn’t create a new product then keep it in the shed, would
you? But the things you need to do to get your site crawled more
frequently are also best practice SEO and could even improve your
position in the search engines - getting even more of that money-making
traffic. All this you can do in just nine steps.
1. Create more content
Creating new content is good for a whole range of reasons. Users love
it, and so do search engines. Every time Google crawls your site, it
looks for stuff it hasn’t seen before. If it can’t find any it might
wait a little longer before visiting you again next time. But if it does
find new stuff, it’ll probably come back sooner (newspaper websites for
example, who update content almost constantly, get crawled several
times a day). Try to update your content as often as it makes sense to
do so. (Remember though: quality is better than quantity. So don’t just
create content for creating content’s sake.)
Yes Google likes seeing lots of content, but it doesn’t like seeing
the same content over and over. That’s just boring. So don’t duplicate
your content.
If there are legitimate reasons for you to duplicate your content (in
your heart of hearts, you know if you’re being naughty or nice), you
might want to deploy a redirect or a canonical tag In fact, if you have duplicate content you should. Whether you want to or not.
Just like you wouldn’t want to drive your car on a road covered in
melted tar, Google doesn’t like sending its spider to crawl slow-loading
sites. And users don’t like visiting them. Site speed is also becoming an increasingly important ranking factor so keep yours up to scratch.
4. You can also adjust the crawl frequency in Google Webmaster Tools.
Go into ‘Configuration’ then ‘Settings’, and you can go from there:
This feature was designed so that webmasters could tell Google that it is crawling their site too aggressively, ie, it's overloading their servers. So if that’s happening, then by all means turn it down a notch here.
Some people however, use this feature to turn their crawl rate up. Personally, I wouldn’t do this because:
- It’s only valid for 90 days (so you’re just creating work for yourself).
- There’s a risk, indeed a temptation, to push the crawl rate
higher than necessary so you might inadvertently overwork your servers
and crash your site. Oops.
Google is pretty clever, and will usually work out an appropriate
frequency for crawling your site. I wouldn’t mess with it. I figure that
it knows what’s it’s doing more than I do, but if you really want to,
now you can. Hooray.
5. Add a sitemap
Google loves them. SEOs have been talking about them for ages. I‘ve
written an (amazing) article on how to create one. You have no excuse. Create a sitemap
6. Make sure your server is working as it should
The easiest way of checking this is to log into the Wordtracker's Keywords tool. Once you've logged in, just click on the campaigns you're working on, and the first thing you'll see will be the dashboard.
Here, where I've put the arrow, you can see how many of your site's
pages have been crawled by Google and Bing. The higher this number, the
better. If the number is very low, there might be a problem. (But treat
it as a rough guide. There are lots of reasons why Google might not be
indexing your site. If only there was an article on crawling and indexing that explained this more fully? I jest, of course there is. It’s the previous link).
7. Check your server responses
Google wants your pages to be where it expects them to be. It doesn’t
like sites whose server keeps throwing back problems. Is your site like
this? Check your server responses by running a site audit in the Keywords tool
8. Get links
As well as improving your rank, bringing in more traffic and
generally being awesome, links potentially get your website crawled more
quickly. Why? Because when Google crawls the web, it will find
(hopefully) lots of links to your site littered throughout it. The more
times it sees other websites pointing to you with links, the harder it
is to stay away.
9. Know your crawl rate
Finally you should monitor your crawl rate. After all, that’s what
we’re talking about. Google probably won’t crawl your site all at once
at regular intervals: it’ll do a few pages one day, a few more the next.
But these are just details. You want Google to crawl as much of your
site as often as possible. So to find out, log into Webmaster Tools. In
the 'Health' section, click ‘Crawl stats’.
And you’ll see a not-very-interesting graph that looks like this:
Most of the time, it will look like this. It will go up a bit and it
will go down a bit. Don’t worry about individual movements of the line,
but do worry about trends.
If there’s a long term upward trend, that’s all good. Keep doing what
you’re doing. If there’s a long term downward trend, re-read this
article. Start at one, go down to seven, then lather, rinse, repeat as
needed. You’ll not turn the ship around overnight, but you will
eventually. We hope.
It’s also worth looking at spikes in the graph. Was there anything
you did on these days? Was there a Google algorithm change? Think about
the method behind Google’s madness and you’ll soon start understanding
how it actually works and what you can do to get in its good books.
Bonus tip! Use Google+
OK, so really this is point number 10, but whatever. 10 is such a overused number.
Your final top tip to get your site crawled more often is quite
simply, to share it on Google+, Google's social network. Remember, one
of Google's motivations for creating a social network was to get access
to real-time data. So, if you share your new content on Google+, it's
likely to get crawled super-fast. Just try it, and you'll see what I
mean. (And, if you're not sure how to do this, check out our Google+ guide
http://googelino.com
Rep ?