There are many factors involved when optimising a website to be discovered in search results on Google. I thought I would share some of the most common reasons I have come across over the years for why a website may not be generating much (or any) visitors from search engines and answer the question why is Google not finding my website?

1. Make sure your website is not being blocked from Google search bots

Often when a website is being built or redesigned, the web developer will block search engines from being able to crawl through the pages and add the content into its database. If a search engine cannot add content to its database, then it is as if the website never existed (as far as search engines go).

It is not uncommon to see a website go live to the world and a web developer has forgotten to stop blocking the site from search engines. Following are some quick checks you can do to see if pages are being blocked from search engines.

Introducing the Robots.txt file…

A Robots.txt file is used to control access to your website from (search) bots – to effectively allow or disallow them to crawl through the web pages by following the hyperlinks throughout the site.

To see if your entire website is being blocked from search engine bots, type example.com/robots.txt into your web browser (replace [example.com] with your website domain name).

The Robots.txt file always lives here in the root directory, so if you get an error page you do not have a Robots.txt file and it will not be blocking anything from your website.

If you do have a Robots.txt file, you will be presented with a text file in your web browser. Take a look and see if you have any of these lines entries appearing:

User-agent: *

Disallow: /

 

The above blocks all types of bots from crawling your website.

User-agent: Googlebot

Disallow: /

 

The above will just stop Googlebot (Google’s web crawler) from crawling your website. If you see any of the above two entries you should contact your web developer or IT support to remove them.

You may also see something like this:

User-agent: *

Disallow: /admin

 

The above tells any web crawler not to crawl the example.com/admin section of the website. This is useful to stop admin areas, login pages, shopping baskets etc. from being indexed by Google and shown in search results and you are likely going to want to leave anything like that.

2. Let Google know your website exists

Now you are sure your website is not being blocked by Google, you may still be wondering how do I make my website show up in Google search?

Start this next step by doing a Google search for the following (make sure there is no space after the “:” and replace [example.com] your site name):

site:example.com

This will show all web pages from your website that Google has in its database. If nothing is showing, then it because Google does not know your website exists yet.

The best way to make Google aware of your site is to create a Google Search Console account. Once you have signed up and followed the on-screen instructions you can go to the Google URL submission page at http://www.google.com/addurl/?continue=/addurl and register your site’s URL (web address) with Google search.

It is not unusual to see a new website or web page appearing in just a few hours after doing the above, so occasionally repeat the site:example.com search to check if your website is now appearing.

3. Optimise web pages for relevant keywords

This section will take up the bulk of my book on search engine optimisation, however, I will try to give you a quick overview here.

First, it is important to understand that you should not be trying to optimise your entire website to be discovered for the same keywords. Instead, treat each individual page as its own source of information and optimise it accordingly for searches that it holds relevant information for.

An important question to ask yourself is “What are the keywords someone would type into Google who would be looking for this web page?”

For example, someone may type “professional marketing courses” or “where to study CIM courses”; if you were trying to get ranked in Google for those phrases, you would be advised to make sure those exact phrases appear on your site.

So, where do you put those keywords?

The most important places are:

  • Page Title: This is what appears in the tab of the web browser window and is shown as the blue heading text in search engine results pages.
  • Meta Description: This is shown as the snippet of text in search results. Although it has no direct impact on search rankings, it does allow you to create an enticing call to action to encourage a click through to your website in search results, rather than your competitors.
  • Page Heading: This is the title of the article, and can be different than the Page Title. It should sit within an <H1> HTML tag and you can check with your web developer to see if that tag has been implemented correctly to wrap around your Page Heading.
  • URL: Having your target keyword(s) in your URL is another place where you can help Google to understand what your page is about. A good URL would look something like http://www.example.com/courses/marketing rather than http://www.example.com/?c=146&p=79
  • Body Copy: Don’t forget to include your keywords (and variations of them) in the main body of your page (either the main article content, service description or product description). Don’t overdo it though, ensure it reads naturally – always write for humans first and edit for search engines afterwards.

Following the above steps will help to get your business to show up on Google search results.

4. Improve the trust and authority of your website

Google was pretty much built on analysing links coming into a website (“backlinks”). A website with backlinks from trusted sources (“authority websites”) will, in turn, become recognised as a trusted resource by Google.

The proprietary algorithm that Google uses to rank individual web pages is called PageRank. A good test to check if a website is likely to rank above competitors on Page 1 of Google used to be to check the PageRank with the Google Toolbar – however, the PageRank is no longer shown publicly.

These days, a popular way to check a domain name’s ability to rank is “Domain Authority” – a proprietary algorithm by MOZ. You can check the Domain Authority of your website by going to MOZ Open Site Explorer and entering your domain name, followed by some of your competitors.

If your competitors have a score of 30+ and your score is in the single digits, it is unlikely you will be ranking above them for the same keywords anytime soon.

To improve your “authority” you will need to increase the number and, most importantly, quality of backlinks to your website. This is the most difficult and time-consuming part of SEO.

Go VERY careful when trying to acquire new links to your site, make sure they come from trusted sources otherwise you could get hit with a Google penalty.

5. Check that your website has not been penalised by Google

If you used to get traffic from Google but it suddenly dropped off one day, you may have been affected by a Google algorithm update. If you can see from your website analytics software the approximate date your website dropped its traffic you can then correlate that with dates of Google algorithm updates to see if you were affected.

Some of the major algorithms to be aware of are Panda, which penalises “thin content” sites and Penguin which penalises sites that violate Google Webmaster Guidelines by manipulating the number of links pointing to a web page (remember, I told you to go careful when acquiring new links!).

If you have been penalised by Google then you should seriously think about hiring an SEO professional to help you, as you could end up making things worse (worst-case scenario: Google de-indexes you, and you are removed from their search results).

Lee Benning is a digital marketing consultant with over 10 years’ experience of working with businesses of all sizes – from start-up to multi-national corporations. He also supports Oxford College of Marketing students with assignment feedback on CIM courses.

If you want to learn more about SEO, our Search Engine Marketing unit from our Diploma in Digital Marketing may be of interest to you. It can be studied as a single award or as part of our wider Diploma. To find out more, get in touch with our student representatives today via email enquiries@oxfordpeg.com or by calling 01865 515255.