Majestic-SEO : FAQ

Some of the questions that either have been frequently asked or we anticipated they would be.

1. What is Anchor Index?
2. Who are you?
3. Where is the data coming from?
4. Why are you doing this?
5. How can I see more detailed information for my site?
6. What's the catch?
7. How big is the index?
8. What software was used to build this index?
9. What are your short term plans?
10. Are you planning to offer SEO services to end user?
11. What is your index update strategy?
12. What is the effect of backlinks and anchor text on relevancy?
13. What are the planned features for the index?
14. Where are the link: and linkdomain: commands?
15. What is the relationship between Majestic-SEO and Majestic-12 community?
16. I can't find my site in your index, what's wrong?
17. Why can't I search using keywords?

1. What is Anchor Index?

Anchor Index is a very big (130 bln+ unique) database of urls from all over the web with identified backlinks, anchor text and some flags from pages (24 bln) that were crawled, analysed, indexed and finally merged into the index that can be queried.

2. Who are you?

Majestic-SEO is a commercial offshoot from Majestic-12, a UK based company founded in 2004.

3. Where is the data coming from?

From the Wild World Web itself for we are the Majestic-12: Distributed Search Engine. We do not meta-search or otherwise query other search engines: we are the search engine! Over a long period of time we developed software capable of crawling and indexing large amounts of web data - this index is a big stepping stone towards relevant full-text search. The purpose of this index is to allow relevancy research as well as help fund continued activites in development of a competitive community driven general purpose web scale search engine.

4. Why are you doing this?

We have been building a web scale search engine for over 3 years now, and we reached the point when it became clear that we need to understand links/anchor text based relevancy much better than we were able to. This has lead us to creation of the anchor index - the tool that will help us experiment with different relevancy algorithms, give something to webmasters whose support we need and ultimately help fund renewed activities in development of the full text search engine that can compete with the best of the best.

5. How can I see more detailed information for my site?

By registering with us you can add your own websites to your profile to see detailed analysis of backlinks and anchor text: this information is prioritised to show most important backlink first, rather than showing a random sample (with probably not-so-randomly removed most important backlinks).

6. What's the catch?

The catch is that in exchange for free basic information about anchor text and backlinks for your site you will have to allow our bot (MJ12bot) to crawl your site for at least 12 months since your last usage of reports. In vast majority of cases (95%+) you already do that, so in this case there is no catch really: only those who specifically disallowed our bot in robots.txt or only allow a handful of other bots to crawl and disallow all the rest, will need to decide if they want to allow us crawl them in exchange for the free information that they can get. We hope your decision will be positive!

7. How big is the index?

Current index was merged on 16/01/08 and it contains backlinks and anchor text from 24 bln crawled pages, in total there are over 130 bln unique urls in the index with approximately 1 trillion (10^12) mapping relationships (url pointing to url) alongside with anchor text that was used in that relationship and some flags. It will grow much bigger.

8. What software was used to build this index?

Proprietary software that was developed by us from scratch using C#/.NET platform: highly parallel stuff that takes advantage of multiple cores and machines to process data in parallel.

9. What are your short term plans?

We have very ambitious plans for 2008. What you have to understand is that what you currently see is about 3 years of work on backend technology, and maybe 1 week of work on analytical reports. We will improve greatly quality of data and tools that we use to analyse it.

10. Are you planning to offer SEO services to the end users?

No, not to the end users. We will offer unique intelligence that will help understand why some pages rank higher than the others, but we won't be actually optimising sites for the end users. We plan to partner with the leading SEO companies who understand the value of anchor text and backlinks insofar as relevancy is concerned and wish to ensure that their clients benefit from this intelligence.

11. What is your index update strategy?

We plan to do next big index update around Aug-Sep 2008. The expectation is that we will grow index considerably as well as refresh many of the existing most important pages. Meanwhile we registered domains (even if they are not yet in the main index) get daily updates that will show new backlinks from current crawling activity (~130 mln crawled pages per day), this includes automatic recrawls of most popular pages backlinks from which are likely to be the most valuable. This is only done for registered domains, so don't delay register yours!

12. What is the effect of backlinks and anchor text on relevancy?

For cases when number of matches is high backlinks and anchor text begin to play very important role. The reason for this is simple - search engine can only show top 10 entries, but when number of matches is in the range of tens or even hundreds of millions, then you will get lots of matches with nearly the same full text matching scores, so you have to use something else to select the "best" of them. Just how important it is? Well, that's exactly why we created anchor index to be able to find good answer to this question!

13. What are the planned features for the index?

Lots of very cool features. For example determining authority sites, types of backlinks, detecting link exchanges and more: you are basically witnessing the kind of secret relevancy tool that any large search engine keeps under wraps away from prying eyes. Here you will have the opportunity to see things that some people would not want you to know: you are being left guessing and using clues that might have been designed to lead you away from truth, here we are actually making a big effort to get to the truth on how relevancy algorithms work in the best search engines out there. So that we could build one ourselves.

14. Where are the link: and linkdomain: commands?

They are not implemented on purpose because we don't want to fight with large scale automated queries trying to get these backlinks. If you want to see backlinks and anchor text then you need to register and then verify your domain: this is when you will get a set of nice reports showing backlinks, anchor text, links from bad neighbourhood and other goodies.

15. What is the relationship between Majestic-SEO and Majestic-12 community?

We are the same company - Majestic-12 Ltd, profits from this commercial offshoot will be shared with the community that helped build it.

16. I can't find my site in your index, what's wrong?

Nothing if your site is very new or not well linked to: our index is based on the links that we followed much like Alice followed the white rabbit. This should be fairly rare situation though as our index is pretty comprehensive. Even though your site might not be in the index, you can still add it and in this case daily updates will take it into account and newly found backlinks will be made available to you - this way you don't have to wait until the big update takes place!

17. Why can't I search using keywords?

There is no full-text index, no HTML cache or anything like this present in the anchor index, which is a collection of factual information about linking on the Web. There are two big reasons for this: first we need to understand relevancy algorithms much better, and secondly we will need a lot more hardware to handle ~30 bln full-text index, so we have to do it in steps - first big one is this anchor index, we hope it will help us move forward to our ultimate goal.

If you have not been satisfied with the information above then feel free to contact us: contact@majesticseo.com.