What Is SEO / Search Engine Optimization?

All major search engines such as Google, Bing and Yahoo have primary search results, where web pages and other content such as videos or local listings are shown and ranked based on what the search engine considers most relevant to users. Payment isn’t involved, as it is with paid search ads.

Search engines “crawl” websites, going from one page to another incredibly quickly, acting like hyperactive speed-readers. They make copies of your pages that get stored in what’s called an “index,” which is like a massive book of the web.

When someone searches, the search engine flips through this big book, finds all the relevant pages and then picks out what it thinks are the very best ones to show first. To be found, you have to be in the book. To be in the book, you have to be crawled.

Most sites generally don’t have crawling issues, but there are things that can cause problems. For example, JavaScript or Flash can potentially hide links; making the pages those links lead to hidden from search engines. And both can potentially cause the actual words on pages to be hidden.

Each site is given a crawl budget, an approximate amount of time or pages a search engine will crawl each day, based on the relative trust and authority of a site. Larger sites may seek to improve their crawl efficiency to ensure that the ‘right’ pages are being crawled more often. The use of robots.txt, internal link structures and specifically telling search engines not to crawl pages with certain URL parameters can all improve crawl efficiency.