Excluding pages from the search database

Every day Yandex indexes millions of pages and adds them to the search database. To avoid filling it with documents that will never show up in the search results, Yandex analyzes each document using a special algorithm. If a page can't provide one of the most relevant answers to a search, it isn't included in the search database.

Excluding these pages from the search base does not affect the number of visits because the users wouldn't see these pages anyway. In addition, Yandex continues to index and analyze these documents in the same way as all the others. If at some point the algorithm reveals that the page can get into the search results, it will be added to the search engine database.

  1. Title, description, keywords meta tags on the page are filled in correctly and the page meets all requirements. Why isn't it in the search results?
  2. Why do pages disappear from the search results and then appear again?
  3. Why do some pages with a similar topic and content get in the search, while others don't?

Title, description, keywords meta tags on the page are filled in correctly and the page meets all requirements. Why isn't it in the search results?

In addition to checking the tags on the page, the algorithm checks if the page content is unique, informative, in-demand and up-to-date, as well as many other factors. However, you should pay attention to meta tags. Often the description and title meta tags are created automatically. They duplicate themselves and don't make the search easier for the users.

If the site contains a lot of similar products that differ only by color, size or configuration, they may be excluded from the search. The same is true about the pagination pages, product selection and comparison pages, and image pages without text content.

Why do pages disappear from the search results and then appear again?

The algorithm checks the pages more than once. It checks the page and calculates the probability of it displaying every time before the search database is updated. The robot indexes excluded pages in the usual way and crawls them regularly. It can detect changes on the requested page and take them into account when calculating the display probability. The algorithm is continually being fine-tuned. This is why pages that were excluded before can appear in the output.

Why do some pages with a similar topic and content get in the search, while others don't?

The algorithm evaluates each page independently from others, even if they belong to the same section. If the page is relevant enough to be displayed in the search results for a specific search in a certain region, it is shown in the search results. The output of the algorithm depends on dozens of indexing factors and hundreds of ranking factors. For example, highly demanded products are displayed in the search, while less suitable products aren't shown in the search output.