Does Search Engine like Google has been indexed unwanted posts, pages or URLs into the search results. And you are unable to comprehend how to remove & prevent them from search results permanently. In this post, I’ll clarify how you can completely get rid of unwanted Search results.
When you start a blog or website, don’t know about what is necessary to index & what not. You Submit your site to Google Webmaster tools (Google Search Console) or other search engines, and after submission, if you have not optimized your site for the search engines. Then some of your site pages will be indexed unnaturally.
So here I’m going to share few methods, that will help you to remove and prevent URLs from Search Engines like Google, Bing, Yahoo and Yandex search.
How to remove URL’s permanently
If some posts have been indexed by search engines then they can be removed using following methods:
1. Set Noindex, follow or nofollow attribute using SEO plugins:
As you know, if your site has duplicate content then Google will penalize your blog and it will affect your site ranking in the search. Duplicate content available on your site in the form of Duplicated Posts, Pages, Categories, Tags, SubPages of archives and Affiliates links.
If You don’t pay attention to this what content is necessary to index or what not. then all the content will be indexed into the search. And the permanent solution for this problem is set a noindex, follow tag on the unnecessary Posts, Pages, Categories, Tags, Subpages of archives, Author archives and nofollow tag on Affiliates links.
So now you’re thinking how to set rel=”noindex, follow” tag then Yoast SEO plugin will help you lot.
NoIndex, follow on Posts or pages
You use WordPress post editor to write or edit a post or page, When you scroll down the editor then you’ll see Yoast SEO settings. To set rel=”noindex, follow” on the post and page, edit your existing post or page and scroll down, click on Settings (gear) icon and you’ll see the Advanced tab. Now Choose noindex option from Meta robots index dropdown menu. And select Meta robots follow radio button. After this click on the “Publish” button to make changes.
If Categories & Tags, SubPages of archives, Author and Date based archives pages have been indexed by search engines then it would be a duplicate content and you can deindex them with Yoast SEO plugin using following method: How to solve WordPress Duplicate content problem.
Set rel= “nofollow” tag attribute on Affiliate Links:
If you’re doing affiliate marketing and redirecting affiliate links using your domain URL trigger:
And you don’t block access to the affiliate links, search engines will index all the affiliate links and source affiliate page content. Then your site can get the huge penalty by Search Engines.
To avoid any affiliate link penalty, you can use these two solutions: one is block URL access using robots.txt and second is use rel= “nofollow” tag attribute to all affiliate links. But here I’m talking about here rel= “nofollow”. So now you’re thinking how to do this? Then again there is two options available:
(a) Add Manually:
If you don’t want to use a plugin then you’ll need to add this manually, read about this: How to add rel= “nofollow” attribute on a link without plugin.
(b) Using plugin:
If you’re using a plugin to redirect affiliate links then check the plugin settings, is there any option available to add nofollow attribute on affiliate links. Mostly redirection plugins provide add rel= “nofollow” attribute. If your redirection plugin doesn’t have the nofollow option. Then I’ll suggest you use a handy tool for all affiliate links and Thirsty-affiliates is a powerful link cloaking plugin. Which is available for free and also has the option to add rel= “nofollow” on all affiliate links without editing the post.
2. Leave 404’s pages as it.
If you’ve removed some URLs from the site recently and the post still available in the Search results then don’t do anything and leave as it is. Google removes the 404’s pages from the search after regular intervals.
3. Block URL’s with Robots.txt
This is an easy solution if you want to block access to the certain posts or pages, Categories, Tags & affiliate links. With the robots.txt file you can also block an entire directory of your site.
You want to block a post URL and URL is: http://www.domain.com/this-is-my-post-url/ then this URL can be blocked for the search engines by adding Disallow: /this-is-my-post-url/ in robots.txt file.
If the Categories and Tags are also indexed by Search Engine and you want to rid them, then add Disallow: /category/ or Disallow /tag/ in the robots.txt file. And all the Categories and tags will be blocked from search engines access. eg:
And the same method will be applied to affiliate links. If you’re not using a plugin which can add nofollow attribute then the affiliate links can be blocked using robots.txt file. But before blocking affiliate links with the robots.txt file you’ll need to know about affiliate link URL trigger. basically all the websites use /go/ or /go/ or /refer/ as an URL trigger which we can easily remember.
For example, my one affiliate URL is: https://www.bloggingglobal.com/go/hostgator and so URL trigger for my affiliate link is /go/ for this in my WordPress robots.txt file, I have added URL trigger as Disallow: /go/
Don’t know how to edit robots.txt file read this : How to create or edit robots.txt file in WordPress.
But I would want to tell you, robots.txt file only blocks the search engines access to the content, URLs will be still indexed by the Search Engines. And the search results will be come with: “A description for this results is not available because of this site’s robots.txt.”
4. Remove URL’s using Webmasters Tools:
Above solutions take some time to deindex URL’s from search but these are the only good method because the content will not be indexed again. However, there is a temporary option available in the Google Search Console (Google webmasters tools) If you want to immediately hide some URLs of your site, this process will help you.
>Login to Google Search Console, Click on Google Index option, which is available on left and choose sub option Remove URLs.
>Now Click on Temporarily hide button and enter the URL on your site that you want to remove from search. After this click on Continue button. Here you can also enter archives Subpages, Category, tag and affiliate link URL trigger. With which you can hide all the subpages. For examples:
- If you want to remove category page and all sub category pages eg: http://www.domain.com/category/adsense/page/2/ & http://www.domain.com/category/wordpress/page/3/. then you can enter only base category URL such as http://www.domain.com/category/ and all the categories that search engines have been indexed, will be removed.
- Same process will be applied on the tag and affiliate links and you’ll need to enter the base URL eg: http://www.domain.com/tag/ or for affiliate links if you’re trigging your URLs with offer then you can enter http://www.domain.com/go/ and all the URL will be deindexed along with subpages of archives.
> On next page Select Request type from the dropdown menu option Temporarily hide directory ( if you’re going to remove all subpages of archives) And if you’re only removing a simple URL (domain.com/this-is-my-post-url/) of your site then choose “Temporarily hide page from search results and remove from cache” option. After choosing an option click on Submit Request button.
>Next you’ll see URL
As I said, it is a temporary option. If you use Google Search Console remove URLs tool to remove URLs from the search results then all the URLs will be re-indexed in a certain period, if you don’t set noindex, follow or nofollow tag attribute on the link. So use this tool only if you’ve already setup noindex, follow or nofollow.
Let me know, If you have any query related to the post.