{"id":4881,"date":"2018-04-23T05:35:33","date_gmt":"2018-04-23T05:35:33","guid":{"rendered":"http:\/\/a1webdesignteam.com\/blog\/?p=4881"},"modified":"2018-04-23T05:35:33","modified_gmt":"2018-04-23T05:35:33","slug":"meta-tags-for-instructing-search-bots","status":"publish","type":"post","link":"https:\/\/a1webdesignteam.com\/blog\/meta-tags-for-instructing-search-bots\/","title":{"rendered":"Meta Tags for Instructing Search Bots"},"content":{"rendered":"

If you want to totally block all search engines:<\/h4>\n
<<\/span>meta name<\/span>=<\/span>\"<\/span>robots\"<\/span><\/span> content<\/span>=<\/span>\"<\/span>noindex, nofollow\"<\/span><\/span>><\/span><\/span>\r\n<!-- or --><\/span>\r\n<<\/span>meta name<\/span>=<\/span>\"<\/span>robots\"<\/span><\/span> content<\/span>=<\/span>\"<\/span>none\"<\/span><\/span>><\/span><\/span>\r\n<!-- or if you only care about Google... --><\/span>\r\n<<\/span>meta name<\/span>=<\/span>\"<\/span>googlebot\"<\/span><\/span> content<\/span>=<\/span>\"<\/span>noindex, nofollow\"<\/span><\/span>><\/span><\/span><\/code><\/pre>\n
#Why?<\/h4>\n
Perhaps it’s a somewhat private URL on your site, but not protected in a such a way that a search bot couldn’t see it. Like if a page requires a log in to see it, that’s blocked from a search bot anyway. But if it’s just an obfuscated URL, who knows how search bots get around, so you might wanna prevent the indexing.<\/p>\n
Search results is another common one. Weird to have a search result be search results, perhaps. WordPress themes tend to include this:<\/p>\n
<<\/span>?<\/span>php if<\/span> (<\/span>is_search<\/span>(<\/span>)<\/span>)<\/span> {<\/span> ?<\/span>><\/span>\r\n  <<\/span>meta name=<\/span>\"robots\"<\/span> content=<\/span>\"noindex, nofollow\"<\/span>><\/span>\r\n<<\/span>?<\/span>php }<\/span> ?<\/span>><\/span><\/code><\/pre>\n
I might alter that a little and tell bots, hey, you can follow links here to help continue indexing the site, but don’t index this page specifically.<\/p>\n
<<\/span>meta name<\/span>=<\/span>\"<\/span>robots\"<\/span><\/span> content<\/span>=<\/span>\"<\/span>noindex, follow\"<\/span><\/span>><\/span><\/span><\/code><\/pre>\n
#Others<\/h4>\n
Search engines tend to use the description tag for text content next to the\u00a0<title><\/code>:<\/p>\n
<<\/span>meta name<\/span>=<\/span>\"<\/span>description\"<\/span><\/span> content<\/span>=<\/span>\"<\/span>A description of the page\"<\/span><\/span> \/><\/span><\/span><\/code><\/pre>\n
See the Google page<\/a>\u00a0for other Google-specific ones like the ability to stop Google-specific UI things like their “site link search box”.<\/p>\n","protected":false},"excerpt":{"rendered":"
If you want to totally block all search engines: <meta name=”robots” content=”noindex, nofollow”> <!– or –> <meta name=”robots” content=”none”> <!– or if you only care about Google… –> <meta name=”googlebot” content=”noindex, nofollow”> #Why? Perhaps it’s a somewhat private URL on your site, but not protected in a such a way that a search bot couldn’t […]<\/p>\n","protected":false},"author":1,"featured_media":4882,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0},"categories":[9,5,22],"tags":[51],"_links":{"self":[{"href":"https:\/\/a1webdesignteam.com\/blog\/wp-json\/wp\/v2\/posts\/4881"}],"collection":[{"href":"https:\/\/a1webdesignteam.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/a1webdesignteam.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/a1webdesignteam.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/a1webdesignteam.com\/blog\/wp-json\/wp\/v2\/comments?post=4881"}],"version-history":[{"count":0,"href":"https:\/\/a1webdesignteam.com\/blog\/wp-json\/wp\/v2\/posts\/4881\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/a1webdesignteam.com\/blog\/wp-json\/wp\/v2\/media\/4882"}],"wp:attachment":[{"href":"https:\/\/a1webdesignteam.com\/blog\/wp-json\/wp\/v2\/media?parent=4881"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/a1webdesignteam.com\/blog\/wp-json\/wp\/v2\/categories?post=4881"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/a1webdesignteam.com\/blog\/wp-json\/wp\/v2\/tags?post=4881"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}