Saturday 21 December 2013

Cannot find the robot.txt anywhere in my file manager but the domain.com/robots.txt is working

In the wordpress website the robots.txt is automatically getting generated by the following script : function do_robots() {
header( 'Content-Type: text/plain; charset=utf-8' );

do_action( 'do_robotstxt' );

if ( '0' == get_option( 'blog_public' ) ) {
echo "User-agent: *\n";
echo "Disallow: /\n";
} else {
echo "User-agent: *\n";
echo "Disallow:\n";
}
}
in the file /functions.php which is under the /wp-includes/ folder. So you can modify the code to change the robots.txt settings.

If you do not wish to show the robots.txt file you can redirect it to the 404 error page using the .htaccess

Redirect 301 /robots.txt http://www.yoursite.com/404

No comments:

Post a Comment

ERR_CONTENT_DECODING_FAILED

If you are getting the error as ERR_CONTENT_DECODING_FAILED while accessing the domain then add the below extension in php.ini  ++++++++...