In
the wordpress website the robots.txt is automatically getting
generated by the following script :
function
do_robots() {
header(
'Content-Type: text/plain; charset=utf-8' );
do_action(
'do_robotstxt' );
if
( '0' == get_option( 'blog_public' ) ) {
echo
"User-agent: *\n";
echo
"Disallow: /\n";
}
else {
echo
"User-agent: *\n";
echo
"Disallow:\n";
}
}
in
the file /functions.php which is under the /wp-includes/ folder. So
you can modify the code to change the robots.txt settings.
If
you do not wish to show the robots.txt file you can redirect it to
the 404 error page using the .htaccess
Redirect
301 /robots.txt http://www.yoursite.com/404