how can I stop this
http://www.mysite.com/ftopic608-0-asc-15.html&sid=fbcf4a8430b5151bb15e911ac4c87ae8
from being indexed
this line in robots.txt
Disallow: /ftopic*asc*.html$
doesn't seem to work
would this work
Disallow: /ftopic*asc*.html*
or this
Disallow: /ftopic*asc*.html*$
Thanks in advance
Hello!
First of all, you have to make sure that the appended sessions.php or the standard nukecops sessions.php has been uploaded.
Then
/ftopic*asc*.html$
is fine, it should work with Google,
BUT,
there is a strange thing about Google. The bot follows every visitor of the side wherever they go, and does not take into consideration the bans put in robots.txt or in the module Adninistration. I have spotted googlebot reading my personal message along with me, or indexing a module that is only for admins.
So, if you want Google to index some particular page, just go there and stay for three minutes, when the bot is crowling your site. :)
PS. Bear also in mind that there is a "natural inertia". It takes time for the bot to "realise" that a particular address has been banned.
yes yes I have already uploaded that sessions.php and when I'm browsing I don't get that &sid....... and most of my site's users because I see it through who-is-where
what's strange is some googlebot still find addresses with the &sid and some the shorter ones (without the sid).
Regarding your observation about googlebot following users, I noticed that too and even the act before the users click on the submit to post, googlebot even indexes it. I don't know whether to laugh or to cry :lol:
Well, thanks anyway and more power to you Dr. Imago :)