robots.txt

[admin]

how can I stop this

http://www.mysite.com/ftopic608-0-asc-15.html&sid=fbcf4a8430b5151bb15e911ac4c87ae8

from being indexed

this line in robots.txt

Disallow: /ftopic*asc*.html$

doesn't seem to work

would this work

Disallow: /ftopic*asc*.html*

or this

Disallow: /ftopic*asc*.html*$

Thanks in advance

[Пламен]

Hello!

First of all, you have to make sure that the appended sessions.php or the standard nukecops sessions.php has been uploaded.

Then
/ftopic*asc*.html$
is fine, it should work with Google,
BUT,
there is a strange thing about Google. The bot follows every visitor of the side wherever they go, and does not take into consideration the bans put in robots.txt or in the module Adninistration. I have spotted googlebot reading my personal message along with me, or indexing a module that is only for admins.

So, if you want Google to index some particular page, just go there and stay for three minutes, when the bot is crowling your site.

PS. Bear also in mind that there is a "natural inertia". It takes time for the bot to "realise" that a particular address has been banned.

[admin]

yes yes I have already uploaded that sessions.php and when I'm browsing I don't get that &sid....... and most of my site's users because I see it through who-is-where

what's strange is some googlebot still find addresses with the &sid and some the shorter ones (without the sid).

Regarding your observation about googlebot following users, I noticed that too and even the act before the users click on the submit to post, googlebot even indexes it. I don't know whether to laugh or to cry  :lol:

Well, thanks anyway and more power to you Dr. Imago