In reply to this post:

@mcc this looks useful: darkvisitors.com/robots-txt-bu h/t