User-agent: gsa-crawler 
Allow: /$
 
User-agent:  *
Disallow: /cjer/
Disallow: /comet/
Disallow: /lre2/