# Exclude Files From All Robots: 

User-agent: baiduspider
Disallow:
Crawl-delay: 400

User-agent: googlebot
Disallow:
Crawl-delay: 400

User-agent: slurp
Disallow:
Crawl-delay: 4000

User-agent: spider
Disallow:/

User-agent: crawl
Disallow:/

User-agent: TencentTraveler
Disallow:/

User-agent: BSpider
Disallow:/