User-agent: * Disallow: /archive/ Disallow: 100M.dat Disallow: 10M.dat