# Robots.txt: discourage well-behaved robots from visiting specified resources # 2011-06-21: Added WAX crawler -ED User-agent: hul-wax Disallow: User-agent: HCL_Ultraseek Disallow: /cfform_scripts/ Disallow: /cgi-bin/ Disallow: /error/ Disallow: /libraries/finearts/faclass/ Disallow: /libraries/finearts/islamicclass/ Disallow: /images/ Disallow: /inc/ Disallow: /technicalservices/images/ Disallow: /technicalservices/includes/ Disallow: /technicalservices/internal/ Disallow: /hours/nowm.cfm Disallow: /hours/unitm.cfm Disallow: /hours/daym.cfm Disallow: /hours/findmm.cfm Disallow: /hours/gen_error.cfm Disallow: /hours/detail.cfm Disallow: /hours/unit.cfm Disallow: /apps/ Disallow: /research/guides/biblio/ Disallow: /reports/ Disallow: /survey/ Disallow: /temp/ Allow: /libraries/finearts/faclass/index.html Allow: /libraries/finearts/islamicclass/index.html User-agent: * # pertains to *all* robots Disallow: /cfform_scripts/ Disallow: /cgi-bin/ Disallow: /error/ Disallow: /info/giving/funds/ Disallow: /libraries/finearts/faclass/ Disallow: /libraries/finearts/islamicclass/ Disallow: /images/ Disallow: /inc/ Disallow: /technicalservices/images/ Disallow: /technicalservices/includes/ Disallow: /technicalservices/internal/ Disallow: /hours/ Disallow: /apps/ Disallow: /survey/ Disallow: /temp/ Disallow: /hours/nowm.cfm Disallow: /hours/unitm.cfm Disallow: /hours/daym.cfm Disallow: /hours/findmm.cfm Disallow: /research/guides/biblio/ Disallow: /reports/ Allow: /libraries/finearts/faclass/index.html Allow: /libraries/finearts/islamicclass/index.html Allow: /hours/week.cfm Allow: /hours/open_now.cfm