Defence.gov.au robots.txt
Dear Department of Defence,
Websites have a "robots.txt" file that helps search engines work out what they should be indexing and shouldn't be.
The Department of Defence's robots.txt file (located at https://www.defence.gov.au/robots.txt) has the below information at the bottom:
++++++++++
# No ducments
User-agent: Googlebot
Disallow: /sites/default/*.pdf$
Disallow: /sites/default/*.doc$
Disallow: /sites/default/*.docx$
Disallow: /sites/default/*.xls$
Disallow: /sites/default/*.xlsx$
++++++++++
I request access to the following documents under the Freedom of Information Act 1982:
1. Documents that informed the decision to block Google's web crawler from indexing pdf, doc, docx, xls, and xlsx files contained on the site.
Yours faithfully,
CR