Defence.gov.au robots.txt

Currently waiting for a response from Department of Defence, they should respond promptly and normally no later than (details).

Dear Department of Defence,

Websites have a "robots.txt" file that helps search engines work out what they should be indexing and shouldn't be.

The Department of Defence's robots.txt file (located at https://www.defence.gov.au/robots.txt) has the below information at the bottom:

++++++++++
# No ducments

User-agent: Googlebot
Disallow: /sites/default/*.pdf$
Disallow: /sites/default/*.doc$
Disallow: /sites/default/*.docx$
Disallow: /sites/default/*.xls$
Disallow: /sites/default/*.xlsx$
++++++++++

I request access to the following documents under the Freedom of Information Act 1982:
1. Documents that informed the decision to block Google's web crawler from indexing pdf, doc, docx, xls, and xlsx files contained on the site.

Yours faithfully,

CR