Server log files
Dear Australian Bureau of Statistics,
This a request under the FOI Act.
Can you please provide all raw server log files for the Census and ABS websites for 9 August 2016. I am able to receive the raw files via USB, ftp or file sharing systems.
I request that fees in relation to this request be waived. The information is in the public interest to determine the severity of the DDoS attack suffered by the ABS and to help other organisations handle similar attacks in future.
Yours faithfully,
Jackson Gothe-Snape
Dear Mr Gothe-Snape,
I refer to your application in which you seek access to documents under
the Freedom of Information Act 1982 (ABS reference FOI
201617/20) regarding:
"all raw server log files for the Census and ABS websites for 9 August
2016"
Your request was received by the ABS on 10 August 2016 and the 30 days
statutory period for processing your request commenced from that date. If
this request will take longer than 30 days to process, we will contact you
to negotiate an extension of time.
You will be notified of any charges in accordance with the Freedom of
Information (Fees and Charges) Regulations, should they apply, in relation
to your request as soon as practicable.
Regards
Keelie Williams
FOI Contact Officer
Australian Bureau of Statistics
(P) (02) 6252 7203
(E) [1][ABS request email] (W) [2]www.abs.gov.au
[3]Inactive hide details for Jackson Gothe-Snape ---10/08/2016 09:41:04
AM---Jackson Gothe-Snape <foi+request-2118-60cf92f7@righttJackson
Gothe-Snape ---10/08/2016 09:41:04 AM---Jackson Gothe-Snape
<[FOI #2118 email]>
*
Basics
Jackson Gothe-Snape 10/08/2016 09:41 AM
<[FOI #2118 email]>
Send To FOI requests at ABS <[ABS request email]>,
cc
Subject Freedom of Information request - Server log files[4]Policy &
legislation WDB
Protective Mark
Categories Freedom of Information\Requests
Visibility Limited Readers
Wolfgang Hertel/Staff/ABS, Policy and Legislation, ELG, SMG,
Christine Williams/Staff/ABS, Duncan Young/Staff/ABS, Jenny
Telford/Staff/ABS, Michelle S Howe/Staff/ABS, Mark
Hindle/Staff/ABS, Tasma Vyver/Staff/ABS, Andrew
Editors Bell/Staff/ABS, Juanita Pettit/Staff/ABS, Lucy
Jones/Staff/ABS, Andrew Begg/Staff/ABS, Kathy
Dawes/Staff/ABS, Kathy Mudford/Staff/ABS, Emma
O'Rourke/Staff/ABS, Caroline Blink/Staff/ABS, Keelie
Williams/Staff/ABS, Emily Covington/Staff/ABS
Wolfgang Hertel/Staff/ABS, Policy and Legislation, ELG, SMG,
Christine Williams/Staff/ABS, Duncan Young/Staff/ABS, Jenny
Telford/Staff/ABS, Michelle S Howe/Staff/ABS, Mark
Hindle/Staff/ABS, Tasma Vyver/Staff/ABS, Andrew
Readers Bell/Staff/ABS, Juanita Pettit/Staff/ABS, Lucy
Jones/Staff/ABS, Andrew Begg/Staff/ABS, Kathy
Dawes/Staff/ABS, Kathy Mudford/Staff/ABS, Emma
O'Rourke/Staff/ABS, Caroline Blink/Staff/ABS, Keelie
Williams/Staff/ABS, Emily Covington/Staff/ABS
Last modified 10/08/2016 09:32:13 PM By Wolfgang Hertel/Staff/ABS
Document Id DCOO-ACNVU7
Dear Australian Bureau of Statistics,
This a request under the FOI Act.
Can you please provide all raw server log files for the Census and
ABS websites for 9 August 2016. I am able to receive the raw files
via USB, ftp or file sharing systems.
I request that fees in relation to this request be waived. The
information is in the public interest to determine the severity of
the DDoS attack suffered by the ABS and to help other organisations
handle similar attacks in future.
Yours faithfully,
Jackson Gothe-Snape
-------------------------------------------------------------------
Please use this email address for all replies to this request:
[FOI #2118 email]
Is [ABS request email] the wrong address for Freedom of
Information requests to Australian Bureau of Statistics? If so,
please contact us using this form:
[5]https://www.righttoknow.org.au/change_re...
This request has been made by an individual using Right to Know.
This message and any reply that you make will be published on the
internet. More information on how Right to Know works can be found
at:
[6]https://www.righttoknow.org.au/help/offi...
If you find this service useful as an FOI officer, please ask your
web manager to link to us from your organisation's FOI page.
-------------------------------------------------------------------
References
Visible links
1. mailto:[email address]
2. http://www.abs.gov.au/
4. file:///tmp/Notes:/DomCO1510/CA257B2800193111/F54534119E524D36CA256D4800227591/B738C3420C785BBCCA25800A008388EB
5. https://www.righttoknow.org.au/change_re...
6. https://www.righttoknow.org.au/help/offi...
Dear Mr Gothe-Snape,
This email refers to your FOI request (201617-20) received on 10 August
2016.
The purpose of this email is to advise you, as required under Section 24AB
of the Freedom of Information Act 1982 (FOI Act) that a practical refusal
reason exists under Section 24AA of the FOI Act in relation to your
request.
The reasons for this practical refusal and the actions required by you are
set out in the attached letter.
(See attached file: 20161720.pdf)
The FOI Act requires that you respond to this notice before the end of the
consultation period, which is 2 September 2016.
Kind regards,
FOI Contact Officer
Australian Bureau of Statistics
(P) (02) 6252 7203
(E) [1][ABS request email] (W) [2]www.abs.gov.au
References
Visible links
1. mailto:[email address]
2. http://www.abs.gov.au/
Dear Keelie Williams,
I just called but your colleague mentioned you weren't working today. I have some follow up enquiries that will help me narrow the scope of the request.
On Monday I sought a response through the ABS media team to the following questions in relation to your correspondence:
-Is the advice provided to me on Friday accurate?
-Why will it take 95 hours to find these server logs?
-How did the ABS come to the figures it provided (95 hours searching, 1900 pages, 159 hours of consideration)?
-Can you provide the worksheet completed in order to provide the estimate for the time required provided in the FOI response?
-Even if there were 1000 server request records printed on each page (far higher than what could be readable), that is only 1,900,000 requests for the day - far lower than the level of traffic understood to have been targeted at the website. Why do these estimates suggest there were so few web requests on Census day?
-Why does the ABS need to consider server logs for 159 hours? What in the logs does the ABS believe might be exempted under the FOI act?
Are you able to assist with the above enquiries to help me reduce the scope? In particular, providing the worksheet would help me understand and avoid the most onerous components of the request.
The ABS media team could not provide much detail on each enquiry, and instead provided me with the following statement:
"The ABS has denied your FOI request in accordance with Section 24AA of the Freedom of Information Act 1982, as it is likely to divert resources away from the ABS’ business. The ABS actively engages with applicants to narrow the scope of documents sought.
"The ABS assessment of your FOI application includes consideration of the time required to search personal holdings and workplaces databases that may contain extracts of raw server logs. Server logs do not readily translate to a document page count estimate. The resource estimate for consideration includes the process required to redact parts or whole server logs for privacy and security reasons.
"The decision to practically refuse this request is consistent with the Freedom of Information Act 1982 and the Freedom of Information Guidelines issued by the Office of the Australian Information Commissioner."
I note in the second paragraph that the ABS assessment included consideration of the time required to search personal holdings and workplaces databases. I believe this may be an error, as Ms Williams has made separate estimates for searching (95 hours) and consideration (159 hours). Can you confirm whether the 95 hours was just for searching and not for consideration?
Finally, can you provide any guidance as to what you believe would be an acceptable scope of the request? My central inquiry here is to investigate the level of traffic received by the ABS over the course of 9 August. In this respect I am seeking the server logs only from the central ABS logs folder, program or repository that will provide an exhaustive record of the requests received. These logs should not raise any exemptions under the FOI act, as each line should resemble something like:
192.168.123.121 - - [10/Jul/2016 15:04:26] "GET /upload/file.zip HTTP/1.1" 200
Look forward to hearing from you.
Yours sincerely,
Jackson Gothe-Snape
Dear Keelie Williams,
Thanks for phoning.
I would like to narrow the scope of my request to:
-The final 100 raw server log rows made by successful visitor requests to the Census (not ABS) website on 9 August 2016, and
-The documents provided by the ABS to the Department of Prime Minister and Cabinet containing advice on what happened on Census day, including traffic charts and information from IBM.
Yours sincerely,
Jackson Gothe-Snape
Dear Mr Gothe-Snape
Please find attached the notification of charges estimate for your Freedom
of Information (FOI) request.
(See attached file: FOI 201617-20.pdf)
If you have any queries on this matter please contact the ABS FOI Contact
Officer at [1][ABS request email] or on (02) 6252 7203.
Kind Regards
FOI Contact Officer
Policy, Legislation and Assurance | Governance and Parliamentary Liaison
Branch | Australian Bureau of Statistics
(P) (02) 6252 7203 (E) [2][ABS request email] (W)
[3]www.abs.gov.au
The [4]ABS Privacy Policy outlines how the ABS handles any personal
information that you provide to us.
References
Visible links
1. mailto:[ABS request email]
2. mailto:[ABS request email]
3. http://www.abs.gov.au/
4. http://www.abs.gov.au/privacy
Dear Mr Gothe-Snape
This email refers to your FOI request (201617/38) received on 31 August
2016.
The purpose of this email is to advise you, as required under Section 24AB
of the Freedom of Information Act 1982 (FOI Act) that I consider a
practical refusal reason exists under Section 24AA of the FOI Act in
relation to your
request.
The reasons and the actions required by you are provided in the attached
letter.
(See attached file: FOI20161738_Practical Refusal.pdf)
The FOI Act requires that you respond to this notice before the end of the
consultation period, which is 14 days from the date of this email.
Please contact the FOI Officer at [1][ABS request email] or on
(02) 6252 7203 if you have any queries.
Kind regards
FOI Contact Officer
Australian Bureau of Statistics
(P) (02) 6252 7203
(E) [2][ABS request email] (W) [3]www.abs.gov.au
References
Visible links
1. mailto:[email address]
2. mailto:[email address]
3. http://www.abs.gov.au/
Dear Emily Covington,
In relation to the first FOI request as part of this correspondence, I have paid the deposit and have sent confirmation of the transfer to [email address].
In relation to the second request subject to the most recent practical refusal, can I please narrow the scope of my request to:
Emails provided on August 10th by David Kalisch or staff in his office to the DPMC explaining what happened to the Census website on August 9th.
Yours sincerely,
Jackson Gothe-Snape
Dear Mr Gothe-Snape
Thank you for your reply.
With regard to your revised scope, could you please clarify what you mean
by 'in his office'.
Kind Regards
FOI Contact Officer
Australian Bureau of Statistics
(P) (02) 6252 7203
(E) [1][ABS request email] (W) [2]www.abs.gov.au
[3]Inactive hide details for Jackson Gothe-Snape ---19/09/2016 12:19:04
PM---Dear Emily Covington, In relation to the first FOI reJackson
Gothe-Snape ---19/09/2016 12:19:04 PM---Dear Emily Covington, In relation
to the first FOI request as part of this correspondence, I have pa
From: Jackson Gothe-Snape <[FOI #2118 email]>
To: Emily Covington <[email address]>,
Date: 19/09/2016 12:19 PM
Subject: Re: FOI 201617/38 [DLM=Sensitive]
--------------------------------------------------------------------------
Dear Emily Covington,
In relation to the first FOI request as part of this correspondence, I
have paid the deposit and have sent confirmation of the transfer to
[ABS request email].
In relation to the second request subject to the most recent practical
refusal, can I please narrow the scope of my request to:
Emails provided on August 10th by David Kalisch or staff in his office to
the DPMC explaining what happened to the Census website on August 9th.
Yours sincerely,
Jackson Gothe-Snape
Dear Emily Covington,
Thanks for getting back to me seeking to clarify the request.
I intend my request to cover emails sent by Mr Kalisch himself or on his behalf, for example by his executive assistants and liaison officers.
Yours sincerely,
Jackson Gothe-Snape
Our reference: RQ16/04150
Agency reference: FOI201617/20
Mr Jackson Gothe-Snape
By email: [1][FOI #2118 email]
Dear Mr Gothe-Snape
Extension of time request under s15AB
On 5 October 2016, the Australian Bureau of Statistics (the ABS) requested
further time to make a decision on your FOI request of 10 August 2016.
This request was on the basis that the processing period is insufficient
to deal adequately with your request, because it is complex or voluminous.
The Information Commissioner has decided to grant the ABS an extension of
time to process your request to 9 November 2016. This decision has been
made under s15AB(2) of the Freedom of Information Act 1982 (Cth) (the FOI
Act).
By granting further time to process your request, it is anticipated that
the ABS will provide a well-reasoned and better managed decision.
Contact
If you have any questions regarding this email please contact the OAIC on
1300 363 992 or via direct email at [2][email address]. Please
quote OAIC reference number RQ16/04150 in all correspondence.
Yours sincerely
Irene Nicolaou | Assistant Investigation and Review Officer | Freedom of
Information Dispute Resolution
Office of the Australian Information Commissioner
Level 3, 175 Pitt Street, SYDNEY NSW 2000
GPO Box 5128 SYDNEY NSW 2001| [3]www.oaic.gov.au
Phone: +61 2 9284 9605 | E-mail: [4][email address]
Protecting information rights – advancing information policy
References
Visible links
1. mailto:[FOI #2118 email]
2. mailto:[email address]
3. http://www.oaic.gov.au/
4. mailto:[email address]
Dear Mr Gothe-Snape
Please find attached the formal response to your Freedom of Information
(FOI) request.
(See attached file: FOI 201617-38 - Decision Letter.pdf)
If you have any queries on this matter please contact the ABS FOI Contact
Officer at [1][ABS request email] or on (02) 6252 7203.
Regards
FOI Contact Officer
Policy, Legislation and Assurance | Governance and Parliamentary Liaison
Branch | Australian Bureau of Statistics
(P) (02) 6252 7203 (E) [2][ABS request email] (W)
[3]www.abs.gov.au
The [4]ABS Privacy Policy outlines how the ABS handles any personal
information that you provide to us.
References
Visible links
1. mailto:[ABS request email]
2. mailto:[ABS request email]
3. http://www.abs.gov.au/
4. http://www.abs.gov.au/privacy
Dear Mr Gothe-Snape
The response to your FOI request (201617/20) is in the final stages of
preparation for release on 9 November 2016 subject to your payment of the
outstanding charges.
To date we have received your deposit of $45 and would ask that you pay
the outstanding amount of $142 within the next 7 days.
If you have any queries on this matter please contact the ABS FOI Contact
Officer at [1][ABS request email] or on (02) 6252 7203.
Kind Regards
FOI Contact Officer
Policy, Legislation and Assurance | Governance and Parliamentary Liaison
Branch | Australian Bureau of Statistics
(P) (02) 6252 7203 (E) [2][ABS request email] (W)
[3]www.abs.gov.au
The [4]ABS Privacy Policy outlines how the ABS handles any personal
information that you provide to us.
References
Visible links
1. mailto:[ABS request email]
2. mailto:[ABS request email]
3. http://www.abs.gov.au/
4. http://www.abs.gov.au/privacy
Dear Caroline Blink,
I have made the payment and forwarded confirmation of the payment to [email address].
Thanks for your assistance.
Yours sincerely,
Jackson Gothe-Snape
Dear Mr Gothe-Snape,
In response to your FOI request received on 31 August 2016, please find
attached the formal response from the ABS.
Should you have any questions regarding this please feel free to contact
the ABS FOI Contact Officer on: 02 6252 7203 or email:
[1][ABS request email]
(See attached file: FOI 201617-20 Documents for release.pdf)
Kind Regards
FOI Contact Officer
Australian Bureau of Statistics
(P) (02) 6252 7203
(E) [2][ABS request email] (W) [3]www.abs.gov.au
The ABS Privacy Policy outlines how the ABS handles any personal
information that you provide to us.
References
Visible links
1. mailto:[ABS request email]
2. mailto:[ABS request email]
3. http://www.abs.gov.au/
Dear Danielle Gillett,
Thank you for this. Can you please provide the spreadsheet in xls, xlsx, or csv format.
Yours sincerely,
Jackson Gothe-Snape
Good Morning Mr Gothe-Snape,
Please find the attached release in xlsx format as requested, noting we
have not included the 'Source IP Address' column as this was redacted in
its entirety in the original release.
(See attached file: FOI_9 November 2016.xlsx)
Should you have any questions regarding this please feel free to contact
the ABS FOI Contact Officer on: 02 6252 7203 or email:
[1][ABS request email]
Kind Regards
FOI Contact Officer
Australian Bureau of Statistics
(P) (02) 6252 7203
(E) [2][ABS request email] (W) [3]www.abs.gov.au
The ABS Privacy Policy outlines how the ABS handles any personal
information that you provide to us.
[4]Inactive hide details for Jackson Gothe-Snape ---09/11/2016 06:28:04
PM---Dear Danielle Gillett, Thank you for this. Can you plJackson
Gothe-Snape ---09/11/2016 06:28:04 PM---Dear Danielle Gillett, Thank you
for this. Can you please provide the spreadsheet in xls, xlsx, or c
From: Jackson Gothe-Snape <[FOI #2118 email]>
To: Danielle Gillett <[email address]>,
Date: 09/11/2016 06:28 PM
Subject: Re: FOI 201617/20 Response
--------------------------------------------------------------------------
Dear Danielle Gillett,
Thank you for this. Can you please provide the spreadsheet in xls, xlsx,
or csv format.
Yours sincerely,
Jackson Gothe-Snape
Locutus Sum left an annotation ()
I have classified this request to say "Some of the information has been received." In fact, it is a little bit difficult to clasify because the applicant has mentioned two different FOI requests on this web page. One request is in the email from 20160810 https://www.righttoknow.org.au/request/s... . This request was also later changed and limited because of a practical refusal reason. I cannot decide if the second FOI request is also on this page or is on another page.
Our reference: RQ16/05402
Agency reference: FOI 12492
By email: [1][FOI #2118 email]
Dear Mr Gothe-Snape
Extension of time request under s15AB
On 13 December 2016, the Department of Veterans’ Affairs (the Department)
requested further time to make a decision on your FOI request of 15
November 2016.
This request was on the basis that the processing period is insufficient
to deal adequately with your request, because it is complex or voluminous.
The Information Commissioner has decided to grant the Department an
extension of time to process your request to 23 December 2016. This
decision has been made under s15AB(2) of the Freedom of Information Act
1982 (Cth) (the FOI Act).
By granting further time it is anticipated that the Department will
provide a well-reasoned and better managed decision.
Contact
If you have any questions regarding this email, please contact me on 02
9284 9605 or via email [email address]. Please quote OAIC
reference number above in all correspondence.
Yours sincerely
Irene Nicolaou | Assistant Investigation and Review Officer | Freedom of
Information Dispute Resolution
Office of the Australian Information Commissioner
Level 3, 175 Pitt Street, SYDNEY NSW 2000
GPO Box 5128 SYDNEY NSW 2001| [2]www.oaic.gov.au
Phone: +61 2 9284 9605 | E-mail: [3][email address]
Protecting information rights – advancing information policy
References
Visible links
1. mailto:[FOI #2118 email]
2. http://www.oaic.gov.au/
3. mailto:[email address]
observant1 left an annotation ()
Evidently the ABS doesn't employ any linux administrators, a simple terminal command should export this data to a text file in around 30 seconds.