UA ccTLD Resiliency During the War Dmitry
Description: UA ccTLD Resiliency During the War Dmitry Kohmanyuk :: Hostmaster.UA UADOM :: Kyiv 2022.12.01 .UA 30th anniversary Disclaimer This story is based on real events Any errors or omissions are mine Russian troops may be harmed during the
Related Topics
Download Presentation
"UA ccTLD Resiliency During the War Dmitry" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. UA ccTLD Resiliency
During the War Dmitry Kohmanyuk :: Hostmaster.UA
UADOM :: Kyiv 2022.12.01
.UA 30th anniversary ‹#›<br>
slide2. Disclaimer This story is based on real events
Any errors or omissions are mine
Russian troops may be harmed during the presentation
Key message is: prepare early, keep your tools sharp ‹#›<br>
slide3. Preparation
2021-12-01 ‹#›<br>
slide4. Preparation Internal discussions started in late 2021
Mostly focused on redundancy, disaster preparedness
Focused on DNS service; most visible and impacting
Considered multiple outsourcing companies bidding
Several were in line up, only one tested
OOB communication: Signal and Google mail/docs
Weekly change routine established ‹#›<br>
slide5. DDOS Attack
2022-02-15 ‹#›<br>
slide6. Impact DNS Service for UA TLD and GOV.UA domains server
Took out one of our anycast nodes…
…That was also zone transfer server
Impact: none of other UA zones did update
Lesson learned: separate public and private
Used Signal chat already established for ops team
Anycast fortunately remained available, mostly ‹#›<br>
slide7. Post-Impact Deployed new anycast service at night…
…which was configured incorrectly…
…which was fixed after I contacted CEO on messenger
Lesson learned: know your CEO’s direct contact
Press release about the attack
Created post-mortem write up, entire team participated
Created spare transfer server on unused host we had ‹#›<br>
slide8. Military Attack
2022-02-24 ‹#›<br>
slide9. Events 04:00 (like in 1941) Kyiv bombings started
I was awake at 06:00, accidentally
First reaction was denial and panic
Next was to call everyone in my team
I assessed the situation and created “to save” list
For major services, I had allocated a backup location
Signal team chat was used to communicate ‹#›<br>
slide10. Priorities ‹#›<br>
slide11. Priorities PEOPLE
DATA
SERVICES
MONEY ‹#›<br>
slide12. Communication ‹#›<br>
slide13. Communication: who, what Team (CHANGES)
Customers/Partners (IMPACT)
Public (SHORT)
VIP: Government, LOA, CERT, … ‹#›<br>
slide14. Components ‹#›<br>
slide15. Components PEOPLE
EPP service, back end database
DNSSEC Signing and key management, zone generation
DNS Service for TLD and our own domains
WHOIS and RDAP services
Websites for public, registrars, government, …
Email, chat, phone*, for support ‹#›<br>
slide16. Components, continued 8. Datacenter space, internet, networking hardware
9. Development infrastructure (Git)
10. DDOS Protection Services **
11. Cloud services ***
12. Business back office (accounting, ticketing system)
13. BACKUPS ‹#›<br>
slide17. Decisions ‹#›<br>
slide18. Outsource or not? Hardware, datacenter: YES and YES
DNS secondary service: YES – we got several
Registry, EPP and WHOIS: NO
Our business and financial operations - NO
Virtual servers - prefer our own virtualization
DNS primary and DNSSEC signing - NO
Calendars, documents, email – YES (Google Workspace) ‹#›<br>
slide19. Lessons ‹#›<br>
slide20. Lessons Must have: server hosting company, multiple locations
Reach out to lots of people, select few to work with
Even with free help, keep track of estimated costs
People are more valuable than computers
Time is more valuable than money
Smaller companies usually react faster
Those that knew us already, were more helpful ‹#›<br>
slide21. Netnod ‹#›<br>
slide22. Netnod contacted Netnod senior employees
received assurance of help, Signal group created
zones setup within 96 hours
Migration of infrastructure in stages
communication: signal chat, Google docs, emails
management and government was involved
Partner domains (Kharkiv.UA, Odesa.UA) included
Nice visibility, statistics ‹#›<br>
slide23. CZ.NIC ‹#›<br>
slide24. CZ.NIC contacted CZNIC CEO
received assurance of help, CTO involved
first servers configured within 72 hours
Migration of infrastructure in many stages
communication: signal chat/calls, Google docs, email
DNS cluster designed, then re-designed again
Partner project (Secondary.net.UA) included
Pure IPv6 uplink to DNS node; dual FRR, VRRP ‹#›<br>
slide25. Gratitude ‹#›<br>
slide26. Acknowledgements 6connect, Anycast DNS: CloudNS, CDNS, Cloudflare, Gransy, Netnod, Packet Clearing House, RcodeZero
CZ.NIC for hosting our core infrastructure
Our colocation partners in Ukraine and abroad
IANA staff, for updating .UA NS on Sunday
CENTR, for suspending .RU membership twice
Netnod, RIPE, CSNOG (+3) for inviting to their meetings ‹#›<br>
slide27. Gratitude My fellow colleagues, all of you
Our hardware and services suppliers, acting quickly
Staff, management, and members of CZ.NIC z.s.p.o.
MFA of Sweden: Utrikesdepartementet
Global NOG Alliance for helping ISPs in Ukraine
6connect staff for deployment of custom anycast cloud
Many members of ccTLD and RIPE communities
Ukrainian armed forces (MIL.UA) ‹#›<br>
slide28. Questions? Dmitry Kohmanyuk <dk@cctld.ua>
Phone/Signal: +380503814604
Hostmaster.UA
Running UA ccTLD since 2001
Under Russian state military attacks since 2014 ‹#›<br>
During the War Dmitry Kohmanyuk :: Hostmaster.UA
UADOM :: Kyiv 2022.12.01
.UA 30th anniversary ‹#›<br>
slide2. Disclaimer This story is based on real events
Any errors or omissions are mine
Russian troops may be harmed during the presentation
Key message is: prepare early, keep your tools sharp ‹#›<br>
slide3. Preparation
2021-12-01 ‹#›<br>
slide4. Preparation Internal discussions started in late 2021
Mostly focused on redundancy, disaster preparedness
Focused on DNS service; most visible and impacting
Considered multiple outsourcing companies bidding
Several were in line up, only one tested
OOB communication: Signal and Google mail/docs
Weekly change routine established ‹#›<br>
slide5. DDOS Attack
2022-02-15 ‹#›<br>
slide6. Impact DNS Service for UA TLD and GOV.UA domains server
Took out one of our anycast nodes…
…That was also zone transfer server
Impact: none of other UA zones did update
Lesson learned: separate public and private
Used Signal chat already established for ops team
Anycast fortunately remained available, mostly ‹#›<br>
slide7. Post-Impact Deployed new anycast service at night…
…which was configured incorrectly…
…which was fixed after I contacted CEO on messenger
Lesson learned: know your CEO’s direct contact
Press release about the attack
Created post-mortem write up, entire team participated
Created spare transfer server on unused host we had ‹#›<br>
slide8. Military Attack
2022-02-24 ‹#›<br>
slide9. Events 04:00 (like in 1941) Kyiv bombings started
I was awake at 06:00, accidentally
First reaction was denial and panic
Next was to call everyone in my team
I assessed the situation and created “to save” list
For major services, I had allocated a backup location
Signal team chat was used to communicate ‹#›<br>
slide10. Priorities ‹#›<br>
slide11. Priorities PEOPLE
DATA
SERVICES
MONEY ‹#›<br>
slide12. Communication ‹#›<br>
slide13. Communication: who, what Team (CHANGES)
Customers/Partners (IMPACT)
Public (SHORT)
VIP: Government, LOA, CERT, … ‹#›<br>
slide14. Components ‹#›<br>
slide15. Components PEOPLE
EPP service, back end database
DNSSEC Signing and key management, zone generation
DNS Service for TLD and our own domains
WHOIS and RDAP services
Websites for public, registrars, government, …
Email, chat, phone*, for support ‹#›<br>
slide16. Components, continued 8. Datacenter space, internet, networking hardware
9. Development infrastructure (Git)
10. DDOS Protection Services **
11. Cloud services ***
12. Business back office (accounting, ticketing system)
13. BACKUPS ‹#›<br>
slide17. Decisions ‹#›<br>
slide18. Outsource or not? Hardware, datacenter: YES and YES
DNS secondary service: YES – we got several
Registry, EPP and WHOIS: NO
Our business and financial operations - NO
Virtual servers - prefer our own virtualization
DNS primary and DNSSEC signing - NO
Calendars, documents, email – YES (Google Workspace) ‹#›<br>
slide19. Lessons ‹#›<br>
slide20. Lessons Must have: server hosting company, multiple locations
Reach out to lots of people, select few to work with
Even with free help, keep track of estimated costs
People are more valuable than computers
Time is more valuable than money
Smaller companies usually react faster
Those that knew us already, were more helpful ‹#›<br>
slide21. Netnod ‹#›<br>
slide22. Netnod contacted Netnod senior employees
received assurance of help, Signal group created
zones setup within 96 hours
Migration of infrastructure in stages
communication: signal chat, Google docs, emails
management and government was involved
Partner domains (Kharkiv.UA, Odesa.UA) included
Nice visibility, statistics ‹#›<br>
slide23. CZ.NIC ‹#›<br>
slide24. CZ.NIC contacted CZNIC CEO
received assurance of help, CTO involved
first servers configured within 72 hours
Migration of infrastructure in many stages
communication: signal chat/calls, Google docs, email
DNS cluster designed, then re-designed again
Partner project (Secondary.net.UA) included
Pure IPv6 uplink to DNS node; dual FRR, VRRP ‹#›<br>
slide25. Gratitude ‹#›<br>
slide26. Acknowledgements 6connect, Anycast DNS: CloudNS, CDNS, Cloudflare, Gransy, Netnod, Packet Clearing House, RcodeZero
CZ.NIC for hosting our core infrastructure
Our colocation partners in Ukraine and abroad
IANA staff, for updating .UA NS on Sunday
CENTR, for suspending .RU membership twice
Netnod, RIPE, CSNOG (+3) for inviting to their meetings ‹#›<br>
slide27. Gratitude My fellow colleagues, all of you
Our hardware and services suppliers, acting quickly
Staff, management, and members of CZ.NIC z.s.p.o.
MFA of Sweden: Utrikesdepartementet
Global NOG Alliance for helping ISPs in Ukraine
6connect staff for deployment of custom anycast cloud
Many members of ccTLD and RIPE communities
Ukrainian armed forces (MIL.UA) ‹#›<br>
slide28. Questions? Dmitry Kohmanyuk <dk@cctld.ua>
Phone/Signal: +380503814604
Hostmaster.UA
Running UA ccTLD since 2001
Under Russian state military attacks since 2014 ‹#›<br>