5 Jun
2022
5 Jun
'22
7:35 a.m.
Stav: -> closed
Popis: Pár slov k výpadku
Na našem blogu jsme uvedli více informací ke čtvrtečnímu výpadku napájení
v Praze. Blog obsahuje také vyjádření, které jsme obdrželi od MasterDC:
https://blog.vpsfree.cz/post-mortem-par-slov-k-vypadku/
Nahlásil: Jakub Skokan
ENGLISH:
State: -> closed
Summary: A few words regarding the outage
We have published a blog post with more information about the power outage
on Thursday. It also contains a communication we've received from MasterDC.
https://blog.vpsfree.cz/post-mortem-par-slov-k-vypadku/
Since the blog post is in Czech, I include a short translation in English.
The outage was connected to power loss in Prague city, not just the datacenter.
While the DC has power backups, they failed and the first power line went down.
At this time, we've continued to run on the second power line. As the DC was
reconnecting selected devices to the second power line, a short circuit has
occurred and the second power line went down as well.
Everything we have in Prague was thus shut down, including our email support
and tools that we otherwise use to communicate with our members. We've immediately
set out to the DC to sort out the issues on the spot. The outage was further
complicated by slowly booting switches. Most of our nodes are booted from PXE
and since the nodes were up faster than the network, they failed to boot. This
led to an additional delay, as we've had to reset them again. We've also lost
one 10G switch during the outage.
We're working on making the PXE server available faster to avoid the boot issues
in the future. We appreciate MasterDC's response, as they were open about the
situation.
Reported by: Jakub Skokan
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6MjM3MywiY2hhbmdlcyI6eyJzdGF0ZSI6eyJmcm9tIjpudWxsLCJ0
byI6ImNsb3NlZCJ9fSwidHJhbnNsYXRpb25zIjp7ImVuIjp7InN1bW1hcnki
OiJBIGZldyB3b3JkcyByZWdhcmRpbmcgdGhlIG91dGFnZSIsImRlc2NyaXB0
aW9uIjoiV2UgaGF2ZSBwdWJsaXNoZWQgYSBibG9nIHBvc3Qgd2l0aCBtb3Jl
IGluZm9ybWF0aW9uIGFib3V0IHRoZSBwb3dlciBvdXRhZ2Vcclxub24gVGh1
cnNkYXkuIEl0IGFsc28gY29udGFpbnMgYSBjb21tdW5pY2F0aW9uIHdlJ3Zl
IHJlY2VpdmVkIGZyb20gTWFzdGVyREMuXHJcblxyXG5odHRwczovL2Jsb2cu
dnBzZnJlZS5jei9wb3N0LW1vcnRlbS1wYXItc2xvdi1rLXZ5cGFka3UvXHJc
blxyXG5TaW5jZSB0aGUgYmxvZyBwb3N0IGlzIGluIEN6ZWNoLCBJIGluY2x1
ZGUgYSBzaG9ydCB0cmFuc2xhdGlvbiBpbiBFbmdsaXNoLlxyXG5cclxuVGhl
IG91dGFnZSB3YXMgY29ubmVjdGVkIHRvIHBvd2VyIGxvc3MgaW4gUHJhZ3Vl
IGNpdHksIG5vdCBqdXN0IHRoZSBkYXRhY2VudGVyLlxyXG5XaGlsZSB0aGUg
REMgaGFzIHBvd2VyIGJhY2t1cHMsIHRoZXkgZmFpbGVkIGFuZCB0aGUgZmly
c3QgcG93ZXIgbGluZSB3ZW50IGRvd24uXHJcbkF0IHRoaXMgdGltZSwgd2Un
dmUgY29udGludWVkIHRvIHJ1biBvbiB0aGUgc2Vjb25kIHBvd2VyIGxpbmUu
IEFzIHRoZSBEQyB3YXNcclxucmVjb25uZWN0aW5nIHNlbGVjdGVkIGRldmlj
ZXMgdG8gdGhlIHNlY29uZCBwb3dlciBsaW5lLCBhIHNob3J0IGNpcmN1aXQg
aGFzXHJcbm9jY3VycmVkIGFuZCB0aGUgc2Vjb25kIHBvd2VyIGxpbmUgd2Vu
dCBkb3duIGFzIHdlbGwuXHJcblxyXG5FdmVyeXRoaW5nIHdlIGhhdmUgaW4g
UHJhZ3VlIHdhcyB0aHVzIHNodXQgZG93biwgaW5jbHVkaW5nIG91ciBlbWFp
bCBzdXBwb3J0XHJcbmFuZCB0b29scyB0aGF0IHdlIG90aGVyd2lzZSB1c2Ug
dG8gY29tbXVuaWNhdGUgd2l0aCBvdXIgbWVtYmVycy4gV2UndmUgaW1tZWRp
YXRlbHlcclxuc2V0IG91dCB0byB0aGUgREMgdG8gc29ydCBvdXQgdGhlIGlz
c3VlcyBvbiB0aGUgc3BvdC4gVGhlIG91dGFnZSB3YXMgZnVydGhlclxyXG5j
b21wbGljYXRlZCBieSBzbG93bHkgYm9vdGluZyBzd2l0Y2hlcy4gTW9zdCBv
ZiBvdXIgbm9kZXMgYXJlIGJvb3RlZCBmcm9tIFBYRVxyXG5hbmQgc2luY2Ug
dGhlIG5vZGVzIHdlcmUgdXAgZmFzdGVyIHRoYW4gdGhlIG5ldHdvcmssIHRo
ZXkgZmFpbGVkIHRvIGJvb3QuIFRoaXNcclxubGVkIHRvIGFuIGFkZGl0aW9u
YWwgZGVsYXksIGFzIHdlJ3ZlIGhhZCB0byByZXNldCB0aGVtIGFnYWluLiBX
ZSd2ZSBhbHNvIGxvc3Rcclxub25lIDEwRyBzd2l0Y2ggZHVyaW5nIHRoZSBv
dXRhZ2UuXHJcblxyXG5XZSdyZSB3b3JraW5nIG9uIG1ha2luZyB0aGUgUFhF
IHNlcnZlciBhdmFpbGFibGUgZmFzdGVyIHRvIGF2b2lkIHRoZSBib290IGlz
c3Vlc1xyXG5pbiB0aGUgZnV0dXJlLiBXZSBhcHByZWNpYXRlIE1hc3RlckRD
J3MgcmVzcG9uc2UsIGFzIHRoZXkgd2VyZSBvcGVuIGFib3V0IHRoZVxyXG5z
aXR1YXRpb24uXHJcbiJ9LCJjcyI6eyJzdW1tYXJ5IjoiUMOhciBzbG92IGsg
dsO9cGFka3UiLCJkZXNjcmlwdGlvbiI6Ik5hIG5hxaFlbSBibG9ndSBqc21l
IHV2ZWRsaSB2w61jZSBpbmZvcm1hY8OtIGtlIMSNdHZydGXEjW7DrW11IHbD
vXBhZGt1IG5hcMOhamVuw61cclxudiBQcmF6ZS4gQmxvZyBvYnNhaHVqZSB0
YWvDqSB2eWrDoWTFmWVuw60sIGt0ZXLDqSBqc21lIG9iZHLFvmVsaSBvZCBN
YXN0ZXJEQzpcclxuXHJcbmh0dHBzOi8vYmxvZy52cHNmcmVlLmN6L3Bvc3Qt
bW9ydGVtLXBhci1zbG92LWstdnlwYWRrdS9cclxuIn19fQ==
-----END BASE64 ENCODED PARSEABLE JSON-----