Datum a čas: 2022-03-31 02:10 CEST
Očekavaná délka: 30 minut
Oznámení se týká serverů: node5.prg
Typ výpadku: vps_reset
Důvod: Kernel panic
Výpadek řeší: Jakub Skokan
node5.prg se v noci restartoval, pravděpodobně kernel panic.
ENGLISH:
Date and time: 2022-03-31 02:10 CEST
Expected duration: 30 minutes
Affected systems: node5.prg
Outage type: vps_reset
Reason: Kernel panic
Handled by: Jakub Skokan
node5.prg rebooted itself during the night, most likely due to a kernel panic.
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6ODgxLCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAyMi0w
My0zMVQwMjoxMDowMCswMjowMCIsImR1cmF0aW9uIjozMCwidHlwZSI6InZw
c19yZXNldCIsImVudGl0aWVzIjpbeyJuYW1lIjoiTm9kZSIsImlkIjoxMDUs
ImxhYmVsIjoibm9kZTUucHJnIn1dLCJoYW5kbGVycyI6WyJKYWt1YiBTa29r
YW4iXSwidHJhbnNsYXRpb25zIjp7ImVuIjp7InN1bW1hcnkiOiJLZXJuZWwg
cGFuaWMiLCJkZXNjcmlwdGlvbiI6Im5vZGU1LnByZyByZWJvb3RlZCBpdHNl
bGYgZHVyaW5nIHRoZSBuaWdodCwgbW9zdCBsaWtlbHkgZHVlIHRvIGEga2Vy
bmVsIHBhbmljLiJ9LCJjcyI6eyJzdW1tYXJ5IjoiS2VybmVsIHBhbmljIiwi
ZGVzY3JpcHRpb24iOiJub2RlNS5wcmcgc2UgdiBub2NpIHJlc3RhcnRvdmFs
LCBwcmF2ZMSbcG9kb2JuxJsga2VybmVsIHBhbmljLiJ9fX0=
-----END BASE64 ENCODED PARSEABLE JSON-----
Datum a čas: 2022-03-29 12:34 CEST
Očekavaná délka: 90 minut
Oznámení se týká serverů: node7.prg, node8.prg, node13.prg, node15.prg, node16.prg, node18.prg, node19.prg, node20.prg, node21.prg, node5.brq, node1.pgnd
Typ výpadku: vps_reset
Důvod: Selhalo live patchovani jadra
Výpadek řeší: Pavel Šnajdr, Jakub Skokan, Martin Myška
pri pokusu zive opravit CVE-2022-1016, se nektere nody livelocknuly, coz vynutilo potrebu jejich resetovani
nebylo to mezi ocekavanymi scenari, plyne z toho nekolik pouceni do budoucna...
omlouvame se :(
ENGLISH:
Date and time: 2022-03-29 12:34 CEST
Expected duration: 90 minutes
Affected systems: node7.prg, node8.prg, node13.prg, node15.prg, node16.prg, node18.prg, node19.prg, node20.prg, node21.prg, node5.brq, node1.pgnd
Outage type: vps_reset
Reason: Failed kernel live patching
Handled by: Pavel Šnajdr, Jakub Skokan, Martin Myška
in an attempt to live-patch CVE-2022-1016, some nodes became live-locked, prompting a reset of those systems
this wasn't expected to happen, lessons are being learned from that...
apologies :(
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6ODgwLCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAyMi0w
My0yOVQxMjozNDowMCswMjowMCIsImR1cmF0aW9uIjo5MCwidHlwZSI6InZw
c19yZXNldCIsImVudGl0aWVzIjpbeyJuYW1lIjoiTm9kZSIsImlkIjoxMDgs
ImxhYmVsIjoibm9kZTcucHJnIn0seyJuYW1lIjoiTm9kZSIsImlkIjoxMDks
ImxhYmVsIjoibm9kZTgucHJnIn0seyJuYW1lIjoiTm9kZSIsImlkIjoxMTQs
ImxhYmVsIjoibm9kZTEzLnByZyJ9LHsibmFtZSI6Ik5vZGUiLCJpZCI6MTE2
LCJsYWJlbCI6Im5vZGUxNS5wcmcifSx7Im5hbWUiOiJOb2RlIiwiaWQiOjEx
NywibGFiZWwiOiJub2RlMTYucHJnIn0seyJuYW1lIjoiTm9kZSIsImlkIjox
MTksImxhYmVsIjoibm9kZTE4LnByZyJ9LHsibmFtZSI6Ik5vZGUiLCJpZCI6
MTIwLCJsYWJlbCI6Im5vZGUxOS5wcmcifSx7Im5hbWUiOiJOb2RlIiwiaWQi
OjEyMSwibGFiZWwiOiJub2RlMjAucHJnIn0seyJuYW1lIjoiTm9kZSIsImlk
IjoxMjIsImxhYmVsIjoibm9kZTIxLnByZyJ9LHsibmFtZSI6Ik5vZGUiLCJp
ZCI6MjE0LCJsYWJlbCI6Im5vZGU1LmJycSJ9LHsibmFtZSI6Ik5vZGUiLCJp
ZCI6MzAwLCJsYWJlbCI6Im5vZGUxLnBnbmQifV0sImhhbmRsZXJzIjpbIlBh
dmVsIMWgbmFqZHIiLCJKYWt1YiBTa29rYW4iLCJNYXJ0aW4gTXnFoWthIl0s
InRyYW5zbGF0aW9ucyI6eyJlbiI6eyJzdW1tYXJ5IjoiRmFpbGVkIGtlcm5l
bCBsaXZlIHBhdGNoaW5nIiwiZGVzY3JpcHRpb24iOiJpbiBhbiBhdHRlbXB0
IHRvIGxpdmUtcGF0Y2ggQ1ZFLTIwMjItMTAxNiwgc29tZSBub2RlcyBiZWNh
bWUgbGl2ZS1sb2NrZWQsIHByb21wdGluZyBhIHJlc2V0IG9mIHRob3NlIHN5
c3RlbXNcclxuXHJcbnRoaXMgd2Fzbid0IGV4cGVjdGVkIHRvIGhhcHBlbiwg
bGVzc29ucyBhcmUgYmVpbmcgbGVhcm5lZCBmcm9tIHRoYXQuLi5cclxuXHJc
bmFwb2xvZ2llcyA6KCJ9LCJjcyI6eyJzdW1tYXJ5IjoiU2VsaGFsbyBsaXZl
IHBhdGNob3ZhbmkgamFkcmEiLCJkZXNjcmlwdGlvbiI6InByaSBwb2t1c3Ug
eml2ZSBvcHJhdml0IENWRS0yMDIyLTEwMTYsIHNlIG5la3RlcmUgbm9keSBs
aXZlbG9ja251bHksIGNveiB2eW51dGlsbyBwb3RyZWJ1IGplamljaCByZXNl
dG92YW5pXHJcblxyXG5uZWJ5bG8gdG8gbWV6aSBvY2VrYXZhbnltaSBzY2Vu
YXJpLCBwbHluZSB6IHRvaG8gbmVrb2xpayBwb3VjZW5pIGRvIGJ1ZG91Y25h
Li4uXHJcblxyXG5vbWxvdXZhbWUgc2UgOigifX19
-----END BASE64 ENCODED PARSEABLE JSON-----
