Datum a čas: 2022-03-29 12:34 CEST Očekavaná délka: 90 minut Oznámení se týká serverů: node7.prg, node8.prg, node13.prg, node15.prg, node16.prg, node18.prg, node19.prg, node20.prg, node21.prg, node5.brq, node1.pgnd Typ výpadku: vps_reset Důvod: Selhalo live patchovani jadra Výpadek řeší: Pavel Šnajdr, Jakub Skokan, Martin Myška

pri pokusu zive opravit CVE-2022-1016, se nektere nody livelocknuly, coz vynutilo potrebu jejich resetovani

nebylo to mezi ocekavanymi scenari, plyne z toho nekolik pouceni do budoucna...

omlouvame se :(

ENGLISH: Date and time: 2022-03-29 12:34 CEST Expected duration: 90 minutes Affected systems: node7.prg, node8.prg, node13.prg, node15.prg, node16.prg, node18.prg, node19.prg, node20.prg, node21.prg, node5.brq, node1.pgnd Outage type: vps_reset Reason: Failed kernel live patching Handled by: Pavel Šnajdr, Jakub Skokan, Martin Myška

in an attempt to live-patch CVE-2022-1016, some nodes became live-locked, prompting a reset of those systems

this wasn't expected to happen, lessons are being learned from that...

apologies :(

-----BEGIN BASE64 ENCODED PARSEABLE JSON----- eyJpZCI6ODgwLCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAyMi0w My0yOVQxMjozNDowMCswMjowMCIsImR1cmF0aW9uIjo5MCwidHlwZSI6InZw c19yZXNldCIsImVudGl0aWVzIjpbeyJuYW1lIjoiTm9kZSIsImlkIjoxMDgs ImxhYmVsIjoibm9kZTcucHJnIn0seyJuYW1lIjoiTm9kZSIsImlkIjoxMDks ImxhYmVsIjoibm9kZTgucHJnIn0seyJuYW1lIjoiTm9kZSIsImlkIjoxMTQs ImxhYmVsIjoibm9kZTEzLnByZyJ9LHsibmFtZSI6Ik5vZGUiLCJpZCI6MTE2 LCJsYWJlbCI6Im5vZGUxNS5wcmcifSx7Im5hbWUiOiJOb2RlIiwiaWQiOjEx NywibGFiZWwiOiJub2RlMTYucHJnIn0seyJuYW1lIjoiTm9kZSIsImlkIjox MTksImxhYmVsIjoibm9kZTE4LnByZyJ9LHsibmFtZSI6Ik5vZGUiLCJpZCI6 MTIwLCJsYWJlbCI6Im5vZGUxOS5wcmcifSx7Im5hbWUiOiJOb2RlIiwiaWQi OjEyMSwibGFiZWwiOiJub2RlMjAucHJnIn0seyJuYW1lIjoiTm9kZSIsImlk IjoxMjIsImxhYmVsIjoibm9kZTIxLnByZyJ9LHsibmFtZSI6Ik5vZGUiLCJp ZCI6MjE0LCJsYWJlbCI6Im5vZGU1LmJycSJ9LHsibmFtZSI6Ik5vZGUiLCJp ZCI6MzAwLCJsYWJlbCI6Im5vZGUxLnBnbmQifV0sImhhbmRsZXJzIjpbIlBh dmVsIMWgbmFqZHIiLCJKYWt1YiBTa29rYW4iLCJNYXJ0aW4gTXnFoWthIl0s InRyYW5zbGF0aW9ucyI6eyJlbiI6eyJzdW1tYXJ5IjoiRmFpbGVkIGtlcm5l bCBsaXZlIHBhdGNoaW5nIiwiZGVzY3JpcHRpb24iOiJpbiBhbiBhdHRlbXB0 IHRvIGxpdmUtcGF0Y2ggQ1ZFLTIwMjItMTAxNiwgc29tZSBub2RlcyBiZWNh bWUgbGl2ZS1sb2NrZWQsIHByb21wdGluZyBhIHJlc2V0IG9mIHRob3NlIHN5 c3RlbXNcclxuXHJcbnRoaXMgd2Fzbid0IGV4cGVjdGVkIHRvIGhhcHBlbiwg bGVzc29ucyBhcmUgYmVpbmcgbGVhcm5lZCBmcm9tIHRoYXQuLi5cclxuXHJc bmFwb2xvZ2llcyA6KCJ9LCJjcyI6eyJzdW1tYXJ5IjoiU2VsaGFsbyBsaXZl IHBhdGNob3ZhbmkgamFkcmEiLCJkZXNjcmlwdGlvbiI6InByaSBwb2t1c3Ug eml2ZSBvcHJhdml0IENWRS0yMDIyLTEwMTYsIHNlIG5la3RlcmUgbm9keSBs aXZlbG9ja251bHksIGNveiB2eW51dGlsbyBwb3RyZWJ1IGplamljaCByZXNl dG92YW5pXHJcblxyXG5uZWJ5bG8gdG8gbWV6aSBvY2VrYXZhbnltaSBzY2Vu YXJpLCBwbHluZSB6IHRvaG8gbmVrb2xpayBwb3VjZW5pIGRvIGJ1ZG91Y25h Li4uXHJcblxyXG5vbWxvdXZhbWUgc2UgOigifX19 -----END BASE64 ENCODED PARSEABLE JSON-----