Datum a čas: 2022-11-18 22:00 CET
Očekavaná délka: 20 minut
Oznámení se týká serverů: node1.stg
Typ výpadku: vps_restart
Důvod: Test Linuxu 5.10.155
Výpadek řeší: Pavel Šnajdr, Jakub Skokan
ENGLISH:
Date and time: 2022-11-18 22:00 CET
Expected duration: 20 minutes
Affected systems: node1.stg
Outage type: vps_restart
Reason: Testing Linux 5.10.155
Handled by: Pavel Šnajdr, Jakub Skokan
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6OTgxLCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAyMi0x
MS0xOFQyMjowMDowMCswMTowMCIsImR1cmF0aW9uIjoyMCwidHlwZSI6InZw
c19yZXN0YXJ0IiwiZW50aXRpZXMiOlt7Im5hbWUiOiJOb2RlIiwiaWQiOjQw
MCwibGFiZWwiOiJub2RlMS5zdGcifV0sImhhbmRsZXJzIjpbIlBhdmVsIMWg
bmFqZHIiLCJKYWt1YiBTa29rYW4iXSwidHJhbnNsYXRpb25zIjp7ImVuIjp7
InN1bW1hcnkiOiJUZXN0aW5nIExpbnV4IDUuMTAuMTU1IiwiZGVzY3JpcHRp
b24iOm51bGx9LCJjcyI6eyJzdW1tYXJ5IjoiVGVzdCBMaW51eHUgNS4xMC4x
NTUiLCJkZXNjcmlwdGlvbiI6bnVsbH19fQ==
-----END BASE64 ENCODED PARSEABLE JSON-----
Datum a čas: 2022-11-16 16:53 CET
Očekavaná délka: 50 minut
Oznámení se týká serverů: node24.prg
Typ výpadku: vps_reset
Důvod: Kernel lockup
Výpadek řeší: Pavel Šnajdr, Jakub Skokan
pricina zatim neznama
ENGLISH:
Date and time: 2022-11-16 16:53 CET
Expected duration: 50 minutes
Affected systems: node24.prg
Outage type: vps_reset
Reason: Kernel lockup
Handled by: Pavel Šnajdr, Jakub Skokan
cause unknown as of yet
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6OTgwLCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAyMi0x
MS0xNlQxNjo1MzowMCswMTowMCIsImR1cmF0aW9uIjo1MCwidHlwZSI6InZw
c19yZXNldCIsImVudGl0aWVzIjpbeyJuYW1lIjoiTm9kZSIsImlkIjoxMjUs
ImxhYmVsIjoibm9kZTI0LnByZyJ9XSwiaGFuZGxlcnMiOlsiUGF2ZWwgxaBu
YWpkciIsIkpha3ViIFNrb2thbiJdLCJ0cmFuc2xhdGlvbnMiOnsiZW4iOnsi
c3VtbWFyeSI6Iktlcm5lbCBsb2NrdXAiLCJkZXNjcmlwdGlvbiI6ImNhdXNl
IHVua25vd24gYXMgb2YgeWV0In0sImNzIjp7InN1bW1hcnkiOiJLZXJuZWwg
bG9ja3VwIiwiZGVzY3JpcHRpb24iOiJwcmljaW5hIHphdGltIG5lem5hbWEi
fX19
-----END BASE64 ENCODED PARSEABLE JSON-----
Datum a čas: 2022-11-15 01:05 CET
Očekavaná délka: 50 minut
Oznámení se týká serverů: node24.prg
Typ výpadku: vps_restart
Důvod: Reboot do 5.10.154
Výpadek řeší: Pavel Šnajdr
Odstranujeme ze zamku optimistic spinning fronty, ktere zpusobovaly, ze se hromady procesu uspavaly pri cekani na zamkach, pri cem se vyrobily dlouhe fronty procesu k probouzeni, coz nakonec pri zatezovych spickach zabijelo uzitecnou zatez
ENGLISH:
Date and time: 2022-11-15 01:05 CET
Expected duration: 50 minutes
Affected systems: node24.prg
Outage type: vps_restart
Reason: Reboot to 5.10.154
Handled by: Pavel Šnajdr
Removing optimistic spinning queue locking, which was causing swaths processes to sleep on locks, creating long wakeup queues, which in the end killed useful workload during the load spikes.
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6OTc5LCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAyMi0x
MS0xNVQwMTowNTowMCswMTowMCIsImR1cmF0aW9uIjo1MCwidHlwZSI6InZw
c19yZXN0YXJ0IiwiZW50aXRpZXMiOlt7Im5hbWUiOiJOb2RlIiwiaWQiOjEy
NSwibGFiZWwiOiJub2RlMjQucHJnIn1dLCJoYW5kbGVycyI6WyJQYXZlbCDF
oG5hamRyIl0sInRyYW5zbGF0aW9ucyI6eyJlbiI6eyJzdW1tYXJ5IjoiUmVi
b290IHRvIDUuMTAuMTU0IiwiZGVzY3JpcHRpb24iOiJSZW1vdmluZyBvcHRp
bWlzdGljIHNwaW5uaW5nIHF1ZXVlIGxvY2tpbmcsIHdoaWNoIHdhcyBjYXVz
aW5nIHN3YXRocyBwcm9jZXNzZXMgdG8gc2xlZXAgb24gbG9ja3MsIGNyZWF0
aW5nIGxvbmcgd2FrZXVwIHF1ZXVlcywgd2hpY2ggaW4gdGhlIGVuZCBraWxs
ZWQgdXNlZnVsIHdvcmtsb2FkIGR1cmluZyB0aGUgbG9hZCBzcGlrZXMuIn0s
ImNzIjp7InN1bW1hcnkiOiJSZWJvb3QgZG8gNS4xMC4xNTQiLCJkZXNjcmlw
dGlvbiI6Ik9kc3RyYW51amVtZSB6ZSB6YW1rdSBvcHRpbWlzdGljIHNwaW5u
aW5nIGZyb250eSwga3RlcmUgenB1c29ib3ZhbHksIHplIHNlIGhyb21hZHkg
cHJvY2VzdSB1c3BhdmFseSBwcmkgY2VrYW5pIG5hIHphbWthY2gsIHByaSBj
ZW0gc2Ugdnlyb2JpbHkgZGxvdWhlIGZyb250eSBwcm9jZXN1IGsgcHJvYm91
emVuaSwgY296IG5ha29uZWMgcHJpIHphdGV6b3Z5Y2ggc3BpY2thY2ggemFi
aWplbG8gdXppdGVjbm91IHphdGV6In19fQ==
-----END BASE64 ENCODED PARSEABLE JSON-----
Datum a čas: 2022-11-12 12:50 CET
Očekavaná délka: 30 minut
Oznámení se týká serverů: node1.stg
Typ výpadku: vps_reset
Důvod: zaseknute NAPI kthready
Výpadek řeší: Pavel Šnajdr
vypada to, ze dedikovane thready nebudou pouzitelne s 5.10 (nejsou zapnute ve vychozim stavu)
ENGLISH:
Date and time: 2022-11-12 12:50 CET
Expected duration: 30 minutes
Affected systems: node1.stg
Outage type: vps_reset
Reason: NAPI kthreads stuck
Handled by: Pavel Šnajdr
looks like dedicated kthreads won't be usable w/ 5.10 (not turned on by default)
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6OTc3LCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAyMi0x
MS0xMlQxMjo1MDowMCswMTowMCIsImR1cmF0aW9uIjozMCwidHlwZSI6InZw
c19yZXNldCIsImVudGl0aWVzIjpbeyJuYW1lIjoiTm9kZSIsImlkIjo0MDAs
ImxhYmVsIjoibm9kZTEuc3RnIn1dLCJoYW5kbGVycyI6WyJQYXZlbCDFoG5h
amRyIl0sInRyYW5zbGF0aW9ucyI6eyJlbiI6eyJzdW1tYXJ5IjoiTkFQSSBr
dGhyZWFkcyBzdHVjayIsImRlc2NyaXB0aW9uIjoibG9va3MgbGlrZSBkZWRp
Y2F0ZWQga3RocmVhZHMgd29uJ3QgYmUgdXNhYmxlIHcvIDUuMTAgKG5vdCB0
dXJuZWQgb24gYnkgZGVmYXVsdCkifSwiY3MiOnsic3VtbWFyeSI6Inphc2Vr
bnV0ZSBOQVBJIGt0aHJlYWR5IiwiZGVzY3JpcHRpb24iOiJ2eXBhZGEgdG8s
IHplIGRlZGlrb3ZhbmUgdGhyZWFkeSBuZWJ1ZG91IHBvdXppdGVsbmUgcyA1
LjEwIChuZWpzb3UgemFwbnV0ZSB2ZSB2eWNob3ppbSBzdGF2dSkifX19
-----END BASE64 ENCODED PARSEABLE JSON-----
Datum a čas: 2022-11-11 01:05 CET
Očekavaná délka: 40 minut
Oznámení se týká serverů: node24.prg
Typ odstávky: vps_restart
Důvod: Upgrade na Linux 5.10.153
Odstávku řeší: Pavel Šnajdr, Jakub Skokan
- PREEMPT_NONE
- vylepseni kernfs (zrychleni proc & sysfs)
- cpuset affinity/sched_setaffinity now ignored by the kernel
- lxcfs zvlast pro skupiny po 50 VPS
ENGLISH:
Date and time: 2022-11-11 01:05 CET
Expected duration: 40 minutes
Affected systems: node24.prg
Maintenance type: vps_restart
Reason: Upgrade to Linux 5.10.153
Handled by: Pavel Šnajdr, Jakub Skokan
- PREEMPT_NONE
- kernfs improvements (proc & sysfs speedup)
- cpuset affinity/sched_setaffinity now ignored by the kernel
- lxcfs per VPS groups of max 50
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6OTc0LCJwbGFubmVkIjp0cnVlLCJiZWdpbnNfYXQiOiIyMDIyLTEx
LTExVDAxOjA1OjAwKzAxOjAwIiwiZHVyYXRpb24iOjQwLCJ0eXBlIjoidnBz
X3Jlc3RhcnQiLCJlbnRpdGllcyI6W3sibmFtZSI6Ik5vZGUiLCJpZCI6MTI1
LCJsYWJlbCI6Im5vZGUyNC5wcmcifV0sImhhbmRsZXJzIjpbIlBhdmVsIMWg
bmFqZHIiLCJKYWt1YiBTa29rYW4iXSwidHJhbnNsYXRpb25zIjp7ImVuIjp7
InN1bW1hcnkiOiJVcGdyYWRlIHRvIExpbnV4IDUuMTAuMTUzIiwiZGVzY3Jp
cHRpb24iOiItIFBSRUVNUFRfTk9ORVxyXG4tIGtlcm5mcyBpbXByb3ZlbWVu
dHMgKHByb2MgXHUwMDI2IHN5c2ZzIHNwZWVkdXApXHJcbi0gY3B1c2V0IGFm
ZmluaXR5L3NjaGVkX3NldGFmZmluaXR5IG5vdyBpZ25vcmVkIGJ5IHRoZSBr
ZXJuZWxcclxuLSBseGNmcyBwZXIgVlBTIGdyb3VwcyBvZiBtYXggNTAifSwi
Y3MiOnsic3VtbWFyeSI6IlVwZ3JhZGUgbmEgTGludXggNS4xMC4xNTMiLCJk
ZXNjcmlwdGlvbiI6Ii0gUFJFRU1QVF9OT05FXHJcbi0gdnlsZXBzZW5pIGtl
cm5mcyAoenJ5Y2hsZW5pIHByb2MgXHUwMDI2IHN5c2ZzKVxyXG4tIGNwdXNl
dCBhZmZpbml0eS9zY2hlZF9zZXRhZmZpbml0eSBub3cgaWdub3JlZCBieSB0
aGUga2VybmVsXHJcbi0gbHhjZnMgenZsYXN0IHBybyBza3VwaW55IHBvIDUw
IFZQUyJ9fX0=
-----END BASE64 ENCODED PARSEABLE JSON-----