Datum a čas: 2022-09-27 01:05 CEST
Očekavaná délka: 45 minut
Oznámení se týká serverů: node22.prg
Typ výpadku: vps_restart
Důvod: Nasazeni opravy ZFS
Výpadek řeší: Pavel Šnajdr
Kdyz jsme menili chovani ZFS, aby nechalo uzivatele prekrocit kvotu, abychom prudce sestrelili pocet async IO pozadavku na disky, vynechali jsme jednu okrajovou situaci, takze nestastna aplikace prepisujici velka mnozstvi dat mohla sem tam dostavat chyby o zaplneni disku.
ENGLISH:
Date and time: 2022-09-27 01:05 CEST
Expected duration: 45 minutes
Affected systems: node22.prg
Outage type: vps_restart
Reason: Deploy ZFS bug fix
Handled by: Pavel Šnajdr
When we modified ZFS behavior to enable users to overshoot quota in order to sharply shoot down the number of async write IO to SSDs, we missed an edge case, so some unlucky application might have received intermittent disk full errors, especially while rewriting huge amounts of existing data.
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6OTM3LCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAyMi0w
OS0yN1QwMTowNTowMCswMjowMCIsImR1cmF0aW9uIjo0NSwidHlwZSI6InZw
c19yZXN0YXJ0IiwiZW50aXRpZXMiOlt7Im5hbWUiOiJOb2RlIiwiaWQiOjEy
MywibGFiZWwiOiJub2RlMjIucHJnIn1dLCJoYW5kbGVycyI6WyJQYXZlbCDF
oG5hamRyIl0sInRyYW5zbGF0aW9ucyI6eyJlbiI6eyJzdW1tYXJ5IjoiRGVw
bG95IFpGUyBidWcgZml4IiwiZGVzY3JpcHRpb24iOiJXaGVuIHdlIG1vZGlm
aWVkIFpGUyBiZWhhdmlvciB0byBlbmFibGUgdXNlcnMgdG8gb3ZlcnNob290
IHF1b3RhIGluIG9yZGVyIHRvIHNoYXJwbHkgc2hvb3QgZG93biB0aGUgbnVt
YmVyIG9mIGFzeW5jIHdyaXRlIElPIHRvIFNTRHMsIHdlIG1pc3NlZCBhbiBl
ZGdlIGNhc2UsIHNvIHNvbWUgdW5sdWNreSBhcHBsaWNhdGlvbiBtaWdodCBo
YXZlIHJlY2VpdmVkIGludGVybWl0dGVudCBkaXNrIGZ1bGwgZXJyb3JzLCBl
c3BlY2lhbGx5IHdoaWxlIHJld3JpdGluZyBodWdlIGFtb3VudHMgb2YgZXhp
c3RpbmcgZGF0YS5cclxuXHJcbiJ9LCJjcyI6eyJzdW1tYXJ5IjoiTmFzYXpl
bmkgb3ByYXZ5IFpGUyIsImRlc2NyaXB0aW9uIjoiS2R5eiBqc21lIG1lbmls
aSBjaG92YW5pIFpGUywgYWJ5IG5lY2hhbG8gdXppdmF0ZWxlIHByZWtyb2Np
dCBrdm90dSwgYWJ5Y2hvbSBwcnVkY2Ugc2VzdHJlbGlsaSBwb2NldCBhc3lu
YyBJTyBwb3phZGF2a3UgbmEgZGlza3ksIHZ5bmVjaGFsaSBqc21lIGplZG51
IG9rcmFqb3ZvdSBzaXR1YWNpLCB0YWt6ZSBuZXN0YXN0bmEgYXBsaWthY2Ug
cHJlcGlzdWppY2kgdmVsa2EgbW5venN0dmkgZGF0IG1vaGxhIHNlbSB0YW0g
ZG9zdGF2YXQgY2h5YnkgbyB6YXBsbmVuaSBkaXNrdS5cclxuXHJcbiJ9fX0=
-----END BASE64 ENCODED PARSEABLE JSON-----
Datum a čas: 2022-09-23 17:30 CEST
Očekavaná délka: 35 minut
Oznámení se týká serverů: node23.prg
Typ výpadku: vps_restart
Důvod: Upgrade Linuxu + ZFS
Výpadek řeší: Pavel Šnajdr
procesy nekterych VPS se zasekavaji, omlouvame se za reboot pres den
ENGLISH:
Date and time: 2022-09-23 17:30 CEST
Expected duration: 35 minutes
Affected systems: node23.prg
Outage type: vps_restart
Reason: Upgrade Linux + ZFS
Handled by: Pavel Šnajdr
processes of some VPSes are getting stuck, sorry for reboot during daytime
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6OTMzLCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAyMi0w
OS0yM1QxNzozMDowMCswMjowMCIsImR1cmF0aW9uIjozNSwidHlwZSI6InZw
c19yZXN0YXJ0IiwiZW50aXRpZXMiOlt7Im5hbWUiOiJOb2RlIiwiaWQiOjEy
NCwibGFiZWwiOiJub2RlMjMucHJnIn1dLCJoYW5kbGVycyI6WyJQYXZlbCDF
oG5hamRyIl0sInRyYW5zbGF0aW9ucyI6eyJlbiI6eyJzdW1tYXJ5IjoiVXBn
cmFkZSBMaW51eCArIFpGUyIsImRlc2NyaXB0aW9uIjoicHJvY2Vzc2VzIG9m
IHNvbWUgVlBTZXMgYXJlIGdldHRpbmcgc3R1Y2ssIHNvcnJ5IGZvciByZWJv
b3QgZHVyaW5nIGRheXRpbWUifSwiY3MiOnsic3VtbWFyeSI6IlVwZ3JhZGUg
TGludXh1ICsgWkZTIiwiZGVzY3JpcHRpb24iOiJwcm9jZXN5IG5la3Rlcnlj
aCBWUFMgc2UgemFzZWthdmFqaSwgb21sb3V2YW1lIHNlIHphIHJlYm9vdCBw
cmVzIGRlbiJ9fX0=
-----END BASE64 ENCODED PARSEABLE JSON-----
Datum a čas: 2022-09-26 01:05 CEST
Očekavaná délka: 40 minut
Oznámení se týká serverů: node22.prg
Typ odstávky: vps_restart
Důvod: Aktualizace Linuxu + ZFS
Odstávku řeší: Pavel Šnajdr
s patchi pro vetsi stroje
ENGLISH:
Date and time: 2022-09-26 01:05 CEST
Expected duration: 40 minutes
Affected systems: node22.prg
Maintenance type: vps_restart
Reason: Upgrade Linux + ZFS
Handled by: Pavel Šnajdr
with patches for bigger boxes
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6OTMyLCJwbGFubmVkIjp0cnVlLCJiZWdpbnNfYXQiOiIyMDIyLTA5
LTI2VDAxOjA1OjAwKzAyOjAwIiwiZHVyYXRpb24iOjQwLCJ0eXBlIjoidnBz
X3Jlc3RhcnQiLCJlbnRpdGllcyI6W3sibmFtZSI6Ik5vZGUiLCJpZCI6MTIz
LCJsYWJlbCI6Im5vZGUyMi5wcmcifV0sImhhbmRsZXJzIjpbIlBhdmVsIMWg
bmFqZHIiXSwidHJhbnNsYXRpb25zIjp7ImVuIjp7InN1bW1hcnkiOiJVcGdy
YWRlIExpbnV4ICsgWkZTIiwiZGVzY3JpcHRpb24iOiJ3aXRoIHBhdGNoZXMg
Zm9yIGJpZ2dlciBib3hlcyJ9LCJjcyI6eyJzdW1tYXJ5IjoiQWt0dWFsaXph
Y2UgTGludXh1ICsgWkZTIiwiZGVzY3JpcHRpb24iOiJzIHBhdGNoaSBwcm8g
dmV0c2kgc3Ryb2plIn19fQ==
-----END BASE64 ENCODED PARSEABLE JSON-----