Datum a čas: 2022-09-27 01:05 CEST Očekavaná délka: 45 minut Oznámení se týká serverů: node22.prg Typ výpadku: vps_restart Důvod: Nasazeni opravy ZFS Výpadek řeší: Pavel Šnajdr Kdyz jsme menili chovani ZFS, aby nechalo uzivatele prekrocit kvotu, abychom prudce sestrelili pocet async IO pozadavku na disky, vynechali jsme jednu okrajovou situaci, takze nestastna aplikace prepisujici velka mnozstvi dat mohla sem tam dostavat chyby o zaplneni disku. ENGLISH: Date and time: 2022-09-27 01:05 CEST Expected duration: 45 minutes Affected systems: node22.prg Outage type: vps_restart Reason: Deploy ZFS bug fix Handled by: Pavel Šnajdr When we modified ZFS behavior to enable users to overshoot quota in order to sharply shoot down the number of async write IO to SSDs, we missed an edge case, so some unlucky application might have received intermittent disk full errors, especially while rewriting huge amounts of existing data. -----BEGIN BASE64 ENCODED PARSEABLE JSON----- eyJpZCI6OTM3LCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAyMi0w OS0yN1QwMTowNTowMCswMjowMCIsImR1cmF0aW9uIjo0NSwidHlwZSI6InZw c19yZXN0YXJ0IiwiZW50aXRpZXMiOlt7Im5hbWUiOiJOb2RlIiwiaWQiOjEy MywibGFiZWwiOiJub2RlMjIucHJnIn1dLCJoYW5kbGVycyI6WyJQYXZlbCDF oG5hamRyIl0sInRyYW5zbGF0aW9ucyI6eyJlbiI6eyJzdW1tYXJ5IjoiRGVw bG95IFpGUyBidWcgZml4IiwiZGVzY3JpcHRpb24iOiJXaGVuIHdlIG1vZGlm aWVkIFpGUyBiZWhhdmlvciB0byBlbmFibGUgdXNlcnMgdG8gb3ZlcnNob290 IHF1b3RhIGluIG9yZGVyIHRvIHNoYXJwbHkgc2hvb3QgZG93biB0aGUgbnVt YmVyIG9mIGFzeW5jIHdyaXRlIElPIHRvIFNTRHMsIHdlIG1pc3NlZCBhbiBl ZGdlIGNhc2UsIHNvIHNvbWUgdW5sdWNreSBhcHBsaWNhdGlvbiBtaWdodCBo YXZlIHJlY2VpdmVkIGludGVybWl0dGVudCBkaXNrIGZ1bGwgZXJyb3JzLCBl c3BlY2lhbGx5IHdoaWxlIHJld3JpdGluZyBodWdlIGFtb3VudHMgb2YgZXhp c3RpbmcgZGF0YS5cclxuXHJcbiJ9LCJjcyI6eyJzdW1tYXJ5IjoiTmFzYXpl bmkgb3ByYXZ5IFpGUyIsImRlc2NyaXB0aW9uIjoiS2R5eiBqc21lIG1lbmls aSBjaG92YW5pIFpGUywgYWJ5IG5lY2hhbG8gdXppdmF0ZWxlIHByZWtyb2Np dCBrdm90dSwgYWJ5Y2hvbSBwcnVkY2Ugc2VzdHJlbGlsaSBwb2NldCBhc3lu YyBJTyBwb3phZGF2a3UgbmEgZGlza3ksIHZ5bmVjaGFsaSBqc21lIGplZG51 IG9rcmFqb3ZvdSBzaXR1YWNpLCB0YWt6ZSBuZXN0YXN0bmEgYXBsaWthY2Ug cHJlcGlzdWppY2kgdmVsa2EgbW5venN0dmkgZGF0IG1vaGxhIHNlbSB0YW0g ZG9zdGF2YXQgY2h5YnkgbyB6YXBsbmVuaSBkaXNrdS5cclxuXHJcbiJ9fX0= -----END BASE64 ENCODED PARSEABLE JSON-----