Datum a čas: 2022-10-28 01:17 CEST Očekavaná délka: 180 minut Oznámení se týká serverů: node1.stg Typ výpadku: vps_restart Důvod: Debugovani problemu se starsi mysql Výpadek řeší: Pavel Šnajdr
ENGLISH: Date and time: 2022-10-28 01:17 CEST Expected duration: 180 minutes Affected systems: node1.stg Outage type: vps_restart Reason: Debugging older mysql issues Handled by: Pavel Šnajdr
-----BEGIN BASE64 ENCODED PARSEABLE JSON----- eyJpZCI6OTYzLCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAyMi0x MC0yOFQwMToxNzowMCswMjowMCIsImR1cmF0aW9uIjoxODAsInR5cGUiOiJ2 cHNfcmVzdGFydCIsImVudGl0aWVzIjpbeyJuYW1lIjoiTm9kZSIsImlkIjo0 MDAsImxhYmVsIjoibm9kZTEuc3RnIn1dLCJoYW5kbGVycyI6WyJQYXZlbCDF oG5hamRyIl0sInRyYW5zbGF0aW9ucyI6eyJlbiI6eyJzdW1tYXJ5IjoiRGVi dWdnaW5nIG9sZGVyIG15c3FsIGlzc3VlcyIsImRlc2NyaXB0aW9uIjpudWxs fSwiY3MiOnsic3VtbWFyeSI6IkRlYnVnb3ZhbmkgcHJvYmxlbXUgc2Ugc3Rh cnNpIG15c3FsIiwiZGVzY3JpcHRpb24iOm51bGx9fX0= -----END BASE64 ENCODED PARSEABLE JSON-----
Popis: Další restarty
Hledáme regresi, která způsobuje pád starších mysqld, zatím bez úspěchu. Neumíme to reprodukovat lokálně, takže možná budeme node1.stg restartovat vícekrát.
Nahlásil: Jakub Skokan
ENGLISH:
Summary: More reboots for debugging
We're trying to find a regression which causes older mysqld to crash, so far without any luck. We cannot reproduce locally, so we might be rebooting node1.stg repeatedly.
Reported by: Jakub Skokan
-----BEGIN BASE64 ENCODED PARSEABLE JSON----- eyJpZCI6MjU1NCwiY2hhbmdlcyI6e30sInRyYW5zbGF0aW9ucyI6eyJlbiI6 eyJzdW1tYXJ5IjoiTW9yZSByZWJvb3RzIGZvciBkZWJ1Z2dpbmciLCJkZXNj cmlwdGlvbiI6IldlJ3JlIHRyeWluZyB0byBmaW5kIGEgcmVncmVzc2lvbiB3 aGljaCBjYXVzZXMgb2xkZXIgbXlzcWxkIHRvIGNyYXNoLCBzbyBmYXIgd2l0 aG91dCBhbnkgbHVjay4gV2UgY2Fubm90IHJlcHJvZHVjZSBsb2NhbGx5LCBz byB3ZSBtaWdodCBiZSByZWJvb3Rpbmcgbm9kZTEuc3RnIHJlcGVhdGVkbHku In0sImNzIjp7InN1bW1hcnkiOiJEYWzFocOtIHJlc3RhcnR5IiwiZGVzY3Jp cHRpb24iOiJIbGVkw6FtZSByZWdyZXNpLCBrdGVyw6EgenDFr3NvYnVqZSBw w6FkIHN0YXLFocOtY2ggbXlzcWxkLCB6YXTDrW0gYmV6IMO6c3DEm2NodS4g TmV1bcOtbWUgdG8gcmVwcm9kdWtvdmF0IGxva8OhbG7EmywgdGFrxb5lIG1v xb5uw6EgYnVkZW1lIG5vZGUxLnN0ZyByZXN0YXJ0b3ZhdCB2w61jZWtyw6F0 LiJ9fX0= -----END BASE64 ENCODED PARSEABLE JSON-----
Stav: -> closed
Popis: Podařilo se
Po dvou dnech zkoušení různých kombinací kernelu/ZFS jsme přišli na to, že změna, která způsobuje pády mysqld na Ubuntu 18.04 je jednoduše navýšení sysctl fs.nr_open. Více info viz
https://github.com/vpsfreecz/vpsadminos/commit/abf1322382bc3c37bc920326031c6...
https://im.vpsfree.cz/archive/irc.libera.chat/#vpsfree/2022/10/28.html#l49
Nahlásil: Jakub Skokan
ENGLISH: State: -> closed
Summary: Success
After almost two days of trying different kernel/ZFS combinations, we found out that the change which caused mysqld on Ubuntu 18.04 to crash was simply increased fs.nr_open sysctl. For more information, see
https://github.com/vpsfreecz/vpsadminos/commit/abf1322382bc3c37bc920326031c6...
https://im.vpsfree.cz/archive/irc.libera.chat/#vpsfree/2022/10/28.html#l49
Reported by: Jakub Skokan
-----BEGIN BASE64 ENCODED PARSEABLE JSON----- eyJpZCI6MjU1NSwiY2hhbmdlcyI6eyJzdGF0ZSI6eyJmcm9tIjpudWxsLCJ0 byI6ImNsb3NlZCJ9fSwidHJhbnNsYXRpb25zIjp7ImVuIjp7InN1bW1hcnki OiJTdWNjZXNzIiwiZGVzY3JpcHRpb24iOiJBZnRlciBhbG1vc3QgdHdvIGRh eXMgb2YgdHJ5aW5nIGRpZmZlcmVudCBrZXJuZWwvWkZTIGNvbWJpbmF0aW9u cywgd2UgZm91bmQgb3V0IHRoYXQgdGhlIGNoYW5nZSB3aGljaCBjYXVzZWQg bXlzcWxkIG9uIFVidW50dSAxOC4wNCB0byBjcmFzaCB3YXMgc2ltcGx5ICBp bmNyZWFzZWQgZnMubnJfb3BlbiBzeXNjdGwuIEZvciBtb3JlIGluZm9ybWF0 aW9uLCBzZWVcclxuXHJcbmh0dHBzOi8vZ2l0aHViLmNvbS92cHNmcmVlY3ov dnBzYWRtaW5vcy9jb21taXQvYWJmMTMyMjM4MmJjM2MzN2JjOTIwMzI2MDMx YzZlZWNlZWRiYWEzNVxyXG5cclxuaHR0cHM6Ly9pbS52cHNmcmVlLmN6L2Fy Y2hpdmUvaXJjLmxpYmVyYS5jaGF0LyN2cHNmcmVlLzIwMjIvMTAvMjguaHRt bCNsNDkifSwiY3MiOnsic3VtbWFyeSI6IlBvZGHFmWlsbyBzZSIsImRlc2Ny aXB0aW9uIjoiUG8gZHZvdSBkbmVjaCB6a291xaFlbsOtIHLFr3puw71jaCBr b21iaW5hY8OtIGtlcm5lbHUvWkZTIGpzbWUgcMWZacWhbGkgbmEgdG8sIMW+ ZSB6bcSbbmEsIGt0ZXLDoSB6cMWvc29idWplIHDDoWR5IG15c3FsZCBuYSBV YnVudHUgMTguMDQgamUgamVkbm9kdcWhZSBuYXbDvcWhZW7DrSBzeXNjdGwg ZnMubnJfb3Blbi4gVsOtY2UgaW5mbyB2aXpcclxuXHJcbmh0dHBzOi8vZ2l0 aHViLmNvbS92cHNmcmVlY3ovdnBzYWRtaW5vcy9jb21taXQvYWJmMTMyMjM4 MmJjM2MzN2JjOTIwMzI2MDMxYzZlZWNlZWRiYWEzNVxyXG5cclxuaHR0cHM6 Ly9pbS52cHNmcmVlLmN6L2FyY2hpdmUvaXJjLmxpYmVyYS5jaGF0LyN2cHNm cmVlLzIwMjIvMTAvMjguaHRtbCNsNDkifX19 -----END BASE64 ENCODED PARSEABLE JSON-----