Datum a čas: 2021-11-26 00:19 CET
Očekavaná délka: 29 minut
Oznámení se týká serverů: Production, Playground, Praha storage, Staging, Praha, Playground, Praha Storage, Staging
Typ výpadku: network
Důvod: Chyba v konfiguraci BGP
Výpadek řeší: Pavel Šnajdr, Martin Myška
zrejme tedy... stale patrame, co se stalo; vypada, ze dell os9 switche neumi vyresit BGP zmeny tak rychle, jak potrebujeme (zrejme problem s CPU QoS nastavenim, od tamtud pak nabalujici se koule zpomalenych updatu)
ENGLISH:
Date and time: 2021-11-26 00:19 CET
Expected duration: 29 minutes
Affected systems: Production, Playground, Praha storage, Staging, Praha, Playground, Praha Storage, Staging
Outage type: network
Reason: BGP configuration error
Handled by: Pavel Šnajdr, Martin Myška
most likely... still investigating; looks like dell os9 based switches don't resolve BGP changes as fast as we'd like (CPU QoS config issues likely, snowball effect from there onwards)
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6ODUwLCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAyMS0x
MS0yNlQwMDoxOTowMCswMTowMCIsImR1cmF0aW9uIjoyOSwidHlwZSI6Im5l
dHdvcmsiLCJlbnRpdGllcyI6W3sibmFtZSI6IkVudmlyb25tZW50IiwiaWQi
OjEsImxhYmVsIjoiUHJvZHVjdGlvbiJ9LHsibmFtZSI6IkVudmlyb25tZW50
IiwiaWQiOjIsImxhYmVsIjoiUGxheWdyb3VuZCJ9LHsibmFtZSI6IkVudmly
b25tZW50IiwiaWQiOjMsImxhYmVsIjoiUHJhaGEgc3RvcmFnZSJ9LHsibmFt
ZSI6IkVudmlyb25tZW50IiwiaWQiOjUsImxhYmVsIjoiU3RhZ2luZyJ9LHsi
bmFtZSI6IkxvY2F0aW9uIiwiaWQiOjMsImxhYmVsIjoiUHJhaGEifSx7Im5h
bWUiOiJMb2NhdGlvbiIsImlkIjo1LCJsYWJlbCI6IlBsYXlncm91bmQifSx7
Im5hbWUiOiJMb2NhdGlvbiIsImlkIjo2LCJsYWJlbCI6IlByYWhhIFN0b3Jh
Z2UifSx7Im5hbWUiOiJMb2NhdGlvbiIsImlkIjo3LCJsYWJlbCI6IlN0YWdp
bmcifV0sImhhbmRsZXJzIjpbIlBhdmVsIMWgbmFqZHIiLCJNYXJ0aW4gTXnF
oWthIl0sInRyYW5zbGF0aW9ucyI6eyJlbiI6eyJzdW1tYXJ5IjoiQkdQIGNv
bmZpZ3VyYXRpb24gZXJyb3IiLCJkZXNjcmlwdGlvbiI6Im1vc3QgbGlrZWx5
Li4uIHN0aWxsIGludmVzdGlnYXRpbmc7IGxvb2tzIGxpa2UgZGVsbCBvczkg
YmFzZWQgc3dpdGNoZXMgZG9uJ3QgcmVzb2x2ZSBCR1AgY2hhbmdlcyBhcyBm
YXN0IGFzIHdlJ2QgbGlrZSAoQ1BVIFFvUyBjb25maWcgaXNzdWVzIGxpa2Vs
eSwgc25vd2JhbGwgZWZmZWN0IGZyb20gdGhlcmUgb253YXJkcykifSwiY3Mi
Onsic3VtbWFyeSI6IkNoeWJhIHYga29uZmlndXJhY2kgQkdQIiwiZGVzY3Jp
cHRpb24iOiJ6cmVqbWUgdGVkeS4uLiBzdGFsZSBwYXRyYW1lLCBjbyBzZSBz
dGFsbzsgdnlwYWRhLCB6ZSBkZWxsIG9zOSBzd2l0Y2hlIG5ldW1pIHZ5cmVz
aXQgQkdQIHptZW55IHRhayByeWNobGUsIGphayBwb3RyZWJ1amVtZSAoenJl
am1lIHByb2JsZW0gcyBDUFUgUW9TIG5hc3RhdmVuaW0sIG9kIHRhbXR1ZCBw
YWsgbmFiYWx1amljaSBzZSBrb3VsZSB6cG9tYWxlbnljaCB1cGRhdHUpIn19
fQ==
-----END BASE64 ENCODED PARSEABLE JSON-----
Datum a čas: 2021-11-16 13:24 CET
Očekavaná délka: 35 minut
Oznámení se týká serverů: node1.brq
Typ výpadku: vps_reset
Důvod: stary zfs bug
Výpadek řeší: Pavel Šnajdr
prosim migrujte na vpsAdminOS, abychom mohli OpenVZ vypnout
ENGLISH:
Date and time: 2021-11-16 13:24 CET
Expected duration: 35 minutes
Affected systems: node1.brq
Outage type: vps_reset
Reason: old zfs bug
Handled by: Pavel Šnajdr
please migrate to vpsAdminOS, so we can shut OpenVZ down
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6ODQ4LCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAyMS0x
MS0xNlQxMzoyNDowMCswMTowMCIsImR1cmF0aW9uIjozNSwidHlwZSI6InZw
c19yZXNldCIsImVudGl0aWVzIjpbeyJuYW1lIjoiTm9kZSIsImlkIjoyMTAs
ImxhYmVsIjoibm9kZTEuYnJxIn1dLCJoYW5kbGVycyI6WyJQYXZlbCDFoG5h
amRyIl0sInRyYW5zbGF0aW9ucyI6eyJlbiI6eyJzdW1tYXJ5Ijoib2xkIHpm
cyBidWciLCJkZXNjcmlwdGlvbiI6InBsZWFzZSBtaWdyYXRlIHRvIHZwc0Fk
bWluT1MsIHNvIHdlIGNhbiBzaHV0IE9wZW5WWiBkb3duIn0sImNzIjp7InN1
bW1hcnkiOiJzdGFyeSB6ZnMgYnVnIiwiZGVzY3JpcHRpb24iOiJwcm9zaW0g
bWlncnVqdGUgbmEgdnBzQWRtaW5PUywgYWJ5Y2hvbSBtb2hsaSBPcGVuVlog
dnlwbm91dCJ9fX0=
-----END BASE64 ENCODED PARSEABLE JSON-----