Datum a čas: 2021-11-26 00:19 CET Očekavaná délka: 29 minut Oznámení se týká serverů: Production, Playground, Praha storage, Staging, Praha, Playground, Praha Storage, Staging Typ výpadku: network Důvod: Chyba v konfiguraci BGP Výpadek řeší: Pavel Šnajdr, Martin Myška zrejme tedy... stale patrame, co se stalo; vypada, ze dell os9 switche neumi vyresit BGP zmeny tak rychle, jak potrebujeme (zrejme problem s CPU QoS nastavenim, od tamtud pak nabalujici se koule zpomalenych updatu) ENGLISH: Date and time: 2021-11-26 00:19 CET Expected duration: 29 minutes Affected systems: Production, Playground, Praha storage, Staging, Praha, Playground, Praha Storage, Staging Outage type: network Reason: BGP configuration error Handled by: Pavel Šnajdr, Martin Myška most likely... still investigating; looks like dell os9 based switches don't resolve BGP changes as fast as we'd like (CPU QoS config issues likely, snowball effect from there onwards) -----BEGIN BASE64 ENCODED PARSEABLE JSON----- eyJpZCI6ODUwLCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAyMS0x MS0yNlQwMDoxOTowMCswMTowMCIsImR1cmF0aW9uIjoyOSwidHlwZSI6Im5l dHdvcmsiLCJlbnRpdGllcyI6W3sibmFtZSI6IkVudmlyb25tZW50IiwiaWQi OjEsImxhYmVsIjoiUHJvZHVjdGlvbiJ9LHsibmFtZSI6IkVudmlyb25tZW50 IiwiaWQiOjIsImxhYmVsIjoiUGxheWdyb3VuZCJ9LHsibmFtZSI6IkVudmly b25tZW50IiwiaWQiOjMsImxhYmVsIjoiUHJhaGEgc3RvcmFnZSJ9LHsibmFt ZSI6IkVudmlyb25tZW50IiwiaWQiOjUsImxhYmVsIjoiU3RhZ2luZyJ9LHsi bmFtZSI6IkxvY2F0aW9uIiwiaWQiOjMsImxhYmVsIjoiUHJhaGEifSx7Im5h bWUiOiJMb2NhdGlvbiIsImlkIjo1LCJsYWJlbCI6IlBsYXlncm91bmQifSx7 Im5hbWUiOiJMb2NhdGlvbiIsImlkIjo2LCJsYWJlbCI6IlByYWhhIFN0b3Jh Z2UifSx7Im5hbWUiOiJMb2NhdGlvbiIsImlkIjo3LCJsYWJlbCI6IlN0YWdp bmcifV0sImhhbmRsZXJzIjpbIlBhdmVsIMWgbmFqZHIiLCJNYXJ0aW4gTXnF oWthIl0sInRyYW5zbGF0aW9ucyI6eyJlbiI6eyJzdW1tYXJ5IjoiQkdQIGNv bmZpZ3VyYXRpb24gZXJyb3IiLCJkZXNjcmlwdGlvbiI6Im1vc3QgbGlrZWx5 Li4uIHN0aWxsIGludmVzdGlnYXRpbmc7IGxvb2tzIGxpa2UgZGVsbCBvczkg YmFzZWQgc3dpdGNoZXMgZG9uJ3QgcmVzb2x2ZSBCR1AgY2hhbmdlcyBhcyBm YXN0IGFzIHdlJ2QgbGlrZSAoQ1BVIFFvUyBjb25maWcgaXNzdWVzIGxpa2Vs eSwgc25vd2JhbGwgZWZmZWN0IGZyb20gdGhlcmUgb253YXJkcykifSwiY3Mi Onsic3VtbWFyeSI6IkNoeWJhIHYga29uZmlndXJhY2kgQkdQIiwiZGVzY3Jp cHRpb24iOiJ6cmVqbWUgdGVkeS4uLiBzdGFsZSBwYXRyYW1lLCBjbyBzZSBz dGFsbzsgdnlwYWRhLCB6ZSBkZWxsIG9zOSBzd2l0Y2hlIG5ldW1pIHZ5cmVz aXQgQkdQIHptZW55IHRhayByeWNobGUsIGphayBwb3RyZWJ1amVtZSAoenJl am1lIHByb2JsZW0gcyBDUFUgUW9TIG5hc3RhdmVuaW0sIG9kIHRhbXR1ZCBw YWsgbmFiYWx1amljaSBzZSBrb3VsZSB6cG9tYWxlbnljaCB1cGRhdHUpIn19 fQ== -----END BASE64 ENCODED PARSEABLE JSON-----