Datum a čas výpadku: 2017-05-26 19:39 CEST Očekavaná délka: 60 minut Výpadek se týká serverů: node4.brq Typ výpadku: reset Důvod: Kernel panic Výpadek řeší: Pavel Šnajdr, Jakub Skokan
.. tak jsme updatnuli jeste kernel a mel by byt klid
ENGLISH: Date and time: 2017-05-26 19:39 CEST Expected duration: 60 minutes Affected systems: node4.brq Outage type: reset Reason: Kernel panic Handled by: Pavel Šnajdr, Jakub Skokan
-----BEGIN BASE64 ENCODED PARSEABLE JSON----- eyJpZCI6Mzc3LCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAxNy0w NS0yNlQxOTozOTowMCswMjowMCIsImR1cmF0aW9uIjo2MCwidHlwZSI6InJl c2V0IiwiZW50aXRpZXMiOlt7Im5hbWUiOiJOb2RlIiwiaWQiOjIxMywibGFi ZWwiOiJub2RlNC5icnEifV0sImhhbmRsZXJzIjpbIlBhdmVsIMWgbmFqZHIi LCJKYWt1YiBTa29rYW4iXSwidHJhbnNsYXRpb25zIjp7ImVuIjp7InN1bW1h cnkiOiJLZXJuZWwgcGFuaWMiLCJkZXNjcmlwdGlvbiI6bnVsbH0sImNzIjp7 InN1bW1hcnkiOiJLZXJuZWwgcGFuaWMiLCJkZXNjcmlwdGlvbiI6Ii4uIHRh ayBqc21lIHVwZGF0bnVsaSBqZXN0ZSBrZXJuZWwgYSBtZWwgYnkgYnl0IGts aWQifX19 -----END BASE64 ENCODED PARSEABLE JSON-----
Očekávaná délka: 60 -> 120 minut
Popis: Zatim se nam nedari import zpoolu, pokousime se to opravit pomoci smartosu, coz nam uz nekolikrat pomohlo
Nahlásil: Jakub Skokan
ENGLISH: Duration: 60 -> 120 minutes
Summary: Zatim se nam nedari import zpoolu, pokousime se to opravit pomoci smartosu, coz nam uz nekolikrat pomohlo
Reported by: Jakub Skokan
-----BEGIN BASE64 ENCODED PARSEABLE JSON----- eyJpZCI6NzY5LCJjaGFuZ2VzIjp7ImR1cmF0aW9uIjp7ImZyb20iOjYwLCJ0 byI6MTIwfX0sInRyYW5zbGF0aW9ucyI6eyJlbiI6eyJzdW1tYXJ5IjoiV2Ug aGF2ZSB0cm91YmxlIGltcG9ydGluZyB0aGUgenBvb2wsIHdlJ3JlIHRyeWlu ZyB0byBmaXggaXQgdXNpbmcgc21hcnRvcywgd2hpY2ggaGFzIGhlbHBlZCB1 cyBpbiB0aGUgcGFzdCIsImRlc2NyaXB0aW9uIjpudWxsfSwiY3MiOnsic3Vt bWFyeSI6IlphdGltIHNlIG5hbSBuZWRhcmkgaW1wb3J0IHpwb29sdSwgcG9r b3VzaW1lIHNlIHRvIG9wcmF2aXQgcG9tb2NpIHNtYXJ0b3N1LCBjb3ogbmFt IHV6IG5la29saWtyYXQgcG9tb2hsbyIsImRlc2NyaXB0aW9uIjpudWxsfX19 -----END BASE64 ENCODED PARSEABLE JSON-----
Popis: Pool se podarilo importovat
Jeste to ale neni za nami. Soubory z nekterych VPS jsou poskozene a nektere VPS tak nestartuji. Spustitelne soubory/knihovny postupne obnovuju ze zalohy, o poskozenych uzivatelskych datech budeme informovat konkretni cleny.
Dalsi vec je, ze se ten pool podarilo importovat jen se ZFS 0.7.0-rc4, coz je problem, protoze send/recv zfs streamu neni kompatibilni se ZFS 0.6.5, ktery mame vsude jinde. V noci tedy muze nasledovat jeste dalsi restart a pravdepodobne neprobehnou dnesni zalohy node4.brq.
Nahlásil: Jakub Skokan
ENGLISH:
Summary: Pool was imported
It's not over yet though. Some files from VPS got corrupted and some of them aren't starting. I'm manually restoring system executables and libraries from backups. We'll be contacting members that have their user data corrupted.
Another problem is that we managed to import the pool only using ZFS 0.7.0-rc4, which has an incompatible send/recv stream with ZFS 0.6.5 that is running on other nodes. It means that we may do another reboot during the night and today's daily backups will probably be cancelled for node4.brq.
Reported by: Jakub Skokan
-----BEGIN BASE64 ENCODED PARSEABLE JSON----- eyJpZCI6NzczLCJjaGFuZ2VzIjp7fSwidHJhbnNsYXRpb25zIjp7ImVuIjp7 InN1bW1hcnkiOiJQb29sIHdhcyBpbXBvcnRlZCIsImRlc2NyaXB0aW9uIjoi SXQncyBub3Qgb3ZlciB5ZXQgdGhvdWdoLiBTb21lIGZpbGVzIGZyb20gVlBT IGdvdCBjb3JydXB0ZWQgYW5kIHNvbWVcclxub2YgdGhlbSBhcmVuJ3Qgc3Rh cnRpbmcuIEknbSBtYW51YWxseSByZXN0b3Jpbmcgc3lzdGVtIGV4ZWN1dGFi bGVzXHJcbmFuZCBsaWJyYXJpZXMgZnJvbSBiYWNrdXBzLiBXZSdsbCBiZSBj b250YWN0aW5nIG1lbWJlcnMgdGhhdCBoYXZlIHRoZWlyXHJcbnVzZXIgZGF0 YSBjb3JydXB0ZWQuXHJcblxyXG5Bbm90aGVyIHByb2JsZW0gaXMgdGhhdCB3 ZSBtYW5hZ2VkIHRvIGltcG9ydCB0aGUgcG9vbCBvbmx5IHVzaW5nIFpGU1xy XG4wLjcuMC1yYzQsIHdoaWNoIGhhcyBhbiBpbmNvbXBhdGlibGUgc2VuZC9y ZWN2IHN0cmVhbSB3aXRoIFpGUyAwLjYuNSB0aGF0XHJcbmlzIHJ1bm5pbmcg b24gb3RoZXIgbm9kZXMuIEl0IG1lYW5zIHRoYXQgd2UgbWF5IGRvIGFub3Ro ZXIgcmVib290IGR1cmluZ1xyXG50aGUgbmlnaHQgYW5kIHRvZGF5J3MgZGFp bHkgYmFja3VwcyB3aWxsIHByb2JhYmx5IGJlIGNhbmNlbGxlZCBmb3Igbm9k ZTQuYnJxLiJ9LCJjcyI6eyJzdW1tYXJ5IjoiUG9vbCBzZSBwb2RhcmlsbyBp bXBvcnRvdmF0IiwiZGVzY3JpcHRpb24iOiJKZXN0ZSB0byBhbGUgbmVuaSB6 YSBuYW1pLiBTb3Vib3J5IHogbmVrdGVyeWNoIFZQUyBqc291IHBvc2tvemVu ZVxyXG5hIG5la3RlcmUgVlBTIHRhayBuZXN0YXJ0dWppLiBTcHVzdGl0ZWxu ZSBzb3Vib3J5L2tuaWhvdm55IHBvc3R1cG5lXHJcbm9ibm92dWp1IHplIHph bG9oeSwgbyBwb3Nrb3plbnljaCB1eml2YXRlbHNreWNoIGRhdGVjaCBidWRl bWUgaW5mb3Jtb3ZhdFxyXG5rb25rcmV0bmkgY2xlbnkuXHJcblxyXG5EYWxz aSB2ZWMgamUsIHplIHNlIHRlbiBwb29sIHBvZGFyaWxvIGltcG9ydG92YXQg amVuIHNlIFpGUyAwLjcuMC1yYzQsXHJcbmNveiBqZSBwcm9ibGVtLCBwcm90 b3plIHNlbmQvcmVjdiB6ZnMgc3RyZWFtdSBuZW5pIGtvbXBhdGliaWxuaSBz ZVxyXG5aRlMgMC42LjUsIGt0ZXJ5IG1hbWUgdnN1ZGUgamluZGUuIFYgbm9j aSB0ZWR5IG11emUgbmFzbGVkb3ZhdCBqZXN0ZVxyXG5kYWxzaSByZXN0YXJ0 IGEgcHJhdmRlcG9kb2JuZSBuZXByb2JlaG5vdSBkbmVzbmkgemFsb2h5IG5v ZGU0LmJycS4ifX19 -----END BASE64 ENCODED PARSEABLE JSON-----
Popis: Presun VPS na node1.brq
Pool se nam nepodarilo importovat s kompatibilni verzi ZFS, takze jsme se rozhodli vsechny VPS presunout na jiny node. Snajpa do DC zavezl opraveny node1.brq a zbyva presunout poslednich ~18 VPS. Behem dneska by VPS mely byt dostupne z vpsAdminu. Poskozenych souboru je pomerne hodne, jakmile budeme mit kompletni seznam, Michal vam bude rozesilat maily.
Nahlásil: Jakub Skokan
ENGLISH:
Summary: Migration to node1.brq
We were unable to import the pool with a compatible ZFS version, so we've decided to migrate all VPS to another node. node1.brq was serviced and snajpa brought it back to DC. We're migrating the last ~18 VPS there. All VPS should be accessible from vpsAdmin by today. There's quite a lot of corrupted files, so as soon as we have a complete list, Michal will start informing you.
Reported by: Jakub Skokan
-----BEGIN BASE64 ENCODED PARSEABLE JSON----- eyJpZCI6Nzc0LCJjaGFuZ2VzIjp7fSwidHJhbnNsYXRpb25zIjp7ImVuIjp7 InN1bW1hcnkiOiJNaWdyYXRpb24gdG8gbm9kZTEuYnJxIiwiZGVzY3JpcHRp b24iOiJXZSB3ZXJlIHVuYWJsZSB0byBpbXBvcnQgdGhlIHBvb2wgd2l0aCBh IGNvbXBhdGlibGUgWkZTIHZlcnNpb24sIHNvXHJcbndlJ3ZlIGRlY2lkZWQg dG8gbWlncmF0ZSBhbGwgVlBTIHRvIGFub3RoZXIgbm9kZS4gbm9kZTEuYnJx IHdhcyBzZXJ2aWNlZFxyXG5hbmQgc25hanBhIGJyb3VnaHQgaXQgYmFjayB0 byBEQy4gV2UncmUgbWlncmF0aW5nIHRoZSBsYXN0IH4xOCBWUFMgdGhlcmUu XHJcbkFsbCBWUFMgc2hvdWxkIGJlIGFjY2Vzc2libGUgZnJvbSB2cHNBZG1p biBieSB0b2RheS4gVGhlcmUncyBxdWl0ZSBhIGxvdFxyXG5vZiBjb3JydXB0 ZWQgZmlsZXMsIHNvIGFzIHNvb24gYXMgd2UgaGF2ZSBhIGNvbXBsZXRlIGxp c3QsIE1pY2hhbCB3aWxsXHJcbnN0YXJ0IGluZm9ybWluZyB5b3UuIn0sImNz Ijp7InN1bW1hcnkiOiJQcmVzdW4gVlBTIG5hIG5vZGUxLmJycSIsImRlc2Ny aXB0aW9uIjoiUG9vbCBzZSBuYW0gbmVwb2RhcmlsbyBpbXBvcnRvdmF0IHMg a29tcGF0aWJpbG5pIHZlcnppIFpGUywgdGFremUganNtZVxyXG5zZSByb3po b2RsaSB2c2VjaG55IFZQUyBwcmVzdW5vdXQgbmEgamlueSBub2RlLiBTbmFq cGEgZG8gREMgemF2ZXpsXHJcbm9wcmF2ZW55IG5vZGUxLmJycSBhIHpieXZh IHByZXN1bm91dCBwb3NsZWRuaWNoIH4xOCBWUFMuIEJlaGVtIGRuZXNrYVxy XG5ieSBWUFMgbWVseSBieXQgZG9zdHVwbmUgeiB2cHNBZG1pbnUuIFBvc2tv emVueWNoIHNvdWJvcnUgamUgcG9tZXJuZVxyXG5ob2RuZSwgamFrbWlsZSBi dWRlbWUgbWl0IGtvbXBsZXRuaSBzZXpuYW0sIE1pY2hhbCB2YW0gYnVkZSBy b3plc2lsYXRcclxubWFpbHkuIn19fQ== -----END BASE64 ENCODED PARSEABLE JSON-----
Stav: announced -> closed
Popis: Migrace na node1.brq dokoncena, VPS pristupne
Vsechny VPS uz jsou na node1.brq a lze je ovladat z vpsAdminu. node4.brq reinstalujeme a vybrane VPS na nej budeme vracet kvuli rozlozeni zateze (node4.brq ma SSD, node1.brq HDD). Tyto migrace probehnou uz standardne pres vpsAdmin, tzn. budete o tom informovani mailem a migrace se provede v nastavenem oknu pro vypadky.
Nahlásil: Jakub Skokan
ENGLISH: State: announced -> closed
Summary: Migration to node1.brq finished, VPS accessible
All VPS have been migrated to node1.brq and are accessible via vpsAdmin. node4.brq will be reinstalled and selected VPS will be migrated back to spread the load (node4.brq uses SSD, node1.brq has HDD). These migrations will be handled by vpsAdmin, i.e. you'll be informed via mail and the migration will take place during configured outage window.
Reported by: Jakub Skokan
-----BEGIN BASE64 ENCODED PARSEABLE JSON----- eyJpZCI6Nzc1LCJjaGFuZ2VzIjp7InN0YXRlIjp7ImZyb20iOiJhbm5vdW5j ZWQiLCJ0byI6ImNsb3NlZCJ9fSwidHJhbnNsYXRpb25zIjp7ImVuIjp7InN1 bW1hcnkiOiJNaWdyYXRpb24gdG8gbm9kZTEuYnJxIGZpbmlzaGVkLCBWUFMg YWNjZXNzaWJsZSIsImRlc2NyaXB0aW9uIjoiQWxsIFZQUyBoYXZlIGJlZW4g bWlncmF0ZWQgdG8gbm9kZTEuYnJxIGFuZCBhcmUgYWNjZXNzaWJsZSB2aWEg dnBzQWRtaW4uXHJcbm5vZGU0LmJycSB3aWxsIGJlIHJlaW5zdGFsbGVkIGFu ZCBzZWxlY3RlZCBWUFMgd2lsbCBiZSBtaWdyYXRlZCBiYWNrXHJcbnRvIHNw cmVhZCB0aGUgbG9hZCAobm9kZTQuYnJxIHVzZXMgU1NELCBub2RlMS5icnEg aGFzIEhERCkuIFRoZXNlIG1pZ3JhdGlvbnNcclxud2lsbCBiZSBoYW5kbGVk IGJ5IHZwc0FkbWluLCBpLmUuIHlvdSdsbCBiZSBpbmZvcm1lZCB2aWEgbWFp bCBhbmQgdGhlXHJcbm1pZ3JhdGlvbiB3aWxsIHRha2UgcGxhY2UgZHVyaW5n IGNvbmZpZ3VyZWQgb3V0YWdlIHdpbmRvdy4ifSwiY3MiOnsic3VtbWFyeSI6 Ik1pZ3JhY2UgbmEgbm9kZTEuYnJxIGRva29uY2VuYSwgVlBTIHByaXN0dXBu ZSIsImRlc2NyaXB0aW9uIjoiVnNlY2hueSBWUFMgdXoganNvdSBuYSBub2Rl MS5icnEgYSBsemUgamUgb3ZsYWRhdCB6IHZwc0FkbWludS4gbm9kZTQuYnJx IHJlaW5zdGFsdWplbWUgYSB2eWJyYW5lIFZQUyBuYSBuZWogYnVkZW1lIHZy YWNldCBrdnVsaSByb3psb3plbmkgemF0ZXplXHJcbihub2RlNC5icnEgbWEg U1NELCBub2RlMS5icnEgSEREKS4gVHl0byBtaWdyYWNlIHByb2JlaG5vdSB1 eiBzdGFuZGFyZG5lXHJcbnByZXMgdnBzQWRtaW4sIHR6bi4gYnVkZXRlIG8g dG9tIGluZm9ybW92YW5pIG1haWxlbSBhIG1pZ3JhY2Ugc2UgcHJvdmVkZVxy XG52IG5hc3RhdmVuZW0gb2tudSBwcm8gdnlwYWRreS4ifX19 -----END BASE64 ENCODED PARSEABLE JSON-----