26 May
2017
26 May
'17
6:14 p.m.
Datum a čas výpadku: 2017-05-26 19:39 CEST
Očekavaná délka: 60 minut
Výpadek se týká serverů: node4.brq
Typ výpadku: reset
Důvod: Kernel panic
Výpadek řeší: Pavel Šnajdr, Jakub Skokan
.. tak jsme updatnuli jeste kernel a mel by byt klid
ENGLISH:
Date and time: 2017-05-26 19:39 CEST
Expected duration: 60 minutes
Affected systems: node4.brq
Outage type: reset
Reason: Kernel panic
Handled by: Pavel Šnajdr, Jakub Skokan
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6Mzc3LCJwbGFubmVkIjpmYWxzZSwiYmVnaW5zX2F0IjoiMjAxNy0w
NS0yNlQxOTozOTowMCswMjowMCIsImR1cmF0aW9uIjo2MCwidHlwZSI6InJl
c2V0IiwiZW50aXRpZXMiOlt7Im5hbWUiOiJOb2RlIiwiaWQiOjIxMywibGFi
ZWwiOiJub2RlNC5icnEifV0sImhhbmRsZXJzIjpbIlBhdmVsIMWgbmFqZHIi
LCJKYWt1YiBTa29rYW4iXSwidHJhbnNsYXRpb25zIjp7ImVuIjp7InN1bW1h
cnkiOiJLZXJuZWwgcGFuaWMiLCJkZXNjcmlwdGlvbiI6bnVsbH0sImNzIjp7
InN1bW1hcnkiOiJLZXJuZWwgcGFuaWMiLCJkZXNjcmlwdGlvbiI6Ii4uIHRh
ayBqc21lIHVwZGF0bnVsaSBqZXN0ZSBrZXJuZWwgYSBtZWwgYnkgYnl0IGts
aWQifX19
-----END BASE64 ENCODED PARSEABLE JSON-----
26 May
26 May
7:05 p.m.
Očekávaná délka: 60 -> 120 minut
Popis: Zatim se nam nedari import zpoolu, pokousime se to opravit pomoci smartosu, coz nam
uz nekolikrat pomohlo
Nahlásil: Jakub Skokan
ENGLISH:
Duration: 60 -> 120 minutes
Summary: Zatim se nam nedari import zpoolu, pokousime se to opravit pomoci smartosu, coz
nam uz nekolikrat pomohlo
Reported by: Jakub Skokan
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6NzY5LCJjaGFuZ2VzIjp7ImR1cmF0aW9uIjp7ImZyb20iOjYwLCJ0
byI6MTIwfX0sInRyYW5zbGF0aW9ucyI6eyJlbiI6eyJzdW1tYXJ5IjoiV2Ug
aGF2ZSB0cm91YmxlIGltcG9ydGluZyB0aGUgenBvb2wsIHdlJ3JlIHRyeWlu
ZyB0byBmaXggaXQgdXNpbmcgc21hcnRvcywgd2hpY2ggaGFzIGhlbHBlZCB1
cyBpbiB0aGUgcGFzdCIsImRlc2NyaXB0aW9uIjpudWxsfSwiY3MiOnsic3Vt
bWFyeSI6IlphdGltIHNlIG5hbSBuZWRhcmkgaW1wb3J0IHpwb29sdSwgcG9r
b3VzaW1lIHNlIHRvIG9wcmF2aXQgcG9tb2NpIHNtYXJ0b3N1LCBjb3ogbmFt
IHV6IG5la29saWtyYXQgcG9tb2hsbyIsImRlc2NyaXB0aW9uIjpudWxsfX19
-----END BASE64 ENCODED PARSEABLE JSON-----
8:31 p.m.
Popis: Pool se podarilo importovat
Jeste to ale neni za nami. Soubory z nekterych VPS jsou poskozene
a nektere VPS tak nestartuji. Spustitelne soubory/knihovny postupne
obnovuju ze zalohy, o poskozenych uzivatelskych datech budeme informovat
konkretni cleny.
Dalsi vec je, ze se ten pool podarilo importovat jen se ZFS 0.7.0-rc4,
coz je problem, protoze send/recv zfs streamu neni kompatibilni se
ZFS 0.6.5, ktery mame vsude jinde. V noci tedy muze nasledovat jeste
dalsi restart a pravdepodobne neprobehnou dnesni zalohy node4.brq.
Nahlásil: Jakub Skokan
ENGLISH:
Summary: Pool was imported
It's not over yet though. Some files from VPS got corrupted and some
of them aren't starting. I'm manually restoring system executables
and libraries from backups. We'll be contacting members that have their
user data corrupted.
Another problem is that we managed to import the pool only using ZFS
0.7.0-rc4, which has an incompatible send/recv stream with ZFS 0.6.5 that
is running on other nodes. It means that we may do another reboot during
the night and today's daily backups will probably be cancelled for node4.brq.
Reported by: Jakub Skokan
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6NzczLCJjaGFuZ2VzIjp7fSwidHJhbnNsYXRpb25zIjp7ImVuIjp7
InN1bW1hcnkiOiJQb29sIHdhcyBpbXBvcnRlZCIsImRlc2NyaXB0aW9uIjoi
SXQncyBub3Qgb3ZlciB5ZXQgdGhvdWdoLiBTb21lIGZpbGVzIGZyb20gVlBT
IGdvdCBjb3JydXB0ZWQgYW5kIHNvbWVcclxub2YgdGhlbSBhcmVuJ3Qgc3Rh
cnRpbmcuIEknbSBtYW51YWxseSByZXN0b3Jpbmcgc3lzdGVtIGV4ZWN1dGFi
bGVzXHJcbmFuZCBsaWJyYXJpZXMgZnJvbSBiYWNrdXBzLiBXZSdsbCBiZSBj
b250YWN0aW5nIG1lbWJlcnMgdGhhdCBoYXZlIHRoZWlyXHJcbnVzZXIgZGF0
YSBjb3JydXB0ZWQuXHJcblxyXG5Bbm90aGVyIHByb2JsZW0gaXMgdGhhdCB3
ZSBtYW5hZ2VkIHRvIGltcG9ydCB0aGUgcG9vbCBvbmx5IHVzaW5nIFpGU1xy
XG4wLjcuMC1yYzQsIHdoaWNoIGhhcyBhbiBpbmNvbXBhdGlibGUgc2VuZC9y
ZWN2IHN0cmVhbSB3aXRoIFpGUyAwLjYuNSB0aGF0XHJcbmlzIHJ1bm5pbmcg
b24gb3RoZXIgbm9kZXMuIEl0IG1lYW5zIHRoYXQgd2UgbWF5IGRvIGFub3Ro
ZXIgcmVib290IGR1cmluZ1xyXG50aGUgbmlnaHQgYW5kIHRvZGF5J3MgZGFp
bHkgYmFja3VwcyB3aWxsIHByb2JhYmx5IGJlIGNhbmNlbGxlZCBmb3Igbm9k
ZTQuYnJxLiJ9LCJjcyI6eyJzdW1tYXJ5IjoiUG9vbCBzZSBwb2RhcmlsbyBp
bXBvcnRvdmF0IiwiZGVzY3JpcHRpb24iOiJKZXN0ZSB0byBhbGUgbmVuaSB6
YSBuYW1pLiBTb3Vib3J5IHogbmVrdGVyeWNoIFZQUyBqc291IHBvc2tvemVu
ZVxyXG5hIG5la3RlcmUgVlBTIHRhayBuZXN0YXJ0dWppLiBTcHVzdGl0ZWxu
ZSBzb3Vib3J5L2tuaWhvdm55IHBvc3R1cG5lXHJcbm9ibm92dWp1IHplIHph
bG9oeSwgbyBwb3Nrb3plbnljaCB1eml2YXRlbHNreWNoIGRhdGVjaCBidWRl
bWUgaW5mb3Jtb3ZhdFxyXG5rb25rcmV0bmkgY2xlbnkuXHJcblxyXG5EYWxz
aSB2ZWMgamUsIHplIHNlIHRlbiBwb29sIHBvZGFyaWxvIGltcG9ydG92YXQg
amVuIHNlIFpGUyAwLjcuMC1yYzQsXHJcbmNveiBqZSBwcm9ibGVtLCBwcm90
b3plIHNlbmQvcmVjdiB6ZnMgc3RyZWFtdSBuZW5pIGtvbXBhdGliaWxuaSBz
ZVxyXG5aRlMgMC42LjUsIGt0ZXJ5IG1hbWUgdnN1ZGUgamluZGUuIFYgbm9j
aSB0ZWR5IG11emUgbmFzbGVkb3ZhdCBqZXN0ZVxyXG5kYWxzaSByZXN0YXJ0
IGEgcHJhdmRlcG9kb2JuZSBuZXByb2JlaG5vdSBkbmVzbmkgemFsb2h5IG5v
ZGU0LmJycS4ifX19
-----END BASE64 ENCODED PARSEABLE JSON-----
27 May
27 May
12:19 p.m.
Popis: Presun VPS na node1.brq
Pool se nam nepodarilo importovat s kompatibilni verzi ZFS, takze jsme
se rozhodli vsechny VPS presunout na jiny node. Snajpa do DC zavezl
opraveny node1.brq a zbyva presunout poslednich ~18 VPS. Behem dneska
by VPS mely byt dostupne z vpsAdminu. Poskozenych souboru je pomerne
hodne, jakmile budeme mit kompletni seznam, Michal vam bude rozesilat
maily.
Nahlásil: Jakub Skokan
ENGLISH:
Summary: Migration to node1.brq
We were unable to import the pool with a compatible ZFS version, so
we've decided to migrate all VPS to another node. node1.brq was serviced
and snajpa brought it back to DC. We're migrating the last ~18 VPS there.
All VPS should be accessible from vpsAdmin by today. There's quite a lot
of corrupted files, so as soon as we have a complete list, Michal will
start informing you.
Reported by: Jakub Skokan
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6Nzc0LCJjaGFuZ2VzIjp7fSwidHJhbnNsYXRpb25zIjp7ImVuIjp7
InN1bW1hcnkiOiJNaWdyYXRpb24gdG8gbm9kZTEuYnJxIiwiZGVzY3JpcHRp
b24iOiJXZSB3ZXJlIHVuYWJsZSB0byBpbXBvcnQgdGhlIHBvb2wgd2l0aCBh
IGNvbXBhdGlibGUgWkZTIHZlcnNpb24sIHNvXHJcbndlJ3ZlIGRlY2lkZWQg
dG8gbWlncmF0ZSBhbGwgVlBTIHRvIGFub3RoZXIgbm9kZS4gbm9kZTEuYnJx
IHdhcyBzZXJ2aWNlZFxyXG5hbmQgc25hanBhIGJyb3VnaHQgaXQgYmFjayB0
byBEQy4gV2UncmUgbWlncmF0aW5nIHRoZSBsYXN0IH4xOCBWUFMgdGhlcmUu
XHJcbkFsbCBWUFMgc2hvdWxkIGJlIGFjY2Vzc2libGUgZnJvbSB2cHNBZG1p
biBieSB0b2RheS4gVGhlcmUncyBxdWl0ZSBhIGxvdFxyXG5vZiBjb3JydXB0
ZWQgZmlsZXMsIHNvIGFzIHNvb24gYXMgd2UgaGF2ZSBhIGNvbXBsZXRlIGxp
c3QsIE1pY2hhbCB3aWxsXHJcbnN0YXJ0IGluZm9ybWluZyB5b3UuIn0sImNz
Ijp7InN1bW1hcnkiOiJQcmVzdW4gVlBTIG5hIG5vZGUxLmJycSIsImRlc2Ny
aXB0aW9uIjoiUG9vbCBzZSBuYW0gbmVwb2RhcmlsbyBpbXBvcnRvdmF0IHMg
a29tcGF0aWJpbG5pIHZlcnppIFpGUywgdGFremUganNtZVxyXG5zZSByb3po
b2RsaSB2c2VjaG55IFZQUyBwcmVzdW5vdXQgbmEgamlueSBub2RlLiBTbmFq
cGEgZG8gREMgemF2ZXpsXHJcbm9wcmF2ZW55IG5vZGUxLmJycSBhIHpieXZh
IHByZXN1bm91dCBwb3NsZWRuaWNoIH4xOCBWUFMuIEJlaGVtIGRuZXNrYVxy
XG5ieSBWUFMgbWVseSBieXQgZG9zdHVwbmUgeiB2cHNBZG1pbnUuIFBvc2tv
emVueWNoIHNvdWJvcnUgamUgcG9tZXJuZVxyXG5ob2RuZSwgamFrbWlsZSBi
dWRlbWUgbWl0IGtvbXBsZXRuaSBzZXpuYW0sIE1pY2hhbCB2YW0gYnVkZSBy
b3plc2lsYXRcclxubWFpbHkuIn19fQ==
-----END BASE64 ENCODED PARSEABLE JSON-----
6:30 p.m.
Stav: announced -> closed
Popis: Migrace na node1.brq dokoncena, VPS pristupne
Vsechny VPS uz jsou na node1.brq a lze je ovladat z vpsAdminu. node4.brq reinstalujeme a
vybrane VPS na nej budeme vracet kvuli rozlozeni zateze
(node4.brq ma SSD, node1.brq HDD). Tyto migrace probehnou uz standardne
pres vpsAdmin, tzn. budete o tom informovani mailem a migrace se provede
v nastavenem oknu pro vypadky.
Nahlásil: Jakub Skokan
ENGLISH:
State: announced -> closed
Summary: Migration to node1.brq finished, VPS accessible
All VPS have been migrated to node1.brq and are accessible via vpsAdmin.
node4.brq will be reinstalled and selected VPS will be migrated back
to spread the load (node4.brq uses SSD, node1.brq has HDD). These migrations
will be handled by vpsAdmin, i.e. you'll be informed via mail and the
migration will take place during configured outage window.
Reported by: Jakub Skokan
-----BEGIN BASE64 ENCODED PARSEABLE JSON-----
eyJpZCI6Nzc1LCJjaGFuZ2VzIjp7InN0YXRlIjp7ImZyb20iOiJhbm5vdW5j
ZWQiLCJ0byI6ImNsb3NlZCJ9fSwidHJhbnNsYXRpb25zIjp7ImVuIjp7InN1
bW1hcnkiOiJNaWdyYXRpb24gdG8gbm9kZTEuYnJxIGZpbmlzaGVkLCBWUFMg
YWNjZXNzaWJsZSIsImRlc2NyaXB0aW9uIjoiQWxsIFZQUyBoYXZlIGJlZW4g
bWlncmF0ZWQgdG8gbm9kZTEuYnJxIGFuZCBhcmUgYWNjZXNzaWJsZSB2aWEg
dnBzQWRtaW4uXHJcbm5vZGU0LmJycSB3aWxsIGJlIHJlaW5zdGFsbGVkIGFu
ZCBzZWxlY3RlZCBWUFMgd2lsbCBiZSBtaWdyYXRlZCBiYWNrXHJcbnRvIHNw
cmVhZCB0aGUgbG9hZCAobm9kZTQuYnJxIHVzZXMgU1NELCBub2RlMS5icnEg
aGFzIEhERCkuIFRoZXNlIG1pZ3JhdGlvbnNcclxud2lsbCBiZSBoYW5kbGVk
IGJ5IHZwc0FkbWluLCBpLmUuIHlvdSdsbCBiZSBpbmZvcm1lZCB2aWEgbWFp
bCBhbmQgdGhlXHJcbm1pZ3JhdGlvbiB3aWxsIHRha2UgcGxhY2UgZHVyaW5n
IGNvbmZpZ3VyZWQgb3V0YWdlIHdpbmRvdy4ifSwiY3MiOnsic3VtbWFyeSI6
Ik1pZ3JhY2UgbmEgbm9kZTEuYnJxIGRva29uY2VuYSwgVlBTIHByaXN0dXBu
ZSIsImRlc2NyaXB0aW9uIjoiVnNlY2hueSBWUFMgdXoganNvdSBuYSBub2Rl
MS5icnEgYSBsemUgamUgb3ZsYWRhdCB6IHZwc0FkbWludS4gbm9kZTQuYnJx
IHJlaW5zdGFsdWplbWUgYSB2eWJyYW5lIFZQUyBuYSBuZWogYnVkZW1lIHZy
YWNldCBrdnVsaSByb3psb3plbmkgemF0ZXplXHJcbihub2RlNC5icnEgbWEg
U1NELCBub2RlMS5icnEgSEREKS4gVHl0byBtaWdyYWNlIHByb2JlaG5vdSB1
eiBzdGFuZGFyZG5lXHJcbnByZXMgdnBzQWRtaW4sIHR6bi4gYnVkZXRlIG8g
dG9tIGluZm9ybW92YW5pIG1haWxlbSBhIG1pZ3JhY2Ugc2UgcHJvdmVkZVxy
XG52IG5hc3RhdmVuZW0gb2tudSBwcm8gdnlwYWRreS4ifX19
-----END BASE64 ENCODED PARSEABLE JSON-----
2546
days inactive
2547
days old
4 comments
1 participants
participants (1)
-
podpora＠vpsfree.cz