Wiederherstellung gelöschter Daten auf einem ZFS-Mirror

Yoda · 4 Juli 2019

Hallo Leute,

ich habe versehentlich auf einem ZFS-Tank (RAID-1 / Mirror) ein Verzeichnis gelöscht.
Anschl. wurde das Verzeichnis wider angelegt, es wurden (meines Wissens nach) keine neuen Daten dadrauf abgespeichert...

Als ich das Desaster bemerkte, habe ich eine Platte (10TB) aus dem Tank schnell aus dem laufenden System entfernt.

Eine Nachfrage bei "www.ontrack.com" ergab, dass eine Datenrettung bei denen Kosten im 4-Stelligen Bereich verursachen würde...! :-(

Jetzt habe ich diese Software gefunden, die das wohl für 50€ auch können sollte: https://www.ufsexplorer.com/de/ufs-explorer-standard-recovery.php

Meine Frage, welche Möglichkeiten habe ich außerdem noch?
Gibt es eine Möglichkeit die gelöschten Daten mit "Boardmitteln" wieder zu bekommen?
Oder kennt einer eine Software bzw. einen erprobten Weg, wie das mit ZFS geht?

Für hilfreiche Antworten wäre ich dankbar.

Gruß
Yoda

Columbo0815 · 4 Juli 2019

Moin,

ich befürchte, dass dir mein Tipp nicht viel hilft. Da du aber ZFS nutzt, sollte man davon ausgehen können, dass du Snapshots angelegt hast. Die Daten sollten dann auf einem Snapshot verfügbar sein. Wenn es keine Snapshots gibt und sonst auch kein Backup, sehe ich keine wirkliche Chance an die Daten zu kommen.

Ansehen kannst du dir mal testdisk (in den Ports und als Package verfügbar). Angeblich [1] hat das auch ZFS Support. Edit: Der Support beschränkt sich wohl nur auf Partitionen und nicht auf Dateien...

HTH!

[1] https://www.cgsecurity.org/wiki/TestDisk

kira12 · 4 Juli 2019

Hallo,

Photorec ist auch sehr hilfreich, allerdings keine Ahnung wie es auch ZFS reagiert.

Gruß ré

mr44er · 4 Juli 2019

testdisk und photorec sind die zu bevorzugenden Werkzeuge, allerdings habe ich damit nur auf ntfs und ufs rumgeschraubt.
Wie das auf ZFS reagiert, würde mich auch interessieren.

Viel Glück!

(Leider riecht das mal wieder, als wäre kein Backup vorhanden...ganz ohne Vorwurf und Belehrung: für ne 4-stellige Summe kann man viel Backupmöglichkeiten einkaufen.)

Columbo0815 · 4 Juli 2019

Photorec ist Bestandteil von testdisk

medV2 · 4 Juli 2019

Ich befürchte Photorec kommt mit ZFS nicht klar. Zum Thema Datenrettung: Es gibt nicht nur Ontrack, ich hatte selbst mal einen Fall da hab ich ein lokales Datenrettungsunternehmen beauftragt. War zwar auch im 4 stelligen Bereich, allerdings knapp über der Tausend + Gratis Kostenvoranschlag. Ontrack ist da wesentlich teurer. Da der Datenträger physisch in ordnung ist sollte es eine Software wohl auch tun, allerdings habe ich da keine Erfahrung, vorallem ZFS könnte kritisch werden, da es da ja auch viele Versionen gibt (SUN ZFS, Orcale ZFS, OpenZFS + verschiedene Features)..

Yoda · 4 Juli 2019

...ja, ich habe aus Speicherplatzmangel weder die eine noch die andere vorbeugende Maßnahme getroffen - hab das seit 20 Jahren so laufen und dachte nicht, das ich mal soeine Dummheit machen würde... :-(

Als ertes probiere ich mal die Test-Version von dem 50€-Teil aus, wenn das funktioniert (die Beschreibung stellt es in Aussicht), dann kaufe ich als erstes mal das Ding und sehe dann weiter...

Danke für Eure Unterstützung! :-)

Gruß
Yoda

Yamagi · 4 Juli 2019

Also... ZFS ist ein Copy on Write Dateisystem und transaktionsbasiert. Es überschreibt sowohl Daten, als auch Metadaten nach frühestens 100 Transaktionen. Eine Transaktion wird ausgeführt, wenn die aktuelle Transaktionsgruppe gefüllt ist oder 5 Sekunden vergangen sind. Wobei leere Transaktionsgruppen nicht zwingend Daten überschreiben. Wenn du die Platte schnell genug aus dem Pool gezogen hast, sind deine Daten also wohl und munter.

ZFS hat nun eine ganze Reihe mehr Funktionen, als in den Manpages steht. Darunter auch die Möglichkeit auf eine ältere Transaktiongruppe zurückzurollen und damit mit etwas Glück dein fatales Löschen rückgängig zu machen. Diese Funktionen sind undokumentiert, da sie Daten zerstören und man das unbedarften Anwendern nicht an die Hand geben will. Daher: Kopiere die Platte vorher einmal mit dd(8) durch.

So, im ersten Schritt musst du herausfinden welche Transaktionen auf deinem Pool (ich nenne ihn hier einfach tank und gehe davon aus, dass er /dev/ada0p1 liegt) vorhanden sind. Dafür machst du einfach ein:

Code:

% zdb -ul /dev/ada0p1

Das Ergebnis ist eine lange Liste. Der Pool hat mehrere Kopien der Metadaten, sogenannte Label. Jedes Laben hat wiederum eine Reihe Transaktionen zuegordnet. Ist der Pool konsistent, sind es bei allen Label die gleichen Transaktionen. Als Beispiel einmal mein Pool vm, auf dem meine virtuellen Maschinen liegen:

Code:

------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'vm'
    state: 0
    txg: 6849
    pool_guid: 4833349645476312860
    hostid: 3551443610
    hostname: 'pine.home.yamagi.org'
    top_guid: 5842668963473420051
    guid: 5842668963473420051
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 5842668963473420051
        path: '/dev/ada0p1.eli'
        whole_disk: 1
        metaslab_array: 67
        metaslab_shift: 32
        ashift: 12
        asize: 480099172352
        is_log: 0
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
Uberblock[0]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699104
    guid_sum = 10676018608949732911
    timestamp = 1562251429 UTC = Thu Jul  4 16:43:49 2019
    checkpoint_txg = 0
Uberblock[1]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694113
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[2]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699106
    guid_sum = 10676018608949732911
    timestamp = 1562251439 UTC = Thu Jul  4 16:43:59 2019
    checkpoint_txg = 0
Uberblock[3]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699107
    guid_sum = 10676018608949732911
    timestamp = 1562251444 UTC = Thu Jul  4 16:44:04 2019
    checkpoint_txg = 0
Uberblock[4]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695492
    guid_sum = 10676018608949732911
    timestamp = 1562090405 UTC = Tue Jul  2 20:00:05 2019
    checkpoint_txg = 0
Uberblock[5]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697669
    guid_sum = 10676018608949732911
    timestamp = 1562176800 UTC = Wed Jul  3 20:00:00 2019
    checkpoint_txg = 0
Uberblock[6]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697670
    guid_sum = 10676018608949732911
    timestamp = 1562176805 UTC = Wed Jul  3 20:00:05 2019
    checkpoint_txg = 0
Uberblock[7]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694119
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[8]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696296
    guid_sum = 10676018608949732911
    timestamp = 1562169901 UTC = Wed Jul  3 18:05:01 2019
    checkpoint_txg = 0
Uberblock[9]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696297
    guid_sum = 10676018608949732911
    timestamp = 1562169906 UTC = Wed Jul  3 18:05:06 2019
    checkpoint_txg = 0
Uberblock[10]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690154
    guid_sum = 10676018608949732911
    timestamp = 1561986301 UTC = Mon Jul  1 15:05:01 2019
    checkpoint_txg = 0
Uberblock[11]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690155
    guid_sum = 10676018608949732911
    timestamp = 1561986306 UTC = Mon Jul  1 15:05:06 2019
    checkpoint_txg = 0
Uberblock[12]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696236
    guid_sum = 10676018608949732911
    timestamp = 1562169600 UTC = Wed Jul  3 18:00:00 2019
    checkpoint_txg = 0
Uberblock[13]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696237
    guid_sum = 10676018608949732911
    timestamp = 1562169605 UTC = Wed Jul  3 18:00:05 2019
    checkpoint_txg = 0
Uberblock[14]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690094
    guid_sum = 10676018608949732911
    timestamp = 1561986000 UTC = Mon Jul  1 15:00:00 2019
    checkpoint_txg = 0
Uberblock[15]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690095
    guid_sum = 10676018608949732911
    timestamp = 1561986005 UTC = Mon Jul  1 15:00:05 2019
    checkpoint_txg = 0
Uberblock[16]
    magic = 0000000000bab10c
    version = 5000
    txg = 1686000
    guid_sum = 10676018608949732911
    timestamp = 1561965458 UTC = Mon Jul  1 09:17:38 2019
    checkpoint_txg = 0
Uberblock[17]
    magic = 0000000000bab10c
    version = 5000
    txg = 1685969
    guid_sum = 10676018608949732911
    timestamp = 1561965303 UTC = Mon Jul  1 09:15:03 2019
    checkpoint_txg = 0
Uberblock[18]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698386
    guid_sum = 10676018608949732911
    timestamp = 1562180400 UTC = Wed Jul  3 21:00:00 2019
    checkpoint_txg = 0
Uberblock[19]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698387
    guid_sum = 10676018608949732911
    timestamp = 1562180405 UTC = Wed Jul  3 21:00:05 2019
    checkpoint_txg = 0
Uberblock[20]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688660
    guid_sum = 10676018608949732911
    timestamp = 1561978800 UTC = Mon Jul  1 13:00:00 2019
    checkpoint_txg = 0
Uberblock[21]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688661
    guid_sum = 10676018608949732911
    timestamp = 1561978805 UTC = Mon Jul  1 13:00:05 2019
    checkpoint_txg = 0
Uberblock[22]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694774
    guid_sum = 10676018608949732911
    timestamp = 1562086800 UTC = Tue Jul  2 19:00:00 2019
    checkpoint_txg = 0
Uberblock[23]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694775
    guid_sum = 10676018608949732911
    timestamp = 1562086805 UTC = Tue Jul  2 19:00:05 2019
    checkpoint_txg = 0
Uberblock[24]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696952
    guid_sum = 10676018608949732911
    timestamp = 1562173200 UTC = Wed Jul  3 19:00:00 2019
    checkpoint_txg = 0
Uberblock[25]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696953
    guid_sum = 10676018608949732911
    timestamp = 1562173205 UTC = Wed Jul  3 19:00:05 2019
    checkpoint_txg = 0
Uberblock[26]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695578
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[27]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695579
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[28]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695580
    guid_sum = 10676018608949732911
    timestamp = 1562166308 UTC = Wed Jul  3 17:05:08 2019
    checkpoint_txg = 0
Uberblock[29]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699101
    guid_sum = 10676018608949732911
    timestamp = 1562184000 UTC = Wed Jul  3 22:00:00 2019
    checkpoint_txg = 0
Uberblock[30]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699102
    guid_sum = 10676018608949732911
    timestamp = 1562184005 UTC = Wed Jul  3 22:00:05 2019
    checkpoint_txg = 0
Uberblock[31]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699103
    guid_sum = 10676018608949732911
    timestamp = 1562251424 UTC = Thu Jul  4 16:43:44 2019
    checkpoint_txg = 0
------------------------------------
LABEL 1
------------------------------------
    version: 5000
    name: 'vm'
    state: 0
    txg: 6849
    pool_guid: 4833349645476312860
    hostid: 3551443610
    hostname: 'pine.home.yamagi.org'
    top_guid: 5842668963473420051
    guid: 5842668963473420051
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 5842668963473420051
        path: '/dev/ada0p1.eli'
        whole_disk: 1
        metaslab_array: 67
        metaslab_shift: 32
        ashift: 12
        asize: 480099172352
        is_log: 0
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
Uberblock[0]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699104
    guid_sum = 10676018608949732911
    timestamp = 1562251429 UTC = Thu Jul  4 16:43:49 2019
    checkpoint_txg = 0
Uberblock[1]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694113
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[2]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699106
    guid_sum = 10676018608949732911
    timestamp = 1562251439 UTC = Thu Jul  4 16:43:59 2019
    checkpoint_txg = 0
Uberblock[3]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699107
    guid_sum = 10676018608949732911
    timestamp = 1562251444 UTC = Thu Jul  4 16:44:04 2019
    checkpoint_txg = 0
Uberblock[4]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695492
    guid_sum = 10676018608949732911
    timestamp = 1562090405 UTC = Tue Jul  2 20:00:05 2019
    checkpoint_txg = 0
Uberblock[5]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697669
    guid_sum = 10676018608949732911
    timestamp = 1562176800 UTC = Wed Jul  3 20:00:00 2019
    checkpoint_txg = 0
Uberblock[6]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697670
    guid_sum = 10676018608949732911
    timestamp = 1562176805 UTC = Wed Jul  3 20:00:05 2019
    checkpoint_txg = 0
Uberblock[7]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694119
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[8]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696296
    guid_sum = 10676018608949732911
    timestamp = 1562169901 UTC = Wed Jul  3 18:05:01 2019
    checkpoint_txg = 0
Uberblock[9]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696297
    guid_sum = 10676018608949732911
    timestamp = 1562169906 UTC = Wed Jul  3 18:05:06 2019
    checkpoint_txg = 0
Uberblock[10]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690154
    guid_sum = 10676018608949732911
    timestamp = 1561986301 UTC = Mon Jul  1 15:05:01 2019
    checkpoint_txg = 0
Uberblock[11]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690155
    guid_sum = 10676018608949732911
    timestamp = 1561986306 UTC = Mon Jul  1 15:05:06 2019
    checkpoint_txg = 0
Uberblock[12]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696236
    guid_sum = 10676018608949732911
    timestamp = 1562169600 UTC = Wed Jul  3 18:00:00 2019
    checkpoint_txg = 0
Uberblock[13]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696237
    guid_sum = 10676018608949732911
    timestamp = 1562169605 UTC = Wed Jul  3 18:00:05 2019
    checkpoint_txg = 0
Uberblock[14]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690094
    guid_sum = 10676018608949732911
    timestamp = 1561986000 UTC = Mon Jul  1 15:00:00 2019
    checkpoint_txg = 0
Uberblock[15]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690095
    guid_sum = 10676018608949732911
    timestamp = 1561986005 UTC = Mon Jul  1 15:00:05 2019
    checkpoint_txg = 0
Uberblock[16]
    magic = 0000000000bab10c
    version = 5000
    txg = 1686000
    guid_sum = 10676018608949732911
    timestamp = 1561965458 UTC = Mon Jul  1 09:17:38 2019
    checkpoint_txg = 0
Uberblock[17]
    magic = 0000000000bab10c
    version = 5000
    txg = 1685969
    guid_sum = 10676018608949732911
    timestamp = 1561965303 UTC = Mon Jul  1 09:15:03 2019
    checkpoint_txg = 0
Uberblock[18]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698386
    guid_sum = 10676018608949732911
    timestamp = 1562180400 UTC = Wed Jul  3 21:00:00 2019
    checkpoint_txg = 0
Uberblock[19]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698387
    guid_sum = 10676018608949732911
    timestamp = 1562180405 UTC = Wed Jul  3 21:00:05 2019
    checkpoint_txg = 0
Uberblock[20]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688660
    guid_sum = 10676018608949732911
    timestamp = 1561978800 UTC = Mon Jul  1 13:00:00 2019
    checkpoint_txg = 0
Uberblock[21]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688661
    guid_sum = 10676018608949732911
    timestamp = 1561978805 UTC = Mon Jul  1 13:00:05 2019
    checkpoint_txg = 0
Uberblock[22]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694774
    guid_sum = 10676018608949732911
    timestamp = 1562086800 UTC = Tue Jul  2 19:00:00 2019
    checkpoint_txg = 0
Uberblock[23]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694775
    guid_sum = 10676018608949732911
    timestamp = 1562086805 UTC = Tue Jul  2 19:00:05 2019
    checkpoint_txg = 0
Uberblock[24]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696952
    guid_sum = 10676018608949732911
    timestamp = 1562173200 UTC = Wed Jul  3 19:00:00 2019
    checkpoint_txg = 0
Uberblock[25]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696953
    guid_sum = 10676018608949732911
    timestamp = 1562173205 UTC = Wed Jul  3 19:00:05 2019
    checkpoint_txg = 0
Uberblock[26]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695578
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[27]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695579
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[28]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695580
    guid_sum = 10676018608949732911
    timestamp = 1562166308 UTC = Wed Jul  3 17:05:08 2019
    checkpoint_txg = 0
Uberblock[29]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699101
    guid_sum = 10676018608949732911
    timestamp = 1562184000 UTC = Wed Jul  3 22:00:00 2019
    checkpoint_txg = 0
Uberblock[30]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699102
    guid_sum = 10676018608949732911
    timestamp = 1562184005 UTC = Wed Jul  3 22:00:05 2019
    checkpoint_txg = 0
Uberblock[31]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699103
    guid_sum = 10676018608949732911
    timestamp = 1562251424 UTC = Thu Jul  4 16:43:44 2019
    checkpoint_txg = 0
------------------------------------
LABEL 2
------------------------------------
    version: 5000
    name: 'vm'
    state: 0
    txg: 6849
    pool_guid: 4833349645476312860
    hostid: 3551443610
    hostname: 'pine.home.yamagi.org'
    top_guid: 5842668963473420051
    guid: 5842668963473420051
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 5842668963473420051
        path: '/dev/ada0p1.eli'
        whole_disk: 1
        metaslab_array: 67
        metaslab_shift: 32
        ashift: 12
        asize: 480099172352
        is_log: 0
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
Uberblock[0]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699104
    guid_sum = 10676018608949732911
    timestamp = 1562251429 UTC = Thu Jul  4 16:43:49 2019
    checkpoint_txg = 0
Uberblock[1]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694113
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[2]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699106
    guid_sum = 10676018608949732911
    timestamp = 1562251439 UTC = Thu Jul  4 16:43:59 2019
    checkpoint_txg = 0
Uberblock[3]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699107
    guid_sum = 10676018608949732911
    timestamp = 1562251444 UTC = Thu Jul  4 16:44:04 2019
    checkpoint_txg = 0
Uberblock[4]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695492
    guid_sum = 10676018608949732911
    timestamp = 1562090405 UTC = Tue Jul  2 20:00:05 2019
    checkpoint_txg = 0
Uberblock[5]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697669
    guid_sum = 10676018608949732911
    timestamp = 1562176800 UTC = Wed Jul  3 20:00:00 2019
    checkpoint_txg = 0
Uberblock[6]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697670
    guid_sum = 10676018608949732911
    timestamp = 1562176805 UTC = Wed Jul  3 20:00:05 2019
    checkpoint_txg = 0
Uberblock[7]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694119
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[8]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696296
    guid_sum = 10676018608949732911
    timestamp = 1562169901 UTC = Wed Jul  3 18:05:01 2019
    checkpoint_txg = 0
Uberblock[9]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696297
    guid_sum = 10676018608949732911
    timestamp = 1562169906 UTC = Wed Jul  3 18:05:06 2019
    checkpoint_txg = 0
Uberblock[10]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690154
    guid_sum = 10676018608949732911
    timestamp = 1561986301 UTC = Mon Jul  1 15:05:01 2019
    checkpoint_txg = 0
Uberblock[11]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690155
    guid_sum = 10676018608949732911
    timestamp = 1561986306 UTC = Mon Jul  1 15:05:06 2019
    checkpoint_txg = 0
Uberblock[12]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696236
    guid_sum = 10676018608949732911
    timestamp = 1562169600 UTC = Wed Jul  3 18:00:00 2019
    checkpoint_txg = 0
Uberblock[13]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696237
    guid_sum = 10676018608949732911
    timestamp = 1562169605 UTC = Wed Jul  3 18:00:05 2019
    checkpoint_txg = 0
Uberblock[14]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690094
    guid_sum = 10676018608949732911
    timestamp = 1561986000 UTC = Mon Jul  1 15:00:00 2019
    checkpoint_txg = 0
Uberblock[15]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690095
    guid_sum = 10676018608949732911
    timestamp = 1561986005 UTC = Mon Jul  1 15:00:05 2019
    checkpoint_txg = 0
Uberblock[16]
    magic = 0000000000bab10c
    version = 5000
    txg = 1686000
    guid_sum = 10676018608949732911
    timestamp = 1561965458 UTC = Mon Jul  1 09:17:38 2019
    checkpoint_txg = 0
Uberblock[17]
    magic = 0000000000bab10c
    version = 5000
    txg = 1685969
    guid_sum = 10676018608949732911
    timestamp = 1561965303 UTC = Mon Jul  1 09:15:03 2019
    checkpoint_txg = 0
Uberblock[18]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698386
    guid_sum = 10676018608949732911
    timestamp = 1562180400 UTC = Wed Jul  3 21:00:00 2019
    checkpoint_txg = 0
Uberblock[19]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698387
    guid_sum = 10676018608949732911
    timestamp = 1562180405 UTC = Wed Jul  3 21:00:05 2019
    checkpoint_txg = 0
Uberblock[20]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688660
    guid_sum = 10676018608949732911
    timestamp = 1561978800 UTC = Mon Jul  1 13:00:00 2019
    checkpoint_txg = 0
Uberblock[21]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688661
    guid_sum = 10676018608949732911
    timestamp = 1561978805 UTC = Mon Jul  1 13:00:05 2019
    checkpoint_txg = 0
Uberblock[22]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694774
    guid_sum = 10676018608949732911
    timestamp = 1562086800 UTC = Tue Jul  2 19:00:00 2019
    checkpoint_txg = 0
Uberblock[23]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694775
    guid_sum = 10676018608949732911
    timestamp = 1562086805 UTC = Tue Jul  2 19:00:05 2019
    checkpoint_txg = 0
Uberblock[24]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696952
    guid_sum = 10676018608949732911
    timestamp = 1562173200 UTC = Wed Jul  3 19:00:00 2019
    checkpoint_txg = 0
Uberblock[25]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696953
    guid_sum = 10676018608949732911
    timestamp = 1562173205 UTC = Wed Jul  3 19:00:05 2019
    checkpoint_txg = 0
Uberblock[26]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695578
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[27]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695579
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[28]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695580
    guid_sum = 10676018608949732911
    timestamp = 1562166308 UTC = Wed Jul  3 17:05:08 2019
    checkpoint_txg = 0
Uberblock[29]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699101
    guid_sum = 10676018608949732911
    timestamp = 1562184000 UTC = Wed Jul  3 22:00:00 2019
    checkpoint_txg = 0
Uberblock[30]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699102
    guid_sum = 10676018608949732911
    timestamp = 1562184005 UTC = Wed Jul  3 22:00:05 2019
    checkpoint_txg = 0
Uberblock[31]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699103
    guid_sum = 10676018608949732911
    timestamp = 1562251424 UTC = Thu Jul  4 16:43:44 2019
    checkpoint_txg = 0
------------------------------------
LABEL 3
------------------------------------
    version: 5000
    name: 'vm'
    state: 0
    txg: 6849
    pool_guid: 4833349645476312860
    hostid: 3551443610
    hostname: 'pine.home.yamagi.org'
    top_guid: 5842668963473420051
    guid: 5842668963473420051
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 5842668963473420051
        path: '/dev/ada0p1.eli'
        whole_disk: 1
        metaslab_array: 67
        metaslab_shift: 32
        ashift: 12
        asize: 480099172352
        is_log: 0
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
Uberblock[0]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699104
    guid_sum = 10676018608949732911
    timestamp = 1562251429 UTC = Thu Jul  4 16:43:49 2019
    checkpoint_txg = 0
Uberblock[1]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694113
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[2]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699106
    guid_sum = 10676018608949732911
    timestamp = 1562251439 UTC = Thu Jul  4 16:43:59 2019
    checkpoint_txg = 0
Uberblock[3]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699107
    guid_sum = 10676018608949732911
    timestamp = 1562251444 UTC = Thu Jul  4 16:44:04 2019
    checkpoint_txg = 0
Uberblock[4]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695492
    guid_sum = 10676018608949732911
    timestamp = 1562090405 UTC = Tue Jul  2 20:00:05 2019
    checkpoint_txg = 0
Uberblock[5]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697669
    guid_sum = 10676018608949732911
    timestamp = 1562176800 UTC = Wed Jul  3 20:00:00 2019
    checkpoint_txg = 0
Uberblock[6]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697670
    guid_sum = 10676018608949732911
    timestamp = 1562176805 UTC = Wed Jul  3 20:00:05 2019
    checkpoint_txg = 0
Uberblock[7]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694119
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[8]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696296
    guid_sum = 10676018608949732911
    timestamp = 1562169901 UTC = Wed Jul  3 18:05:01 2019
    checkpoint_txg = 0
Uberblock[9]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696297
    guid_sum = 10676018608949732911
    timestamp = 1562169906 UTC = Wed Jul  3 18:05:06 2019
    checkpoint_txg = 0
Uberblock[10]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690154
    guid_sum = 10676018608949732911
    timestamp = 1561986301 UTC = Mon Jul  1 15:05:01 2019
    checkpoint_txg = 0
Uberblock[11]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690155
    guid_sum = 10676018608949732911
    timestamp = 1561986306 UTC = Mon Jul  1 15:05:06 2019
    checkpoint_txg = 0
Uberblock[12]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696236
    guid_sum = 10676018608949732911
    timestamp = 1562169600 UTC = Wed Jul  3 18:00:00 2019
    checkpoint_txg = 0
Uberblock[13]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696237
    guid_sum = 10676018608949732911
    timestamp = 1562169605 UTC = Wed Jul  3 18:00:05 2019
    checkpoint_txg = 0
Uberblock[14]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690094
    guid_sum = 10676018608949732911
    timestamp = 1561986000 UTC = Mon Jul  1 15:00:00 2019
    checkpoint_txg = 0
Uberblock[15]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690095
    guid_sum = 10676018608949732911
    timestamp = 1561986005 UTC = Mon Jul  1 15:00:05 2019
    checkpoint_txg = 0
Uberblock[16]
    magic = 0000000000bab10c
    version = 5000
    txg = 1686000
    guid_sum = 10676018608949732911
    timestamp = 1561965458 UTC = Mon Jul  1 09:17:38 2019
    checkpoint_txg = 0
Uberblock[17]
    magic = 0000000000bab10c
    version = 5000
    txg = 1685969
    guid_sum = 10676018608949732911
    timestamp = 1561965303 UTC = Mon Jul  1 09:15:03 2019
    checkpoint_txg = 0
Uberblock[18]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698386
    guid_sum = 10676018608949732911
    timestamp = 1562180400 UTC = Wed Jul  3 21:00:00 2019
    checkpoint_txg = 0
Uberblock[19]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698387
    guid_sum = 10676018608949732911
    timestamp = 1562180405 UTC = Wed Jul  3 21:00:05 2019
    checkpoint_txg = 0
Uberblock[20]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688660
    guid_sum = 10676018608949732911
    timestamp = 1561978800 UTC = Mon Jul  1 13:00:00 2019
    checkpoint_txg = 0
Uberblock[21]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688661
    guid_sum = 10676018608949732911
    timestamp = 1561978805 UTC = Mon Jul  1 13:00:05 2019
    checkpoint_txg = 0
Uberblock[22]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694774
    guid_sum = 10676018608949732911
    timestamp = 1562086800 UTC = Tue Jul  2 19:00:00 2019
    checkpoint_txg = 0
Uberblock[23]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694775
    guid_sum = 10676018608949732911
    timestamp = 1562086805 UTC = Tue Jul  2 19:00:05 2019
    checkpoint_txg = 0
Uberblock[24]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696952
    guid_sum = 10676018608949732911
    timestamp = 1562173200 UTC = Wed Jul  3 19:00:00 2019
    checkpoint_txg = 0
Uberblock[25]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696953
    guid_sum = 10676018608949732911
    timestamp = 1562173205 UTC = Wed Jul  3 19:00:05 2019
    checkpoint_txg = 0
Uberblock[26]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695578
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[27]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695579
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[28]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695580
    guid_sum = 10676018608949732911
    timestamp = 1562166308 UTC = Wed Jul  3 17:05:08 2019
    checkpoint_txg = 0
Uberblock[29]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699101
    guid_sum = 10676018608949732911
    timestamp = 1562184000 UTC = Wed Jul  3 22:00:00 2019
    checkpoint_txg = 0
Uberblock[30]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699102
    guid_sum = 10676018608949732911
    timestamp = 1562184005 UTC = Wed Jul  3 22:00:05 2019
    checkpoint_txg = 0
Uberblock[31]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699103
    guid_sum = 10676018608949732911
    timestamp = 1562251424 UTC = Thu Jul  4 16:43:44 2019
    checkpoint_txg = 0

Also 4 Label (Metadatenkopien) mit 31 Transaktionen. Da sieht man nun auch schön an den Timestamps, dass die Kiste in den letzten Tagen eher weniger benutzt wurde. Nun musst eine Transaktion suchen, auf die du zurück möchtest. Hangele dich an den Timestamps lang und beachtet, dass der Pool durch das gewaltsame ziehen der Platte eventuell inkonsistent sein wird. Also nicht jedes Label alle Transaktionen enthalten muss.

Hast du dich für eine Transaktion entschieden, braucht du ihre ID. Das ist das Feld txg in der Ausgabe. Also wenn ich auf die Transaktion vom timestamp = 1562184005 UTC = Wed Jul 3 22:00:05 2019 zurück möchte, wähle ich txg = 1699102

Mit diesen Informationen kann der Pool nun importiert werden. Das Importieren ist zerstörend, es verwirft alle Transaktionen, die neuer als die gewünschte Transaktion sind. Unwiderbringlich, für immer, nada. Daher das Backup oben. Ich würde ihn read-only importieren, aber an sich ist es nicht notwendig.

Code:

% pool import -R /mnt -o readonly=on -f -T $txg tank

-R /mnt um Pfadkollissionen zu verhindern, -f da der Pool noch als importiert gekennzeichnet sein wird und -T $txg ist die undokumentierte Brechstange. Das kann ein paar Minuten dauern, keine Panik. Anschließend bist du idealerweise auf einem Stand vor dem Desaster, kannst deine Daten rauskopieren und bist mir ewig dankbar.

medV2 · 4 Juli 2019

@Yamagi unser ZFS Guru

Cooles Feature das mir auch unbekannt war, schade, dass sowas nicht dokumentiert ist. Ein zfs destroy -R oder so kann ja auch im Zweifel Daten zerstören und ist in der Doc.

Kann man die Anzahl der Transaktionen irgendwie beeinflussen die hier "aufgehoben" werden? An meinem Pool werden z.b. die letzten 128 Transaktionen gelistet. Mein txg.timeout ist auf 12, ansonsten habe ich bei den txgs nichts herumgespielt.

Yamagi · 5 Juli 2019

medV2 schrieb:
Kann man die Anzahl der Transaktionen irgendwie beeinflussen die hier "aufgehoben" werden?

Das weiß ich nicht. Google findet da auf die Schnelle auch nichts zu, vielleicht geht das einfach nicht.

medV2 · 5 Juli 2019

Hab auch schon gesucht

zum Glück ist ja Freitag.
Ist wohl fix ein Ringbuffer mit 128 Einträgen. Ich hab heute auch im Büro die bsd Kisten gecheckt, alle 128 Einträge. Fraglich wieso du nur 32 hattest, vielleicht ganz neuer Pool oder Platz am Ende?

Yamagi · 5 Juli 2019

Gute Frage. Auf der Kiste haben alle Pools nur 32 Transaktionen. Allerdings sind die Pools auch wirklich alt, die habe ich irgendwann um und bei FreeBSD 9.0 angelegt und seitdem immer nur mit zpool upgrade -a aktualisiert. Vielleicht wurde das später man erhöht und betrifft nur neu angelegte Pools oder so.

Yoda · 5 Juli 2019

Yamagi schrieb:

Also... ZFS ist ein Copy on Write Dateisystem und transaktionsbasiert. Es überschreibt sowohl Daten, als auch Metadaten nach frühestens 100 Transaktionen. Eine Transaktion wird ausgeführt, wenn die aktuelle Transaktionsgruppe gefüllt ist oder 5 Sekunden vergangen sind. Wobei leere Transaktionsgruppen nicht zwingend Daten überschreiben. Wenn du die Platte schnell genug aus dem Pool gezogen hast, sind deine Daten also wohl und munter.

ZFS hat nun eine ganze Reihe mehr Funktionen, als in den Manpages steht. Darunter auch die Möglichkeit auf eine ältere Transaktiongruppe zurückzurollen und damit mit etwas Glück dein fatales Löschen rückgängig zu machen. Diese Funktionen sind undokumentiert, da sie Daten zerstören und man das unbedarften Anwendern nicht an die Hand geben will. Daher: Kopiere die Platte vorher einmal mit dd(8) durch.

So, im ersten Schritt musst du herausfinden welche Transaktionen auf deinem Pool (ich nenne ihn hier einfach tank und gehe davon aus, dass er /dev/ada0p1 liegt) vorhanden sind. Dafür machst du einfach ein:

Code:

% zdb -ul /dev/ada0p1

Das Ergebnis ist eine lange Liste. Der Pool hat mehrere Kopien der Metadaten, sogenannte Label. Jedes Laben hat wiederum eine Reihe Transaktionen zuegordnet. Ist der Pool konsistent, sind es bei allen Label die gleichen Transaktionen. Als Beispiel einmal mein Pool vm, auf dem meine virtuellen Maschinen liegen:

Code:

------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'vm'
    state: 0
    txg: 6849
    pool_guid: 4833349645476312860
    hostid: 3551443610
    hostname: 'pine.home.yamagi.org'
    top_guid: 5842668963473420051
    guid: 5842668963473420051
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 5842668963473420051
        path: '/dev/ada0p1.eli'
        whole_disk: 1
        metaslab_array: 67
        metaslab_shift: 32
        ashift: 12
        asize: 480099172352
        is_log: 0
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
Uberblock[0]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699104
    guid_sum = 10676018608949732911
    timestamp = 1562251429 UTC = Thu Jul  4 16:43:49 2019
    checkpoint_txg = 0
Uberblock[1]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694113
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[2]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699106
    guid_sum = 10676018608949732911
    timestamp = 1562251439 UTC = Thu Jul  4 16:43:59 2019
    checkpoint_txg = 0
Uberblock[3]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699107
    guid_sum = 10676018608949732911
    timestamp = 1562251444 UTC = Thu Jul  4 16:44:04 2019
    checkpoint_txg = 0
Uberblock[4]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695492
    guid_sum = 10676018608949732911
    timestamp = 1562090405 UTC = Tue Jul  2 20:00:05 2019
    checkpoint_txg = 0
Uberblock[5]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697669
    guid_sum = 10676018608949732911
    timestamp = 1562176800 UTC = Wed Jul  3 20:00:00 2019
    checkpoint_txg = 0
Uberblock[6]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697670
    guid_sum = 10676018608949732911
    timestamp = 1562176805 UTC = Wed Jul  3 20:00:05 2019
    checkpoint_txg = 0
Uberblock[7]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694119
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[8]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696296
    guid_sum = 10676018608949732911
    timestamp = 1562169901 UTC = Wed Jul  3 18:05:01 2019
    checkpoint_txg = 0
Uberblock[9]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696297
    guid_sum = 10676018608949732911
    timestamp = 1562169906 UTC = Wed Jul  3 18:05:06 2019
    checkpoint_txg = 0
Uberblock[10]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690154
    guid_sum = 10676018608949732911
    timestamp = 1561986301 UTC = Mon Jul  1 15:05:01 2019
    checkpoint_txg = 0
Uberblock[11]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690155
    guid_sum = 10676018608949732911
    timestamp = 1561986306 UTC = Mon Jul  1 15:05:06 2019
    checkpoint_txg = 0
Uberblock[12]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696236
    guid_sum = 10676018608949732911
    timestamp = 1562169600 UTC = Wed Jul  3 18:00:00 2019
    checkpoint_txg = 0
Uberblock[13]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696237
    guid_sum = 10676018608949732911
    timestamp = 1562169605 UTC = Wed Jul  3 18:00:05 2019
    checkpoint_txg = 0
Uberblock[14]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690094
    guid_sum = 10676018608949732911
    timestamp = 1561986000 UTC = Mon Jul  1 15:00:00 2019
    checkpoint_txg = 0
Uberblock[15]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690095
    guid_sum = 10676018608949732911
    timestamp = 1561986005 UTC = Mon Jul  1 15:00:05 2019
    checkpoint_txg = 0
Uberblock[16]
    magic = 0000000000bab10c
    version = 5000
    txg = 1686000
    guid_sum = 10676018608949732911
    timestamp = 1561965458 UTC = Mon Jul  1 09:17:38 2019
    checkpoint_txg = 0
Uberblock[17]
    magic = 0000000000bab10c
    version = 5000
    txg = 1685969
    guid_sum = 10676018608949732911
    timestamp = 1561965303 UTC = Mon Jul  1 09:15:03 2019
    checkpoint_txg = 0
Uberblock[18]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698386
    guid_sum = 10676018608949732911
    timestamp = 1562180400 UTC = Wed Jul  3 21:00:00 2019
    checkpoint_txg = 0
Uberblock[19]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698387
    guid_sum = 10676018608949732911
    timestamp = 1562180405 UTC = Wed Jul  3 21:00:05 2019
    checkpoint_txg = 0
Uberblock[20]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688660
    guid_sum = 10676018608949732911
    timestamp = 1561978800 UTC = Mon Jul  1 13:00:00 2019
    checkpoint_txg = 0
Uberblock[21]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688661
    guid_sum = 10676018608949732911
    timestamp = 1561978805 UTC = Mon Jul  1 13:00:05 2019
    checkpoint_txg = 0
Uberblock[22]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694774
    guid_sum = 10676018608949732911
    timestamp = 1562086800 UTC = Tue Jul  2 19:00:00 2019
    checkpoint_txg = 0
Uberblock[23]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694775
    guid_sum = 10676018608949732911
    timestamp = 1562086805 UTC = Tue Jul  2 19:00:05 2019
    checkpoint_txg = 0
Uberblock[24]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696952
    guid_sum = 10676018608949732911
    timestamp = 1562173200 UTC = Wed Jul  3 19:00:00 2019
    checkpoint_txg = 0
Uberblock[25]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696953
    guid_sum = 10676018608949732911
    timestamp = 1562173205 UTC = Wed Jul  3 19:00:05 2019
    checkpoint_txg = 0
Uberblock[26]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695578
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[27]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695579
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[28]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695580
    guid_sum = 10676018608949732911
    timestamp = 1562166308 UTC = Wed Jul  3 17:05:08 2019
    checkpoint_txg = 0
Uberblock[29]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699101
    guid_sum = 10676018608949732911
    timestamp = 1562184000 UTC = Wed Jul  3 22:00:00 2019
    checkpoint_txg = 0
Uberblock[30]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699102
    guid_sum = 10676018608949732911
    timestamp = 1562184005 UTC = Wed Jul  3 22:00:05 2019
    checkpoint_txg = 0
Uberblock[31]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699103
    guid_sum = 10676018608949732911
    timestamp = 1562251424 UTC = Thu Jul  4 16:43:44 2019
    checkpoint_txg = 0
------------------------------------
LABEL 1
------------------------------------
    version: 5000
    name: 'vm'
    state: 0
    txg: 6849
    pool_guid: 4833349645476312860
    hostid: 3551443610
    hostname: 'pine.home.yamagi.org'
    top_guid: 5842668963473420051
    guid: 5842668963473420051
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 5842668963473420051
        path: '/dev/ada0p1.eli'
        whole_disk: 1
        metaslab_array: 67
        metaslab_shift: 32
        ashift: 12
        asize: 480099172352
        is_log: 0
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
Uberblock[0]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699104
    guid_sum = 10676018608949732911
    timestamp = 1562251429 UTC = Thu Jul  4 16:43:49 2019
    checkpoint_txg = 0
Uberblock[1]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694113
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[2]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699106
    guid_sum = 10676018608949732911
    timestamp = 1562251439 UTC = Thu Jul  4 16:43:59 2019
    checkpoint_txg = 0
Uberblock[3]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699107
    guid_sum = 10676018608949732911
    timestamp = 1562251444 UTC = Thu Jul  4 16:44:04 2019
    checkpoint_txg = 0
Uberblock[4]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695492
    guid_sum = 10676018608949732911
    timestamp = 1562090405 UTC = Tue Jul  2 20:00:05 2019
    checkpoint_txg = 0
Uberblock[5]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697669
    guid_sum = 10676018608949732911
    timestamp = 1562176800 UTC = Wed Jul  3 20:00:00 2019
    checkpoint_txg = 0
Uberblock[6]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697670
    guid_sum = 10676018608949732911
    timestamp = 1562176805 UTC = Wed Jul  3 20:00:05 2019
    checkpoint_txg = 0
Uberblock[7]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694119
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[8]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696296
    guid_sum = 10676018608949732911
    timestamp = 1562169901 UTC = Wed Jul  3 18:05:01 2019
    checkpoint_txg = 0
Uberblock[9]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696297
    guid_sum = 10676018608949732911
    timestamp = 1562169906 UTC = Wed Jul  3 18:05:06 2019
    checkpoint_txg = 0
Uberblock[10]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690154
    guid_sum = 10676018608949732911
    timestamp = 1561986301 UTC = Mon Jul  1 15:05:01 2019
    checkpoint_txg = 0
Uberblock[11]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690155
    guid_sum = 10676018608949732911
    timestamp = 1561986306 UTC = Mon Jul  1 15:05:06 2019
    checkpoint_txg = 0
Uberblock[12]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696236
    guid_sum = 10676018608949732911
    timestamp = 1562169600 UTC = Wed Jul  3 18:00:00 2019
    checkpoint_txg = 0
Uberblock[13]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696237
    guid_sum = 10676018608949732911
    timestamp = 1562169605 UTC = Wed Jul  3 18:00:05 2019
    checkpoint_txg = 0
Uberblock[14]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690094
    guid_sum = 10676018608949732911
    timestamp = 1561986000 UTC = Mon Jul  1 15:00:00 2019
    checkpoint_txg = 0
Uberblock[15]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690095
    guid_sum = 10676018608949732911
    timestamp = 1561986005 UTC = Mon Jul  1 15:00:05 2019
    checkpoint_txg = 0
Uberblock[16]
    magic = 0000000000bab10c
    version = 5000
    txg = 1686000
    guid_sum = 10676018608949732911
    timestamp = 1561965458 UTC = Mon Jul  1 09:17:38 2019
    checkpoint_txg = 0
Uberblock[17]
    magic = 0000000000bab10c
    version = 5000
    txg = 1685969
    guid_sum = 10676018608949732911
    timestamp = 1561965303 UTC = Mon Jul  1 09:15:03 2019
    checkpoint_txg = 0
Uberblock[18]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698386
    guid_sum = 10676018608949732911
    timestamp = 1562180400 UTC = Wed Jul  3 21:00:00 2019
    checkpoint_txg = 0
Uberblock[19]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698387
    guid_sum = 10676018608949732911
    timestamp = 1562180405 UTC = Wed Jul  3 21:00:05 2019
    checkpoint_txg = 0
Uberblock[20]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688660
    guid_sum = 10676018608949732911
    timestamp = 1561978800 UTC = Mon Jul  1 13:00:00 2019
    checkpoint_txg = 0
Uberblock[21]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688661
    guid_sum = 10676018608949732911
    timestamp = 1561978805 UTC = Mon Jul  1 13:00:05 2019
    checkpoint_txg = 0
Uberblock[22]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694774
    guid_sum = 10676018608949732911
    timestamp = 1562086800 UTC = Tue Jul  2 19:00:00 2019
    checkpoint_txg = 0
Uberblock[23]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694775
    guid_sum = 10676018608949732911
    timestamp = 1562086805 UTC = Tue Jul  2 19:00:05 2019
    checkpoint_txg = 0
Uberblock[24]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696952
    guid_sum = 10676018608949732911
    timestamp = 1562173200 UTC = Wed Jul  3 19:00:00 2019
    checkpoint_txg = 0
Uberblock[25]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696953
    guid_sum = 10676018608949732911
    timestamp = 1562173205 UTC = Wed Jul  3 19:00:05 2019
    checkpoint_txg = 0
Uberblock[26]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695578
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[27]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695579
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[28]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695580
    guid_sum = 10676018608949732911
    timestamp = 1562166308 UTC = Wed Jul  3 17:05:08 2019
    checkpoint_txg = 0
Uberblock[29]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699101
    guid_sum = 10676018608949732911
    timestamp = 1562184000 UTC = Wed Jul  3 22:00:00 2019
    checkpoint_txg = 0
Uberblock[30]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699102
    guid_sum = 10676018608949732911
    timestamp = 1562184005 UTC = Wed Jul  3 22:00:05 2019
    checkpoint_txg = 0
Uberblock[31]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699103
    guid_sum = 10676018608949732911
    timestamp = 1562251424 UTC = Thu Jul  4 16:43:44 2019
    checkpoint_txg = 0
------------------------------------
LABEL 2
------------------------------------
    version: 5000
    name: 'vm'
    state: 0
    txg: 6849
    pool_guid: 4833349645476312860
    hostid: 3551443610
    hostname: 'pine.home.yamagi.org'
    top_guid: 5842668963473420051
    guid: 5842668963473420051
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 5842668963473420051
        path: '/dev/ada0p1.eli'
        whole_disk: 1
        metaslab_array: 67
        metaslab_shift: 32
        ashift: 12
        asize: 480099172352
        is_log: 0
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
Uberblock[0]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699104
    guid_sum = 10676018608949732911
    timestamp = 1562251429 UTC = Thu Jul  4 16:43:49 2019
    checkpoint_txg = 0
Uberblock[1]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694113
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[2]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699106
    guid_sum = 10676018608949732911
    timestamp = 1562251439 UTC = Thu Jul  4 16:43:59 2019
    checkpoint_txg = 0
Uberblock[3]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699107
    guid_sum = 10676018608949732911
    timestamp = 1562251444 UTC = Thu Jul  4 16:44:04 2019
    checkpoint_txg = 0
Uberblock[4]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695492
    guid_sum = 10676018608949732911
    timestamp = 1562090405 UTC = Tue Jul  2 20:00:05 2019
    checkpoint_txg = 0
Uberblock[5]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697669
    guid_sum = 10676018608949732911
    timestamp = 1562176800 UTC = Wed Jul  3 20:00:00 2019
    checkpoint_txg = 0
Uberblock[6]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697670
    guid_sum = 10676018608949732911
    timestamp = 1562176805 UTC = Wed Jul  3 20:00:05 2019
    checkpoint_txg = 0
Uberblock[7]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694119
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[8]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696296
    guid_sum = 10676018608949732911
    timestamp = 1562169901 UTC = Wed Jul  3 18:05:01 2019
    checkpoint_txg = 0
Uberblock[9]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696297
    guid_sum = 10676018608949732911
    timestamp = 1562169906 UTC = Wed Jul  3 18:05:06 2019
    checkpoint_txg = 0
Uberblock[10]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690154
    guid_sum = 10676018608949732911
    timestamp = 1561986301 UTC = Mon Jul  1 15:05:01 2019
    checkpoint_txg = 0
Uberblock[11]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690155
    guid_sum = 10676018608949732911
    timestamp = 1561986306 UTC = Mon Jul  1 15:05:06 2019
    checkpoint_txg = 0
Uberblock[12]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696236
    guid_sum = 10676018608949732911
    timestamp = 1562169600 UTC = Wed Jul  3 18:00:00 2019
    checkpoint_txg = 0
Uberblock[13]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696237
    guid_sum = 10676018608949732911
    timestamp = 1562169605 UTC = Wed Jul  3 18:00:05 2019
    checkpoint_txg = 0
Uberblock[14]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690094
    guid_sum = 10676018608949732911
    timestamp = 1561986000 UTC = Mon Jul  1 15:00:00 2019
    checkpoint_txg = 0
Uberblock[15]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690095
    guid_sum = 10676018608949732911
    timestamp = 1561986005 UTC = Mon Jul  1 15:00:05 2019
    checkpoint_txg = 0
Uberblock[16]
    magic = 0000000000bab10c
    version = 5000
    txg = 1686000
    guid_sum = 10676018608949732911
    timestamp = 1561965458 UTC = Mon Jul  1 09:17:38 2019
    checkpoint_txg = 0
Uberblock[17]
    magic = 0000000000bab10c
    version = 5000
    txg = 1685969
    guid_sum = 10676018608949732911
    timestamp = 1561965303 UTC = Mon Jul  1 09:15:03 2019
    checkpoint_txg = 0
Uberblock[18]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698386
    guid_sum = 10676018608949732911
    timestamp = 1562180400 UTC = Wed Jul  3 21:00:00 2019
    checkpoint_txg = 0
Uberblock[19]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698387
    guid_sum = 10676018608949732911
    timestamp = 1562180405 UTC = Wed Jul  3 21:00:05 2019
    checkpoint_txg = 0
Uberblock[20]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688660
    guid_sum = 10676018608949732911
    timestamp = 1561978800 UTC = Mon Jul  1 13:00:00 2019
    checkpoint_txg = 0
Uberblock[21]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688661
    guid_sum = 10676018608949732911
    timestamp = 1561978805 UTC = Mon Jul  1 13:00:05 2019
    checkpoint_txg = 0
Uberblock[22]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694774
    guid_sum = 10676018608949732911
    timestamp = 1562086800 UTC = Tue Jul  2 19:00:00 2019
    checkpoint_txg = 0
Uberblock[23]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694775
    guid_sum = 10676018608949732911
    timestamp = 1562086805 UTC = Tue Jul  2 19:00:05 2019
    checkpoint_txg = 0
Uberblock[24]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696952
    guid_sum = 10676018608949732911
    timestamp = 1562173200 UTC = Wed Jul  3 19:00:00 2019
    checkpoint_txg = 0
Uberblock[25]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696953
    guid_sum = 10676018608949732911
    timestamp = 1562173205 UTC = Wed Jul  3 19:00:05 2019
    checkpoint_txg = 0
Uberblock[26]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695578
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[27]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695579
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[28]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695580
    guid_sum = 10676018608949732911
    timestamp = 1562166308 UTC = Wed Jul  3 17:05:08 2019
    checkpoint_txg = 0
Uberblock[29]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699101
    guid_sum = 10676018608949732911
    timestamp = 1562184000 UTC = Wed Jul  3 22:00:00 2019
    checkpoint_txg = 0
Uberblock[30]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699102
    guid_sum = 10676018608949732911
    timestamp = 1562184005 UTC = Wed Jul  3 22:00:05 2019
    checkpoint_txg = 0
Uberblock[31]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699103
    guid_sum = 10676018608949732911
    timestamp = 1562251424 UTC = Thu Jul  4 16:43:44 2019
    checkpoint_txg = 0
------------------------------------
LABEL 3
------------------------------------
    version: 5000
    name: 'vm'
    state: 0
    txg: 6849
    pool_guid: 4833349645476312860
    hostid: 3551443610
    hostname: 'pine.home.yamagi.org'
    top_guid: 5842668963473420051
    guid: 5842668963473420051
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 5842668963473420051
        path: '/dev/ada0p1.eli'
        whole_disk: 1
        metaslab_array: 67
        metaslab_shift: 32
        ashift: 12
        asize: 480099172352
        is_log: 0
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
Uberblock[0]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699104
    guid_sum = 10676018608949732911
    timestamp = 1562251429 UTC = Thu Jul  4 16:43:49 2019
    checkpoint_txg = 0
Uberblock[1]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694113
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[2]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699106
    guid_sum = 10676018608949732911
    timestamp = 1562251439 UTC = Thu Jul  4 16:43:59 2019
    checkpoint_txg = 0
Uberblock[3]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699107
    guid_sum = 10676018608949732911
    timestamp = 1562251444 UTC = Thu Jul  4 16:44:04 2019
    checkpoint_txg = 0
Uberblock[4]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695492
    guid_sum = 10676018608949732911
    timestamp = 1562090405 UTC = Tue Jul  2 20:00:05 2019
    checkpoint_txg = 0
Uberblock[5]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697669
    guid_sum = 10676018608949732911
    timestamp = 1562176800 UTC = Wed Jul  3 20:00:00 2019
    checkpoint_txg = 0
Uberblock[6]
    magic = 0000000000bab10c
    version = 5000
    txg = 1697670
    guid_sum = 10676018608949732911
    timestamp = 1562176805 UTC = Wed Jul  3 20:00:05 2019
    checkpoint_txg = 0
Uberblock[7]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694119
    guid_sum = 10676018608949732911
    timestamp = 1562083510 UTC = Tue Jul  2 18:05:10 2019
    checkpoint_txg = 0
Uberblock[8]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696296
    guid_sum = 10676018608949732911
    timestamp = 1562169901 UTC = Wed Jul  3 18:05:01 2019
    checkpoint_txg = 0
Uberblock[9]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696297
    guid_sum = 10676018608949732911
    timestamp = 1562169906 UTC = Wed Jul  3 18:05:06 2019
    checkpoint_txg = 0
Uberblock[10]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690154
    guid_sum = 10676018608949732911
    timestamp = 1561986301 UTC = Mon Jul  1 15:05:01 2019
    checkpoint_txg = 0
Uberblock[11]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690155
    guid_sum = 10676018608949732911
    timestamp = 1561986306 UTC = Mon Jul  1 15:05:06 2019
    checkpoint_txg = 0
Uberblock[12]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696236
    guid_sum = 10676018608949732911
    timestamp = 1562169600 UTC = Wed Jul  3 18:00:00 2019
    checkpoint_txg = 0
Uberblock[13]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696237
    guid_sum = 10676018608949732911
    timestamp = 1562169605 UTC = Wed Jul  3 18:00:05 2019
    checkpoint_txg = 0
Uberblock[14]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690094
    guid_sum = 10676018608949732911
    timestamp = 1561986000 UTC = Mon Jul  1 15:00:00 2019
    checkpoint_txg = 0
Uberblock[15]
    magic = 0000000000bab10c
    version = 5000
    txg = 1690095
    guid_sum = 10676018608949732911
    timestamp = 1561986005 UTC = Mon Jul  1 15:00:05 2019
    checkpoint_txg = 0
Uberblock[16]
    magic = 0000000000bab10c
    version = 5000
    txg = 1686000
    guid_sum = 10676018608949732911
    timestamp = 1561965458 UTC = Mon Jul  1 09:17:38 2019
    checkpoint_txg = 0
Uberblock[17]
    magic = 0000000000bab10c
    version = 5000
    txg = 1685969
    guid_sum = 10676018608949732911
    timestamp = 1561965303 UTC = Mon Jul  1 09:15:03 2019
    checkpoint_txg = 0
Uberblock[18]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698386
    guid_sum = 10676018608949732911
    timestamp = 1562180400 UTC = Wed Jul  3 21:00:00 2019
    checkpoint_txg = 0
Uberblock[19]
    magic = 0000000000bab10c
    version = 5000
    txg = 1698387
    guid_sum = 10676018608949732911
    timestamp = 1562180405 UTC = Wed Jul  3 21:00:05 2019
    checkpoint_txg = 0
Uberblock[20]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688660
    guid_sum = 10676018608949732911
    timestamp = 1561978800 UTC = Mon Jul  1 13:00:00 2019
    checkpoint_txg = 0
Uberblock[21]
    magic = 0000000000bab10c
    version = 5000
    txg = 1688661
    guid_sum = 10676018608949732911
    timestamp = 1561978805 UTC = Mon Jul  1 13:00:05 2019
    checkpoint_txg = 0
Uberblock[22]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694774
    guid_sum = 10676018608949732911
    timestamp = 1562086800 UTC = Tue Jul  2 19:00:00 2019
    checkpoint_txg = 0
Uberblock[23]
    magic = 0000000000bab10c
    version = 5000
    txg = 1694775
    guid_sum = 10676018608949732911
    timestamp = 1562086805 UTC = Tue Jul  2 19:00:05 2019
    checkpoint_txg = 0
Uberblock[24]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696952
    guid_sum = 10676018608949732911
    timestamp = 1562173200 UTC = Wed Jul  3 19:00:00 2019
    checkpoint_txg = 0
Uberblock[25]
    magic = 0000000000bab10c
    version = 5000
    txg = 1696953
    guid_sum = 10676018608949732911
    timestamp = 1562173205 UTC = Wed Jul  3 19:00:05 2019
    checkpoint_txg = 0
Uberblock[26]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695578
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[27]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695579
    guid_sum = 10676018608949732911
    timestamp = 1562166303 UTC = Wed Jul  3 17:05:03 2019
    checkpoint_txg = 0
Uberblock[28]
    magic = 0000000000bab10c
    version = 5000
    txg = 1695580
    guid_sum = 10676018608949732911
    timestamp = 1562166308 UTC = Wed Jul  3 17:05:08 2019
    checkpoint_txg = 0
Uberblock[29]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699101
    guid_sum = 10676018608949732911
    timestamp = 1562184000 UTC = Wed Jul  3 22:00:00 2019
    checkpoint_txg = 0
Uberblock[30]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699102
    guid_sum = 10676018608949732911
    timestamp = 1562184005 UTC = Wed Jul  3 22:00:05 2019
    checkpoint_txg = 0
Uberblock[31]
    magic = 0000000000bab10c
    version = 5000
    txg = 1699103
    guid_sum = 10676018608949732911
    timestamp = 1562251424 UTC = Thu Jul  4 16:43:44 2019
    checkpoint_txg = 0

Also 4 Label (Metadatenkopien) mit 31 Transaktionen. Da sieht man nun auch schön an den Timestamps, dass die Kiste in den letzten Tagen eher weniger benutzt wurde. Nun musst eine Transaktion suchen, auf die du zurück möchtest. Hangele dich an den Timestamps lang und beachtet, dass der Pool durch das gewaltsame ziehen der Platte eventuell inkonsistent sein wird. Also nicht jedes Label alle Transaktionen enthalten muss.

Hast du dich für eine Transaktion entschieden, braucht du ihre ID. Das ist das Feld txg in der Ausgabe. Also wenn ich auf die Transaktion vom timestamp = 1562184005 UTC = Wed Jul 3 22:00:05 2019 zurück möchte, wähle ich txg = 1699102

Mit diesen Informationen kann der Pool nun importiert werden. Das Importieren ist zerstörend, es verwirft alle Transaktionen, die neuer als die gewünschte Transaktion sind. Unwiderbringlich, für immer, nada. Daher das Backup oben. Ich würde ihn read-only importieren, aber an sich ist es nicht notwendig.

Code:

% pool import -R /mnt -o readonly=on -f -T $txg tank

-R /mnt um Pfadkollissionen zu verhindern, -f da der Pool noch als importiert gekennzeichnet sein wird und -T $txg ist die undokumentierte Brechstange. Das kann ein paar Minuten dauern, keine Panik. Anschließend bist du idealerweise auf einem Stand vor dem Desaster, kannst deine Daten rauskopieren und bist mir ewig dankbar.

Danke!
Na, das macht doch Hoffnung! :-)

Gruß
Yoda

Yoda · 5 Juli 2019

weil ich nur höchstens 10TB-Platten besitze, muss ich wohl darauf warten, bis meine nächste Lieferung mit 2x10TB-Platten eintrifft und dann damit einen RAID-0 bauen, auf dem ich dann das 10TB-Image ablegen kann für die Wiederherstellung...

Also werde ich in ca. einer Woche damit beginnen können, zugesagt wurden die erst für Do. :grumble:

satriani · 6 Juli 2019

Na dann, good luck!
Bitte, halte uns auf dem Laufenden. Interessiert mich brennend wie es ausgeht

Yoda · 8 Juli 2019

So, gestern Abend ist der Kopiervorgang des Images fertig geworden (Kopierdauer: mehr als 48h). Leider habe ich wohl die Platte doch zu spät abgezogen... Die TimeStamps der Transaktionen reichen nicht weit genug zurück... :-(
Aber wasfür Transaktionen sind denn das? Ich hab doch in der Zeit nix auf der Platte gemacht?

Ach ich habe auch nur 32, meine Tanks sind auch schon älter und wurden nur mit "zpool upgrade -a" aktualisiert.

Gruß
QuarzSnoopy

Yoda · 8 Juli 2019

hab mir gerade dieses Tool runtergeladen und probiere mal die Testversion aus:
https://www.ufsexplorer.com/de/ufs-explorer-standard-recovery.php
der sagt gearde, dass er ca. 11 Stunden für die Suche nach Daten braucht...

medV2 · 8 Juli 2019

Schade das es nicht geklappt hat @Yoda, viel Erfolg mit dem Tool.

Ad 32 vs 128 Uberblocks: Ashift 12 -> 32, Ashift 9 -> 128

LG

Yoda · 9 Juli 2019

Leider hat das Tool meine gesuchten Daten auch nicht gefunden. Offensichtlich kann das Tool (auf ZFS) auch nicht mehr als Yamagi oben beschtieben hat... :confused:

Mir bleibt wohl wirklich nichts anderes übrig als die Platte einzuschicken. :eek:

@medV2
was meinst Du mit "Ashift 12 -> 32, Ashift 9 -> 128" genau?

EDIT:
hab es gefunden: https://de.wikibooks.org/wiki/ZFS_auf_Linux/_ashift

medV2 · 9 Juli 2019

nochmal blöd nachgefragt um sicher zu gehen bevor du einschickst: du hast dir alle timestamps der uberblocks angesehen? Der letzte Block (nr 31)ist nicht zwangsweise der älteste.

Yoda · 9 Juli 2019

medV2 schrieb:
nochmal blöd nachgefragt um sicher zu gehen bevor du einschickst: du hast dir alle timestamps der uberblocks angesehen? Der letzte Block (nr 31)ist nicht zwangsweise der älteste.

Ja, ich hab mir alle angesehen. Und bei mir war auch der "31" nicht der älteste... :-)

Yoda · 9 Juli 2019

ich werde die jetzt wohl zu www.daten-klinik.com schicken, die sind nicht ganz soweit weg. Und die garrantieren, dass nur Kosten anfallen, wenn ich meine Daten bekomme.

medV2 · 9 Juli 2019

Ich weiß auf jeden Fall, wer in Zukunft ein gutes Backup haben wird

Yoda · 9 Juli 2019

medV2 schrieb:
Ich weiß auf jeden Fall, wer in Zukunft ein gutes Backup haben wird

...nun, von 20TB ist es garnicht so einfach ein Backup zu machen aber von diesen Wichtigen Daten wird es auf jeden Fall eines geben (wenn sie wieder da sind)... - Ich habe mehr als 15 TB "weniger wichtige" Daten, ausgerechnet die wichtigsten wurden gelöscht... :grumble:

Yamagi · 9 Juli 2019

medV2 schrieb:
Ad 32 vs 128 Uberblocks: Ashift 12 -> 32, Ashift 9 -> 128

Ha. Wenn man drüber nachdenkt, ist es eigentlich logisch. Wieder was gelernt.

Und vielleicht noch ein Tipp. Eine Snapshot-Rotation mitlaufen lassen. Ich mache es z.B. mit sysutils/zfsnap. Da kommt sowas in die Root-Crontab und schon ist man gegen die meisten Dummheitsfehler (ist nicht böse gemeint, ich glaube jedem von uns ist schon mal ein rm oder ähnliches ausgerutscht) gewappnet:

Code:

# Snapshotrotation
0 * * * *  /usr/local/sbin/zfSnap -a 6h -r system
@reboot    /usr/local/sbin/zfSnap -a 6h -r system

5 * * * *  /usr/local/sbin/zfSnap -d
@reboot    /usr/local/sbin/zfSnap -d

Es erstellt zur Bootzeit und dann jede volle Stunde einen Snapshot auf allen Datasets des Pools system, ist ein Snapshot mehr als 6 Stunden alt, wird er automatisch wieder gelöscht. So kann man in Schritten von jeweils einer Stunde insgesamt 6 Stunden zurückgehen.

Wiederherstellung gelöschter Daten auf einem ZFS-Mirror

[Linux|FreeBSD] - User

Kaffeemann

Well-Known Member

moderater Moderator

Kaffeemann

Well-Known Member

[Linux|FreeBSD] - User

Possessed With Psi Powers

Well-Known Member

Possessed With Psi Powers

Well-Known Member

Possessed With Psi Powers

[Linux|FreeBSD] - User

[Linux|FreeBSD] - User

SysLion

[Linux|FreeBSD] - User

[Linux|FreeBSD] - User

Well-Known Member

[Linux|FreeBSD] - User

Well-Known Member

[Linux|FreeBSD] - User

[Linux|FreeBSD] - User

Well-Known Member

[Linux|FreeBSD] - User

Possessed With Psi Powers

Wir schützen deine Privatsphäre