Discussion:
RaidZ Zpool Recovery, possible or am I SOL?
(too old to reply)
s***@public.gmane.org
2014-01-05 16:51:46 UTC
Permalink
Am hoping this may be recoverable, but I fear I may be SOL...

I had a failed drive on one of my zpools and messed up when I tried
replacing it.
Rather than remove the device from the zpool, add the new device and then
resilver, I instead powered down the system, added the new drive, updated
the vdev description to point to the new drive and thought that would sort
it.
When that failed, I then did a zpool add for the new drive.

Unfortunately, this means I now have two devices with the same name in the
zpool.
After trying for a few days with the failed drive removed, I re-added it
into the system today and it seems to have (temporarily) come back to life
and my zpool now looks like this:

~ # zpool list -v midtank1
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
midtank1 - - - - - FAULTED /mnt/zfs
raidz1 - - - -
m1d2 - - - -
m1d3 - - - -
c1d3 - - - -
c2d3 - - - -
c2d3 - - - 16.0E


~ # zpool status -v midtank1
pool: midtank1
state: UNAVAIL
status: One or more devices could not be used because the label is missing
or invalid. There are insufficient replicas for the pool to
continue
functioning.
action: Destroy and re-create the pool from
a backup source.
see: http://zfsonlinux.org/msg/ZFS-8000-5E
scan: none requested
config:

NAME STATE READ WRITE CKSUM
midtank1 UNAVAIL 0 0 0 insufficient replicas
raidz1-0 DEGRADED 0 0 0
m1d2 ONLINE 0 0 0
m1d3 ONLINE 0 0 0
c1d3 ONLINE 0 0 0
c2d3 OFFLINE 0 0 0
c2d3 UNAVAIL 0 0 0 corrupted data



As the second device was never added to the raidz1 configuration, I can't
think of any "technical" reason why I shouldn't be able to recover my data
and resolve this, however after a few days of Googling and performing
arcane rituals, I can't seem to find a solution.

I'm running a home rolled linux 3.8.8 kernel, spl/zfs 0.6.2, pool version
5000 and FS version 5.0

Is there any hope for recovery, or do I just chalk this down to a hard
lesson learned?

Thanks for any pointers!

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Tim Chase
2014-01-05 17:38:00 UTC
Permalink
Post by s***@public.gmane.org
Am hoping this may be recoverable, but I fear I may be SOL...
I had a failed drive on one of my zpools and messed up when I tried
replacing it.
Rather than remove the device from the zpool, add the new device and
then resilver, I instead powered down the system, added the new drive,
updated the vdev description to point to the new drive and thought
that would sort it.
When that failed, I then did a zpool add for the new drive.
Unfortunately, this means I now have two devices with the same name in
the zpool.
After trying for a few days with the failed drive removed, I re-added
it into the system today and it seems to have (temporarily) come back
...
~ # zpool status -v midtank1
pool: midtank1
state: UNAVAIL
status: One or more devices could not be used because the label is missing
or invalid. There are insufficient replicas for the pool
to continue
functioning.
action: Destroy and re-create the pool from
a backup source.
see: http://zfsonlinux.org/msg/ZFS-8000-5E
scan: none requested
NAME STATE READ WRITE CKSUM
midtank1 UNAVAIL 0 0 0 insufficient replicas
raidz1-0 DEGRADED 0 0 0
m1d2 ONLINE 0 0 0
m1d3 ONLINE 0 0 0
c1d3 ONLINE 0 0 0
c2d3 OFFLINE 0 0 0
c2d3 UNAVAIL 0 0 0 corrupted data
Don't be too concerned about the names. Your problem is that the "c2d3"
vdev you added (zpool add) doesn't appear to look like a proper vdev.
You should be able to recover your data but you'll ultimately want to
make a new pool because there's no way to remove the c2d3 vdev which was
inadvertently as a stripe component. Here's my suggestion:

1. Rename /etc/zfs/zpool.cache to something else in order to hide it
2. Create an empty directory somewhere
3. Create symlinks in the new directory that point to the proper device
nodes for your pool
4. See whether it's importable with a simple "zpool import" command

As a specific off-the-cuff example:

# mv /etc/zfs/zpool.cache /etc/zfs/zpool.cache.save
# mkdir /mydev
# ln -s /dev/<whatever>/m1d2 /mydev
<make symlinks for the other devices in the pool. make sure you include
the actual device which you "zpool add"ed>
# zpool import -p /mydev

The final zpool command will scan the directory and will show you
whether the pool is importable. If you're not sure whether you've got
the proper devices linked into the directory, you can check them
individually with "zdb -l /mydev/<devnode>" and it should display the 4
ZFS labels. Each device you symlink should produce sensible output when
tested with "zdb -l". Obviously you don't need a symlink that
represents the offline vdev of the raidz.

If you get it to the point where a "dry run" zpool command shows it as
being importable, you can import it for real with "zpool import -d
/mydev midtank1" and begin your recovery.

- Tim


To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
s***@public.gmane.org
2014-01-05 17:55:36 UTC
Permalink
Thanks for the reply and giving me a small glimmer of hope :)

I ran through the steps you outlined, however, when I got the part about
performing the "dry run" import I get the message:

~ # zpool import -p /tmp/zfsrecovery
invalid option 'p'

usage:
import [-d dir] [-D]
import [-d dir | -c cachefile] [-F [-n]] <pool | id>
import [-o mntopts] [-o property=value] ...
[-d dir | -c cachefile] [-D] [-f] [-m] [-N] [-R root] [-F [-n]]
-a
import [-o mntopts] [-o property=value] ...
[-d dir | -c cachefile] [-D] [-f] [-m] [-N] [-R root] [-F [-n]]
<pool | id> [newpool]


I ran the "zdb -l" command against each of the 4 devices I had symlinked
and they all displayed 4 ZFS labels, so that's at least a little but
positive...

Any thoughts?

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Tim Chase
2014-01-05 17:59:35 UTC
Permalink
Post by s***@public.gmane.org
Thanks for the reply and giving me a small glimmer of hope :)
I ran through the steps you outlined, however, when I got the part
~ # zpool import -p /tmp/zfsrecovery
invalid option 'p'
Doh! Sorry, I meant "-d". I've been doing too much work with zdb lately
which uses "-p" for the same purpose.

- Tim


To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
s***@public.gmane.org
2014-01-05 18:14:57 UTC
Permalink
-d just returns me back to the prompt, I also tried with a few -V's for
some additional verbosity but received just the same:

~ # zpool import -d /tmp/zfsrecovery
~ #
~ # zpool import -V -V -V -d /tmp/zfsrecovery
~ #


Any thoughts?

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Tim Chase
2014-01-05 18:25:44 UTC
Permalink
Post by s***@public.gmane.org
-d just returns me back to the prompt, I also tried with a few -V's
~ # zpool import -d /tmp/zfsrecovery
~ #
~ # zpool import -V -V -V -d /tmp/zfsrecovery
~ #
Any thoughts?
I suspect your alread-loaded module has consumed the contents of your
(previously existing) /etc/zfs/zpool.cache file. That is exactly the
you'd get it does and if the module has read a cached configuration.
Try these steps to get a clean import:

* Unload the modules: modprobe -v -r zfs
* Make sure it's really gone: lsmod | grep zfs
* Don't continue until it is gone
* Clean or the cache file if it still exists: rm /etc/zfs/zpool.cache
* Re-load the module: modprobe zfs
* Try the test import: zpool import -d /tmp/zfsrecovery

I was going to suggest those steps in my original note but wanted to
keep it simple.

If you can't unload the modules because you've got additional pools
created, you'll have to take a different approach depending on your
distro, root-on-zfs status, contents of initrd, etc.

- Tim

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
s***@public.gmane.org
2014-01-05 18:50:50 UTC
Permalink
Tim, first of all, thanks for taking the time to try and help me with this,
it really is appreciated.

I did have another ZFS pool (bigtank1) but I closed any services that had
files open in it and unmounted it first.
I then cleanly removed the ZFS module.
Deleted the zfs.cache file.
Inserted the ZFS module.
Re-ran the zpool import -d /tmp/zfsrecovery

Unfortunately with exactly the same response:

~ # rmmod zfs
~ # dmesg -c
[ 9588.866577] ZFS: Unloaded module v0.6.2-1
~ # lsmod | grep zfs
~ # rm /etc/zfs/zpool.cache
~ # modprobe zfs
~ # dmesg
[ 9650.163528] ZFS: Loaded module v0.6.2-1, ZFS pool version 5000, ZFS
filesystem version 5
~ # zpool import -d /tmp/zfsrecovery
~ # zpool list
no pools available
~ #


Any more thoughts?

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
s***@public.gmane.org
2014-01-05 19:14:06 UTC
Permalink
I've also removed the SPL module and associated sub modules, performed a
complete system reboot (with no zpool.cache file) and tried again.

What is interesting is that now if I just run a "zfs -import" command, the
message is a little different:


~ # zpool import
pool: midtank1
id: 12240748602613404447
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://zfsonlinux.org/msg/ZFS-8000-6X
config:

midtank1 UNAVAIL missing device
raidz1-0 DEGRADED
sdb ONLINE
sdc ONLINE
sdd ONLINE
sdi OFFLINE
cache
c1s1-part3
c2s1-part3

Additional devices are known to be part of this pool, though their
exact configuration cannot be determined.

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Tim Chase
2014-01-05 19:30:11 UTC
Permalink
Post by s***@public.gmane.org
I've also removed the SPL module and associated sub modules, performed
a complete system reboot (with no zpool.cache file) and tried again.
What is interesting is that now if I just run a "zfs -import" command,
~ # zpool import
pool: midtank1
id: 12240748602613404447
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://zfsonlinux.org/msg/ZFS-8000-6X
midtank1 UNAVAIL missing device
raidz1-0 DEGRADED
sdb ONLINE
sdc ONLINE
sdd ONLINE
sdi OFFLINE
cache
c1s1-part3
c2s1-part3
Additional devices are known to be part of this pool, though their
exact configuration cannot be determined.
The previous note I sent outlined some additional things to try. Based
on this output, it does seem the labels may be inconsistent between the
vdevs it's finding in the normal search path (all the various /dev
directories). I think we'll need to see the ZFS labels of each device
that really should be in the pool (the 3 good components of the raidz
and the 1 extra stripe component).

Are the cache devices supposed to be there? They didn't show up in your
original posting.

- Tim

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
s***@public.gmane.org
2014-01-05 19:39:03 UTC
Permalink
Yes, the cache's are correct, I'm unsure why they didn't show up when the
system was using the zpool.cache file, however it's highly likely that it's
a result of one of the things I had done when using google to try and help
resolve this issue.
In my system I have (had?) two zpools, "bigtank1", which is a RaidZ1 pool
of approximately 10.9TB with two 128Gb SSD cache's and another, "midtank1"
which was a RaidZ1 pool of approximately 6Tb also with two 128Gb SSD
cache's.

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Tim Chase
2014-01-05 19:24:56 UTC
Permalink
Post by s***@public.gmane.org
Tim, first of all, thanks for taking the time to try and help me with
this, it really is appreciated.
I did have another ZFS pool (bigtank1) but I closed any services that
had files open in it and unmounted it first.
I then cleanly removed the ZFS module.
Deleted the zfs.cache file.
Inserted the ZFS module.
Re-ran the zpool import -d /tmp/zfsrecovery
~ # rmmod zfs
~ # dmesg -c
[ 9588.866577] ZFS: Unloaded module v0.6.2-1
~ # lsmod | grep zfs
~ # rm /etc/zfs/zpool.cache
~ # modprobe zfs
~ # dmesg
[ 9650.163528] ZFS: Loaded module v0.6.2-1, ZFS pool version 5000,
ZFS filesystem version 5
~ # zpool import -d /tmp/zfsrecovery
~ # zpool list
no pools available
~ #
Can you run "zdb -l" on each of the symlinks in /tmp/zfsrecovery:

# for i in /tmp/zfsrecovery; do zdb -l $i; done

If there's even a single device in there which shows the 4 ZFS labels,
then a plain "zpool import -d /tmp/zfsrecovery" should print some pool
information. I have a feeling the symlinks aren't being created correctly.

If they are all correct, then there /may/ be some sort of conflict
between them causing the import dry-run to fail. In that case, you'll
have to hide the bogus device entry by removing it or by moving it
within the normal /dev area. Please check your symlinks with "zdb -l"
on each one and if each reports the proper set of 4 labels, please email
me personally the output of the "zdb -l" for each one and I can likely
come up with a formula to get it imported.

- Tim

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
s***@public.gmane.org
2014-01-05 19:33:54 UTC
Permalink
Tim, I modified your bashline a little (/tmp/zfsrecovery/*) and the output
is this:

~ # for disk in /tmp/zfsrecovery/*; do zdb -l $disk; done
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Tim Chase
2014-01-05 20:06:59 UTC
Permalink
Post by s***@public.gmane.org
Tim, I modified your bashline a little (/tmp/zfsrecovery/*) and the
~ # for disk in /tmp/zfsrecovery/*; do zdb -l $disk; done
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
etc...

That's obviously a problem. Could you please share what "ls -l
/tmp/zfsrecovery" looks like? Whatever you're symlinking into there
must not be working. The basic strategy to get a pool imported when
there are conflicting devices laying around in the /dev area is to
create a fresh directory with either symlinks or actual device nodes for
the (working) devices that are part of the pool. One complication is
that the labels (when you can actually see them) do actually have the
Post by s***@public.gmane.org
type: 'disk'
id: 0
guid: 4176143715270407871
path: '/dev/sda1'
whole_disk: 0
DTL: 127
create_txg: 4
Once you get a handle on the actual device nodes involved with your
pool, you want to arrange things so that the "zpool import" can see
*only* those devices and nothing else. The typical trick is to create a
directory full of appropriate symlinks. In some cases, however, you
need to hide/delete/rename the actual device nodes within the /dev
hierarchy because the import process *will* look at the devices
indicated in the labels.

Hopefully this gives you the information you need to get your pool
imported. It certainly sounds like you do have an importable pool.

- Tim

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
s***@public.gmane.org
2014-01-05 20:33:29 UTC
Permalink
Tim, your comments were enough to guide me to what I was doing wrong.
I had created symlinks to the raw base device (ln -s /etc/disk/by-vdev/c2d3
/tmp/zfsrecovery/c2d3, etc), however, I needed to create them to the
partition that ZFS seems to have created on each disk.

Now I have:

~ # ls -la /tmp/zfsrecovery
total 8
drwxr-xr-x 2 root root 4096 Jan 5 20:27 .
drwxrwxrwt 5 root root 4096 Jan 5 19:11 ..
lrwxrwxrwx 1 root root 28 Jan 5 20:27 c1d3-part1 ->
/dev/disk/by-vdev/c1d3-part1
lrwxrwxrwx 1 root root 28 Jan 5 20:27 c2d3-part1 ->
/dev/disk/by-vdev/c2d3-part1
lrwxrwxrwx 1 root root 28 Jan 5 20:27 m1d2-part1 ->
/dev/disk/by-vdev/m1d2-part1
lrwxrwxrwx 1 root root 28 Jan 5 20:27 m1d3-part1 ->
/dev/disk/by-vdev/m1d3-part1


But running a zpool import -d /tmp/zfsrecovery still gives me:

~ # zpool import -d /tmp/zfsrecovery/
pool: midtank1
id: 12240748602613404447
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://zfsonlinux.org/msg/ZFS-8000-6X
config:

midtank1 UNAVAIL missing device
raidz1-0 DEGRADED
m1d2 ONLINE
m1d3 ONLINE
c1d3 ONLINE
c2d3 OFFLINE
cache
c1s1-part3
c2s1-part3

Additional devices are known to be part of this pool, though their
exact configuration cannot be determined.


Any thoughts on next steps?

Thanks again!
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Niels de Carpentier
2014-01-06 06:36:59 UTC
Permalink
Post by s***@public.gmane.org
Tim, your comments were enough to guide me to what I was doing wrong.
I had created symlinks to the raw base device (ln -s
/etc/disk/by-vdev/c2d3
/tmp/zfsrecovery/c2d3, etc), however, I needed to create them to the
partition that ZFS seems to have created on each disk.
~ # ls -la /tmp/zfsrecovery
total 8
drwxr-xr-x 2 root root 4096 Jan 5 20:27 .
drwxrwxrwt 5 root root 4096 Jan 5 19:11 ..
lrwxrwxrwx 1 root root 28 Jan 5 20:27 c1d3-part1 ->
/dev/disk/by-vdev/c1d3-part1
lrwxrwxrwx 1 root root 28 Jan 5 20:27 c2d3-part1 ->
/dev/disk/by-vdev/c2d3-part1
lrwxrwxrwx 1 root root 28 Jan 5 20:27 m1d2-part1 ->
/dev/disk/by-vdev/m1d2-part1
lrwxrwxrwx 1 root root 28 Jan 5 20:27 m1d3-part1 ->
/dev/disk/by-vdev/m1d3-part1
~ # zpool import -d /tmp/zfsrecovery/
pool: midtank1
id: 12240748602613404447
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://zfsonlinux.org/msg/ZFS-8000-6X
midtank1 UNAVAIL missing device
raidz1-0 DEGRADED
m1d2 ONLINE
m1d3 ONLINE
c1d3 ONLINE
c2d3 OFFLINE
cache
c1s1-part3
c2s1-part3
Additional devices are known to be part of this pool, though their
exact configuration cannot be determined.
Any thoughts on next steps?
You need the device that you added to the pool. The problem is that this
device wasn't added to the raidz, but as another (single drive) vdev that
is now striped with your raidz. So your failed drive shouldn't be required
to import the pool (as the redundancy of the raidz1 will be enough to
recover the data), but the replacement drive is critical.
There might not yet be any critical data on that replacement drive, but
once it's part of the pool you cannot remove it anymore, and the pool is
not usable without it.

Niels


To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Loading...