Discussion:
I have lost a mountain of data using ZFS on Linux. Would appreciate some ideas how to recover.
(too old to reply)
tom-/sOA9cmYp6rgYzRL+
2014-07-05 05:39:13 UTC
Permalink
I have been using zfs-fuze, happily, for a lot of years. I've never lost a
byte, using it. When I moved to advanced format disks, it got slow. I
tolerated that because my priority is 99% reliability, 1% performance. It
was sufficient.

I am running two identical Ubuntu 14.04 systems with 10 x 4TB Seagate NAS
drives in each. Every day, the backup system turns itself on (BIOS
setting), rsync's the entire array, and then cleanly shuts down.

For some reason, I thought it was time to try ZoL on the backup system. It
was an OS upgrade, ZoL, and a fresh array format. I ran it for about three
months with one of those months being spent copying data from the old
array. Using ashift=12, the performance has been really good. Write
performance roughly quadrupled while read performance went up 11x. Pretty
nice.

It ran fine for a few months so I figured I'd work it out a bit. I did a
couple of scrubs and ran some massive read/write cycles to the array. It
worked great.

... so I reinstalled the primary server identically to the backup. Ubuntu
14.04, ZoL, fresh array format (I don't trust it to upgrade).

As you can imagine, at roughly 2TB into the copy (very early on), the
backup lost the majority of it's data. A partition that had 14.5TB now
shows 4.5TB and all of those directories now show empty. ZPOOL status
shows no errors of any kind.

I've left it to continue copying and I have double backups of the most
important stuff, such as my photography, key personal data, etc. so I'm not
entirely screwed but there is a ton of stuff that I'd like to get back.
They are mostly large files of 1~40GB in size (disk images of a bunch of
workstations). Most of those images are backups but a few of those systems
are long gone but I still need to have the data that was on them, just in
case. I could really use that data.

So far, I have only tried exporting and importing the array. I'm worried
that trying too much will ruin my chances of recovering the data.

Any ideas of what to do next? I'd like to recover the data and convert the
backup system back to zfs-fuse.

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Richard Yao
2014-07-05 05:49:03 UTC
Permalink
Post by tom-/sOA9cmYp6rgYzRL+
... so I reinstalled the primary server identically to the backup. Ubuntu
14.04, ZoL, fresh array format (I don't trust it to upgrade).
As you can imagine, at roughly 2TB into the copy (very early on), the
backup lost the majority of it's data. A partition that had 14.5TB now
shows 4.5TB and all of those directories now show empty. ZPOOL status
shows no errors of any kind.
Does this system have ECC memory? It sounds like a bitflip in rsync
caused it to mess up.

It would have likely been better to do incremental zfs send/recv. That
would not address the ECC memory issue, but it should at least avoid
giving a bitflip the power to say "wipe everything". Did you use
snapshots at all?
tom-/sOA9cmYp6rgYzRL+
2014-07-05 06:31:41 UTC
Permalink
The backup system does not have ECC RAM. It is a low power system, not a
server.

It was not an rsync issue. RSYNC on the primary system is reading from a
read-only NFS mount. I have not written anything or done much to the
backup system, nor will I until it is finished copying what it can.

I have not used snapshots on these systems.
Post by Richard Yao
Does this system have ECC memory? It sounds like a bitflip in rsync
caused it to mess up.
It would have likely been better to do incremental zfs send/recv. That
would not address the ECC memory issue, but it should at least avoid
giving a bitflip the power to say "wipe everything". Did you use
snapshots at all?
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Michael Kjörling
2014-07-05 12:30:18 UTC
Permalink
The backup system does not have ECC RAM. It is a low power system,
not a server.
Unfortunately, that could be a real problem, and potentially the
source of your woes. ZFS (the file system, not any particular
implementation such as zfs-fuse or ZFS On Linux) taxes the system a
fair bit more than most other file systems do. It is designed to be
able to withstand disk problems, _but not RAM problems_. All of those
fancy parity and checksum calculations depend on the assumption that
the outcome can be trusted to be correct for a given input, and the
way that is done is by implicitly trusting the system's RAM to do what
it says on the tin. That trust is potentially broken when you are
using non-ECC RAM, since a RAM error in that case will go undetected.
(Technically, you probably could get away with parity RAM and an
immediate system halt on any error detected much like it was done for
a fair while on IBM PCs, but pretty much everyone who needs this sort
of assurances these days use ECC RAM.)

The trust ZFS places in the system itself and distrust it places in
the storage, gives another possibility that at least FreeBSD's
implementation of ZFS appears to take advantage of (and I doubt ZoL is
different, because they should trace their lineage back to the same
codebase). Namely, if an error is detected and sufficient redundancy
exists to correct that error, _the error is corrected on disk even
when you are nominally working in "read only" mode_. If an error is
_incorrectly_ detected, or if the "corrected" data actually introduces
an error, this means that you are _introducing_ errors that were not
previously there; again, the way to protect against that is by using
ECC RAM.

ZFS also relies quite a lot on write caching. Anything that is about
to be written is dumped into the intent log, but unless a system crash
necessitates a log replay, it relies on the write cached data in RAM
to be correct as that is _at some later point in time_ written out to
disk. An inopportunate single flipped bit in that data can potentially
cause all sorts of mayhem on disk, including rendering all that fancy
redundancy moot (they will all carry the corrupted data, and there
will be no way to tell what is correct).

There was a link posted here not long ago that went into some detail
about the issue of ECC vs non-ECC RAM with ZFS, from a FreeNAS
(FreeBSD) perspective. Let me see if I can dig it up... ah, here it
is. [1A] The important part is the very first post (though you can
read further if you are curious about the ensuing discussion) and it
shows by example why non-ECC RAM is much more of a problem with ZFS
than with with virtually every other file system. Also note [1B] in
the same thread that says _in part_ (by user "cyberjock" [2] April 4
Sun's own documentation said to use ECC with ZFS, without question.
In the PDF copy I had it was the only text that was in red in the 70
or so pages I had read. This used to be pretty easy to deal with and
required no "enforcement" or actual research to make sure you had it
because the only OSes that had ZFS were server OSes. And those
servers always supported ECC and used ECC anyway. So it was one of
those things you never thought about. Windows doesn't have to say
"x86 based CPU" because you know better. ZFS back in the day was no
different. But today, thanks to open source, ZFS is now available
for so many devices it's astonishing. That doesn't change the
requirements at all, but it does give us a false sense of security
and complacency that "it's okay".
Note that essentially all other major subsystems in a PC has some form
of parity checking or error correction built in. PCIe does (I think
someone mentioned here that PCIe uses 8/10 FEC, and a quick Google
search provides ample evidence that PCIe does error detection and
correction on the bus), SATA does on the wire (that's the very basis
for SMART attribute 199), hard drives do (look at SMART attribute 195,
for example), USB does on the protocol level ([3] says USB uses CRC
for error detection), and so on and so forth. Ironically, the only
major component of most PCs today that does not have any form of error
detection or correction is also one of the most critical components:
the system RAM.

[1A] http://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/

[1B] http://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/page-10#post-112729

[2] http://forums.freenas.org/index.php?members/cyberjock.22572/

[3] http://www.beyondlogic.org/usbnutshell/usb3.shtml
--
Michael Kjörling • http://michael.kjorling.se • ***@kjorling.se
OpenPGP B501AC6429EF4514 http://michael.kjorling.se/public-keys/pgp
“People who think they know everything really annoy
those of us who know we don’t.” (Bjarne Stroustrup)

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Richard Yao
2014-07-05 15:25:44 UTC
Permalink
What you have described is consistent with a problem in rsync. The ZFS source code does not have the capability to start purging directories en masse without being explicitly told by userland to do it, but rsync does. If rsync believes that the other end has no files, it will start removing files en masse. A bitflip is all that is needed to alter its behavior.

If you had snapshots, you could rollback, but without them, the data is likely gone. ZFS cannot protect against a malfunctioning userland program doing something similiar to `rm -rf /path/to/files`. I suppose it might be possible to try to import at a historical point in time, but that would require finding an old uberblock on the disk and getting the ZFS code to verbatim import it. I had an idea on how to make this easier (storing the previous N uberblocks in the MOS for these occasions), but it requires disk format changes such that it could only help on future pools once it is available.
The backup system does not have ECC RAM. It is a low power system, not a server.
It was not an rsync issue. RSYNC on the primary system is reading from a read-only NFS mount. I have not written anything or done much to the backup system, nor will I until it is finished copying what it can.
I have not used snapshots on these systems.
Post by Richard Yao
Does this system have ECC memory? It sounds like a bitflip in rsync
caused it to mess up.
It would have likely been better to do incremental zfs send/recv. That
would not address the ECC memory issue, but it should at least avoid
giving a bitflip the power to say "wipe everything". Did you use
snapshots at all?
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Peter M
2014-07-05 06:47:21 UTC
Permalink
Hi,
Post by tom-/sOA9cmYp6rgYzRL+
Post by tom-/sOA9cmYp6rgYzRL+
... so I reinstalled the primary server identically to the backup.
Ubuntu
Post by tom-/sOA9cmYp6rgYzRL+
14.04, ZoL, fresh array format (I don't trust it to upgrade).
As you can imagine, at roughly 2TB into the copy (very early on), the
backup lost the majority of it's data. A partition that had 14.5TB now
shows 4.5TB and all of those directories now show empty. ZPOOL status
shows no errors of any kind.
Does this system have ECC memory? It sounds like a bitflip in rsync
caused it to mess up.
It would have likely been better to do incremental zfs send/recv. That
would not address the ECC memory issue, but it should at least avoid
giving a bitflip the power to say "wipe everything". Did you use
snapshots at all?
Wow, that's scary. Really scary.

Can you elaborate how this can happen that really everything is gone?

I am in progress of migrating to ZFS.
I would have a main server A, a backup server B and an archive server C.

If I would go for ECC I would need to buy old hardware (DDR1) for expensive
27EUR/GB.
Each server has currently 8GB of RAM, that would add up to 650 EUR just for
ECC. Not an option for my private system!

What can be done to reduce the risk and how would it help?

Also, as soon as ZFS is involved, would it mean that everything needs to be
ECC?

In my example above, would it at least be possible to just buy ECC for one
of the boxes?

Or would it help to leave one of the boxes as non-ZFS (and unfortunately
loosing so many advantages)?

System A has RAID1, system B has no redundancy and system C has RAID5
(RAIDz1). Additionally I plan to copy most important data monthly to an
external drive which is brought to a different location.
So I do a bunch of effort to have lots of redundancy because it's just all
data I have (of mostly personal value).
I really a single bit flip could cause me to loose everything despite this
large amount of redundancy then ZFS is indeed not an option. But is it
really the case?
I mean, a bit flip can not only be caused by RAM, it can happen in the
controller, the cache, the cable, ... so if ZFS sounds so sensitive to
that, how can I be sure that it won't loose any data if I use ECC RAM?


Peter


To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Gordan Bobic
2014-07-05 11:50:13 UTC
Permalink
Post by tom-/sOA9cmYp6rgYzRL+
Hi,
Post by tom-/sOA9cmYp6rgYzRL+
... so I reinstalled the primary server identically to the
backup. Ubuntu
Post by tom-/sOA9cmYp6rgYzRL+
14.04, ZoL, fresh array format (I don't trust it to upgrade).
As you can imagine, at roughly 2TB into the copy (very early on),
the
Post by tom-/sOA9cmYp6rgYzRL+
backup lost the majority of it's data. A partition that had
14.5TB now
Post by tom-/sOA9cmYp6rgYzRL+
shows 4.5TB and all of those directories now show empty. ZPOOL
status
Post by tom-/sOA9cmYp6rgYzRL+
shows no errors of any kind.
Does this system have ECC memory? It sounds like a bitflip in rsync
caused it to mess up.
It would have likely been better to do incremental zfs send/recv. That
would not address the ECC memory issue, but it should at least avoid
giving a bitflip the power to say "wipe everything". Did you use
snapshots at all?
Wow, that's scary. Really scary.
Can you elaborate how this can happen that really everything is gone?
I am in progress of migrating to ZFS.
I would have a main server A, a backup server B and an archive server C.
If I would go for ECC I would need to buy old hardware (DDR1) for
expensive 27EUR/GB.
Each server has currently 8GB of RAM, that would add up to 650 EUR just
for ECC. Not an option for my private system!
DDR1? Seriously?

For €650 you could buy 3 new motherboards, CPUs and 8GB of ECC RAM for each.
Post by tom-/sOA9cmYp6rgYzRL+
What can be done to reduce the risk and how would it help?
If you're not sufficiently worried yet, read this:
http://www.zdnet.com/blog/storage/dram-error-rates-nightmare-on-dimm-street/638
Post by tom-/sOA9cmYp6rgYzRL+
Also, as soon as ZFS is involved, would it mean that everything needs to
be ECC?
No, but ZFS won't help if the dta gets corrupted in RAM before it gets
flushed to the disk.
Post by tom-/sOA9cmYp6rgYzRL+
In my example above, would it at least be possible to just buy ECC for
one of the boxes?
It's up to you to weigh off the risks. I have long stopped using non-ECC
RAM-ed machines for any important task. Laptops and gaming machines
don't matter, but servers and workstations do.
Post by tom-/sOA9cmYp6rgYzRL+
Or would it help to leave one of the boxes as non-ZFS (and unfortunately
loosing so many advantages)?
The non-ECC situation doesn't get _worse_ with ZFS, IMO.
Post by tom-/sOA9cmYp6rgYzRL+
System A has RAID1, system B has no redundancy and system C has RAID5
(RAIDz1). Additionally I plan to copy most important data monthly to an
external drive which is brought to a different location.
So I do a bunch of effort to have lots of redundancy because it's just
all data I have (of mostly personal value).
I really a single bit flip could cause me to loose everything despite
this large amount of redundancy then ZFS is indeed not an option. But is
it really the case?
I think the implication was that it might have happened in rsync.
Post by tom-/sOA9cmYp6rgYzRL+
I mean, a bit flip can not only be caused by RAM, it can happen in the
controller, the cache, the cable, ... so if ZFS sounds so sensitive to
that, how can I be sure that it won't loose any data if I use ECC RAM?
I don't subscribe that ZFS without ECC is any more dangerous than any
other FS without ECC.

For the other things mentioned, cable carries SATA information which has
2-bit parity per byte. Caches typically have ECC, both on the CPU and on
disk controllers.


To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Richard Yao
2014-07-05 15:57:32 UTC
Permalink
Post by Peter M
Hi,
Post by Richard Yao
Post by tom-/sOA9cmYp6rgYzRL+
... so I reinstalled the primary server identically to the backup. Ubuntu
14.04, ZoL, fresh array format (I don't trust it to upgrade).
As you can imagine, at roughly 2TB into the copy (very early on), the
backup lost the majority of it's data. A partition that had 14.5TB now
shows 4.5TB and all of those directories now show empty. ZPOOL status
shows no errors of any kind.
Does this system have ECC memory? It sounds like a bitflip in rsync
caused it to mess up.
It would have likely been better to do incremental zfs send/recv. That
would not address the ECC memory issue, but it should at least avoid
giving a bitflip the power to say "wipe everything". Did you use
snapshots at all?
Wow, that's scary. Really scary.
Can you elaborate how this can happen that really everything is gone?
0. There were no snapshots.
1. A bit flip in rsync causes it to do the equivalent of `rm -rf`
2. All of the data is removed by rsync
Post by Peter M
I am in progress of migrating to ZFS.
I would have a main server A, a backup server B and an archive server C.
If I would go for ECC I would need to buy old hardware (DDR1) for expensive 27EUR/GB.
Each server has currently 8GB of RAM, that would add up to 650 EUR just for ECC. Not an option for my private system!
What can be done to reduce the risk and how would it help?
Use snapshots so you can rollback if a userland program goes haywire.
Post by Peter M
Also, as soon as ZFS is involved, would it mean that everything needs to be ECC?
As soon as you care about data, you need ECC (and possibly also an IOMMU). ZFS does not change this. What ZFS does change is the number of ways things can go wrong. Organizing all data into a self-healing merkle tree prevents data loss from trivial corruption by misdirected writes and other storage system oddities.

ZFS does not prevent data loss from bit flips, software bugs, malicious intrusion, Acts of God or malfunctioning devices. For bitflips, you want to have ECC. For software bugs, you want to have snapshots. For malicious intrusion, you want to have cold backups. For Acts of God, you want those cold backups to be offsite, among other things. For malfunctioning devices, you want to have an IOMMU.
Post by Peter M
In my example above, would it at least be possible to just buy ECC for one of the boxes?
Which motherboards and processors do these systems have? It will depend on that.
Post by Peter M
Or would it help to leave one of the boxes as non-ZFS (and unfortunately loosing so many advantages)?
No. Bitflips will cause problems no matter what filesystem is in use. The problem with rsync would have happened had the system run ext4. If it happened there (and it likely does), it is likely that the person would have likely blamed ext4 too. Since people expect ext4 to mess up, we would not hear about it because it is not news. In this instance, ZFS was blamed for an issue in rsync. ZFS messing up would be news, which is why we are hearing about it. Sadly, this incident was not ZFS' fault.
Post by Peter M
System A has RAID1, system B has no redundancy and system C has RAID5 (RAIDz1). Additionally I plan to copy most important data monthly to an external drive which is brought to a different location.
So I do a bunch of effort to have lots of redundancy because it's just all data I have (of mostly personal value).
I really a single bit flip could cause me to loose everything despite this large amount of redundancy then ZFS is indeed not an option. But is it really the case?
If a bitflip causes a userland program to run the equivalent of `rm -rf` and you have no snapshots, you will lose data and ZFS will not protect you. Protecting against malfunctioning userland programs wiping out unsnapshotted data is beyond the scope of ZFS' ability to protect data.
Post by Peter M
I mean, a bit flip can not only be caused by RAM, it can happen in the controller, the cache, the cable, ... so if ZFS sounds so sensitive to that, how can I be sure that it won't loose any data if I use ECC RAM?
ZFS can protect against bitflips that occur on the controller. It cannot protect against bitflips that occur in main memory. You need ECC to protect against those. ZFS does not make things more likely to go wrong without ECC. It just eliminates many possibilities where things go wrong that do not involve ECC.

Analogously, antibiotics did not cause incidents of cancer to rise, but they did eliminate a class of untreatable ailments such that if you have an untreatable ailment in a country where antibiotics are available, it is more likely to be cancer.
Post by Peter M
Peter
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Tom D
2014-07-05 19:32:17 UTC
Permalink
Durval,

Yes, only files have been lost. Subdirectories remain.

I do not outright eliminate the possibility of operator error, however, I
feel the odds are extremely slim. Everything was going fine with me
occasionally glancing at a terminal window with files slowly scrolling by
(very large files). All I was doing was the occasional "zfs list".

One thing I will add, even though it will probably end rational thought in
this discussion, is that we had a power outage Tuesday evening during the
copy. I have a UPS but it only lasts about 40 minutes and the computers do
not shut down gracefully when the UPS goes flat. When power was restored,
the machines came back up and I re-started copying the files. That was
late in the evening on Tuesday. The file loss happened either late
Thursday evening or Friday, as gauged by "zfs list"

I will point out, do not feel I can blame ZoL with any certainty. It would
be just as easy to believe the issue is a drive controller, drive, or
system malfunction. I know memory errors exist but I do not feel they are
the primary suspicion in this event.

It's clear that Richard does not consider ZoL to be fallible. I suppose
that's fair, as he has built up a trust in the system and probably has
extensive positive experience. As well as either not reading or not
properly understanding, it is clear you are going through a process of not
willing to consider it as a potential point of failure. I totally get
that. ... but without objectivity, I need to return to zfs-fuse as that is
a system *I* can trust. Trusting your backup storage is what it's all
about.

This is not an rsync issue. I have made that clear, with my detailed
history of the events.

Once again, this could be user error but I consider that less likely than a
failure of another component, as the copy had been running since late
Tuesday evening, when I noticed the data loss on Friday morning. I monitor
these systems frequently throughout the day and there was no problem on
Thursday around midnight, when the data was last reported to be intact.


- Tom

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Gregor Kopka
2014-07-06 11:04:16 UTC
Permalink
Tom,

iirc rsync first creates the directory structure on the target, then transfers the files.

Your systems rebooted because of a power outage while your primary system was half empty.

Could it be that after the power outage your backup server started to backup the primary (through cron), thus dropping all the files still missing there?

Gregor
Post by Tom D
Durval,
Yes, only files have been lost. Subdirectories remain.
I do not outright eliminate the possibility of operator error, however, I
feel the odds are extremely slim. Everything was going fine with me
occasionally glancing at a terminal window with files slowly scrolling by
(very large files). All I was doing was the occasional "zfs list".
One thing I will add, even though it will probably end rational thought in
this discussion, is that we had a power outage Tuesday evening during the
copy. I have a UPS but it only lasts about 40 minutes and the
computers do
not shut down gracefully when the UPS goes flat. When power was restored,
the machines came back up and I re-started copying the files. That was
late in the evening on Tuesday. The file loss happened either late
Thursday evening or Friday, as gauged by "zfs list"
I will point out, do not feel I can blame ZoL with any certainty. It would
be just as easy to believe the issue is a drive controller, drive, or
system malfunction. I know memory errors exist but I do not feel they are
the primary suspicion in this event.
It's clear that Richard does not consider ZoL to be fallible. I suppose
that's fair, as he has built up a trust in the system and probably has
extensive positive experience. As well as either not reading or not
properly understanding, it is clear you are going through a process of not
willing to consider it as a potential point of failure. I totally get
that. ... but without objectivity, I need to return to zfs-fuse as that is
a system *I* can trust. Trusting your backup storage is what it's all
about.
This is not an rsync issue. I have made that clear, with my detailed
history of the events.
Once again, this could be user error but I consider that less likely than a
failure of another component, as the copy had been running since late
Tuesday evening, when I noticed the data loss on Friday morning. I monitor
these systems frequently throughout the day and there was no problem on
Thursday around midnight, when the data was last reported to be intact.
- Tom
To unsubscribe from this group and stop receiving emails from it, send
--
Facebook is a plague to the Internet and needs to be exterminated.

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Will Rouesnel
2014-07-06 14:44:51 UTC
Permalink
<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
In lieu of a scrub showing checksum errors (and there's no reason
you can't do that while the rsync is running, it'll just run slower)
it's staggeringly unlikely that ZoL has lost data from a filesystem
error with no other events.<br>
<br>
Can you post your logs to pastebin from around the time, if
possible?<br>
<br>
You seem fairly driven that this is a ZoL error, but there's a lot
of moving parts in what you were doing *and* systems going down from
unclean shutdowns. Moreover, it's <i>extremely</i> suspicious that
you lost a chunk of files in alphabetical and numerical order. <br>
<br>
What's in your crontab and cron.weekly directories? That would fit
the approximate timeframe of something starting up.<br>
<br>
<div class="moz-cite-prefix">On 06/07/14 05:32, Tom D wrote:<br>
</div>
<blockquote
cite="mid:5562720a-233c-46a1-b731-359982917195-VKpPRiiRko4/***@public.gmane.org"
type="cite">
<meta http-equiv="Context-Type" content="text/html; charset=UTF-8">
<div dir="ltr">Durval,<br>
<br>
Yes, only files have been lost.  Subdirectories remain.<br>
<br>
I do not outright eliminate the possibility of operator error,
however, I feel the odds are extremely slim.  Everything was
going fine with me occasionally glancing at a terminal window
with files slowly scrolling by (very large files).  All I was
doing was the occasional "zfs list".<br>
<br>
One thing I will add, even though it will  probably end rational
thought in this discussion, is that we had a power outage
Tuesday evening during the copy.  I have a UPS but it only lasts
about 40 minutes and the computers do not shut down gracefully
when the UPS goes flat.  When power was restored, the machines
came back up and I re-started copying the files.  That was late
in the evening on Tuesday.  The file loss happened either late
Thursday evening or Friday, as gauged by "zfs list"<br>
<br>
I will point out, do not feel I can blame ZoL with any
certainty.  It would be just as easy to believe the issue is a
drive controller, drive, or system malfunction.  I know memory
errors exist but I do not feel they are the primary suspicion in
this event.<br>
<br>
It's clear that Richard does not consider ZoL to be fallible.  I
suppose that's fair, as he has built up a trust in the system
and probably has extensive positive experience.  As well as
either not reading or not properly understanding, it is clear
you are going through a process of not willing to consider it as
a potential point of failure.  I totally get that.  ... but
without objectivity, I need to return to zfs-fuse as that is a
system *I* can trust.  Trusting your backup storage is what it's
all about.<br>
<br>
This is not an rsync issue.  I have made that clear, with my
detailed history of the events.<br>
<br>
Once again, this could be user error but I consider that less
likely than a failure of another component, as the copy had been
running since late Tuesday evening, when I noticed the data loss
on Friday morning.  I monitor these systems frequently
throughout the day and there was no problem on Thursday around
midnight, when the data was last reported to be intact.<br>
<br>
<br>
- Tom<br>
</div>
To unsubscribe from this group and stop receiving emails from it,
send an email to <a moz-do-not-send="true"
href="mailto:zfs-discuss+unsubscribe-VKpPRiiRko4/***@public.gmane.org">zfs-discuss+unsubscribe-VKpPRiiRko4/***@public.gmane.org</a>.<br>
</blockquote>
<br>
</body>
</html>

<p></p>

To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:zfs-discuss+unsubscribe-VKpPRiiRko4/***@public.gmane.org">zfs-discuss+unsubscribe-VKpPRiiRko4/***@public.gmane.org</a>.<br />
Daniel Brooks
2014-07-06 14:51:26 UTC
Permalink
When ZFS detects lost data, it's always at the block level and not the file
level, and it's equally likely to be in a directory entry as a file. Since
you've lost whole files but no directories, and zfs hasn't reported any
errors, it looks entirely like a problem outside of ZFS.

How are changes on your primary system normally distributed to your
secondary? I assume you disabled this process before you began copying from
secondary back to primary. Could have started back up after the reboot?

db48x
Post by Tom D
Durval,
Yes, only files have been lost. Subdirectories remain.
I do not outright eliminate the possibility of operator error, however, I
feel the odds are extremely slim. Everything was going fine with me
occasionally glancing at a terminal window with files slowly scrolling by
(very large files). All I was doing was the occasional "zfs list".
One thing I will add, even though it will probably end rational thought
in this discussion, is that we had a power outage Tuesday evening during
the copy. I have a UPS but it only lasts about 40 minutes and the
computers do not shut down gracefully when the UPS goes flat. When power
was restored, the machines came back up and I re-started copying the
files. That was late in the evening on Tuesday. The file loss happened
either late Thursday evening or Friday, as gauged by "zfs list"
I will point out, do not feel I can blame ZoL with any certainty. It
would be just as easy to believe the issue is a drive controller, drive, or
system malfunction. I know memory errors exist but I do not feel they are
the primary suspicion in this event.
It's clear that Richard does not consider ZoL to be fallible. I suppose
that's fair, as he has built up a trust in the system and probably has
extensive positive experience. As well as either not reading or not
properly understanding, it is clear you are going through a process of not
willing to consider it as a potential point of failure. I totally get
that. ... but without objectivity, I need to return to zfs-fuse as that is
a system *I* can trust. Trusting your backup storage is what it's all
about.
This is not an rsync issue. I have made that clear, with my detailed
history of the events.
Once again, this could be user error but I consider that less likely than
a failure of another component, as the copy had been running since late
Tuesday evening, when I noticed the data loss on Friday morning. I monitor
these systems frequently throughout the day and there was no problem on
Thursday around midnight, when the data was last reported to be intact.
- Tom
To unsubscribe from this group and stop receiving emails from it, send an
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Chris Dunlop
2014-07-07 05:42:11 UTC
Permalink
Hi Tom,

It's likely way too late, but it's possible you could recover
data by rolling back your pool to a transaction (txg) prior to
the deletions. See 'zpool import -T' (documentation added in
zfs-0.6.3) - but heed the warnings in the docs!!. Unfortunately
I've no idea how to go about identifying whether it's too late
nor how to identify an appropriate txg to roll back to.

The rest of my response isn't addressing your question of how to
recover, but...
Post by Tom D
Durval,
Yes, only files have been lost. Subdirectories remain.
I do not outright eliminate the possibility of operator error, however, I
feel the odds are extremely slim. Everything was going fine with me
occasionally glancing at a terminal window with files slowly scrolling by
(very large files). All I was doing was the occasional "zfs list".
One thing I will add, even though it will probably end rational thought in
this discussion, is that we had a power outage Tuesday evening during the
copy. I have a UPS but it only lasts about 40 minutes and the computers do
not shut down gracefully when the UPS goes flat. When power was restored,
the machines came back up and I re-started copying the files. That was
late in the evening on Tuesday. The file loss happened either late
Thursday evening or Friday, as gauged by "zfs list"
If I understand correctly, your secondary system was originally
configured to periodically automatically power on and perform a
backup of the primary system using rsync, including the --delete
option.

To my mind the most likely explanation for your data loss is
that this "backup from primary to secondary" kicked in when the
power was restored to the secondary, and so the backup started
happily deleting files from the secondary because they didn't
exist on the primary.

Are your primary to secondary backups logged, with sufficient
detail, to show that this wasn't the case?

As has been stated by others, it's very very difficult (read:
nigh on impossible) to conceive of a case where a problem within
ZoL would cause the pattern of data loss you've described.

Cheers,

Chris

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Fajar A. Nugraha
2014-07-07 06:53:57 UTC
Permalink
On Sun, Jul 6, 2014 at 2:32 AM, Tom D <tom-/sOA9cmYp6rgYzRL+***@public.gmane.org> wrote:

only files have been lost. Subdirectories remain.
Post by Tom D
I do not outright eliminate the possibility of operator error, however, I
feel the odds are extremely slim.
This is not an rsync issue. I have made that clear, with my detailed
history of the events.
I have quoted only three lines from your previous mail.

For the first one, IIRC any error in zfs side (e.g. checksum error) should
show up in "zpool status". If you have tons of files missing and nothing
about corrupted files on zpool status, then I doubt it's a problem in zfs
(or even in hardware, for that matter)

For the second and third one, I'm more inclined to think it's operator
error. It's the only explanation that make sense (assuming zfs didn't
complain about any errors).

In any case, I think the biggest take away from this is "always have
automatic snapshots on". It could've helped you rollback to the point when
you last see the files. Plus, it's also a good way to estimate at first
glance whether it's operator/backup script problem or not (i.e. if current
files are gone but older snapshot still have it, or if all snapshots are
suddenly gone, then it's definitely operator/backup script fault).
--
Fajar

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Andrew Barnes
2014-07-05 06:19:02 UTC
Permalink
Hi Tom,
Post by tom-/sOA9cmYp6rgYzRL+
As you can imagine, at roughly 2TB into the copy (very early on), the
backup lost the majority of it's data. A partition that had 14.5TB now
shows 4.5TB and all of those directories now show empty. ZPOOL status
shows no errors of any kind.
Just to clarify what is it that that lead you to believe the backup lost
the majority of it's data? Did the used column from "zfs list" change for
no reason from 14.5TB to 4.5TB, or is a directory that should contain
files/dirs now appear empty? Have you checked that all your filesystems are
mounted: zfs list -r -o name,mounted,mountpoint

I've left it to continue copying and I have double backups of the most
Post by tom-/sOA9cmYp6rgYzRL+
important stuff, such as my photography, key personal data, etc. so I'm not
entirely screwed but there is a ton of stuff that I'd like to get back.
They are mostly large files of 1~40GB in size (disk images of a bunch of
workstations). Most of those images are backups but a few of those systems
are long gone but I still need to have the data that was on them, just in
case. I could really use that data.
So far, I have only tried exporting and importing the array. I'm worried
that trying too much will ruin my chances of recovering the data.
Any ideas of what to do next? I'd like to recover the data and convert
the backup system back to zfs-fuse.
To unsubscribe from this group and stop receiving emails from it, send an
Cheers
Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
tom-/sOA9cmYp6rgYzRL+
2014-07-05 06:34:48 UTC
Permalink
Post by Andrew Barnes
Hi Tom,
Just to clarify what is it that that lead you to believe the backup lost
the majority of it's data? Did the used column from "zfs list" change for
no reason from 14.5TB to 4.5TB, or is a directory that should contain
files/dirs now appear empty? Have you checked that all your filesystems are
mounted: zfs list -r -o name,mounted,mountpoint
As well as losing 10TB from the "zfs list", I have many hundreds of empty
directories that formerly housed system images.
Post by Andrew Barnes
Cheers
Andrew
The only change I have recently made to the system is moving from the
RealTek r8169 driver to the r8168 driver. When this copy completes, I will
roll that back and see what happens.


Thanks for the responses. I appreciate the input. :)

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Gordan Bobic
2014-07-05 11:38:03 UTC
Permalink
Post by tom-/sOA9cmYp6rgYzRL+
I have been using zfs-fuze, happily, for a lot of years. I've never
lost a byte, using it. When I moved to advanced format disks, it got
slow. I tolerated that because my priority is 99% reliability, 1%
performance. It was sufficient.
FYI, the zfs-fuse fork that supports pool v26 had recently acquired the
ashift configuration option. I'm still using zfs-fuse on my ARM machines
(because they are 32-bit).
Post by tom-/sOA9cmYp6rgYzRL+
I am running two identical Ubuntu 14.04 systems with 10 x 4TB Seagate
NAS drives in each. Every day, the backup system turns itself on (BIOS
setting), rsync's the entire array, and then cleanly shuts down.
[...]
Post by tom-/sOA9cmYp6rgYzRL+
It ran fine for a few months so I figured I'd work it out a bit. I did
a couple of scrubs and ran some massive read/write cycles to the array.
It worked great.
... so I reinstalled the primary server identically to the backup.
Ubuntu 14.04, ZoL, fresh array format (I don't trust it to upgrade).
As you can imagine, at roughly 2TB into the copy (very early on), the
backup lost the majority of it's data. A partition that had 14.5TB now
shows 4.5TB and all of those directories now show empty. ZPOOL status
shows no errors of any kind.
Did your cron job kick off and rsync the empty FS to the backup side? Do
you have snapshots on the backup side?


To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Michael Kjörling
2014-07-05 12:32:33 UTC
Permalink
Do you have snapshots on the backup side?
I have not used snapshots on these systems.
--
Michael Kjörling • http://michael.kjorling.se • ***@kjorling.se
OpenPGP B501AC6429EF4514 http://michael.kjorling.se/public-keys/pgp
“People who think they know everything really annoy
those of us who know we don’t.” (Bjarne Stroustrup)

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
tom-/sOA9cmYp6rgYzRL+
2014-07-05 15:56:41 UTC
Permalink
Thanks for the input. I particularly appreciate reading that zfs-fuse now
has ashift capability for advanced format disks.

I hate to be snarky to a group that I am asking help from. For that
reason, I have been brief in my replies. I promise you my intentions are
positive and I respect the work done on ZoL. Thank you.

... but let me point out a few things.

1 - Has anyone read my original post? If so, you have not comprehended it.
2 - This problem was not caused by rsync. I know it sounds exactly like
someone running rsync with "--delete" but rsync was run on the target
machine with the source being a ro NFS mount. rsync did not cause this.
3 - I have been running zfs-fuse since 2008 with no loss of data, although
I've had some scary moments when a disk starts to show errors, I replace
it, it resilvers, and then another disk starts to show errors shortly
afterward. Fortunately, I've been OK but I've since moved to RAIDZ2. I
have been running zfs-fuse on these exact systems (with a hard disk upgrade
in 2013) for two years. Again, no loss of data. I tried to convert to ZoL
and lost 70% of my data in a very short time. The suspect here, for me, is
ZoL.
4 - I also understand NAS drives are not data center quality but I'm not
talking about a flipped bit knocking out a file somewhere. I'm talking
about losing thousands of files.

Peter M - I've loved ZFS on Solaris forever. That's why I run it on
Linux. ECC is better but it is not absolutely required for ZFS to provide
a pretty reliable system.

I do not expect to convert ECC at this time, although I might in the
future. I understand the benefits. I wouldn't dream of using a system at
work without ECC. If I return to zfs-fuse and have another issue, I will
definitely upgrade to ECC, regardless of cost.

With the gem of knowledge that zfs-fuse supports ashift (thank you Gordon
Bobic!), I will convert back to zfs-fuse. I care only about reliability.
zfs-fuse is fast enough for me and I trust it with my life.

At the end of the day, I take responsibility for my own data loss. Shame
on me for moving too quickly to a new system. I should have tested it for
a year or even two, prior to committing my data to it. There was no need
for me to jump in with both feet. I was intoxicated by the quadruple write
speed over zfs-fuse. As a home user, it's tough to justify a third system
for the purpose of testing.

Thank you, gentlemen. I am sincere in that wish. I am grateful for the
project and help offered.


Kind regards,

Tom

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Richard Yao
2014-07-05 16:39:51 UTC
Permalink
Thanks for the input. I particularly appreciate reading that zfs-fuse now has ashift capability for advanced format disks.
I hate to be snarky to a group that I am asking help from. For that reason, I have been brief in my replies. I promise you my intentions are positive and I respect the work done on ZoL. Thank you.
... but let me point out a few things.
1 - Has anyone read my original post? If so, you have not comprehended it.
2 - This problem was not caused by rsync. I know it sounds exactly like someone running rsync with "--delete" but rsync was run on the target machine with the source being a ro NFS mount. rsync did not cause this.
If I sat down to look at the binaries long enough, I am certain that I could write a program that uses the system debugging facilities to flip a single bit in a running rsync to cause it to start operating with --delete midway through its execution. Does this require a demonstration?
3 - I have been running zfs-fuse since 2008 with no loss of data, although I've had some scary moments when a disk starts to show errors, I replace it, it resilvers, and then another disk starts to show errors shortly afterward. Fortunately, I've been OK but I've since moved to RAIDZ2. I have been running zfs-fuse on these exact systems (with a hard disk upgrade in 2013) for two years. Again, no loss of data. I tried to convert to ZoL and lost 70% of my data in a very short time. The suspect here, for me, is ZoL.
4 - I also understand NAS drives are not data center quality but I'm not talking about a flipped bit knocking out a file somewhere. I'm talking about losing thousands of files.
Peter M - I've loved ZFS on Solaris forever. That's why I run it on Linux. ECC is better but it is not absolutely required for ZFS to provide a pretty reliable system.
I do not expect to convert ECC at this time, although I might in the future. I understand the benefits. I wouldn't dream of using a system at work without ECC. If I return to zfs-fuse and have another issue, I will definitely upgrade to ECC, regardless of cost.
With the gem of knowledge that zfs-fuse supports ashift (thank you Gordon Bobic!), I will convert back to zfs-fuse. I care only about reliability. zfs-fuse is fast enough for me and I trust it with my life.
At the end of the day, I take responsibility for my own data loss. Shame on me for moving too quickly to a new system. I should have tested it for a year or even two, prior to committing my data to it. There was no need for me to jump in with both feet. I was intoxicated by the quadruple write speed over zfs-fuse. As a home user, it's tough to justify a third system for the purpose of testing.
You will likely have bit flips cause something else go wrong, but I doubt it will manifest itself as this particular event. Bitflips are far more likely to crash a program than they are to start wiping out data. I used to run without ECC until I detected programs crashing due to a bitflip in a library. I initially blamed ZFS, but being one of the developers, I investigated and discovered that a single bit had flipped in an ARC buffer caching libpython. In that situation, I was able to determine the exact bit in the file.

The event you encountered is something that does happen (and you are the second that I believe to have hit it), but having it happen is like winning the mega millions lottery. Having it occur twice to the same person has never happened to the best of my knowledge. If you do have rsync mess up like this again (assuming the on-disk binaries binary are uncorrupted and your memory passes memtest), then I would start to wonder if something extraordinary were happening in a bad way.
Thank you, gentlemen. I am sincere in that wish. I am grateful for the project and help offered.
One more word of advice. Use snapshots so that you can recover from errant userland programs.
Kind regards,
Tom
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Michael Kjörling
2014-07-05 17:11:29 UTC
Permalink
Post by Richard Yao
One more word of advice. Use snapshots so that you can recover from
errant userland programs.
Or user error. It wasn't that long ago that having zfs-auto-snap run
on a regular basis saved me after I had managed to mess up a local git
repository (the ZFS/SPL source code, as it happens). It probably would
have been possible to "back out" the borked merge from upstream and
end up with a good copy, but it was _so_ much easier to just roll back
to a snapshot taken a few hours earlier (to be sure there was no trace
of my mistakes in there) and retrace the known good steps.

And lest someone thinks otherwise given my talking about ECC RAM, keep
in mind that it wasn't until _very_ recently that I switched to ECC
RAM myself. I figure I've been lucky; even since moving the bulk of my
data storage to ZFS I've never seen any indications of a problem; no
reported errors, no I/O errors from ZFS refusing to hand me data, no
corrected blocks reported. Even so, ECC gives me that additional
safety net of knowing that data in RAM is correct. (Now if there was
only a good way to tell that it was actually _working_, that could be
trusted to report the correct status on all systems...)
--
Michael Kjörling • http://michael.kjorling.se • ***@kjorling.se
OpenPGP B501AC6429EF4514 http://michael.kjorling.se/public-keys/pgp
“People who think they know everything really annoy
those of us who know we don’t.” (Bjarne Stroustrup)

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Gordan Bobic
2014-07-05 16:51:31 UTC
Permalink
Post by tom-/sOA9cmYp6rgYzRL+
... but let me point out a few things.
1 - Has anyone read my original post? If so, you have not comprehended it.
2 - This problem was not caused by rsync. I know it sounds exactly like
someone running rsync with "--delete" but rsync was run on the target
machine with the source being a ro NFS mount. rsync did not cause this.
3 - I have been running zfs-fuse since 2008 with no loss of data,
although I've had some scary moments when a disk starts to show errors,
I replace it, it resilvers, and then another disk starts to show errors
shortly afterward. Fortunately, I've been OK but I've since moved to
RAIDZ2. I have been running zfs-fuse on these exact systems (with a
hard disk upgrade in 2013) for two years. Again, no loss of data. I
tried to convert to ZoL and lost 70% of my data in a very short time.
The suspect here, for me, is ZoL.
While theoretically possible, it seems very unlikely, since there are
people on this list with orders of magnitude more data and nobody has to
the best of my knowledge reported similar loss of data. I have been
using ZoL since 2010, when the KQ Infotech posix layer was first
released, before ZoL project had a complete implementation and never
experienced actual data loss.
Post by tom-/sOA9cmYp6rgYzRL+
4 - I also understand NAS drives are not data center quality but I'm not
talking about a flipped bit knocking out a file somewhere. I'm talking
about losing thousands of files.
The disks don't sound like they are the cause of your problems, and most
of us run ZoL with desktop or NAS drives (I only use cheap drives, and
most of my 1TB drives still in service are over 5 years old (my rolling
upgrade cycle releases disks for recycling at a similar rate to failures
occurring, which keeps me with enough spares of required size).
Post by tom-/sOA9cmYp6rgYzRL+
Peter M - I've loved ZFS on Solaris forever. That's why I run it on
Linux. ECC is better but it is not absolutely required for ZFS to
provide a pretty reliable system.
Well, personally I didn't think it's an absolute necessity since I ran
my backup ZFS server with 16GB of non-ECC RAM (couldn't get 4GB DDR2
unbuffered ECC DIMMs) for several years, but having read some of the
links provided in this thread I can see where they are coming from with
the idea that there is little point in bothering with ZFS if you don't
have ECC RAM.
Post by tom-/sOA9cmYp6rgYzRL+
I do not expect to convert ECC at this time, although I might in the
future. I understand the benefits. I wouldn't dream of using a system
at work without ECC. If I return to zfs-fuse and have another issue, I
will definitely upgrade to ECC, regardless of cost.
zfs-fuse is _much_ stingier with memory. IIRC the default settings are
for 256MB of ARC which is tiny, and a bit flip is much less likely to
occur in a smaller block of RAM. ZoL OTOH defaults to something like
half the RAM for ARC, and often exceeds it. This could increase the
changes of bit flip in it's RAM without ECC by orders of magnitude (e.g.
with a 2GB of 20GB ARC)
Post by tom-/sOA9cmYp6rgYzRL+
With the gem of knowledge that zfs-fuse supports ashift (thank you
Gordon Bobic!), I will convert back to zfs-fuse. I care only about
reliability. zfs-fuse is fast enough for me and I trust it with my life.
It is worth noting that the zfs-fuse version that supports it is forked
from the mainline zfs-fuse (which is now essentially abandoned). Also,
it was forked BEFORE the zfs-fuse 0.7.0 official release, so there are
some patches in the official release that aren't in the fork, and there
are a number of patches in the fork that aren't in the official release.
The unofficial fork is the one with support for pool v26 and ashift
adjustment. As I said before, I use it on my ARM machines because ZoL
doesn't play well with memory constraints of 32-bit systems, let alone
32-bit systems with RAM in the 0.5-1GB range.

The compromise is that my 2GHz single-core ARM tops out at allowing the
scrub to proceed no faster than about 50MB/s, with disks operating at
about 50% I/O capacity. But I can live with that.
Post by tom-/sOA9cmYp6rgYzRL+
At the end of the day, I take responsibility for my own data loss.
Shame on me for moving too quickly to a new system. I should have
tested it for a year or even two, prior to committing my data to it.
Or maybe even just kept snapshots and used zfs send | receive to copy
the data. If you were using rsync to copy the data back, the most
plausible explanation remains that an rsync issue was the root cause.
Post by tom-/sOA9cmYp6rgYzRL+
There was no need for me to jump in with both feet. I was intoxicated
by the quadruple write speed over zfs-fuse. As a home user, it's tough
to justify a third system for the purpose of testing.
The way I look at it is by weighing the cost of a spare system against
the value I put on my data.


To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Richard Yao
2014-07-05 16:59:51 UTC
Permalink
Post by tom-/sOA9cmYp6rgYzRL+
Peter M - I've loved ZFS on Solaris forever. That's why I run it on
Linux. ECC is better but it is not absolutely required for ZFS to
provide a pretty reliable system.
Well, personally I didn't think it's an absolute necessity since I ran my backup ZFS server with 16GB of non-ECC RAM (couldn't get 4GB DDR2 unbuffered ECC DIMMs) for several years, but having read some of the links provided in this thread I can see where they are coming from with the idea that there is little point in bothering with ZFS if you don't have ECC RAM.
Those recommending ECC do *NOT* claim that. However, we do claim that bitflips in system memory form a class of problems that ZFS cannot address. Protecting against those problems requires ECC RAM.

Some people read this and then misinterpret it to say that "ZFS needs ECC to work", but that cannot be further from the truth. Other filesystems are equally affected. They are just so much less reliable that you cannot distinguish problems caused by bitflips from other problems that affect them, but not ZFS.

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
tom-/sOA9cmYp6rgYzRL+
2014-07-05 16:33:32 UTC
Permalink
Let me provide a bit more information:

Base Board Information
Manufacturer: ASUSTeK COMPUTER INC.
Product Name: E45M1-M PRO
Version: Rev X.0x
Serial Number: 120700372000200
Asset Tag: To be filled by O.E.M.
Features:
Board is a hosting board
Board is replaceable
Location In Chassis: To be filled by O.E.M.
Chassis Handle: 0x0003
Type: Motherboard
Contained Object Handles: 0

RAM = 8GB of Mushkin RAM, underclocked, default voltage.

Further, I have not lost all of my data. I've lost about 70% of the data
on one ZFS filesystem. It simply started showing less data in "zfs list"
and now has a ton of empty directories.

The stuff I absolutely require was doubly backed up to external hard drives
prior to this procedure so it's OK but, oddly, it is the first thing my
script copies from the backup so it came over to the primary intact. Most
of the stuff I really would like to have has been lost but this is not the
majority of the data. Over the next few months, these disk arrays will be
re-loaded with system-image backups of running systems and my life will
mostly be OK. I like to back up systems immediately upon receiving them,
so I can restore to the original image if necessary, and I've lost most of
those. Some of those will be available elsewhere. Also, I have images of
some systems that no longer exist but could have been rebuilt, if the data
on those systems were deemed necessary. Now, those systems appear to be
lost.

I'm still running ZoL. I just realized this problem, yesterday. The
primary system is still in the process of copying the last of the remaining
data from the backup system. I will let this run to completion which I
expect to finish sometime this afternoon.

There are still a few things I intend to try. I was hoping for more
ideas. I will report back with any success, if I am able to have some.

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
ray vantassle
2014-07-05 17:55:21 UTC
Permalink
Tom, not to brag, but I was the one who gave zfs-fuse the ashift
capability. I was and still am astonished that people have been wanting
this for years but nobody added it. The patch was so small it took me only
an afternoon to do it.

That aside ....
I echo your concerns about ZOL. I first looked at it several years ago and
bypassed it because at that time it was far from ready for prime-time. I
went with zfs-fuse on both my systems, the main repository and the backup
system. Several months ago I switched by backup system to ZOL -- and
discovered that it is functionally un-usable on a 32-bit system so I had to
upgrade my OS to 64-bit Debian. I still use zfs-fuse on my main
repository and don't expect to ever change it over to ZOL.

I recently upgraded my ZOL from 0.6.0-rc13 to 0.6.2. After reading all the
trouble that people were having with 0.6.3 I decided to stay away from that.
You know how you can identify a pioneer? From all the arrows in his back.
;-)

Some of the ZOL folks are, IMHO, living in an echo chamber. The ECC
business being a case in point. The attitude of "if you don't believe in
ECC you might as well just copy your data to /dev/null" doesn't help, it
just exhibits arrogant snottiness. Regardless of the theoretical advantages
of ECC, the entire computer industry has de-facto concluded that the
theoretical benefits are are not real and that ECC is realistically not
needed in 99+% of the real world, only in highly specific situations at
best. Heck, the industry even dropped parity RAM decades ago.

I wish they'd stop wasting time on a battle that they can never win. The
industry has decided that 8-bit non-ECC non-parity RAM is all that is
needed and will not even listen to the arguments of the proponents of a
small offbeat filesystem.
Sorry, </rant>

I'm not sure if snapshots would have saved your data, but this
(unexpectedly losing data) is the very reason I started doing automatic
hourly, daily, and monthly snapshots. Now if there were only a easy and
transparent way to access them....

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Richard Yao
2014-07-05 18:12:50 UTC
Permalink
Tom, not to brag, but I was the one who gave zfs-fuse the ashift capability. I was and still am astonished that people have been wanting this for years but nobody added it. The patch was so small it took me only an afternoon to do it.
I told the Mac ZFS guys how to do it before O3X was made. The only thing being added here is support for an override. Importing pools that have different ashift values for top level vdevs has *always* been supported.
That aside ....
I echo your concerns about ZOL. I first looked at it several years ago and bypassed it because at that time it was far from ready for prime-time. I went with zfs-fuse on both my systems, the main repository and the backup system. Several months ago I switched by backup system to ZOL -- and discovered that it is functionally un-usable on a 32-bit system so I had to upgrade my OS to 64-bit Debian. I still use zfs-fuse on my main repository and don't expect to ever change it over to ZOL.
This is because of design flaws in Linux. The Linux upstream developers believe that virtual memory is a user land affair and have crippled it in such a way that it is unusable on 32-bit and barely usable on 64-bit. Addressing it requires addressing a fundamental incompatibility between how Linux manages kernel memory and how Solaris manages it. While I and others have been working on that, it is *NOT* an easy thing to fix.
I recently upgraded my ZOL from 0.6.0-rc13 to 0.6.2. After reading all the trouble that people were having with 0.6.3 I decided to stay away from that.
You know how you can identify a pioneer? From all the arrows in his back. ;-)
The problems have to do with the kernel-userland interface (/dev/zfs) being changed in a backward incompatible way. If your user land utilities and modules are in sync, things will be fine. I had foreseen this problem years ago, but unfortunately, ZFS developers on other platforms did *NOT* care. They only care now that they have seen the consequences of making backward-incompatible changes to /dev/zfs interface.

In hindsight, I wish that I had made more of a point to Brian that we should co-author the release announcement for 0.6.3. Had we done that, I would have included information about the kernel-userland interface in the release announcement, but that ship has sailed.
Some of the ZOL folks are, IMHO, living in an echo chamber. The ECC business being a case in point. The attitude of "if you don't believe in ECC you might as well just copy your data to /dev/null" doesn't help, it just exhibits arrogant snottiness. Regardless of the theoretical advantages of ECC, the entire computer industry has de-facto concluded that the theoretical benefits are are not real and that ECC is realistically not needed in 99+% of the real world, only in highly specific situations at best. Heck, the industry even dropped parity RAM decades ago.
Not a single person who recommends ECC for reliable operation made such assertions. Those who do not like the idea that ECC is necessary for a reliability system on the other hand assert that is what is being said. I did a good write-up on this on the Open ZFS website:

http://open-zfs.org/wiki/Hardware#ECC_Memory
I wish they'd stop wasting time on a battle that they can never win. The industry has decided that 8-bit non-ECC non-parity RAM is all that is needed and will not even listen to the arguments of the proponents of a small offbeat filesystem.
Sorry, </rant>
You have the code. If you can do better, by all means go ahead. I and others would be more than happy to step aside if you can do better than we did.
I'm not sure if snapshots would have saved your data, but this (unexpectedly losing data) is the very reason I started doing automatic hourly, daily, and monthly snapshots. Now if there were only a easy and transparent way to access them....
It is not the file system's fault if your user land program decides to do the equivalent of `rm -rf`.

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Richard Yao
2014-07-05 18:15:47 UTC
Permalink
Post by Richard Yao
Tom, not to brag, but I was the one who gave zfs-fuse the ashift capability. I was and still am astonished that people have been wanting this for years but nobody added it. The patch was so small it took me only an afternoon to do it.
I told the Mac ZFS guys how to do it before O3X was made. The only thing being added here is support for an override. Importing pools that have different ashift values for top level vdevs has *always* been supported.
Since my words are likely going to be twisted, I should probably clarify that the changes to add this are identical in both ZFS-FUSE and Mac-ZFS. If you can do them in one, then you can do them in another.

Not that the details of what I say matter as you have demonstrated that others are capable of asserting that I said something completely different no matter what I say.

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Linda Kateley
2014-07-05 18:18:38 UTC
Permalink
Richard,

I have been searching for a good doc on the zfs and ecc ram and I was so
happy to read yours. I also would like to suggest to the open-zfs
community to have an "office hours" on the subject as I am not sure, but
believe this is a very misunderstood concept in zfs.

linda
Post by Richard Yao
Post by ray vantassle
Tom, not to brag, but I was the one who gave zfs-fuse the ashift
capability. I was and still am astonished that people have been
wanting this for years but nobody added it. The patch was so small
it took me only an afternoon to do it.
I told the Mac ZFS guys how to do it before O3X was made. The only
thing being added here is support for an override. Importing pools
that have different ashift values for top level vdevs has *always*
been supported.
Post by ray vantassle
That aside ....
I echo your concerns about ZOL. I first looked at it several years
ago and bypassed it because at that time it was far from ready for
prime-time. I went with zfs-fuse on both my systems, the main
repository and the backup system. Several months ago I switched by
backup system to ZOL -- and discovered that it is functionally
un-usable on a 32-bit system so I had to upgrade my OS to 64-bit
Debian. I still use zfs-fuse on my main repository and don't expect
to ever change it over to ZOL.
This is because of design flaws in Linux. The Linux upstream
developers believe that virtual memory is a user land affair and have
crippled it in such a way that it is unusable on 32-bit and barely
usable on 64-bit. Addressing it requires addressing a fundamental
incompatibility between how Linux manages kernel memory and how
Solaris manages it. While I and others have been working on that, it
is *NOT* an easy thing to fix.
Post by ray vantassle
I recently upgraded my ZOL from 0.6.0-rc13 to 0.6.2. After reading
all the trouble that people were having with 0.6.3 I decided to stay
away from that.
You know how you can identify a pioneer? From all the arrows in his back. ;-)
The problems have to do with the kernel-userland interface (/dev/zfs)
being changed in a backward incompatible way. If your user land
utilities and modules are in sync, things will be fine. I had foreseen
this problem years ago, but unfortunately, ZFS developers on other
platforms did *NOT* care. They only care now that they have seen the
consequences of making backward-incompatible changes to /dev/zfs
interface.
In hindsight, I wish that I had made more of a point to Brian that we
should co-author the release announcement for 0.6.3. Had we done that,
I would have included information about the kernel-userland interface
in the release announcement, but that ship has sailed.
Post by ray vantassle
Some of the ZOL folks are, IMHO, living in an echo chamber. The ECC
business being a case in point. The attitude of "if you don't believe
in ECC you might as well just copy your data to /dev/null" doesn't
help, it just exhibits arrogant snottiness. Regardless of the
theoretical advantages of ECC, the entire computer industry has
de-facto concluded that the theoretical benefits are are not real and
that ECC is realistically not needed in 99+% of the real world, only
in highly specific situations at best. Heck, the industry even
dropped parity RAM decades ago.
Not a single person who recommends ECC for reliable operation made
such assertions. Those who do not like the idea that ECC is necessary
for a reliability system on the other hand assert that is what is
http://open-zfs.org/wiki/Hardware#ECC_Memory
Post by ray vantassle
I wish they'd stop wasting time on a battle that they can never win.
The industry has decided that 8-bit non-ECC non-parity RAM is all
that is needed and will not even listen to the arguments of the
proponents of a small offbeat filesystem.
Sorry, </rant>
You have the code. If you can do better, by all means go ahead. I and
others would be more than happy to step aside if you can do better
than we did.
Post by ray vantassle
I'm not sure if snapshots would have saved your data, but this
(unexpectedly losing data) is the very reason I started doing
automatic hourly, daily, and monthly snapshots. Now if there were
only a easy and transparent way to access them....
It is not the file system's fault if your user land program decides to
do the equivalent of `rm -rf`.
To unsubscribe from this group and stop receiving emails from it, send
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
'pete' via zfs-discuss
2014-07-05 18:47:00 UTC
Permalink
Tom, what Version of rsync did you use, latest is from 22, june. http://rsync.samba.org/ftp/unpacked/rsync/
Rsync has lots of problems.

Is there any chance to repeat the copy? If it is caused by some odd tool, it may be repeatable and we can find the root cause together.

Pete

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Michael Kjörling
2014-07-05 19:09:47 UTC
Permalink
Post by ray vantassle
That aside ....
I echo your concerns about ZOL. I first looked at it several years ago and
bypassed it because at that time it was far from ready for prime-time. I
went with zfs-fuse on both my systems, the main repository and the backup
system. Several months ago I switched by backup system to ZOL -- and
discovered that it is functionally un-usable on a 32-bit system so I had to
upgrade my OS to 64-bit Debian. I still use zfs-fuse on my main
repository and don't expect to ever change it over to ZOL.
I recently upgraded my ZOL from 0.6.0-rc13 to 0.6.2.
I don't know what your definition of "recently" is, but if you are
running a year-and-a-half old version (0.6.0-rc13 being dated Dec 20
2012) of such rapidly maturing code, IMO you really shouldn't
complain. From what I can tell, 0.6.1 was the first version termed
"ready for wide scale deployment"[1] and was released late March 2013.

I dare say that the vast majority of people who had issues with the
0.6.2 to 0.6.3 kernel interface changes only needed to reboot their
systems for things to start working as intended again. A very few that
I've seen on this list have had problems after a reboot, problems
which could generally be tracked down to the same root cause:
userland/kernel version mismatches. As far as I am aware, nobody's
data has been at risk from those changes, even though I'm sure a few
people have had to clean some beverages off their computer screens and
keyboards.

The requirement of ZFS On Linux for a 64-bit system has been well
documented in the FAQ for a very long time; since September 2010 at
least, putting it well before the ZoL 0.6.0 RCs. [2] [3]

[1] https://groups.google.com/a/zfsonlinux.org/forum/?_escaped_fragment_=topic/zfs-announce/ZXADhyOwFfA#!topic/zfs-announce/ZXADhyOwFfA

[2] http://web.archive.org/web/20100926104451/http://zfsonlinux.org/faq.html#WhyShouldIUseA64BitSystem

[3] http://zfsonlinux.org/faq.html#WhyShouldIUseA64BitSystem
--
Michael Kjörling • http://michael.kjorling.se • ***@kjorling.se
OpenPGP B501AC6429EF4514 http://michael.kjorling.se/public-keys/pgp
“People who think they know everything really annoy
those of us who know we don’t.” (Bjarne Stroustrup)

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Gordan Bobic
2014-07-06 10:00:16 UTC
Permalink
Post by ray vantassle
Some of the ZOL folks are, IMHO, living in an echo chamber. The ECC
business being a case in point. The attitude of "if you don't believe in
ECC you might as well just copy your data to /dev/null" doesn't help, it
just exhibits arrogant snottiness. Regardless of the theoretical
advantages of ECC, the entire computer industry has de-facto concluded
that the theoretical benefits are are not real and that ECC is
realistically not needed in 99+% of the real world, only in highly
specific situations at best. Heck, the industry even dropped parity RAM
decades ago.
It was a cost cutting measure and it was never dropped on server and
workstation grade hardware. It was dropped on gaming and home desktop
grade hardware where people run Windows, which meant that Windows' long
history of instability meant that an extra crash or error every few
weeks was never going to get noticed.

IMO, despite my deeply rooted dislike of Windows (every time I have to
use it for anything more advanced than being a Steam boot loader I feel
like I'm trying to read braille with hooks), I think the lack of ECC did
Window a great disservice because the improvements in the software made
little difference when run on unstable hardware.
Post by ray vantassle
I wish they'd stop wasting time on a battle that they can never win.
The industry has decided that 8-bit non-ECC non-parity RAM is all that
is needed and will not even listen to the arguments of the proponents of
a small offbeat filesystem.
Sorry, </rant>
How many servers and workstations come without ECC?
Post by ray vantassle
I'm not sure if snapshots would have saved your data, but this
(unexpectedly losing data) is the very reason I started doing automatic
hourly, daily, and monthly snapshots. Now if there were only a easy
and transparent way to access them....
.zfs doesn't work for you?


To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Linda Kateley
2014-07-06 19:57:54 UTC
Permalink
I think the point that people are missing is that zfs just lets you know
there is a problem/error. Those same errors occur everywhere else, you
just don't see them and are not aware. It's called silent data
corruption.. With zfs it will let you know you have an error. It is
noisy data corruption.

If you are using zfs for any data that matters to you, and IMHO, if you
are using zfs you probably using it because your data matters.. eg
checksums, snapshots zfs send/recv. Use it with ecc ram. The price you
are paying for a free/open enterprise grade filesystem, should make up
for the cost of ram. If you don't care about your data eg. home server,
gaming, scratch, than it shouldn't matter. Just make sure you have a
good backup.

lk
Post by Gordan Bobic
Post by ray vantassle
Some of the ZOL folks are, IMHO, living in an echo chamber. The ECC
business being a case in point. The attitude of "if you don't believe in
ECC you might as well just copy your data to /dev/null" doesn't help, it
just exhibits arrogant snottiness. Regardless of the theoretical
advantages of ECC, the entire computer industry has de-facto concluded
that the theoretical benefits are are not real and that ECC is
realistically not needed in 99+% of the real world, only in highly
specific situations at best. Heck, the industry even dropped parity RAM
decades ago.
It was a cost cutting measure and it was never dropped on server and
workstation grade hardware. It was dropped on gaming and home desktop
grade hardware where people run Windows, which meant that Windows'
long history of instability meant that an extra crash or error every
few weeks was never going to get noticed.
IMO, despite my deeply rooted dislike of Windows (every time I have to
use it for anything more advanced than being a Steam boot loader I
feel like I'm trying to read braille with hooks), I think the lack of
ECC did Window a great disservice because the improvements in the
software made little difference when run on unstable hardware.
Post by ray vantassle
I wish they'd stop wasting time on a battle that they can never win.
The industry has decided that 8-bit non-ECC non-parity RAM is all that
is needed and will not even listen to the arguments of the proponents of
a small offbeat filesystem.
Sorry, </rant>
How many servers and workstations come without ECC?
Post by ray vantassle
I'm not sure if snapshots would have saved your data, but this
(unexpectedly losing data) is the very reason I started doing automatic
hourly, daily, and monthly snapshots. Now if there were only a easy
and transparent way to access them....
.zfs doesn't work for you?
To unsubscribe from this group and stop receiving emails from it, send
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
devsk
2014-07-06 20:06:58 UTC
Permalink
Post by Linda Kateley
I think the point that people are missing is that zfs just lets you know
there is a problem/error. Those same errors occur everywhere else, you
just don't see them and are not aware. It's called silent data
corruption.. With zfs it will let you know you have an error. It is
noisy data corruption.
If you are using zfs for any data that matters to you, and IMHO, if you
are using zfs you probably using it because your data matters.. eg
checksums, snapshots zfs send/recv. Use it with ecc ram. The price you
are paying for a free/open enterprise grade filesystem, should make up
for the cost of ram. If you don't care about your data eg. home server,
gaming, scratch, than it shouldn't matter. Just make sure you have a
good backup.
There is a hole in that logic. The backup on what FS? It will suffer the
same fate as your regular FS.

For any data that matters (home server, personal laptop where you edit and
store docs or keep photos until they are archived on your homeserver etc.),
I think ZFS is the only way to go at this time.

-devsk


To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Linda Kateley
2014-07-06 20:09:45 UTC
Permalink
Yes you're right. I had my sunday logic on .. If it doesn't matter, why
have a backup?

lk
Post by Linda Kateley
I think the point that people are missing is that zfs just lets you know
there is a problem/error. Those same errors occur everywhere else, you
just don't see them and are not aware. It's called silent data
corruption.. With zfs it will let you know you have an error. It is
noisy data corruption.
If you are using zfs for any data that matters to you, and IMHO, if you
are using zfs you probably using it because your data matters.. eg
checksums, snapshots zfs send/recv. Use it with ecc ram. The price you
are paying for a free/open enterprise grade filesystem, should make up
for the cost of ram. If you don't care about your data eg. home server,
gaming, scratch, than it shouldn't matter. Just make sure you have a
good backup.
There is a hole in that logic. The backup on what FS? It will suffer
the same fate as your regular FS.
For any data that matters (home server, personal laptop where you edit
and store docs or keep photos until they are archived on your
homeserver etc.), I think ZFS is the only way to go at this time.
-devsk
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Will Rouesnel
2014-07-07 08:22:35 UTC
Permalink
<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Redundancy. You can add checksum and parity protection to any
archive structure you want.<br>
<br>
What you don't want to do is make byte-for-byte copies of the actual
filesystem - which is why I'm opposed to using zfs-send-streams for
long term storage, because they're rather literal copies of the
filesystem structure.<br>
<br>
<div class="moz-cite-prefix">On 07/07/14 06:09, Linda Kateley wrote:<br>
</div>
<blockquote cite="mid:53B9AD09.6040107-***@public.gmane.org" type="cite">
<meta http-equiv="Context-Type" content="text/html; charset=UTF-8">
<div class="moz-cite-prefix">Yes you're right. I had my sunday
logic on .. If it doesn't matter, why have a backup? <br>
<br>
lk<br>
<br>
On 7/6/14, 3:06 PM, devsk wrote:<br>
</div>
<blockquote
cite="mid:ef433ff0-c07a-41b2-b23f-26fdd8432b2a-VKpPRiiRko4/***@public.gmane.org"
type="cite">
<div dir="ltr">On Sunday, July 6, 2014 12:57:59 PM UTC-7, Linda
Kateley wrote:
<blockquote class="gmail_quote">I think the point that people
are missing is that zfs just lets you know <br>
there is a problem/error. Those same errors occur everywhere
else, you <br>
just don't see them and are not aware. It's called silent
data <br>
corruption.. With zfs it will let you know you have an
error. It is <br>
noisy data corruption. <br>
<br>
If you are using zfs for any data that matters to you, and
IMHO, if you <br>
are using zfs you probably using it because your data
matters.. eg <br>
checksums, snapshots zfs send/recv. Use it with ecc ram. The
price you <br>
are paying for a free/open enterprise grade filesystem,
should make up <br>
for the cost of ram. If you don't care about your data eg.
home server, <br>
gaming, scratch, than it shouldn't matter. Just make sure
you have a <br>
good backup. <br>
</blockquote>
<div> </div>
<div>There is a hole in that logic. The backup on what FS? It
will suffer the same fate as your regular FS.<br>
<br>
For any data that matters (home server, personal laptop
where you edit and store docs or keep photos until they are
archived on your homeserver etc.), I think ZFS is the only
way to go at this time.<br>
<br>
-devsk<br>
 </div>
</div>
</blockquote>
<br>
To unsubscribe from this group and stop receiving emails from it,
send an email to <a moz-do-not-send="true"
href="mailto:zfs-discuss+unsubscribe-VKpPRiiRko4/***@public.gmane.org">zfs-discuss+unsubscribe-VKpPRiiRko4/***@public.gmane.org</a>.<br>
</blockquote>
<br>
</body>
</html>

<p></p>

To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:zfs-discuss+unsubscribe-VKpPRiiRko4/***@public.gmane.org">zfs-discuss+unsubscribe-VKpPRiiRko4/***@public.gmane.org</a>.<br />
Tom D
2014-07-05 17:16:17 UTC
Permalink
OK, guys. Thanks for the help. If I come up with anything positive, I
will post back.

I've only tested snapshots but I'm not interested in them. If I determine
that more backup redundancy is required, I will acquire more systems and
run entirely independent file store, preferably with a different file
system.

Running only 8GB, I had tuned the cache as far as I dare in zfsrc, on
zfs-fuse. To be honest, I could see RAM use increase but it did not
consume RAM to the limit until quite a bit of transfer took place, as
in... a couple of days.

Now I'm posting for future people who may search on a similar problem.

This is not an rsync issue. It simply isn't plausible for a bit flip on
one machine to cause rsync to run as with the "--delete" option and a bit
flip on another machine to cause NFS to flip from "ro" to "rw".

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Blake Dunlap
2014-07-05 17:25:53 UTC
Permalink
I'm missing something, how do you think the source NFS being ro/rw has
any affect on the receiving side?

Let's be clear, does the rsync command you run have "--delete"
enabled? You've been ambiguous on this point.

Withholding further comment until I see the answer to the above
question out of the benefit of the doubt.

-Blake
OK, guys. Thanks for the help. If I come up with anything positive, I will
post back.
I've only tested snapshots but I'm not interested in them. If I determine
that more backup redundancy is required, I will acquire more systems and run
entirely independent file store, preferably with a different file system.
Running only 8GB, I had tuned the cache as far as I dare in zfsrc, on
zfs-fuse. To be honest, I could see RAM use increase but it did not consume
RAM to the limit until quite a bit of transfer took place, as in... a
couple of days.
Now I'm posting for future people who may search on a similar problem.
This is not an rsync issue. It simply isn't plausible for a bit flip on one
machine to cause rsync to run as with the "--delete" option and a bit flip
on another machine to cause NFS to flip from "ro" to "rw".
To unsubscribe from this group and stop receiving emails from it, send an
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Richard Yao
2014-07-05 17:31:40 UTC
Permalink
I agree. I do not see how the mountpoint being ro/rw matters here.
Post by Blake Dunlap
I'm missing something, how do you think the source NFS being ro/rw has
any affect on the receiving side?
Let's be clear, does the rsync command you run have "--delete"
enabled? You've been ambiguous on this point.
Withholding further comment until I see the answer to the above
question out of the benefit of the doubt.
-Blake
OK, guys. Thanks for the help. If I come up with anything positive, I will
post back.
I've only tested snapshots but I'm not interested in them. If I determine
that more backup redundancy is required, I will acquire more systems and run
entirely independent file store, preferably with a different file system.
Running only 8GB, I had tuned the cache as far as I dare in zfsrc, on
zfs-fuse. To be honest, I could see RAM use increase but it did not consume
RAM to the limit until quite a bit of transfer took place, as in... a
couple of days.
Now I'm posting for future people who may search on a similar problem.
This is not an rsync issue. It simply isn't plausible for a bit flip on one
machine to cause rsync to run as with the "--delete" option and a bit flip
on another machine to cause NFS to flip from "ro" to "rw".
To unsubscribe from this group and stop receiving emails from it, send an
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Tom D
2014-07-05 17:53:00 UTC
Permalink
OK. I suppose I can understand the ambiguity. Here's more detail.

2 x identical systems:

- Asus E45M1-M PRO
- 8 GB Mushkin (non-ECC) RAM, underclocked, standard voltage
- 1TB boot drive using EXT4
- Ubuntu 14.4 server
- 10 x Seagate 4TB NAS drives
- drives on system in question currently configured in 2 x 5 drive RAIDZ
arrays
- case is Norco 4020
- power supply is Pico linear power supply

ii dkms 2.2.0.3-1.1ubuntu5+zfs9~trusty
all Dynamic Kernel Module Support Framework
ii libzfs2 0.6.3-2~trusty
amd64 Native ZFS filesystem library for Linux
ii mountall 2.53-zfs1
amd64 filesystem mounting tool
ii ubuntu-zfs 8~trusty
amd64 Native ZFS filesystem metapackage for Ubuntu.
ii zfs-dkms 0.6.3-2~trusty
amd64 Native ZFS filesystem kernel modules for Linux
ii zfsutils 0.6.3-2~trusty
amd64 Native ZFS management utilities for Linux


System names: primary, secondary

Step 1:

primary = zfs-fuse
secondary = zfs-fuse
Note - ran this way for years

Step 2:

primary = zfs-fuse
secondary = ZoL
Note 1 - Installed Ubuntu server 14.04 and ZoL, created a new secondary
ZPOOL with "-o ashift=12" for both VDEVs, and copied the entire contents of
the primary system to create a redundant system
Note 3 - Performance tests show: zfs-fuse = 20 MBps write, 80MBps read ;
ZoL = 240MBps write, 1.1GBps read
Note 3 - Ran since release of Ubuntu server 14.04. Total data stored was
roughly 16TB. Was stable until yesterday morning.
Note 4 - Ran "zpool scrub secondary" prior to upgrading primary. I have
never seen a disk error since converting to Seagate NAS drives to the point
that I am confident they are being masked in the drives.

Step3 (started 6 days ago):

primary = ZoL
secondary = ZoL
Note 1: Secondary NFS shares are "ro" in exports file.
Note 2: rsync run without "--delete" on primary, pulling data through NFS
from secondary (command = "***@primary:-# rsync -av /secondary/sysimages/
/primary/sysimages/")
Note 3: I was monitoring closely with everything going great until Friday
morning when I noticed rsync errors. It was the third day of copying,
about 4TB in, and rsync was indicating files had moved or been deleted. On
the secondary system, "zfs list" now shows 4.5TB where it used to show
14.5TB. Secondary system also has many hundreds of empty directories.
Directory structure appears to be intact but most files are now gone.
Note 4: Files seem to be missing in alphabetical order from about "C*" out
to "P*". As I list the directories, the ones with missing files are
sequential. I now have empty sub-directories in about a dozen
directories. The top level directories that are have lost files are not in
order, but sub-directories under the top level have lost files in
alphabetical order.



To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Durval Menezes
2014-07-05 19:15:14 UTC
Permalink
Hi Tom,
Post by Tom D
OK. I suppose I can understand the ambiguity. Here's more detail.
- Asus E45M1-M PRO
- 8 GB Mushkin (non-ECC) RAM, underclocked, standard voltage
- 1TB boot drive using EXT4
- Ubuntu 14.4 server
- 10 x Seagate 4TB NAS drives
- drives on system in question currently configured in 2 x 5 drive RAIDZ
arrays
- case is Norco 4020
- power supply is Pico linear power supply
ii dkms 2.2.0.3-1.1ubuntu5+zfs9~trusty
all Dynamic Kernel Module Support Framework
ii libzfs2 0.6.3-2~trusty
amd64 Native ZFS filesystem library for Linux
ii mountall 2.53-zfs1
amd64 filesystem mounting tool
ii ubuntu-zfs 8~trusty
amd64 Native ZFS filesystem metapackage for Ubuntu.
ii zfs-dkms 0.6.3-2~trusty
amd64 Native ZFS filesystem kernel modules for Linux
ii zfsutils 0.6.3-2~trusty
amd64 Native ZFS management utilities for Linux
System names: primary, secondary
primary = zfs-fuse
secondary = zfs-fuse
Note - ran this way for years
primary = zfs-fuse
secondary = ZoL
Note 1 - Installed Ubuntu server 14.04 and ZoL, created a new secondary
ZPOOL with "-o ashift=12" for both VDEVs, and copied the entire contents of
the primary system to create a redundant system
Note 3 - Performance tests show: zfs-fuse = 20 MBps write, 80MBps read ;
ZoL = 240MBps write, 1.1GBps read
Note 3 - Ran since release of Ubuntu server 14.04. Total data stored was
roughly 16TB. Was stable until yesterday morning.
Note 4 - Ran "zpool scrub secondary" prior to upgrading primary. I have
never seen a disk error since converting to Seagate NAS drives to the point
that I am confident they are being masked in the drives.
primary = ZoL
secondary = ZoL
Note 1: Secondary NFS shares are "ro" in exports file.
Note 2: rsync run without "--delete" on primary, pulling data through NFS
/primary/sysimages/")
I agree that this (plus the NFS being mounted ro) makes it really really
hard (as in, "almost impossible") for rsync being responsible for the
deletion, regardless of uncaught RAM errors. First of all, as you stated,
there would have to be a bit flip inside NFS code or data in the secondary
machine in order for it to change the system from ro to rw (or allow writes
to ro). Second, there would have to be *another* bit flip in the primary
machine to make rsync turn on the "--remove-source-files" option (NOT any
of the "--delete" options as these only affect rsync's *destination*).
Also, this would *not* explain what you saw because "--remove-source-files"
only removes *files*, never any directories or subdirectories... and if I
understand you correctly, you have lost subdirectories too.

One very important question: have you run "zpool scrub" on the secondary
machine? The only chance of this being due to RAM errors would (IMHO) be if
scrub detects some kind of corruption... if not, we have operator error (no
offense meant, just listing the possibilities) or else some other program
that decided to delete files on the secondary.
Post by Tom D
Note 3: I was monitoring closely with everything going great until Friday
morning when I noticed rsync errors. It was the third day of copying,
about 4TB in, and rsync was indicating files had moved or been deleted. On
the secondary system, "zfs list" now shows 4.5TB where it used to show
14.5TB. Secondary system also has many hundreds of empty directories.
Directory structure appears to be intact but most files are now gone.
Note 4: Files seem to be missing in alphabetical order from about "C*" out
to "P*". As I list the directories, the ones with missing files are
sequential. I now have empty sub-directories in about a dozen
directories. The top level directories that are have lost files are not in
order, but sub-directories under the top level have lost files in
alphabetical order.
Humrmrmr.... only files? No (sub)directory lost anywhere? See above...

Cheers,
--
Durval.
Post by Tom D
To unsubscribe from this group and stop receiving emails from it, send an
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Richard Yao
2014-07-06 16:25:04 UTC
Permalink
I thought you said that the destination had files unlinked. In this case, source that was a readonly NFS mount had files unlinked. In the former case, that would clearly be a bit flip. In the latter case, there is no clear cause. So far, all I know is that it looks like something meticulously unlinked files from a system that had neither snapshots nor ECC. There is not enough information to be able to say what happened, but here are some possibilities that occur to me:

- Something went wrong with the NFS daemon that caused it to start unlinking files upon each access. If this happened, it likely tried to unlink directories too, which would have failed because it would need to handle those differently. Check your system logs to for diagnostic messages from the NFS daemon. If it did go haywire, maybe it might have logged a potentially insightful diagnostic messages.

- You were compromised and the intruder decided to unlink files. Check logs for signs of a possible intrusion.

- Maybe a bit flip caused some kind of extreme case of metadata corruption in ZFS. It will be very hard to check for this, but you could at least try using zdb to verify the free space maps. If the metadata is corrupt, then the free space maps should be messed up.

These are stabs in the dark, but that seems like all that we can do. Sadly, there is no way to definitively rule out any possibility in the event that they all come back negative. I wish that you would reconsider your policy of not using snapshots and forgoing ECC memory for the future. If you had either of them in place, this would be *much* easier to troubleshoot. With neither snapshots nor ECC, the possibilities for ways that things could have gone wrong are enormous. Recovery is likely impossible since you have no snapshots to use to ensure that the metadata/data blocks for these files have not been overridden.
Post by Tom D
OK. I suppose I can understand the ambiguity. Here's more detail.
- Asus E45M1-M PRO
- 8 GB Mushkin (non-ECC) RAM, underclocked, standard voltage
- 1TB boot drive using EXT4
- Ubuntu 14.4 server
- 10 x Seagate 4TB NAS drives
- drives on system in question currently configured in 2 x 5 drive RAIDZ arrays
- case is Norco 4020
- power supply is Pico linear power supply
ii dkms 2.2.0.3-1.1ubuntu5+zfs9~trusty all Dynamic Kernel Module Support Framework
ii libzfs2 0.6.3-2~trusty amd64 Native ZFS filesystem library for Linux
ii mountall 2.53-zfs1 amd64 filesystem mounting tool
ii ubuntu-zfs 8~trusty amd64 Native ZFS filesystem metapackage for Ubuntu.
ii zfs-dkms 0.6.3-2~trusty amd64 Native ZFS filesystem kernel modules for Linux
ii zfsutils 0.6.3-2~trusty amd64 Native ZFS management utilities for Linux
System names: primary, secondary
primary = zfs-fuse
secondary = zfs-fuse
Note - ran this way for years
primary = zfs-fuse
secondary = ZoL
Note 1 - Installed Ubuntu server 14.04 and ZoL, created a new secondary ZPOOL with "-o ashift=12" for both VDEVs, and copied the entire contents of the primary system to create a redundant system
Note 3 - Performance tests show: zfs-fuse = 20 MBps write, 80MBps read ; ZoL = 240MBps write, 1.1GBps read
Note 3 - Ran since release of Ubuntu server 14.04. Total data stored was roughly 16TB. Was stable until yesterday morning.
Note 4 - Ran "zpool scrub secondary" prior to upgrading primary. I have never seen a disk error since converting to Seagate NAS drives to the point that I am confident they are being masked in the drives.
primary = ZoL
secondary = ZoL
Note 1: Secondary NFS shares are "ro" in exports file.
Note 3: I was monitoring closely with everything going great until Friday morning when I noticed rsync errors. It was the third day of copying, about 4TB in, and rsync was indicating files had moved or been deleted. On the secondary system, "zfs list" now shows 4.5TB where it used to show 14.5TB. Secondary system also has many hundreds of empty directories. Directory structure appears to be intact but most files are now gone.
Note 4: Files seem to be missing in alphabetical order from about "C*" out to "P*". As I list the directories, the ones with missing files are sequential. I now have empty sub-directories in about a dozen directories. The top level directories that are have lost files are not in order, but sub-directories under the top level have lost files in alphabetical order.
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Blake Dunlap
2014-07-06 18:36:50 UTC
Permalink
Yeah, so I just don't see how this could be ZFS based on the symptoms.
If you had had metadata issues, it would not have been in ordered
sections, nor would it have been files only in structure.

Rsync however would very easily create this situation, and although
what you are sating happened couldn't have caused this, you have
intentionally left out data in previous reports. I absolutely don't
assume ZFS is infallible, but I just don't see how ZFS issues could
have caused your specific problem, while I can easily see how user
error or forgetfulness with rsync could have coupled with the
potential issues related to your set up (using NFS source means there
is not positive assurance in the case of the source device being
unavailable, the origin dir could still be there and empty).

This is really sounding to me like a case for Occam's razor barring
other information or someone else experiencing a similar issue.

-Blake
Post by Richard Yao
- Something went wrong with the NFS daemon that caused it to start unlinking files upon each access. If this happened, it likely tried to unlink directories too, which would have failed because it would need to handle those differently. Check your system logs to for diagnostic messages from the NFS daemon. If it did go haywire, maybe it might have logged a potentially insightful diagnostic messages.
- You were compromised and the intruder decided to unlink files. Check logs for signs of a possible intrusion.
- Maybe a bit flip caused some kind of extreme case of metadata corruption in ZFS. It will be very hard to check for this, but you could at least try using zdb to verify the free space maps. If the metadata is corrupt, then the free space maps should be messed up.
These are stabs in the dark, but that seems like all that we can do. Sadly, there is no way to definitively rule out any possibility in the event that they all come back negative. I wish that you would reconsider your policy of not using snapshots and forgoing ECC memory for the future. If you had either of them in place, this would be *much* easier to troubleshoot. With neither snapshots nor ECC, the possibilities for ways that things could have gone wrong are enormous. Recovery is likely impossible since you have no snapshots to use to ensure that the metadata/data blocks for these files have not been overridden.
Post by Tom D
OK. I suppose I can understand the ambiguity. Here's more detail.
- Asus E45M1-M PRO
- 8 GB Mushkin (non-ECC) RAM, underclocked, standard voltage
- 1TB boot drive using EXT4
- Ubuntu 14.4 server
- 10 x Seagate 4TB NAS drives
- drives on system in question currently configured in 2 x 5 drive RAIDZ arrays
- case is Norco 4020
- power supply is Pico linear power supply
ii dkms 2.2.0.3-1.1ubuntu5+zfs9~trusty all Dynamic Kernel Module Support Framework
ii libzfs2 0.6.3-2~trusty amd64 Native ZFS filesystem library for Linux
ii mountall 2.53-zfs1 amd64 filesystem mounting tool
ii ubuntu-zfs 8~trusty amd64 Native ZFS filesystem metapackage for Ubuntu.
ii zfs-dkms 0.6.3-2~trusty amd64 Native ZFS filesystem kernel modules for Linux
ii zfsutils 0.6.3-2~trusty amd64 Native ZFS management utilities for Linux
System names: primary, secondary
primary = zfs-fuse
secondary = zfs-fuse
Note - ran this way for years
primary = zfs-fuse
secondary = ZoL
Note 1 - Installed Ubuntu server 14.04 and ZoL, created a new secondary ZPOOL with "-o ashift=12" for both VDEVs, and copied the entire contents of the primary system to create a redundant system
Note 3 - Performance tests show: zfs-fuse = 20 MBps write, 80MBps read ; ZoL = 240MBps write, 1.1GBps read
Note 3 - Ran since release of Ubuntu server 14.04. Total data stored was roughly 16TB. Was stable until yesterday morning.
Note 4 - Ran "zpool scrub secondary" prior to upgrading primary. I have never seen a disk error since converting to Seagate NAS drives to the point that I am confident they are being masked in the drives.
primary = ZoL
secondary = ZoL
Note 1: Secondary NFS shares are "ro" in exports file.
Note 3: I was monitoring closely with everything going great until Friday morning when I noticed rsync errors. It was the third day of copying, about 4TB in, and rsync was indicating files had moved or been deleted. On the secondary system, "zfs list" now shows 4.5TB where it used to show 14.5TB. Secondary system also has many hundreds of empty directories. Directory structure appears to be intact but most files are now gone.
Note 4: Files seem to be missing in alphabetical order from about "C*" out to "P*". As I list the directories, the ones with missing files are sequential. I now have empty sub-directories in about a dozen directories. The top level directories that are have lost files are not in order, but sub-directories under the top level have lost files in alphabetical order.
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
devsk
2014-07-06 20:02:17 UTC
Permalink
I think we all have ignored one aspect of this situation: the sudden power
loss during rsync copy. If both systems (source and destination of rsync)
went down because of power loss, there can be interesting play when the
power comes back up and the same command is fired. Think about what happens
when the source side is not (NFS) mounted yet and is empty...:-) rsync can
be a real b**ch sometimes.

If ZFS were to die on you Tom, I have every bit of faith in the fact that
you won't be able to either import or scrub will throw errors. I have used
zfs-fuse as well ZoL from the very very early days, and I have never ever
seen anything like this. ZFS has its quirks but losing data is definitely
not one of them.

And what's with this 'no snapshots for me' kind of attitude? They have
saved my skin more number of times than I would like to admit.

-devsk
Post by Richard Yao
I thought you said that the destination had files unlinked. In this case,
source that was a readonly NFS mount had files unlinked. In the former
case, that would clearly be a bit flip. In the latter case, there is no
clear cause. So far, all I know is that it looks like something
meticulously unlinked files from a system that had neither snapshots nor
ECC. There is not enough information to be able to say what happened, but
- Something went wrong with the NFS daemon that caused it to start
unlinking files upon each access. If this happened, it likely tried to
unlink directories too, which would have failed because it would need to
handle those differently. Check your system logs to for diagnostic messages
from the NFS daemon. If it did go haywire, maybe it might have logged a
potentially insightful diagnostic messages.
- You were compromised and the intruder decided to unlink files. Check
logs for signs of a possible intrusion.
- Maybe a bit flip caused some kind of extreme case of metadata corruption
in ZFS. It will be very hard to check for this, but you could at least try
using zdb to verify the free space maps. If the metadata is corrupt, then
the free space maps should be messed up.
These are stabs in the dark, but that seems like all that we can do.
Sadly, there is no way to definitively rule out any possibility in the
event that they all come back negative. I wish that you would reconsider
your policy of not using snapshots and forgoing ECC memory for the future.
If you had either of them in place, this would be *much* easier to
troubleshoot. With neither snapshots nor ECC, the possibilities for ways
that things could have gone wrong are enormous. Recovery is likely
impossible since you have no snapshots to use to ensure that the
metadata/data blocks for these files have not been overridden.
Post by Tom D
OK. I suppose I can understand the ambiguity. Here's more detail.
- Asus E45M1-M PRO
- 8 GB Mushkin (non-ECC) RAM, underclocked, standard voltage
- 1TB boot drive using EXT4
- Ubuntu 14.4 server
- 10 x Seagate 4TB NAS drives
- drives on system in question currently configured in 2 x 5 drive RAIDZ
arrays
Post by Tom D
- case is Norco 4020
- power supply is Pico linear power supply
ii dkms 2.2.0.3-1.1ubuntu5+zfs9~trusty
all Dynamic Kernel Module Support Framework
Post by Tom D
ii libzfs2 0.6.3-2~trusty
amd64 Native ZFS filesystem library for Linux
Post by Tom D
ii mountall 2.53-zfs1
amd64 filesystem mounting tool
Post by Tom D
ii ubuntu-zfs 8~trusty
amd64 Native ZFS filesystem metapackage for Ubuntu.
Post by Tom D
ii zfs-dkms 0.6.3-2~trusty
amd64 Native ZFS filesystem kernel modules for Linux
Post by Tom D
ii zfsutils 0.6.3-2~trusty
amd64 Native ZFS management utilities for Linux
Post by Tom D
System names: primary, secondary
primary = zfs-fuse
secondary = zfs-fuse
Note - ran this way for years
primary = zfs-fuse
secondary = ZoL
Note 1 - Installed Ubuntu server 14.04 and ZoL, created a new secondary
ZPOOL with "-o ashift=12" for both VDEVs, and copied the entire contents of
the primary system to create a redundant system
Post by Tom D
Note 3 - Performance tests show: zfs-fuse = 20 MBps write, 80MBps read ;
ZoL = 240MBps write, 1.1GBps read
Post by Tom D
Note 3 - Ran since release of Ubuntu server 14.04. Total data stored
was roughly 16TB. Was stable until yesterday morning.
Post by Tom D
Note 4 - Ran "zpool scrub secondary" prior to upgrading primary. I have
never seen a disk error since converting to Seagate NAS drives to the point
that I am confident they are being masked in the drives.
Post by Tom D
primary = ZoL
secondary = ZoL
Note 1: Secondary NFS shares are "ro" in exports file.
Note 2: rsync run without "--delete" on primary, pulling data through
/secondary/sysimages/ /primary/sysimages/")
Post by Tom D
Note 3: I was monitoring closely with everything going great until
Friday morning when I noticed rsync errors. It was the third day of
copying, about 4TB in, and rsync was indicating files had moved or been
deleted. On the secondary system, "zfs list" now shows 4.5TB where it used
to show 14.5TB. Secondary system also has many hundreds of empty
directories. Directory structure appears to be intact but most files are
now gone.
Post by Tom D
Note 4: Files seem to be missing in alphabetical order from about "C*"
out to "P*". As I list the directories, the ones with missing files are
sequential. I now have empty sub-directories in about a dozen directories.
The top level directories that are have lost files are not in order, but
sub-directories under the top level have lost files in alphabetical order.
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
r***@public.gmane.org
2014-07-06 22:41:01 UTC
Permalink
I agree with Richard here. Minor addendums below...
Post by Richard Yao
I thought you said that the destination had files unlinked. In this case,
source that was a readonly NFS mount had files unlinked. In the former
case, that would clearly be a bit flip. In the latter case, there is no
clear cause. So far, all I know is that it looks like something
meticulously unlinked files from a system that had neither snapshots nor
ECC. There is not enough information to be able to say what happened, but
- Something went wrong with the NFS daemon that caused it to start
unlinking files upon each access. If this happened, it likely tried to
unlink directories too, which would have failed because it would need to
handle those differently. Check your system logs to for diagnostic messages
from the NFS daemon. If it did go haywire, maybe it might have logged a
potentially insightful diagnostic messages.
- You were compromised and the intruder decided to unlink files. Check
logs for signs of a possible intrusion.
- Maybe a bit flip caused some kind of extreme case of metadata corruption
in ZFS. It will be very hard to check for this, but you could at least try
using zdb to verify the free space maps. If the metadata is corrupt, then
the free space maps should be messed up.
A zpool scrub is advised to verify all data and metadata.
Post by Richard Yao
These are stabs in the dark, but that seems like all that we can do.
Sadly, there is no way to definitively rule out any possibility in the
event that they all come back negative. I wish that you would reconsider
your policy of not using snapshots and forgoing ECC memory for the future.
If you had either of them in place, this would be *much* easier to
troubleshoot. With neither snapshots nor ECC, the possibilities for ways
that things could have gone wrong are enormous. Recovery is likely
impossible since you have no snapshots to use to ensure that the
metadata/data blocks for these files have not been overridden.
If there is a "single bit flip" event then only a very specific series of
events would lead
to undetectable corruption. RAM bit flips fall into this category, as well
as bit flips in
the CPU's datapath.

If the scrub advised above is clear, then the exercise becomes one of
forensics.
Unless you had some sort of auditing enabled you might not have the data
needed
to find out exactly who did what to the files.
Post by Richard Yao
Post by Tom D
OK. I suppose I can understand the ambiguity. Here's more detail.
- Asus E45M1-M PRO
- 8 GB Mushkin (non-ECC) RAM, underclocked, standard voltage
- 1TB boot drive using EXT4
- Ubuntu 14.4 server
- 10 x Seagate 4TB NAS drives
- drives on system in question currently configured in 2 x 5 drive RAIDZ
arrays
Post by Tom D
- case is Norco 4020
- power supply is Pico linear power supply
ii dkms 2.2.0.3-1.1ubuntu5+zfs9~trusty
all Dynamic Kernel Module Support Framework
Post by Tom D
ii libzfs2 0.6.3-2~trusty
amd64 Native ZFS filesystem library for Linux
Post by Tom D
ii mountall 2.53-zfs1
amd64 filesystem mounting tool
Post by Tom D
ii ubuntu-zfs 8~trusty
amd64 Native ZFS filesystem metapackage for Ubuntu.
Post by Tom D
ii zfs-dkms 0.6.3-2~trusty
amd64 Native ZFS filesystem kernel modules for Linux
Post by Tom D
ii zfsutils 0.6.3-2~trusty
amd64 Native ZFS management utilities for Linux
Post by Tom D
System names: primary, secondary
primary = zfs-fuse
secondary = zfs-fuse
Note - ran this way for years
primary = zfs-fuse
secondary = ZoL
Note 1 - Installed Ubuntu server 14.04 and ZoL, created a new secondary
ZPOOL with "-o ashift=12" for both VDEVs, and copied the entire contents of
the primary system to create a redundant system
Post by Tom D
Note 3 - Performance tests show: zfs-fuse = 20 MBps write, 80MBps read ;
ZoL = 240MBps write, 1.1GBps read
Post by Tom D
Note 3 - Ran since release of Ubuntu server 14.04. Total data stored
was roughly 16TB. Was stable until yesterday morning.
Post by Tom D
Note 4 - Ran "zpool scrub secondary" prior to upgrading primary. I have
never seen a disk error since converting to Seagate NAS drives to the point
that I am confident they are being masked in the drives.
Post by Tom D
primary = ZoL
secondary = ZoL
Note 1: Secondary NFS shares are "ro" in exports file.
Note 2: rsync run without "--delete" on primary, pulling data through
/secondary/sysimages/ /primary/sysimages/")
Post by Tom D
Note 3: I was monitoring closely with everything going great until
Friday morning when I noticed rsync errors. It was the third day of
copying, about 4TB in, and rsync was indicating files had moved or been
deleted. On the secondary system, "zfs list" now shows 4.5TB where it used
to show 14.5TB. Secondary system also has many hundreds of empty
directories. Directory structure appears to be intact but most files are
now gone.
Post by Tom D
Note 4: Files seem to be missing in alphabetical order from about "C*"
out to "P*". As I list the directories, the ones with missing files are
sequential. I now have empty sub-directories in about a dozen directories.
The top level directories that are have lost files are not in order, but
sub-directories under the top level have lost files in alphabetical order.
This eliminates ZFS as the source of corruption or missing files. ZFS does
nothing
in alphabetical order. It operates temporally.
-- richard

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Graham Perrin
2014-07-20 18:18:46 UTC
Permalink
On Saturday, 5 July 2014 18:53:01 UTC+1, Tom D wrote:



Post by Tom D
Note 4: Files seem to be missing in alphabetical order from about "C*" out
to "P*". As I list the directories, the ones with missing files are
sequential. I now have empty sub-directories in about a dozen
directories. The top level directories that are have lost files are not in
order, but sub-directories under the top level have lost files in
alphabetical order.
Is it worth thinking about the normalization property on affected datasets?

Linux aside, for a moment: in some situations, *disappearance* (a symptom)
can be a result of the normalization property being not set appropriately
(appropriately for the OS).

An extremely remote possibility 
 long shot:

* none of the names of Tom's directory files included a character that can
be affected by a normalization property

* for each normal file that disappeared (from within directories), the name
included a character that was affected by a normalization property.

I admit to not getting my head around the whole of this thread, so if this
reply is way off: my apologies.

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/***@public.gmane.org
Loading...