Try before you buy

Crashplan: Part Two

Since my last post about Crashplan there have been some developments.

In a nutshell (and, like before, I'll start with the conclusion for those that want the Cliff Notes version, and then provide the details after that) Code 42 (the developers who produce Crashplan):

Continued to ignore my concerns and questions until after I'd written the blog post, and started linking to it from the comment sections of various online reviews of the software.
Repeatedly claimed that I've got facts wrong, when I'm just reiterating what their support engineer told me.
Changed their story about what the root cause of the problem was.
Denigrated their competitors in their replies dealing with this topic.
Silently deleted comments from their support forum
Continued to refuse to answer questions posed by customers (myself, and others)

To understand what happened you have to follow three distinct lines of communication that opened after I'd written the original post.

In the first, Matthew Dornquast, who identifies himself as "one of the developers/founding partners of Code 42 Software" opened a new Crashplan support ticket with me, and started a discussion there.

In the second, "Brian" starts a thread on Crashplan's support forums, "Silently corrupted data" which links to my blog post. This attracts numerous comments.

And in the third, several people leave comments under my original blog post. I notice these somewhat after the fact, as I didn't have the "Send me e-mail when people comment" option turned on in Blogger. That's now remedied.

What follows is the unedited exchange between myself and Matthew in the new support ticket, along with a description and links to relevant entries in the Crashplan support forum thread.

One thing I want to call out up-front. You'll note that midway through the exchange between myself and Matthew he asks that I keep his comments private, and not repost them. I have chosen not to respect this requst for several reasons.

First, in the support thread there was concern expressed several times that I may have been paraphrasing what Code 42 staff had said. I want there to be no doubt that this a full, unedited copy of the conversation.

Second, I don't believe it's appropriate for someone to say something and then note, after the fact, "Oh, by the way, that was off the record". If you want to have those sorts of conversations then clearly establish that up front, not half way through.

Third, Matthew's comments are clearly intended to represent Code 42 -- he's writing in his professional capacity, not a private one.

Fourth, Matthew's comments in private contradict what another Code 42 employee wrote and that had already been quoted.

So, with that said, Matthew's first message, sent on March 30th, is as follows (misspellings and other grammar oddities retained):

Hi Nik,

I'm one of the developers/founding partners of Code 42 Software, we make CrashPlan. I read your blog post and saw several inaccuracies based on your interaction with one of our support agents. I wanted to track down the source, so I re-read the correspondence between our support staff and yourself. As I suspected, it's a case of miscommunication, we're to blame.

Rather than follow up on your blog post directly, I'd rather write you in hopes you can revise/clean up your initial blog post. I'm not trying to do damage control, you can write what you want. What I'm trying to do is share facts with you in hopes your personal trust in CrashPlan is restored.

If you have any questions about what I've written below, please give me a call! I'd love to chat with you. It's much faster that way- [number redacted -- Nik].
If you can agree with my points below, I'd appreciate you updating your blog entry to properly reflect the situation.. what you saw is real, but the cause is misdirected and frankly, you're kicking dirt on the very feature that makes us better than everyone else! :-)

Here are my points with your blog post.

re:"Previous versions of Crashplan have silently corrupted data that has been backed up."

This is absolutely not true. CrashPlan does not have a bug that silently corrupts data. Previous version of CrashPlan failed to detect all types of corruption that occur "in the wild" – bad disks, corrupt volume info, rebooting a machine instead of cleanly shutting down, etc. Again - CrashPlan did NOT corrupt your data, it just failed to detect the TYPE of corruption that had occurred. If we had corruption in our product, all your archives would have been corrupted. (And frankly, yours wouldn't be the ONLY negative post EVER on the internet.) This misunderstanding isn't your fault - it's ours. (Read my comments on support ticket below) The support engineer on our side was not communicating well.

Not only are we the only company that's so paranoid we're verifying your data at destinations (and trying to heal!) but we're also one of the few that verify every aspect of the restore as well! That part worked - it logged each and every file that didn't work. It's another failsafe we have. Other backup products don't verify the integrity of the restores! We actually detected and communicated to you the issues. Most products just write them out. Imagine if we'd just written out all your data - you would have trusted it worked. Months, maybe never, you would have attributed those few files failure to something else. Verifying everything at the mathematical level we do is expensive and time consuming. It took a lot of engineering to do.

re:"The team at Crashplan are aware of this. More recent versions of the software do not have this problem."

This statement is false because the first one is. The software does not have a backup issue, and does not corrupt data. The team at CrashPlan is aware of the fact we've improved our healing technology to include scenarios we otherwise had not considered.

re:"However, more recent versions of the software do not fix, acknowledge, or in any way indicate that some of the files in the backup are corrupt."

That's actually not true either. Every version of our software fixes, acknowledges, and repairs files that are corrupt. Do they detect, repair and fix every conceivable corruption possible? We can't say that for certain. What we can say for certain is every day we heal hundreds of thousands of issues discovered do to bad disks, reboots, corrupted file systems, etc. No other backup product works as hard as CrashPlan to make the inherently unreliable, reliable.

re:"Crashplan support appear to wholly unconcerned with this in a manner that means I no longer have faith in the product or their support. I leave you to determine the course of action that's right for you."

I agree with this if you limit it to the one agent. It's unfair to say our entire support group isn't concerned. (Hey, I'm writing you!) The reality is, there are a lot of agents here. Some new, some experienced. You got a new agent that did not properly recognize the seriousness of your event. He should have escalated this internally. This misunderstanding could have been avoided. I apologize for that. I would like to point out that we hire really great people, and we haven't outsourced our support overseas like our two main competitors have. Typically, we receive accolades for our support. Your experience (which is absolutely valid) is the exception, not the rule.

Here are my points on your support ticket with us. (Sorry for verbosity, I'm just talking informally.)

1. Mis communication by support engineer.

There was a misunderstanding as to what the source/cause was. CrashPlan is not the cause of the corruption, however it failed to heal around it. Healing around every possible situation is difficult to imagine, every release we have improves this healing. The corruption that occurred was not due to crashplan, it's just because it was a really old version, it had not been healed. The proof in what I'm saying is look at your other archive, it was fine! Your drive, your computer, the cable, something caused data corruption. It was NOT crashplan. Again, we just failed to discover and heal around that particular form of corruption that affected a very small % of the archive. I'm not making light of it, we want to be bulletproof on our healing technology. It's been improved many times since the issue you faced occurred. I strongly suspect you have a corrupt VTOC on that drive, have you run a disk repair utility on it?

2. You would not be able to reproduce the situation unless you reproduced the failure at the exact same moment. (i.e. disconnected drive, corrupted a block of data, etc.) Since the corruption was very small, my guess is it was a perfectly timed reboot as it was writing to drive. There is a good chance if you check integrity of the filesystem on that disk, there are issues.

3. I don't feel our support engineer properly conveyed a sense of urgency around this issue. My guess is he felt it wasn't as big of a deal as you had another destination and you had the data, but that doesn't make it any less serious that if it were your only source. Our agent should have treated this with a greater sense of urgency and spent more time explaining the details of this to you. Your faith in CrashPlan was unnecessarily shaken.

CrashPlan is the only product that automatically verifies destinations and attempts to heal around issues discovered through bad hardware (i.e. disks, disconnected cables, etc.) We've learned a lot over the last 3 years, we're continuously improving the feature. Please don't confuse the failure of a defensive feature with the core backup/restore engine. Our engine is 100% solid. Unfortunately, you had some corruption from an older backup that we did not heal around. You can't reproduce this, as we improved the way we store & verify data several times since then.

In summary - I'm sorry you had to post a blog entry to get the attention this deserves. I always tell support - "support is our marketing. Each person has the power to undo years of hard work." Already, your blog entry was linked to a recommendation, where now the guy might use something else other than us.. which I believe is the wrong call. This person wont get a more reliable product than CrashPlan. Who else supports multiple destinations, multiple levels of integrity checks, and attempts to heal around any and all corruption automatically?

I send the following reply on April 13th:

Matthew,

Sorry it's taken some time to reply to you -- a combination of training courses and vacation have left me away from the computer for an extended period of time.

On 30 March 2010 17:01, CrashPlan Support wrote:

Here are my points with your blog post.

re:"Previous versions of Crashplan have silently corrupted data that has been backed up."

This is absolutely not true. CrashPlan does not have a bug that silently corrupts data. Previous version of CrashPlan failed to detect all types of corruption that occur "in the wild" – bad disks, corrupt volume info, rebooting a machine instead of cleanly shutting down, etc. Again - CrashPlan did NOT corrupt your data, it just failed to detect the TYPE of corruption that had occurred.
This is semantics. It doesn't matter if it's the raw data that's corrupt or the checksum that is corrupt (and/or computed incorrectly).

It's also at complete odds with what your support engineer wrote.

re:"However, more recent versions of the software do not fix, acknowledge, or in any way indicate that some of the files in the backup are corrupt."

That's actually not true either. Every version of our software fixes, acknowledges, and repairs files that are corrupt. Do they detect, repair and fix every conceivable corruption possible? We can't say that for certain. What we can say for certain is every day we heal hundreds of thousands of issues discovered do to bad disks, reboots, corrupted file systems, etc.
You go out of your way to say this for certain on the Crashplan website. To quote this text again:

Once your files are backed up, CrashPlan continuously checks that your files are 100% healthy and ready to restore when you need them. If it finds any problems, CrashPlan fixes them.

Not "99.9x% healthy".

Not "If it finds some problems".

If you can't stand by these statements in private then don't make them in public. And I don't expect to find out that files are unrestorable at the point when I do the restore -- note that the affected files date from 2008 and have not been modified since then, so there's been plenty of time for Crashplan's "continuous" file check to determine that they're not restorable and alert me.

To be clear -- software has bugs, hardware is not error free, I know this. However, if your promotional material tells me that using Crashplan means I don't have to worry about performing test restores then I'm going to be upset as a customer if it turns out that I have to.

Here are my points on your support ticket with us. (Sorry for verbosity, I'm just talking informally.)

1. Mis communication by support engineer.

There was a misunderstanding as to what the source/cause was. CrashPlan is not the cause of the corruption, however it failed to heal around it. Healing around every possible situation is difficult to imagine, every release we have improves this healing. The corruption that occurred was not due to crashplan, it's just because it was a really old version, it had not been healed.

The proof in what I'm saying is look at your other archive, it was fine!
No -- the other archive was never tested, I just copied the original files from the original PC.

Since the other (remote) archive was created by unplugging the USB drive that contained the corrupt archive and physically handing it to the person who hosts the remote archive, whereupon they imported it in to their Crashplan instance I don't see that there's any evidence for making any claims, positive or negative, about the health of the remote archive.

Your drive, your computer, the cable, something caused data corruption. It was NOT crashplan. Again, we just failed to discover and heal around that particular form of corruption that affected a very small % of the archive. I'm not making light of it, we want to be bulletproof on our healing technology. It's been improved many times since the issue you faced occurred. I strongly suspect you have a corrupt VTOC on that drive, have you run a disk repair utility on it?
Yes. No issues found.

2. You would not be able to reproduce the situation unless you reproduced the failure at the exact same moment. (i.e. disconnected drive, corrupted a block of data, etc.) Since the corruption was very small, my guess is it was a perfectly timed reboot as it was writing to drive. There is a good chance if you check integrity of the filesystem on that disk, there are issues.
Again, to be completely clear -- are you saying that the support engineer's statement that:

they were stored with an older version of the CrashPlan Application that has a known issue with incorrectly checksum-ing stored files in a backup archive.
is false, and that older versions of Crashplan did not have a known issue when checksumming stored files?

Matthew replies the same day.

no worries - i figured as much. I'm disappointed in your response - I was hoping you'd see the severity of your accusation and while a lot of your presumptions were based on a single miswritten statement from a support engineer, realize you jumped the gun on a few conclusions.

You're publicly saying "Crashplan corrupts data" and "they know about it" and "aren't communicating it."

That's ridiculous. That's false. You should have confirmed that understanding before writing them publicly as fact. Bloggers should fact check just like journalists if they're going to publish. (This is my opinion, hey, don't agree.. but as soon as you started promoting your blog by hijacking other threads, you crossed that line. IMHO.)

Why did you not try your other backup? It could have confirmed the checksum issue wasn't present in the product and was due to something else. Maybe we'll make progress on the source of the issue rather than assuming the worst?

Taking a junior support guys communication mistakes (he's a new hire, hasn't been here all that long) and blowing it up into "crashplan corrupts data" then hijacking our public positive threads about us is a bit over the top. If you had simply said, "Hey, can I talk to your supervisor? This doesn't make sense. It seems inconsistent with everything I'm reading/have heard including your website" it would have saved us a lot of pain and time.

I wouldn't hang google out to dry on what a single support person said in an email.. at least, not without agreement from their higher ups!

Finally, let's not loose perspective on how great CrashPlan is. Do you know of another backup product (free or otherwise) that backs up to multiple destinations, has multiple levels of checksums and protections, encrypts before transmission, and then ultimately tries to identify and heal around corruption at destinations asynchronously? One that proactively sends backup status reports to prevent silent failure? Do you seriously think spreading FUD about CrashPlan helps the consumer? What else will they use? Mozy? Carbonite? Give me a break.. While yours is the only negative thread I know of like this, literally thousands have failed to restore data with those products. They're not 1,000 times bigger.. or 50.. or even 10.

We're a market leader for a reason. We try really hard, and we care a ton. It's beyond frustrating that you take a junior guys mis-step in communication and blow it out like this. We're engineers that care, we're engineers that work really hard to make a great product, that which (I personally) think is far more reliable and secure than anything else out there. Where we fail, we have a culture of fixing things, of not accepting anything else less than perfect.

I'll post on our forums as much as I can about all the safeties we employ for your data.. hopefully you'll agree it's ridiculous how far we go to do a great job.. certainly farther than anyone else out there.. and ultimately.. what a disservice you're doing with your post.. again.. IMHO.

Sorry for the fragmented thread - jamming fast between meetings.

Also.. if I come across as harsh.. sorry.. I'm mostly frustrated at how this situation even developed.. it could have been avoided with better communication at the start. Had the support guy said, "we can't heal around all types of loss, you hit one, we have a theory on why and think we've improved it".. it would have went a better way.. you might have asked what do we do.. and then been satisfied with it.

And while you're free to post whatever you wont.. please don't? I'm writing you personally.. please respect my privacy.

As always, you're free to call me.. it can save a lot of time.. and confusion around typing.

And my reply -- which is, at the time of writing, the final one on this ticket.

On 13 April 2010 22:29, CrashPlan Support wrote:

no worries - i figured as much. I'm disappointed in your response - I was hoping you'd see the severity of your accusation and while a lot of your presumptions were based on a single miswritten statement from a support engineer, realize you jumped the gun on a few conclusions.
You're publicly saying "Crashplan corrupts data" and "they know about it" and "aren't communicating it."

That's ridiculous. That's false. You should have confirmed that understanding before writing them publicly as fact.
I did confirm it -- with the support engineer. See my message of Mar 24 4:32,

Are you saying that backups that were started with the older version of Crashplan may have this problem, and that simply using the newer version is not sufficient to correct the issue -- the corrupt backups need to be wiped, and the backup started afresh?
Correct. [...]
See also my message of Mar 25 4:01, where I write "Are you saying that the following sequence of events [...] is sufficient to cause this corruption?"

To which your engineer replies "Yes, that is correct".

In two different messages there I said that Crashplan was causing corruption, and your engineer confirmed what I was saying, and did not quibble with my use of "corrupt" and "corruption".

While you may feel that this is one of your junior engineers speaking out of turn (I note that he was the third engineer that replied to the ticket, so from my perspective it's been escalated to more senior engineers twice) I think it is unreasonable of you to expect me to know the ins and outs of your staffing organisation.

Why did you not try your other backup?
Because it's at the other end of a very slow Internet connection, and the original files were sitting on a PC a few feet away from the one I was trying to restore the files to.

Taking a junior support guys communication mistakes (he's a new hire, hasn't been here all that long) and blowing it up into "crashplan corrupts data" then hijacking our public positive threads about us is a bit over the top. If you had simply said, "Hey, can I talk to your supervisor? This doesn't make sense. It seems inconsistent with everything I'm reading/have heard including your website" it would have saved us a lot of pain and time.
As I say:

1. Support agreed with the use of the term "corrupt".

2. This was the third person who'd chimed in on the support ticket -- from my perspective I was already talking to someone senior.

3. The final response to the ticket left several questions that I'd asked open, with a complete refusal to answer them. As far as I was concerned Crashplan was done talking to me.

Finally, let's not loose perspective on how great CrashPlan is. Do you know of another backup product (free or otherwise) that backs up to multiple destinations, has multiple levels of checksums and protections, encrypts before transmission, and then ultimately tries to identify and heal around corruption at destinations asynchronously? One that proactively sends backup status reports to prevent silent failure? Do you seriously think spreading FUD about CrashPlan helps the consumer? What else will they use? Mozy? Carbonite? Give me a break.. While yours is the only negative thread I know of like this, literally thousands have failed to restore data with those products. They're not 1,000 times bigger.. or 50.. or even 10.
With respect, this is irrelevant to the issue at hand. I also find it crass that a significant part of your response to my concerns involves denigrating your competitors.

And you still haven't answered the questions that I've asked. Again, to be completely clear -- are you saying that the support engineer's statement that:

they were stored with an older version of the CrashPlan Application that has a known issue with incorrectly checksum-ing stored files in a backup archive.

is false, and that older versions of Crashplan did not have a known issue when checksumming stored files?

And while you're free to post whatever you wont.. please don't? I'm writing you personally.. please respect my privacy.
I reserve the right to

a) Post copies of this to the existing support thread (https://crashplan.zendesk.com/entries/140286-silently-corrupted-data) and

b) Excerpt some or all of the text for what I'm sure will be a followup post to the blog. One, I hope, that I will be writing after this is resolved satisfactorily.

Having read that it may be instructive to read the Crashplan support forum thread sparked by the initial blog post. I didn't contribute to the thread until page two, because until that point I was unaware of its existence. I post some entries to the thread clarifying that the quote acknowledging that this was caused by a Crashplan bug is a direct quote from a Crashplan employee, and two further messages that are copies of the ongoing correspondance with Crashplan (the messages quoted above).

In response to this, Code 42 silently deleted those two messages (by which I mean -- they are removed from the forum, and there is no indication that they were ever there, no placeholder that says something like "This message removed by a moderator" or similar). Handily, their forum software (optionally) e-mails participants in a thread a copy of new posts, so at least some other contributors there saw them (and quoted them in replies).

Matthew then posts a message headed "SITUATION SUMMARY" in the thread (I can't link to it directly, their forum software does not allow you to link to individual posts, it's about half way down page 2). I'm not going to reproduce it here, partly because it'll make a lengthy post even lengthier, and partly because you can see the responses and questions that other customers asked that forum (assuming they haven't been deleted).

You'll also note that those questions haven't been answered.

So, I'm still looking for alternative backup software. Some cursory searching has turned up Wuala, Spider Oak, Jungle Disk, as well as those two competitors that Matthew was so quick to trash, Mozy and Carbonite.

Does anyone have any positive experiences about them to relate?

Crashplan

[See Crashplan Part Two for the follow-up]

Three weeks ago I started to write a blog post about Crashplan. This is not how I expected it to turn out.

This is likely to be quite long, so I'll put the conclusions at the front, and then the information I've used to draw those conclusions follows.

If you're a Crashplan user (quite possibly because I've recommended it to you in the past) you need to be aware that.

Previous versions of Crashplan have silently corrupted data that has been backed up.
The team at Crashplan are aware of this. More recent versions of the software do not have this problem.
However, more recent versions of the software do not fix, acknowledge, or in any way indicate that some of the files in the backup are corrupt.
Crashplan support appear to wholly unconcerned with this in a manner that means I no longer have faith in the product or their support. I leave you to determine the course of action that's right for you.

With that out of the way, some background, and the events that lead me to the four points above.

I've been an enthusiastic user of the Crashplan backup software for something like two and a half years. I forget how I found it -- probably some blog post or mailing list -- but it seemed to me to be a great example of software that just works. It was flexible enough to handle my backup needs, and easy enough to use that I recommended it to family. friends, and work colleagues. I'm a paying customer, and have purchased Crashplan licenses to give to other people as gifts to encourage them to back up their important data safely.

So for more than two years my main computer at home has been backed up using Crashplan, initially to a locally attached USB drive, and latterly also to a colleague who I convinced to run Crashplan for his backup needs.

One of Crashplan's more useful features is that the software will auto-update, prompting you when a new version is released. So during this period I've very closely tracked whatever the most recent version of Crashplan is.

A couple of weeks ago I purchased a new PC, and the plan was once I'd gone through the somewhat tedious business of reinstalling my software, restoring all my data, and so forth I was going to decommission the old one. To that end, once the new PC was up and running one of the first things I did was install Crashplan on the new PC, make sure the old PC was 100% backed up to the USB drive, and then plug the USB drive in to the new PC.

When you do this, Crashplan can "attach" to the backup. Even though the files in the backup weren't from the new PC I just had to enter the password for the backup so it could decrypt them and restore them to the new PC. I thought this would be the simplest (and probably fastest) way of migrating my data from the old to the new PC.

I let Crashplan chug along doing the restore, which took several hours because of the volume of data. And then, at the end of the process, I saw a warning that 140 files have failed the "integrity check" during the restore, and couldn't be properly restored. All of them were digital photos.

Now this is a bit odd. One of the things that the Crashplan team champion on the website is the following claim:

Once your files are backed up, CrashPlan continuously checks that your files are 100% healthy and ready to restore when you need them. If it finds any problems, CrashPlan fixes them.
Source: http://www.crashplan.com/consumer/features.html

For me, this is a big benefit. One of the things you should do when backing up data is periodically try and restore it, to ensure that the backup is actually working. The fact that Crashplan tries to do this in the background was an important part of choosing the software.

Now I knew the backup was complete -- I'd verified it before I unplugged it from the old PC, so this is one of those things that should just never happen.

I sent an e-mail to the Crashplan support address. This generated ticket #20145 in their queue, and my message went like this:

Hi,

I'm migrating to a replacement PC. I decided to migrate my data across by plugging the external hard drive that the original PC backs up to using Crashplan+, and then restoring from that archive on the new PC, running version 3.8.2010.

29,742 files restored correctly. 140 failed, listing in the History tab as - Integrity check failed for

First, it would be very useful if I could cut/paste the contents of the History tab. It would make it much easier to figure out which files I'll need to copy over by hand.

Second, and much more importantly, I'm very concerned by this. From http://b1.crashplan.com/consumer/features.html:

Once your files are backed up, CrashPlan continuously checks that your files are 100% healthy and ready to restore when you need them. If it finds any problems, CrashPlan fixes them.

This does not appear to have happened. How do I find out what went wrong in this instance, and how do I fix it?

About 5h30m later (which is, by the way, fine, we're in very different time zones, so that sort of response time is not only perfectly acceptable it's probably above and beyond what I would normally expect) I get a reply from Renee at Crashplan, asking if I can send logs from the destination computer, and instructions on how to do that. I do so, and over the course of a few days (a short vacation intervened) I send logs from the source computer (i.e., the one that's been doing all the backups over the last few years) as well.

A day and a half after I send the necessary logs I get a reply from Bret at Crashplan. He says:

Unfortunately these logs don't point to a clear source of this error. A copy of the restored file was preserved with a modified name; it may be useful for you to review this modified file and let us know if the file that was restored appears to be correct or is non-functional. For example, the following file:

C:/Documents and Settings/Nik Clayton/My Documents/My Pictures/2006/2006 07 14 All Things Gothic/IMG_2301.JPG

was restored to the following location:

C:\Users\nik\Documents\My Documents\My Pictures\2006\2006 07 14 All Things Gothic\restore.failed-checksum.IMG_2301.JPG

Can you attempt to open this file and verify that it is a well-formed JPEG file?

I do some digging, and reply about five hours later, with

It's not a valid file. Windows Photo Viewer refuses to open it.

The restore.failed-checksum.* files have suspicious file sizes:

09/09/2008 19:42 786,432 restore.failed-checksum.IMG_2296.JPG
09/09/2008 19:42 786,432 restore.failed-checksum.IMG_2297.JPG
09/09/2008 19:42 786,432 restore.failed-checksum.IMG_2298.JPG
09/09/2008 19:42 786,432 restore.failed-checksum.IMG_2299.JPG
09/09/2008 19:42 786,432 restore.failed-checksum.IMG_2301.JPG
09/09/2008 19:42 786,432 restore.failed-checksum.IMG_2302.JPG
09/09/2008 19:42 786,432 restore.failed-checksum.IMG_2303.JPG
09/09/2008 19:42 917,504 restore.failed-checksum.IMG_2305.JPG
09/09/2008 19:42 917,504 restore.failed-checksum.IMG_2306.JPG
09/09/2008 19:42 786,432 restore.failed-checksum.IMG_2307.JPG
09/09/2008 19:42 917,504 restore.failed-checksum.IMG_2308.JPG
09/09/2008 19:42 786,432 restore.failed-checksum.IMG_2309.JPG
09/09/2008 19:42 786,432 restore.failed-checksum.IMG_2310.JPG
09/09/2008 19:42 655,360 restore.failed-checksum.IMG_2311.JPG

They're all exact multiples of 1,024, and far too small. Compare and contrast with the same files that I restored by direct sync from the source PC to the target PC:

09/09/2008 19:42 2,999,745 IMG_2296.JPG
09/09/2008 19:42 3,029,664 IMG_2297.JPG
09/09/2008 19:42 3,102,390 IMG_2298.JPG
09/09/2008 19:42 2,923,048 IMG_2299.JPG
09/09/2008 19:42 2,939,522 IMG_2301.JPG
09/09/2008 19:42 3,077,000 IMG_2302.JPG
09/09/2008 19:42 2,707,091 IMG_2303.JPG
09/09/2008 19:42 3,478,028 IMG_2305.JPG
09/09/2008 19:42 3,509,851 IMG_2306.JPG
09/09/2008 19:42 2,627,625 IMG_2307.JPG
09/09/2008 19:42 3,169,280 IMG_2308.JPG
09/09/2008 19:42 2,859,546 IMG_2309.JPG
09/09/2008 19:42 2,924,675 IMG_2310.JPG
09/09/2008 19:42 2,518,022 IMG_2311.JPG

It goes quiet for two days, and then Matt Genelin takes over the ticket, saying:

Let me step in here. Thank you for the log files. After checking with several engineers on our staff, our best causation of the 140 files missing / corrupt is as follows:

The 140 files stored on your external hard drive are inaccessible because they were stored with an older version of the CrashPlan Application that has a known issue with incorrectly checksum-ing stored files in a backup archive. We have corrected this issue in the last 12 months, and the current version of the CrashPlan Client Application backs up files with the correct checksum information.

Moving forward here, the best recommendation we can make is:

1. Restore your complete archive from your other backup destination (I believe this is [redacted]).
(verify that your restore is successful.) Then proceed to step 2:

2. Shutdown the CrashPlan Backup engine on [redacted] like this:
http://support.crashplan.com/doku.php/recipe/stop_and_start_engine

3. Erase, delete or replace the backup archive that is stored on your external drive named "Folder: External 320G". Simply perform a file copy from [redacted] to your external drive.

Please note that since your external drive was created on 12/12/2008 and your archive on [redacted] was created on 2/7/2009, you will loose any file version information that was made between December 2008 and Feb. 2009.

4. Restart (start) the backup on [redacted], again:
http://support.crashplan.com/doku.php/recipe/stop_and_start_engine
If this seems like an unreasonable fix to this issue, please let me know.

"[redacted]" was the name of the remote destination I also back up to -- since it's a colleague's name I've removed it from the above.

I should mention at this point that none of my data has been irretrievably lost. My original PC is still here, and with some faffing around I can retrieve the missing files from it (or download them from the [redacted] offsite backup). But that is purely by luck. If all my backups had the same problem, which is not an unreasonable assumption, this data (140 digital photos) would have been lost forever.

I wasn't sure that I'd quite understood Matt correctly. In particular, with the reference to an older version of Crashplan I thought that perhaps he'd misunderstood, and assumed that the backup I was restoring from was only created with an older version of Crashplan. So we had the following exchange. First, me:

While it is the case that I first started backing up to "External 320G" using an older version of Crashplan, the Crashplan version has been regularly (auto)updated since then. The specific set of steps I carried out to do the restore was:

1. Power up PC #1 (runs XP SP3, Crashplan+, and is the machine that "External 320G" has been plugged in to for the last few years).

2. Verify (through the Crashplan UI) that Crashplan thinks that the backup of PC #1 to "External 320G" is complete. This is using the latest version of Crashplan (3.8.2010) because it auto updated earlier in the month.

3. Power down PC #1, power off the external drive, power up PC #2 (Windows 7), plug the external drive in to PC #2 and power it up. Install the latest version of Crashplan from crashplan.com, import the backup from the external drive, and attempt the restore.

That then generated the checksum errors for 140 files upon restore.

Are you saying that backups that were started with the older version of Crashplan may have this problem, and that simply using the newer version is not sufficient to correct the issue -- the corrupt backups need to be wiped, and the backup started afresh?

I've just reviewed the release notes going back to 12.10.2008, and don't see this mentioned.

Matt's reply:

Correct. The Backups being the backup archive on your External Drive. I am recommending:

1. Verifying the [redacted] Backup.
2. Wiping the External Drive.
3. Coping the [redacted] backup archive over to the External Drive.

Seem reasonable?

At this point I'm still not quite convinced that I have this right. In particular, he's not correcting my assertion that this is a problem they've known about, and fixed with no notice in the release notes, and no mechanism to fix existing-but-broken backups. After all, this is a company that sells backup software (and sells an optional service whereby they'll host external backups for you). They wouldn't be that cavalier about the integrity of their customers' data, would they?

So I replied:

Well, I don't need to do that, because I've moved the data from the old machine by other means -- restoring from the backup was (supposed) to be the simplest way to do this.

However, I want to make sure that I understand you correctly. Are you saying that the following sequence of events:

1. Install Crashplan in 2008.
2. Tell Crashplan to backup to an external drive.
3. Let Crashplan autoupdate throughout 2008, 2009, and 2010, and continue to backup to the external drive throughout this period.

is sufficient to cause this corruption? This was not an external backup that I created once using an old version of Crashplan, and then put away -- the external drive has been attached to this PC almost continuously, and Crashplan (from the earlier 2008 version to the most recent March 2010 version) has been backing up to it on pretty much a daily basis.

I must ask why Crashplan doesn't warn about this -- big red flashing letters saying "Warning: You created this backup with a version of Crashplan that had checksum errors. You must delete this backup and start afresh".

Better still, why don't newer versions of Crashplan detect this and correct it automatically? http://b4.crashplan.com/consumer/features.html is quite explicit:

Once your files are backed up, CrashPlan continuously checks that your files are 100% healthy and ready to restore when you need them. If it finds any problems, CrashPlan fixes them.

This does not appear to have happened here.

I'm very concerned that based on what I've been told so far it seems as though an older version of Crashplan corrupted my backup, you released a fixed version without noting the fix in the release notes, but the fixed version does not correct prior instances of the problem.

Right now I do not have a warm fuzzy feeling about continuing to trust Crashplan with my data.

Matt's reply:

Yes, that is correct. This is what I am stating here.

I am also explaining that an older version of CrashPlan has a known issue -- that has been corrected in our newer versions of the CrashPlan Client. This known issue appears to have passed our nightly archive maint. check:

And only appears when you attempt to restore files.

Normally our website is correct; once you back a file up, there is no need to worry about your files. In your case, it appears from your archive that some of your files were backed up with a version of the CrashPlan Client with a known issue, and the newer versions CrashPlan Client's nightly archive maint. did not detect the problem in your archive. The problem here surfaced when you went to restore your external drive's archive, that is 99.996% fine, but 0.004% corrupted.

I am suggesting a course of action that brings you back to 100% fine, and throws away the archive that is 0.004% corrupted.

I can understand. Your feelings on CrashPlan are a conclusion you will need to come to on your own.

Let's keep in mind the facts here:

* Only one of our multiple-destination archives is having issues here.
* The one archive that has issues restored 29,742 correctly and failed to restore 140 files. That's a failure rate of 0.004%.

I agree -- this is not perfect. Perfection would be 100% data recovery. This is why CrashPlan Allows you to backup to multiple destinations. You should be able to achieve perfection of recovery by using your second archive; on your [redacted] computer.

A couple of points here. Matt's skipped over my "Why doesn't Crashplan warn about this, and/or fix the problem automatically?" question. He also seems to think that you can quantify the effectiveness of a backup solution by taking the number of files, and divide that by the number that failed to restore as some sort of useful metric. That takes no account of the relative importance of the files -- these were photos, and irreplaceable, nor the absolute volume of data lost.

He also assumes that I can restore the files from the [redacted] site. While that may be possible (and I haven't tried, I haven't needed to) that backup was created by taking a copy of my local backup archive and giving it to my colleague, so it's entirely possible that that archive has the same problem as my local one.

And finally, the Crashplan site is quite explicit, "100% healthy and ready to restore". There's no equivocating around some-number-of-9s availability. They claim 100%.

My final message to Matt asked:

1. Will the next release of Crashplan detect this problem and fix it.

If not, when will it be fixed?

2. Why wasn't this problem called out in any of the release notes for versions released after the problem was detected?

3. Will you inform existing customers of this problem, and the need to wipe and restart existing backups if they're older than date ?

All, I think, reasonable questions, which are ducked in Matt's final reply.

It has been a pleasure working with you. It's clear that the technical recommendation I have made for you will correct the issue at hand here, and that the quoting of text back and fourth is leading our conversation in a circle. I want to bring you to a place that moves you forward, and the best way to do this is to end our conversation now.

I believe I have answered your questions repeatedly, and your questions are deviating away from solving your technical problem. By closing this conversation, I am hoping that you will take my recommendation in good faith, and apply it to your unique situation to move your backups with CrashPlan forward.

Looking back through this discussion those three questions are not answered.

There's no commitment that future versions of Crashplan will detect and fix this problem.
There's no answer as to why Crashplan weren't honest about this problem in the release notes of the software once they detected and fixed it.
And there's nothing to suggest that they'll inform existing customers of the problem.

So, if you're using Crashplan you should definitely make sure that your backup is 100% readable by the current client. And if it isn't you'll need to wipe it and start the backup from scratch again.

You might also want to start thinking about trusting your data to a different organisation; and in particular one that values honesty when it notices and fixes a mistake that leads to data loss.

Has anyone got any recommendations?

Useful Android Apps: Listen

Podcasts are something I only recently started to listen to. I can read information far faster than I can listen to it, so spending time listening to (what I imagined would be) not much more than someone reading their blog entries didn't really appeal. And then some friends started recommending a few podcasts, and I decided it was time to find out more about it.

I'll cheerfully admit that this can be a bit tricky on Android phones, certainly trickier than on an iPhone or iPod. There's only really one way to get audio on to an iPhone (and associated iPods), and that's via iTunes. I use iTunes to manage my own music library, but it doesn't sync to Android, so getting music and podcasts on to the phone involves a bit of drag-and-drop, and trying to remember what you have and haven't already synced. That's more effort than I really want to put in to ephemeral entertainment.

Enter Listen, from Google Labs. It lets you subscribe to, download, and listen to arbitrary podcasts without needing to sync the content from a desktop or laptop.

Once installed you'll see this screen. I can search for new podcasts to subscribe to, review the episodes I can listen to ("My listen items") and see the podcasts I'm subscribed to ("My subscriptions"), as well as see popular searches to find new and hopefully interesting podcasts. At the bottom of the screen you can see I'm part way though listening to an episode of "Wait Wait... Don't Tell Me".

Subscribing to new podcasts is easy. I can either search from the main screen, or tap through to "Manage my subscriptions" and enter a feed URL directly, as shown here.

Tapping "My listen items" shows me items I've queued up as well as items that are freshly downloaded. Listen normally plays through the queued items first, before moving on to fresh content, so you can control the order in which episodes are played by adding them to the queue. In this example many of these podcasts are new to me, so I've queued up a number of their back-episodes which I'm slowly working my way through.

Pressing through to any of the existing subscriptions allows you to unsubscribe, or queue entries, or mark them all listened or unlistened as necessary. Tapping on an episode lets you add it to the queue, listen to it immediately, and manage the subscription.

Actually playing an episode shows a screen like this, where as well as playing or otherwise navigating through the episode some of the options to manage the subscription to this podcast are also provided.

Finally, Listen has a number of thoughtful configuration options, which amongst other things can be used to ensure that you don't rack up a large bill downloading content without a WiFi connection, or consume too much power doing so.

Listen certainly makes it easy to keep up with your podcasts. It's very refreshing to start the morning commute knowing that there's almost certainly something new to listen to that's come in overnight, and the UI does a good job of getting out of your way.

In a stroke of genius, or idiocy, depending on which way you look at it, recent versions of Listen store your podcast feed subscriptions in a folder in Google Reader. This has the distinct advantage that your subscriptions are not tied to the phone, they're tied to your Google account, so they'll follow you as you upgrade your phone, and you can use Reader as another client to manage your subscriptions. You can also use the recommendation feature in Reader to find new podcasts to listen to. The downside to this is that if you're not careful you may inadvertently mark one of these podcasts as 'read' in Reader, at which point Listen will ignore it.

The only thing I still have to get used to is the notion that these aren't automatically available on my home media system. The rest of my audio collection is, and so at the moment I've subscribed to these podcasts twice, once to download so that I can play them when I'm at home, and again for when I'm on the move.

Listen is available for free. Search for it in Android Market, or scan this QR code with your phone to install it. And for more details (including FAQ and discussion groups) see http://listen.googlelabs.com/.

And if you'd like to listen to what I'm listening to, here are the podcasts I'm subscribed to at the moment.

Useful Android Apps: NewsRob

Do you have an Android phone? I've been using one for about 15 months now -- first it was a G1, and now the Google¹ Nexus 1 -- and in that time I've discovered a number of applications that have proven indispensable, and over the coming entries I'll share my recommendations.

The first of these is NewsRob. It's a Google Reader client, syncing the content of your news feeds from Google Reader to the phone. It can also sync the web page content of the articles so that they are easily accessible when you're out and about -- handy for RSS feeds that don't include the full content of a post in the feed, or where you want to see comments that were left on the post on the original site.

With a slick, simple UI that gets out of the way, and a wealth of battery and wallet friendly syncing options ("only on WiFi", "only when charging", and so on) that can be set on a per-feed basis if necessary it's easy to stay in control of your feeds, and NewsRob can also share and star interesting posts via Google Reader.

On opening NewsRob you're presented with a list of your Google Reader top-level categories that have unread items. Tapping a category (in this case, "tech news") shows you the web site sources in that category that contain unread items. Note the handy "all articles" items at the top of the list, making it easy to just work through all unread articles, or all unread articles in a particular category.

On the left is the default article view, showing the article from the selected source. Notice the lack of UI elements for moving through the articles. Tapping an unused portion of the screen brings up 'floating' arrows that allow you to navigate the feed, without constantly consuming precious screen real estate. Notice, too, the star in the top navigation bar. Tapping that marks this entry with a star in Google Reader.

Tapping the article title takes you to the article as it appears on the originating web site, the right-hand screen shot in this example. This is done within NewsRob, your default web browser is not launched, so you can jump to the original site, read the content in context there while still being able to star the item, or use the floating UI to quickly move to the next article.

NewsRob is available for free (ad-supported), or you can pay for the ad-free version. Search for it in Android Market, or scan this QR code with your phone to install it.

NewsRob QR Code

¹ My employer. This and other entries are my opinion, not theirs.

EuroBSDCon 2009

A couple of weekends ago I attended the 2009 European BSD Conference, in Cambridge.

My trip report has just been posted on the official Google Open Source Blog, and includes a few of my photos from the event.

Productions

Since the start of this year I've been involved with the Zurich Comedy Club, an English speaking theatre group that's been running in Zurich for 50+ years (back when "Comedy Club" meant "We tend to put on comedies" and not the more modern "Stand up at a mic and deliver a routine for 15 minutes").

Anyway, I've got two updates that may be of interest.

First, our November production is "GamePlan", by Alan Ayckbourn.

Following the dot-com crash a mother and her daughter are abandoned by their husband. With money running out, and determined not to have to move away from her senior school, the daughter, Sorrel, hatches a plan to raise money by advertising as a prostitute on the Internet. Things are going swimmingly, until one of her clients dies of a heart attack mid-coitus. Then things start to unravel.

If you're interested in acting then you should know that auditions for GamePlan are this Monday, 7th September, starting at 7.30pm, at St Andrew's Community Centre, Promenadengasse 9, 8001 Zurich.

View Larger Map

If backstage work is more your thing (and it is mine) you should still come along on the Monday to meet people, find out what needs to be done, and volunteer. Or, drop me a note on here.

The second thing is that we have a production going up sooner than November. "Three Couples and a Goldfish" consists of three one-act plays, being performed from Tuesday 29th September through to Sunday 4th October at the Theatre STOK in Zurich. Performed almost in the round each evening will consist of a performance of "Henna Night", "Abortive", and "The Extraordinary Revelations of Orca the Goldfish¹". All three shows explore the nature of relationships.

View Larger Map

Henna Night: Judith leaves her ex-boyfriend a desperate message on his answerphone, saying that she is not coping with their break-up and that she has brought some razor blades and henna. She adds that she hasn't decided whether to dye her hair or slash her wrists. However it is his new partner, Ros, who hears the message and comes rushing to Judith's bedsit.

Abortive: Three weeks after her abortion, Roz - having been raped by Billy - and Colin find themselves in bed wondering how to carry on. Through the reproaches that are traded and the incidents that come up for discussion, the picture of a disturbing relationship emerges. Bed is no longer a safe island.

The Extraordinary Revelations of Orca the Goldfish: Henry and Alice have been married for 15 years. They are bored and have lost the zest for life and their relationship. Their lives have become a blur between fantasy and reality. Secret passions and desires explode and two seemingly separate fantasy worlds are on a collision course with reality. And it all started with a goldfish they named Orca.

Tickets are available now from Ticketcorner, and are CHF 28.

I'll be running around backstage each night with my Stage Manager hat on, trying to keep order and making sure everything runs smoothly. So don't expect to see me on stage; unless that is, it's a blackout, and we need to move some of the set...

¹ Keep this one in mind the next time you're playing charades...

Worldwide photo walk

July 18th was the second Worldwide Photo Walk, and I took part in the Zurich walk. It was touch-and-go in the morning; with the walk scheduled to start at 2pm the rain didn't let up until about 1.50.

This was also a good opportunity to catch up with Jessica, Kerrin, and Karl, who were also out and about with their cameras.

Over the course of three hours I took about 150 or shots. Weeding through them afterwards reduced that to 30 that I quite liked, and here are three of them.

There also more photos from other members of the walk on the Zurich photo walk Flickr pool page.