HN Reader

Photos does a lot of extra work on import (merging RAW+JPEG pairs, generating previews, database indexing, optional deletion), so my guess is a concurrency bug where a buffer gets reused or a file handle is closed before the copy finishes.

Rare, nondeterministic corruption fits the profile.

But good gravy that troubleshooting path got expensive real fast. Replacing the laptop and the camera? Why not start by trying something other than Photos? It doesn’t even need to be a paid product; the Olympus software is free not to mention a good baseline since it - of all the applications - should be able to import photos without corrupting them.

Edit to add: delete on import seems pretty risky. My workflow is to import and only delete from the camera after 1) the imported photos are backed up 2) I’ve done a first pass culling.

A better way to have further narrowed down the problem down to Actually iPhoto would've been to do the same tests with a USB-C card reader plugged directly into the Mac, which would've eliminated cables, hubs, and camera hardware/software/firmware as possibilities.

It's worth noting that searches show that OM-1 USB support is imperfect, the camera manual addresses that "USB transfers aren’t guaranteed in some setups", and user consensus seems to be to use a card reader for reliable file transfers.

Tangential story - 12-13 years ago I was a burgeoning and super eager software dev that moved to Seattle to be closer to "the scene." tenderlove's content was a major reason for me going there and I poured through his posts learning way too much about Nokogiri, Active Record, and much much more.

I went to every Ruby meetup I could get to out there and I remember one in particular, a Seattle RB meetup, in the Substantial office. It was a pretty small group, at most 15-20 people.

I was with a coder buddy but knew nobody else. We were all just drinking pints of Manny's beer and eating pizza from Big Mario's or something. Ryan Davis (the creator of minitest among other things) was doing a presentation on Unicode.

Aaron Patterson (tenderlove) was cracking jokes at every opportunity. At one point I asked a relatively naive question and Aaron _tenderly_ answered in joke-form response. I felt such a _part_ of the scene then. Aja Hammerly was super engaged in the presentation, I think even Ryan Bates and/or Geoffrey Grosenbach were there.

It was quite surreal to be in this dream-like state around giants and heroes just doing what they were doing and being so inclusive. It seemed so normal but became a core memory.

Thanks for everything Aaron, you've truly been an inspiration!

They finally recognized there is an issue, but there is no fix, as of a few weeks ago :(

Sure security is important but integrity is too.

I’ve used Olympus cameras for over a decade. Well, the same camera to be honest, a PEN E-PM2. This has only appeared in the past couple of years.

I haven’t seen it on photos from my Canon EOS 80D yet, but I guess it’s time to change my workflow. And maybe OS.

For the longest time my process while traveling was importing onto my iPad or occasionally my iPhone since I didn't have a personal laptop, just a Mac Mini at home.

All my photos are managed using Digikam and developed using Darktable. They are also visualized via immich, but immich only has access via a read-only mountpoint.

Everything is hosted locally of course.

That's because my worry is corruption of the entire Library, which Photos stores as one gigantic opaque file/directory abomination. My .photoslibrary file is currently 70gb in size, and I'm terrified of what would happen if it becomes corrupted. The Photos app crashes not infrequently.

https://www.cgsecurity.org/wiki/photoRec

So much wasted time, now I just use image capture to import and organize directly.

Still get errors from time to time.

Apple needs to hire more quality control, their software integration is going from a positive to a negative.

Seamless integration was a large part of Apple's initial hook, and continues to be a part of their drive to push services, it should be a priority.

Then for the current files you’ll want to see what happened. Often with this class of problem either the bytes are zeroed or shifted. Since the size is the same, perhaps they’re zeroed or perhaps bytes are LE to BE or dumb shit like that (don’t know why it would be but weird world right).

Just diff and see if you see anything (I wrote off memory but you get the idea)

   diff -aui <(xxd -r file1) <(xxd -r file2)

From that bare start you can see what’s up.

When flash fails it returns garbage or zeros instead of (what was) your data. It can be tranient or persistent. And without any error codes from the storage device or the file system.

If storage returns garbage for filesystem metadata, all bets are off how the OS filesystem driver will behave.

Reformat should be done in camera. And that card used only in that camera. And only that camera gets to write to that card. And don't delete individual images.

On day 7 or so the import failed and all files on the pad got corrupted. But also the SD card got corrupted.

I stopped using the device and the card because I knew not all is lost. I had to buy a new card in SF as replacement. Back home I used a recovery software to check if data is still on the card (I used the same software before on a card that got deleted by another person and I was able to get all images back). I was able to get most of the images recovered and also recovered a few from the iPad. All in all I lost maybe 10 out of a few 100. Now I travel with multiple cards and backup already each night while in the hotel. And I don’t delete the images on the SD Card. I format only when I’m sure I have everything copied and secured.

It writes the metadata into the audio file. Badly.

In particular, it's that "Play Count": iTunes rewrites the audio file every time you play the song.

Usually it just corrupts the metadata enough to forget the album art. But it's perfectly willing to destroy the audio data.

I've been importing raws for years from Sony and earlier from Pentax and didn't experience it.

In fact searching for "OM image corruption" shows bunch of results not related to Apple Photo.

My guess is that OM has buggy SD driver which starts deletion before actual read finished.

Recently, importing via image capture has resulted in recurrent crashes. Files appear on Image Capture that do not appear on the iPhone, nor can they be downloaded via image capture, or deleted. I wish I knew wtfudge was going on.

I learned the hard way to never delete photos from the SD. Just buy a new one it's so cheap anyway.

Great article by the way, sounds like my kind of rabbit hole :)

I think he was just looking for an excuse to buy new kit

For the 99.999% of us who just use our phone, nothing to worry about.

Scared me for no reason! =P

Last time I looked (pre-COVID) there wasn't a lot of promising options, and some didn't support HEIF images

https://www.businessinsider.com/steve-jobs-mobileme-failure-...

That's a mistake no mater what application you're importing to, else we'll be graced with another blog post, "Darktable app Corrupts Photos".

What's the purpose of RAW+jpg though? Seems rather redundant?

Personally, I have seen a row of green pixels on the top or bottom + vertically flipped photos on import.

Good sleuthing!

I've stopped buying apple stuff

(sorry not sorry)

https://cdfinder.de/blog/files/image_capture_bug.html

(I'm not sure whether this bug has been fixed or not yet, though I think it has been fixed.)

I got similar symptoms as mentioned, I suspected AFS+, but what do I know. It has happened on at least 3 iphones (pros), now when I think about it, I don’t remember any iphone I haven’t have troubles with. Having 5000+ images (non-raw) where 5-10% are corrupt is infuriating, but I just stupidly buy another iphone every year (the most expensive one).

Re-importing images 10-100 times could sometimes extract a few additional images, but the phone just disconnects after a while when running such scripts.