Ensuring Your Digital Photo Archive Survives for Generations (i.e. to DNG or not DNG)
Whenever you visit your parents’ home and you dust off a few old family albums, you take it for granted that those photos are a generation or two old. You assume they’ll always be around. Other than printing a few good digital photos you’ve taken lately, what are you actively doing to ensure YOUR photos will truly last for many future generations?
Seriously, how exactly do you plan on ensuring your unprinted digital photos will survive the test of time? You need to be sure you’ll NEVER lose the original files (i.e. due to poor backup or a home fire) and you must ensure the files will be accessible by future computer systems we can’t even yet envision today.
If you’re worried about physically protecting your photo files, checkout the post I wrote about my photo backup and storage strategy. Assuming you don’t lose the physical files, the next issue is selecting a file format to archive.
UPDATE (Aug 21, 2012): With regard to long-term archival-grade digital storage, Amazon Glacier has been announced with a price that’s 1/5th those of Google Drive and is only $0.01/GB per month (12 cents/yr per GB). This is like cold storage for data files as they’re not immediately accessible once archived this way. You effectively request an “archive” or “vault” (that’s one file or collection of files) and must wait 3-5 hours for it to be available again. Once available, it’s online for 24 hours for you to use/download. They brag that the annual durability of these files is 99.999999999%. That’s pretty good . They will soon launch an automated system so that files you store in Amazon S3 can be archived into Amazon Glacier automatically (e.g. based on age).
Longevity Issues with Proprietary Camera Raw Formats
In 20 years, a DVD drive will be about as hard to find as a 5 1/2″ floppy drive is today. 20 years is not that far away. If you know that, you won’t bother using DVDs for long term storage. Why then would you choose a file format that’s entirely specific to your camera make AND model which may also go extinct?
If you’re already getting serious about DSLR photography, you’re probably shooting in the camera’s raw format. This is your camera’s native and proprietary file format which has serious advantages over the limited JPEG file format option. Since raw formats are specific to each camera (both make and model), who’s going to keep track of that raw format conversion a century from now? How many unique camera models are there yet to come? We already have well over 350 raw formats to contend with and thousands are on their way.
It’s quite possible that JPEG support may survive for a very, very long time given its popularity and standardization, but do you think in 100 years or more that anyone is going to keep track of the Canon EOS Rebel XTi and its format? It’s possible but it’s less likely. Even the latest Canon EOS 5D Mark III and Nikon D800 will be forgotten dinosaurs by then.
Introducing the Digital Negative (DNG)
What’s needed is an industry-standard file format that maintains all the fidelity we enjoy about raw files without the dependence on a single company or camera brand. Storage “cost” should be the same or better (i.e. large TIFF files won’t do) and it should be an independent and well supported standard.
The good news is that such a format already exists and it’s called “DNG” (Digital Negative). It was pioneered by Adobe but it’s free and open to everyone and every company to use.
- DNG is an open specification created in 2004 in response to this need for a common raw format.
- It’s lossless, meaning there’s no degradation in quality (as found with formats like JPEG)
- It typically takes about 20% less space than comparable raw files
- It integrates all metadata directly into the one file (no need for extra “sidecar” .xmp files)
DNG is an open specification without royalties to Adobe. That means that any company can use and support this format in hardware or software without cost or concern of patent infringement. Every product from Adobe since 2004 has supported the DNG format and it’s been submitted to an external standards body (ISO) for future management of the specification.
Objectives of the DNG Format
Adobe has cited the following objectives for the DNG format. This is taken directly from the Wikipedia article on DNG:
- Digital image preservation (sometimes known as “archiving”): to be suitable for the purpose of preserving digital images as an authentic resource for future generations. Assessment: The US Library of Congress states that DNG is a recommended alternative to other raw image formats: “Less desirable file formats: RAW; Suggested alternatives: DNG”. The Digital Photography Best Practices and Workflow (dpBestflow) project, funded by the United States Library of Congress and run by the American Society of Media Photographers (ASMP), singles out DNG, and states “DNG files have proven to be significantly more useful than the proprietary raw files in our workflow”.
- Easy and/or comprehensive exploitation by software developers: to enable software to be developed without the need for reverse engineering; and to avoid the need for frequent software upgrades and re-releases to cater for new cameras. Assessment: Software could support raw formats without DNG, by using reverse engineering and/or dcraw; DNG makes it easier, and many software products can handle, via DNG, images from cameras that they have no specific knowledge of. An unresolved restriction is that any edit/development settings stored in the DNG file by a software product are unlikely to be recognized by a product from a different company (this problem is not specific to DNG).
- In-camera use by camera manufacturers: to be suitable for many camera manufacturers to use as a native or optional raw image format in many cameras. Assessment: About 12 camera manufacturers have used DNG in-camera. About 38 camera models have used DNG. Raw image formats for more than 230 camera models can be converted to DNG.
- Multi-vendor interoperability: to be suitable for workflows where different hardware and software components share raw image files and/or transmit and receive them.
Characteristics of the DNG Format
All of the above objectives are facilitated or enabled by most of these characteristics:
- Freely-available specification: this can be downloaded from the Adobe website without negotiation or needing justification.
- Format based on open specifications and/or standards: DNG is compatible with TIFF/EP, and various open formats and/or standards are used, including Exif metadata, XMP metadata,IPTC metadata, CIE XYZ coordinates, ICC profiles, and JPEG.
- Self-contained file format: a DNG file contains the data (raw image data and metadata) needed to render an image without needing additional knowledge of the characteristics of the camera.
- Version control scheme: it has a version scheme built into it that allows the DNG specification, DNG writers, and DNG readers, to evolve at their own paces.
- Freely-available source-code-based software development kit (SDK): there are 3 aspects – there is an SDK; it is source-code-based (as can be verified by examination); and it can be downloaded from the Adobe website without needing justification.
- Documented to have no known intellectual property encumbrances or license requirements: there is both a “Digital Negative (DNG) Specification Patent License” which says that anyone can exploit DNG, and a statement that there are no known intellectual property encumbrances or license requirements for DNG.
Converting Raw to DNG
If you’re sold on the idea of adopting DNG, you can download a tool from Adobe called the “Adobe Digital Negative Converter“. As you’d expect, it’s for converting about 350 different raw image formats into DNG in one big batch process. It’s a pretty simple interface as you can see below:
Get the converter from the Adobe website here:
If you use Adobe Photoshop Lightroom, then you will be familiar with the DNG format as it’s well integrated into its workflow. It’s an option you see during the “Import” flow. Simple select “Copy as DNG” and be a bit more patient. I highly recommend Lightroom regardless but this will ensure you keep your entire archive or raw files as DNG from the get go.
Shooting in DNG
A handful of camera manufacturers allow shooting directly in the DNG format. Here’s the full list form the Adobe website as of June 2012:
To DNG or not to DNG?
There’s obviously going to be some debate in the industry when one company positions itself as the industry standard. That said, I think Adobe has made it clear that their intentions are to create an open and non-proprietary format that solves a number of important issues created by the proliferation of camera-specific raw formats. Some opposition is to be expected. At least Adobe doesn’t manufacturer any camera hardware and is relatively neutral in that regard. It helps that they create industry-standard tools like Photoshop.
After hesitating for a couple years myself, I gave it more thought and decided the fear of the “unknown” about DNG can’t be as bad as the unknowns about camera raw file issues of the future. The longer I waited, the more “work” there would be to convert my digital archive. This year, I decided to include DNG conversion as part of my standard workflow. I now convert all my Canon .CR2 raw files into DNG as part of my file import routine. I do not embed the CR2 into the DNG to save space and I never shoot JPEG.
I use Adobe Lightroom so it’s not really an extra step. When I pull down my photos off the memory cards, I choose “Copy as DNG” instead of “Copy”. Yes, it does slow down the import process but it speeds up post-processing and the resulting files should be about 20% smaller overall without any loss of quality or flexibility.
If you’re very nervous about all this, you can have your cake and eat it too as DNG supports the embedding of the full original raw file inside the DNG file. You effectively have both versions but you’ll have huge files.
If there were viable alternatives to DNG I’d bring them up here but there are none. Other than a older attempt to create an “OpenRAW” format, it seems nothing has had the same momentum, backing and cross-vendor support as Adobe DNG. I’ll put my money on open standards with deep corporate backing.
If only Canon and Nikon would give users the option to choose DNG in-camera as done by the likes of Leica and Hassleblad. Perhaps that will come in time.
What do you think? Do you DNG or not? Will you now? Why or why not?