Showing posts with label RAW. Show all posts
Showing posts with label RAW. Show all posts

Friday, July 29, 2011

Adobe Lightroom 2.7 Error - Unable to Export

I am in the final stages of the first phase of restructuring the Herbarium's digital image archive. Of course, this is when I hit a snag. I am sorting through a 2 terabyte mirror of the first archive structure, built who knows when. It contains TIFFs, NEFs, CRWs, and Kodak DCS 660 TIFF RAWs. There were 161,658 files hidden in a very opaque folder structure consisting of 6,201 folders.

Using Adobe Lightroom, I sorted the files into collections according to type: TIFF, RAW, and KodakRAW (the Kodak TIFF RAWs were tricky, because they show up as TIFF files, so I segregated these from the RAW files).

Out of the 60,396 TIFF files, I made a collection of 21,142 TIFFs for which there were no corresponding RAW files and exported them to an external hard drive. These will be returned to the archive, along with the RAW files. The remaining 45,564 TIFF files will be backed up on to tape and deleted from our servers.

I successfully converted the Kodak TIFF RAW files into DNG.

Now, I just have to move the RAW files. I could do this using a Windows search and just moving them on the sevrer, but I want to unify the file formats, so I figured I would export them from Lightroom as DNG. Unfortunately, and not surprisingly, I encountered the following error:

Unable to Export
An internal error occurred: XMP parse threw an unknown exception

Now what?


Monday, June 13, 2011

DNG Archive

Today I am taking the first step towards tidying up the New York Botanical Garden Herbarium image archive.

I have made the leap. I will no longer be producing TIFF Production Master files. I will only be archiving DNG Archive Master files.

Now the real work begins, sorting through all the TIFF files and moving them offline.

Friday, May 27, 2011

How To: Searching Multiple Files In Lightroom

I am going through ten years of archive photos at The New York Botanical Garden Herbarium and creating Adobe Lightroom catalogs in preparation for a massive file type migration.

Catalog of CD Archive

We have over 800,000 images taking up about 27 terabytes of space. Many of the files are duplicates in various formats. There are about 400,000 legacy small derivative files (500GB) in a format that is no longer supported by our web client. While they are duplicates, we will not throw them away, they will simply be taken offline and put on tape. There are also about 300,000 TIFFs which are production masters. As discussed in an earlier post, these files are rarely if ever accessed and there are corresponding RAW archive masters. These TIFFs will also be moved offline on to tape. This will free up close to 8 terabytes.

The challenge I have created for myself, is to have at least one of every image remain on our server, so I am looking for files that do not exist in RAW format (there are about 3000 files - 1% - that were not archived properly). So, how to find 3000 files out of 300,000?

I tried a Windows search of the image server using a semicolon delimited list. Unfortunately, this produced poor results for reasons I can not explain. So then, in my newly minted Lightroom catalog, I tried a filename search "Contains" semicolon delimited list.

Semicolon delimited search

This was a test. I knew those files were in there. So, I tried a Filename "Contains" comma delimited list.

The winner

Monday, April 4, 2011

Digitization Special Interests Group Meeting - METRO.org

Digitization Special Interests Group

Last week I attended Metro's Digitization Special Interests Group meeting. The discussion panel included  John Mignault from The New York Botanical Garden, Andrea Buchner from the Gruss Lipper Digital Laboratory at the Center for Jewish History, and Jennifer Vinopal from New York University.

It was a very informative meeting and there are some take away thoughts posted on Metro's tech blog.

Jennifer Vinopal mentioned that at NYU they make a distinction between archive and preservation and I find the distinction to be helpful. The archive guarantees bits. You put bits in, you get the same bits out in the same bit order. There is no guarantee of the viability of the file format. Preservation, she said, is an institutional commitment that requires human vigilance. 

I like that. It makes me feel brave.

Friday, March 25, 2011

Preview RAW Formats in Windows 7 and XP

Here's a useful tool for anyone who wants to preview RAW image files in Windows 7: FastPicture Viewer Professional.

Once you install the trial, you will see you will be able to see your RAW images as thumbnails in Windows.

Digital Photo Archiving Practices

As I contemplate what to do about the growing and vastly underutilized collection of TIFF files residing on our server, I thought it would be a good idea to review the goals and intentions of an image archive. There are a number of great resources online that have informed my observations, including the National Archives Technical Guidelines for Digitizing Archival Materials for Electronic Access:   Creation of Production Master Files - Raster Images, the Federal Agencies Digitization Guidelines Initiative, and ASMP's Digital Photography Best Practices and Workflow.

While I may not follow the recommendations to a tee, the following is my interpretation of the similarities between the three sources on the subject of file types.

Archive Master
The purpose of an Archive Master image is to preserve the original recording. All subsequent images are derived from the Archive Master. The image processing and file format of derivatives is determined by their intended use.

Production Master
A Production Master image is a rendition of the Archive Master. The Production Master is intended to show the image when reproduced in ideal form. Image processing may be applied to Production Masters.

Derivative
It is common practice to save a compressed version of the Production Master to serve as an access Derivative. Derivative images are usually saved as 8-bit Jpegs. For ease of reproduction, Derivatives may be stored as sRGB images.

Thursday, March 10, 2011

Obsolete File Formats

With regard to my earlier post regarding the archiving of RAW files or TIFFs, I now have a very good case study of what happens when you use a proprietary file format.

The Virtual Herbarium at one point used a proprietary file format (that shall remain anonymous) to display high resolution images online. The images are excellent, but there are downsides: the files are expensive, they cost money to generate and they are large, eating server and tape backup resources; the anonymous files require a plugin to view them online; and being proprietary, it makes them harder to share with others.

Our new online image viewer, FSI by NeptuneLabs, accesses our Jpeg derivative files. It's a Adobe Flash based viewer, and what computer these days doesn't come with Flash (except iPads and iPhones, but who uses those anyway)?

So, we have over 200,000 files eating 291 GB on our image server that aren't being used. They are now backed up and will be stored offline (probably next to my 8-track tape collection).

We still have all the images archived as RAW files (and TIFFs) and can therefore migrate the images to new file formats if need be. And isn't this the purpose of an archive?

Thursday, March 3, 2011

Archiving RAW Files

I am beginning to wonder about digital archiving practices, specifically when it comes to RAW file formats.

Currently, The New York Botanical Garden Herbarium archives RAW files (Canon's CR2) and a Production Master File (TIFF). Jpegs are derived from the Production Master file for access in the Virtual Herbarium.

I am not certain what role the TIFFs play and why we should be archiving them. They consume a lot of server space, and therefore cost a lot of money to store. Isn't the RAW format sufficient for the purpose of archiving?

The traditional belief I hear a lot is that since RAW formats are proprietary to the camera manufacturer they are subject to the whims of the market place and may not be supported in the future. Aren't all archives subject to technological change? There is currently a collection of 3000 35mm slides in a closet upstairs. A slide archive is not very much good without a slide viewer and hardly convenient or easily accessed by the public.

It is commonly believed that TIFF is a more broadly supported format and that the decoding of TIFFs is well documented. Of course, DNG, Adobe's RAW format could solve this problem. It is well documented and the instructions for decoding the files are readily available.

I guess, just like the slide collection, should the day come that RAW formats are no longer supported the images will have to be migrated to the next format.