TDWG Conference - Standardized Digital Imaging and Archiving Procedures

I have recently had several requests for information regarding the new Herbarium imaging stations.

I am in the early stages of preparing a step by step guide to building one, but as you can see from the images below, it's a simple design (and fairly easy to set up, once you have the right parts).

I hope to present this, along with a more comprehensive discussion about imaging and archiving procedures at the upcoming TDWG Conference in New Orleans.

Side view

Front view

In the Herbarium

I will be posting all of this information online. Please stay tuned.

1 Billion Gigabytes = 1 Exabyte

This tidbit comes to me by way of the Proverbial Lone Wolf Librarian:

"But there are thousands of TIFFs on that server!" - (Deleted)

Did you feel that? I just deleted 119,570 TIFF files off of one of our archive servers.

5.1 TB just vanished.



(Don't worry -they have been moved offline onto tape)

Obsolete File Formats

With regard to my earlier post regarding the archiving of RAW files or TIFFs, I now have a very good case study of what happens when you use a proprietary file format.

The Virtual Herbarium at one point used a proprietary file format (that shall remain anonymous) to display high resolution images online. The images are excellent, but there are downsides: the files are expensive, they cost money to generate and they are large, eating server and tape backup resources; the anonymous files require a plugin to view them online; and being proprietary, it makes them harder to share with others.

Our new online image viewer, FSI by NeptuneLabs, accesses our Jpeg derivative files. It's a Adobe Flash based viewer, and what computer these days doesn't come with Flash (except iPads and iPhones, but who uses those anyway)?

So, we have over 200,000 files eating 291 GB on our image server that aren't being used. They are now backed up and will be stored offline (probably next to my 8-track tape collection).

We still have all the images archived as RAW files (and TIFFs) and can therefore migrate the images to new file formats if need be. And isn't this the purpose of an archive?

Archiving RAW Files

I am beginning to wonder about digital archiving practices, specifically when it comes to RAW file formats.

Currently, The New York Botanical Garden Herbarium archives RAW files (Canon's CR2) and a Production Master File (TIFF). Jpegs are derived from the Production Master file for access in the Virtual Herbarium.

I am not certain what role the TIFFs play and why we should be archiving them. They consume a lot of server space, and therefore cost a lot of money to store. Isn't the RAW format sufficient for the purpose of archiving?

The traditional belief I hear a lot is that since RAW formats are proprietary to the camera manufacturer they are subject to the whims of the market place and may not be supported in the future. Aren't all archives subject to technological change? There is currently a collection of 3000 35mm slides in a closet upstairs. A slide archive is not very much good without a slide viewer and hardly convenient or easily accessed by the public.

It is commonly believed that TIFF is a more broadly supported format and that the decoding of TIFFs is well documented. Of course, DNG, Adobe's RAW format could solve this problem. It is well documented and the instructions for decoding the files are readily available.

I guess, just like the slide collection, should the day come that RAW formats are no longer supported the images will have to be migrated to the next format.