Storage rapidly becomes a problem for a digital photographer. Check out this storage workflow as a way to deal with it!
Photographers love to talk about their editing workflows, but I haven’t seen nearly as much discussion about their storage workflow, which I believe to be just as important. Here is the storage workflow that I recommend (WARNING: lots of acronyms and nerdery in here, check out the long answer if it doesn’t make sense):
- Make two copies locally. I recommend the first thing you do is copy images from your memory cards to TWO hard drives (not to two spots on the same drive). One should be connected using the very fastest connection available on your computer. The second drive can be something cheaper and slower. I also wouldn’t erase the photos from the card until you have them actually written out to both drives. I personally use Microsoft SyncToy to do this for me each night, copying from my fast drive to a slower drive, so I don’t format the memory card in my camera (DON’T delete them from the card using your computer) until that process has happened over night.
- Process on fast copy. Process the photos from the drive connected through the fast connection so that Lightroom, Photoshop, and other processing tools can work well. NAS and USB 2.0 storage is too slow for this. I think an internal SSD is ideal here, although direct attached storage (DAS) solutions like a Drobo Mini or G-Technology G-RAID can be sufficiently fast as well for those who have more being processed at once than can fit practically on an SSD. Next to internal SATA, Thunderbolt and Thunderbolt2 are both good options (can be faster with DAS RAID), eSATA does pretty well, and USB 3.0 is just barely enough to make you not want to scream at the computer in between each photo (check out speed comparisons in the long answer). New things like USB 3.1, Thunderbolt 3, and M.2 drives are coming in this space very soon (end of 2015), promising even better options! So stay tuned.
- Move to long-term storage. Once the processing of a shoot has been completed, use Lightroom to move the photos from the faster/expensive/smaller drive to a slower/cheaper/larger drive. The 3-2-1 backup strategy that I highly recommend, says that this long term storage disk should be different from the second, slower drive used in step 1 (2 copies on local but physically different disks). But if you use a NAS with RAID then that is really happening inside the NAS. Therefore, I highly recommend the use of a NAS device here using RAID 5. The Drobo 5N or the Western Digital MyCloud Expert Series are two very good options.
- Backup to offsite. Now for the “1” portion of the 3-2-1 backup strategy – one copy offsite. I wish cloud could be the simple choice for everyone. But, pretty much no matter which cloud storage provider you choose, backup to the cloud is going to be SLOOOOOOW (my cloud backup just recently completed and it took 8 months to do 3TB). ISPs can tell that you are talking to a backup provider and they slow that portion of your Internet down. I still recommend setting one up, probably backing up from your long-term storage, but you may have to be pretty selective on what you backup. If you are one of those lucky photographers generating 4TB of data a year (or more) you will probably want to drop a hard drive off at the house of family or friend periodically.
- Keep as little as possible. This one is really hard to do. I totally understand that as a photographer you want to spend as little time as possible thinking about this, but the better you can contain your storage use, the easier it will be. Consider incorporating some “space saving” techniques into your workflow. For example, I do not have more than 1 copy of any of my exports from Lightroom. I have the original photo backed up in 3 spots, along with the Lightroom Catalog, so if I lose that image I can easily export it again.
It is really easy to think that storage is cheap so you don’t have to conserve or worry about it. I know I have heard that statement over and over from many a photographer. But the reality is that it does get hard to maintain is your library grows. The average hobbyist photographer generates between 200GB and 400GB of data per year – which is actually a very manageable rate. However, the numbers get much larger for professional photographers who on average generate between 1TB and 4TB per year. Those numbers don’t reflect much use of video, which of course is significantly larger and makes this problem much worse. Bottom line is that storage can really become a serious problem VERY quickly.
To make matters worse, if you have your precious photo library in a single spot, you WILL lose it at some point. Therefore, I highly recommend the 3-2-1 backup strategy prescribed by Carnegie Mellon to ensure that you keep all of those valuable shots. You should have 3 copies of each photo, 2 of them on local (i.e. in your home or office) but physically separate drives, and 1 offsite (i.e. cloud or friends/family house).
That’s a great strategy to make sure you don’t lose your extremely valuable photos, but it means that you have to multiply your rapidly expanding storage needs by 3! So what is a photographer to do? I thought I would walk you through my recommendation on a storage workflow similar to how photographers share their editing workflows.
Step 1: Two Local Copies
This first step is about keeping your digital photos safe from being lost when that ticking time bomb of a hard drive (HDD and SSD) randomly decides it is done. You want to get your 3 copies of the photo made as soon as possible, but being practical about it I recommend only one of them actually be manual. To me, unless the backup strategy is happening automatically, it isn’t going to happen.
So, the first step in my recommended storage workflow is to copy the images from your memory card to a drive connected to your computer. As soon as you have done this you have technically met the “2 copies on different drives” requirement of the 3-2-1 backup solution because one is on your computer’s drive and one is on the card. But you don’t want to keep your cards around with old photo shoots, so I’ll recommend when it is time to put the card back into service in a moment. First, some information about this drive you are copying the photo to here.
I recommend you manually copy your images from the memory card to a drive with enough space to hold at least 2-3 photo shoots, connected to your computer the very fastest way possible (see details in the next step). For PC users, I highly recommend an internal SSD like the super speedy Samsung 850 Pro 512GB ($260) for this “fast” storage. If the images from 2-3 shoots won’t fit there, then use a 7200 RPM HDD like the HGST Deskstar 6TB. For Mac users, get the SSD or Fusion drive option when you order your Mac. If your “fast” storage needs are bigger than the drives in your computer, for either Mac or PC you could also go with direct attached storage (DAS) devices like the Drobo Mini ($660 for 4TB) or a G-Technology G-RAID ($330 for 4TB). Network attached storage (NAS) solutions, where an ethernet cable is the only way to really talk to them, is NOT fast enough for this storage.
OK, so now you have your shots on your memory card and on your “fast” storage. Yeah! They are in two places and you can feel better that you won’t lose them. However, I don’t recommend keeping the images on the memory cards for more than a few days. You need those cards available for future shoots. So you need to put the images on some secondary “slow” storage to get that memory card back in service. For this, I recommend some form of automated copying that happens every night from the “fast” storage off to some physically different, larger/slower/cheaper storage.
This “slow” storage should be much larger as it will also play a role in my storage workflow recommendation for long-term storage. You could use internal or external hard drives for this (avoid Seagate 1.5TB and 3TB drives as they have had very high failure rates), although professional photographers will quickly outgrow the 4TB, 6TB, or 8TB drives that are available today. Alternatively, if your storage needs are larger, I recommend for both PC and Mac using a network attached storage (NAS) device like the Western Digital MyCloud Expert Series. The 4-Bay model can provide 8TB ($700), 16TB ($1,000), or 24TB ($1,500) of fault tolerant storage – meaning a hard drive can fail within the device and you won’t lose any data! The Drobo 5N is a similar kind of solution with 5 bays that supports multiple configurations including 6TB ($870), 10TB ($1,100), 20TB ($1,500), and 27TB ($2,300) to show a few.
As a PC user, I have personally set up the free Microsoft SyncToy 2.1 combined with the Task Scheduler built into Windows to copy files from my fast drive over to a backup drive every night (listen to my Photo Taco podcast about SyncToy here). I have seen recommendations for using ChronoSync ($40) or a combination of the crontab and rsync utilities (free but very geeky) built into OSX for Mac to do this.
To summarize a bit, my recommendation is to copy (do not move or delete from the memory card using your computer, only format it in the camera) the images from your shoot onto “fast” storage and then have an automated process copy them from that drive to different “slow” storage nightly. As soon as this nightly copy has happened, get your memory card(s) back into service by putting them into the camera and formatting them so they are ready for the next shoot.
Step 2: Process From “Fast” Storage
Now that you can rest a little easier because you have 2 copies of your images on two physically different storage, I recommend you do your image processing from your “fast” storage. I outlined some of my recommended solutions for the actual drives to be used, but promised more detail on what I meant by the fastest connection to your computer. So let’s get into that detail now.
The RAW files from my beloved Canon 7D Mark II are on average about 20MB (megabytes), which I will use to illustrate the need to have images on “fast” storage. When you pull up an image in Lightroom to process it, the computer has to read it from the storage as well as any of the settings you have specific to that image from your catalog.
Below is a chart that shows the maximum speed for the most common ways you can connect storage today. There are two speed measurements shown in the chart as well as how long it would take to load one of my 20MB RAW files over that speed. Bandwidth is usually measured in megabits per second, but files are usually measured in megabytes. Your Internet speed is usually measured in megabits, while computer files like your photos are measured in megabytes.I am showing both in hopes it can be the least confusing and you can better identify with the speeds.
Looking at that chart is should be obvious that Thunderbolt 2 is by far the fastest connection available as of the writing of this article (Thunderbolt 3 and USB 3.1 will both offer more speed!). However, there are three things to remember here:
- These numbers are the theoretical maximums. They are what the engineers targeted in their designs as they created the connection type. These numbers are NEVER fully realized in the real-world. Thunderbolt 2 has been seen to go as fast as 1375 MB/s in a real-world test rather than all the way to 2500, only about 55% of the theoretical maximum speed. Not all of the different connection types suffer from as large a gap in real-world vs theoretical limits, and some are worse (Wireless has always functioned WAY under the theoretical max). But none of them actually hit the theoretical maximum.
- We have another component here that is a huge part of the speed equation, the speed of the drive itself. If you put a single 7200 RPM SATA drive into an enclosure that had a Thunderbolt 2 connection, you will only going read speeds of about 130MB/s even though the connection will allow as much as 1375MB/s to go through it. Think of it like a water pipe. The Thunderbolt 2 connection allows 1375MB worth of data per second to go through it, but the 7200 RPM hard drive can only supply 130MB worth of data in a single second. In order to get more data into the Thunderbolt/USB3.0 pipe, you have to use RAID. RAID 0 (zero) and RAID 5 both combine the speed of the drives together in order to get more data in the pipe. With RAID 0 you can get 400MB/s, RAID 5 can combine 8 drives to fill up the Thunderbolt 2 connection entirely!
- Cost is a factor. A RAID 5 array of high performing SSDs is capable of filling the pipe of a Thunderbolt 2 connection, but such a configuration will also cost a bundle. The price to performance is a key consideration to remember. Wouldn’t you rather spend money on better lenses than better hard drives?
The point to take away from all this nerdiness here is that connecting to network attached storage (NAS) or external USB drives over USB 1.1 or 2.0 will noticeably slow down how quickly you can process your photos – especially when you are going between photos. If you are using a Mac, connect a fast external drive or RAID array through Thunderbolt (1 or 2) is a great option (see above for specific recommendations). For a PC, use either USB 3.0 or eSATA to connect your external storage (again see above for specific recommendations). I strongly recommend against editing photos on storage connected through either a wired or wireless network connection.
Step 3: Move To Long-Term Storage
If you are following my storage workflow you have now taken good steps towards keeping enough backups of your images to keep them safe, and you have used “fast” storage while processing to make that go well. Now it is time to move them off of that “fast” storage to make room for processing future photo shoots. The 3-2-1 backup strategy says you should have 2 copies of your photos on local storage (drives located in your home or office) at all times and that each copy must be on a physically separate drive. So how do you do this when you have your storage needs growing as much as 4TB a year?!?!
Before I get to that, let me outline my current solution, fit for a hobbyist generating much less data per year. I am using 3 internal drives to get this far in my storage workflow. I have one SSD that I copy my photos to from my memory cards. Each night an automatic process copies them from the SSD to a 3TB internal drive (at which point I have 2 copies on different drives). When I am done processing the photos in Lightroom, I use Lightroom to move them from the SSD to a third 4TB internal drive for long-term storage (still keeping 2 copies on different drives). I use Lightroom to move the files because I only want Lightroom pointed at 1 of the images, not 2 or 3. You could just as easily have the second and third drives be connected externally. Remember, the speed of these other two drives is not nearly as important because you aren’t working on them in Lightroom while they are there (or at least would be very rarely).
The 3-2-1 backup strategy doesn’t just cover drive failures, it is also meant to protect you from yourself. Having your photos on 2 different drives keeps them safe should one fail. Or, if you take the advice I offered in Step 1 and use a NAS for long-term storage, then the image is as safe as is practical to do from individual drives failing thanks to how RAID works. However, what if for some reason you accidentally delete a file from your NAS? The NAS looks like a single drive to Windows or OSX, so if you delete the image, it is gone. From all the drives in the RAID array. It is just as gone as if a drive failure had taken it out. Something to think about.
I am saving my pennies to buy a NAS solution because I think that is much better for long-term storage. I hope that my current storage solution holds out long enough that I can continue investing in lenses right now. My 3 drives installed in my computer are meeting my hobbyist photography needs, although my long-term storage drive is already 1/3 of the way full and it won’t work forever. I could replace it with a 6TB or even 8TB drive later, but I would rather get a NAS in place that has the ability to have 20TB+ of storage.
Once I have a NAS, the question then is what I will doing about the second copy of the image? The 3-2-1 backup strategy doesn’t just cover drive failures, it is also meant to protect you from yourself. Having your photos on 2 different drives keeps them safe should one fail. But once I have a NAS for long-term storage, then the image is as safe as is practical to do from individual drives failing thanks to how RAID works. However, what if for some reason you accidentally delete a file from your NAS? The NAS looks like a single drive to Windows or OSX, so if you delete the image, it is gone from every drive it was on in the RAID. It is just as gone as if you had them on a single drive. I think that I will personally take the risk that I won’t somehow end up deleting my photos by accident. After all, I am a hobbyist. If I were to delete some old photos I would be sad, but my livelihood wouldn’t be threatened. Plus, there is the “1” part of the 3-2-1 backup in the next section that is there to cover me too. My practical recommendation then is that if you use a RAID 5 NAS for your long-term storage, you don’t need another copy on storage in your home or office, just be careful about deleting anything from that NAS! But you’ll have to decide how much you can trust yourself. If you don’t, then you may need two NAS devices with the same amount of storage and copy images between them.
Step 4: Backup Offsite
As I just alluded to, the “1” portion of the 3-2-1 backup strategy says your photo needs to also be stored somewhere outside of where your other storage is located. If you have 3 drives connected to your computer at an office or at your home, you need to have a copy of the image not at the same physical location. The idea here is that if your house or office were to be flooded or burned to the ground, you have your 1 copy that wasn’t there. Or even if your home/office had a theft of all your equipment, you have your 1 copy that they couldn’t get to. It is a good idea, one I recommend doing.
You could do this by having 2 copies at your office/studio and 1 at home. You could have 2 copies at home and 1 at the house of a friend or family member. Or you can have 2 at home/office and 1 in the cloud. The challenge with this offsite storage is automating it, and again my feeling is that if it isn’t automated it isn’t going to happen.
Every major technology company seems to be vying to get you as their cloud storage customer right now. Many are offering free storage, but not in the quantities photographers need. You have solutions from well known companies like Dropbox, Amazon, Microsoft, Apple, and Google to name a few. But you need to check out how much it costs for the huge amount of storage you need to backup ALL of your photos. Many of them become REALLY expensive after 1TB. You have probably heard that many companies have recently offered free, “unlimited” storage, but there are caveats as to the file sizes allowed that make that free storage not an option. I have decided personally to use Backblaze for my cloud backup, but Crashplan and others are good options as well.
The good thing about nearly all of the cloud storage providers is that they offer software you can run on your computer to automatically backup your photos. The bad news is your ISP won’t like you doing that and they will slow down your Internet connection when your computer talks to that cloud provider (they keep the rest moving fast, they only throttle it when the data is going out to the cloud storage provider). I just recently had my 3TB backup to the cloud complete, and it took 8 months because of the ISP throttling my connection to Backblaze.
Even with the evil ISP throttling slowing down your ability to backup to the cloud, I think it is one of the better options because it can happen automatically. How great would it be to know that soon as you put your photos out to your long-term storage (NAS or external drive), it is also going to get backed up to the cloud as fast as your ISP will allow! I know it really helps me sleep better at night. However, if you are one of those professional photographers generating 1 to 4TB of data in a single year, your backup will feel like it will never finish. Even worse if you do much of anything with video. I still recommend doing it, but if that is your situation you should consider putting all of your photo shoot data onto a drive every 6 months and putting it at the house of a friend or family member. Or, you can decide to only backup certain portions of your shoots, like the flagged images rather than all of them, to lessen how much data you need to go to the cloud.
Getting the backup of your photos offsite isn’t very easy, but you need to do something here or a catastrophe that takes out your equipment becomes even worse with you losing all of your photos entirely.
Step 5: Keep As Little As Possible
This is mostly a continuation of Step 4, attempting to make it more practical. This is something I am currently failing miserably at myself. Given how hard it is to get the offsite storage to work well, consider adding to your processing workflow steps that can reduce the amount of space needed to backup your shoots.
It is a tough problem here because the post-processing capabilities are ever improving, and a shot that is not worth saving today could turn out to be good enough to use tomorrow. Plus, I don’t routinely go back through my photos and actually delete the ones that I didn’t pick as keepers. That would be additional time spent on my shoots, time I would much rather spend on shooting more.
Still, I think you should spend some time thinking through any practical steps you could add to your processing workflow to limit the number of shots you keep long-term. I have thought for some time I need to do a better job of using the “reject” flag in Lightroom on shots I am confident I don’t want around (i.e. eyes closed on a person) and delete all of those from the disk when I am fully done processing the shoot. I am confident there would be a lot of those and I would really save myself a lot on both the local and cloud storage.
I hope you have found my storage workflow helpful. Hopefully you have considered how it is your are currently using storage and have enough information here to get yourself into a better situation than you are in today. Be sure to stay tuned to improvephotography.com and the podcasts on the network as new options become available, some exciting things are coming very soon that are likely to change my recommendations.
About the Author
Jeff Harmon is a hobbyist photographer living in Salt Lake City, Utah. He loves to photograph anything he make time for, but mainly shoots landscapes. He also does some family portrait shoots. Check out his portfolio and find other contact information at http://jhpics.zenfolio.com.