How to manage your data in Lightroom securely and efficiently

In Post-processing, Uncategorized by Rick McEvoy8 Comments

This article arose out of problems I have been having with Lightroom and Photoshop, the slow speed of my PC and ultimately error messages when trying to save files to my external hard drive.

All these problems have been fixed, so there is a happy ending.

In this article I will explain how I manage my data in Lightroom, and how I recovered from a potentially serious hard drive and computer failure without losing any data.

Were the issues I was having with Lightroom software related? If you make it all the way to the end you will find out.

But I will say this – how I have set everything up helped me to quickly recover and get back to work.

Let me ask you a question

Have you ever had that moment when you fear that you may have lost all your data? I had that feeling last week.

I felt sick. If I could not fix this I was worried that I would not be able to work commercially without significant investment in new hardware, and having done that I was not sure if that would sort the problems I was having.

I thought that the hard drive which holds all my data had failed, and that my PC was about to grind to a slow painful end.

Happily, I managed to sort the problems without losing a single image.

There are a few strands to this that I would like to write about here, which will hopefully help people manage these huge amounts of data we all have these days.

I am not saying this is the best way to do this, this is my way, and I have managed to survive a hard drive becoming faulty without losing any data whatsoever.

The areas I will write about are

  • Lightroom
  • File Structure
  • Simplicity
  • Lightroom Catalogue
  • Importing into Lightroom
  • Backup
  • Maintenance
  • When something goes wrong
  • Future proofing
  • What went wrong
  • Who fault was it?
  • Summary

What is Lightroom then?

To start at the very beginning, Lightroom is the most popular software for organising, editing and outputting digital photographs. Lightroom is produced by Adobe, and has been around since 2007. I have been using Lightroom since the very beginning, with version 1.0, which you had to buy on a CD which was posted to you in a box!

Lightroom is designed for photographers, unlike Photoshop which has an amount of graphic design in its origins.

There is a strong possibility therefore that as you are reading this you are a Lightroom user.

  • I use Lightroom Classic 7.2 on a Dell Windows 10 PC.
  • I do all my image processing in Lightroom, with the exception of final cleaning up of images.
  • I probably spend 95% of my time working in Lightroom, and 5% working in Photoshop.

I do not use any other software at the moment.

File Structure

I have circa 60,000 images in Lightroom. How you organise your files can seem a daunting challenge.

If you are starting from scratch with Lightroom you can just let Lightroom organise your files for you. In some ways I wish I had done this, but my file structure has evolved over the years.

Simplicity

I now have a simple file structure. This is it. I know where everything is, and more importantly where everything should be!

Lightroom Catalogue

This is an important point. All my photos are in a single folder. My Lightroom Catalogue is in a separate folder.

I use one Lightroom Catalogue. The Lightroom Catalogue has a file structure which is replicated in windows.

The Lightroom Catalogue references the photos which are in a separate place.

Before the problems with my PC and Lightroom I used to have my images in a folder on my external hard drive, and my Lightroom Catalogue on my PC hard drive.

Adobe contacted me and enquired about my problems – oh the power of a hashtag! They assigned me a specialist adviser who started me off down the road to recovering the situation.

Adobe advise that the Lightroom Catalogue and the actual photos are on the same drive.

Mine are now. This helps Lightroom work more quickly.

Check out this article with advice on how to make Lightroom work more quickly.

Importing into Lightroom

When I import into Lightroom I make a duplicate set of images, which are a backup. These are located on my PC hard drive.

Every month I backup everything to an external hard drive. Everything on my PC hard drive and my external hard drive. Until I have done that I leave the original data on the memory cards which are stored in my desk. I have never had to use one of these import backups before until now, but I like the security this gives me.

And yes, I have enough SD cards to last a month – they are not expensive, and I am not one for taking lots of images.

Backup

This is what saved me. My set was as follows

  • Lightroom Catalogue on my PC hard drive.
  • Import backups on my PC hard drive.
  • Photos on my external hard drive.
  • Cloud backup using Backblaze (not expensive and always running in the background).
  • Once a month I copy everything to another external hard drive and store this drive off site.
  • Documents sit on my hard drive and in the cloud. Not ideal.

Now my Lightroom Catalogue and photos are on the same external drive.

All my documents are in a single main folder. Working files are on the cloud – I use Office 365 for this, which allows me to jump from iPad to iPhone to PC working on the same documents.

Like this article here.

I now have an additional external hard drive. I am going to use this to introduce another backup, which will be fortnightly, and alternate with the other backup. Basically, I will have a fortnightly backup stored off site.

Maintenance

This was part of the problem.

I deleted nearly 200GB of data from my PC hard drive.

I deleted more than 15 programs that I no longer use.

Although I use one main Lightroom Catalogue I had legacy Lightroom Catalogues on my hard drive, and to be honest I am not sure why they were there. I have moved them all onto an external hard drive. I had copies of Catalogues I had created to try and sort the problems I was experiencing.

It was all a bit of a mess and I was in danger of losing the plot!

Now I have one folder containing all my photos, and one Lightroom Catalogue. And no photos anywhere else.

And no other catalogues anywhere else.

And believe me – this make backing up and maintenance so much easier.

I had a Lightroom Catalogue with no photos in it that took up 120GB of hard drive space – there was clearly something wrong there – I have no idea what was going on, but this problem has been removed.

When something goes wrong

Panic.

No seriously don’t (OK I admit  I did). But before something goes wrong have your plans in place to deal with any problems, should this happen.

I have actually written down my data management plan, what sits where, what gets backed up and when.

I know this sounds incredibly dull but spending the time getting yourselves organised whilst things are working fine will give you reassurance, and help massively should things ever go wrong.

Future proofing

If you are taking any amount of photos you will fill a PC hard drive. And when you fill a PC hard drive it slows and slows until it is painful. I have been here once before, and it is not pleasant.

My external hard drive has 2 TB of data on it, so it is more than half full. But there is another massive 2TB to fill.

OK – a 4TB hard drive does not actually have 4TB of capacity – mine have 3.6TB which is ridiculous!

But these things are not expensive – I had to get a new Western Digital 4TB external hard drive, which I picked up from PC World for £99.99.

Less than £100 for a hard drive to store all the photos I have ever taken?

What went wrong

This is really the origins of this article.

  • Lightroom had been getting slower and slower.
  • The latest update, 7.2, did not speed up Lightroom at all. It made it worse.
  • I was beginning to get errors in moving files.
  • I was beginning to get errors importing files.
  • Photoshop was slowing to a painful grinding halt.
  • Word and Excel would open up eventually, but files would not open.
  • Everything was spiralling into slow painfulness.

The final thing that told me that something was not right was this. After editing a photo in Photoshop, I could not save it back into Lightroom.

I had to manually save it to my desktop, rename the file in Windows Explorer and then import into Lightroom.

And Lightroom was giving me all sorts of weird messages.

And then I noticed that Lightroom was telling me that all my files were missing. Yes – all my files were missing.

I was resigned to the fact that I was going to have to buy a new PC, and the spec I was looking at would cost me £1500. That is without a monitor.

And the PC I have is only 18 months old.

And recent articles suggested that upgrading your PC might not even speed up Lightroom.

I was also considering upgrading the RAM to my PC. I did not do this as I was not convinced that this was the problem, and I was right. I would be spending £200 to boost my RAM to 16GB to take advantage of the advice from Adobe about the Lightroom 7.2 release. To benefit from this release, you need a PC with a minimum of 12 GB of RAM.

But in the back of my mind I knew there were things that I could do.

This is what I did.

  • Copied all my photos to a new, external hard drive. This was a simple copy and paste exercise using Windows Explorer.
  • Copied my Lightroom Catalogue to the same external hard drive that my photos were on.
  • Linked the Catalogue to the pictures. A straight forward task.
  • Moved everything else to another external hard drive (I was not ready to delete everything there and then in case this didn’t work).
  • Deleted loads of things I did not need, old programs, duplicate files, backups etc. And when I say lots I mean lots.
  • Disconnected the other hard drives leaving just the new external hard drive connected.
  • This took a number of days to do, as I was moving large amounts of data around.
  • Rebooted my PC with just the brand new external hard drive connected.

And held my breath and waited.

And did it work?

Yes. All is fine and lovely again. Lightroom works. Photoshop works. My PC is running much quicker. Word and Excel are fast, responsive apps.

And my PC fires up much quicker.

And I am more organised than I was.

There were some files missing from Lightroom, but I managed to recover these from an import backup folder and reimport them into Lightroom.

Whose fault was it then?

Mine. Not Adobes. Not Dells.

I have proved that – all these that things I have done (one of my favourite songs by the way) have got Lightroom and Photoshop working how they should. I have not done anything to the actual software. There have been no Adobe Creative Cloud updates whilst I have gone through this pain.

So, this one is down to me, and I must say than you to Adobe for getting in touch with me and assigning someone to help me.

Summary

The problems I was having with Lightroom have been largely dealt with. By taking a step back and thinking about how I was working, carrying out some much-needed maintenance and simplifying how I organise my digital data I have managed to get Lightroom back to where it should be.

Sure, it has the odd wobble, and could be faster, but Lightroom and Photoshop are working for me like they did a couple of years ago.

I am enjoying using Lightroom again.

And I have managed to simplify my file structure and backup processes.

And all the time there was the cloud backup which I could have fallen back on if needed.

When I get a new PC, which I will at some point, it will be a straightforward task to move my data over.

My PC has a 1TB hard drive, with 700GB of space!

I hope that this article has been helpful, and would be delighted to answer any questions you may have.


About the Author

Rick McEvoy

Twitter

I am an architectural photographer based in the lovely county of Dorset in England.I have worked in construction all my life, so combining my knowledge of buildings with my photography naturally led to down the architectural photographer route. I use Canon DSLRs and lenses (no film cameras). I specialise in architectural photography – well anything to do with buildings, and extend this to industrial and commercial photography which have similar requirements – stationery subjects, no people, no animals. I also enjoy landscape and travel photography.My dream job is photographing buildings in nice places, which I am still working on. My Website, and I have a daily photography Blog.

Comments

  1. Rick, Many thanks for sharing your experience. Question: does your LR catalogue file structure exactly image your external disc file structure where your raw images are stored?

    1. Author

      Victor

      Thanks very much for getting in touch. I created the file structure and folders using Lightroom, which adds the folders as they are in Lightroom so yes – they are exactly the same.

      With regards from England

      Rick

  2. This is one of the most poorly written articles I’ve ever read. The grammar is so bad it detracts from the point of the article-so much so that I had to stop reading. Plus, did you really delete 200TB of data? Doubtful…

    1. Author

      Jefferson

      I am sorry that you did not enjoy reading my article, but thank you for taking the time to leave a comment.

      It was actually 200GB, not 200TB.

      Regards

      Rick

  3. Managing your Lightroom catalogue from the beginning is so important so kudos for the article. I never managed my files properly and ended up with 500gb of mess in my LR catalogue – Something which I still haven’t found the time to sort out!

    1. Author

      Hi and thank you very much for your comments. All I can say is that I feel a whole lot better now that I have sorted everything out, and that Lightroom is running so much better than it has in a long time!

      With regards from England

      Rick

Leave a Comment