PDA

View Full Version : The perfect storage quest



Pietro Mingotti
06-16-2014, 12:21 PM
Hello everybody!

Writing from Italy here, so I hope my English is good enough for opening a techincal topic like this.

first of all: of course I've used the search function, and even though I found some interesting posts and answers, they didn't really fit the question I'm going to ask. So please, I know for the most advanced users it may be a no brainer, for me the following situation is becoming a huge problem...and I'm sure I'm not alone.

Ok, so here it is: I've been working with photography and film making for over 5 years now, which means that my first works where (and still are) stored on tape; I actually have a closet full of my Master tapes etc. The it came the time of the DSLR videomaking and in general, of the digital data stream. I've been working with several 2/3 TB external hard drives for many years, dividing them in categories like "work-in use" (and its backup), "work storage" (and its backup), "Scratch Disks", "Personal", "Stock", "Computer Backups"...etc. I perfectly knew that a horrible day would have come. That day where I start getting old (lol) and getting confused of which file goes where, and if I did make a backup or not, and when one of the hard drives gets full and I have to swap projects between hard drives waiting to buy a new one..... yes. A HUGE mess.

So I finally came up with submitting to the idea of investing money on buying a large array of units to connect in RAID (yes, I should have done it before...), and also the time has come to start buying some SSD drives. I've been working with Black Magic Cameras from time to time, renting them, and I am absolutely in love with the workflow and the image "taste". So I'm going to buy one. But with those huge DNG files, my situation is getting even worse.

So, going to my questions to you old pros of intelligent storage:

Is it a good idea what I have in mind? -------> keeping the "old drives" for "old backups", buying a large (maybe 16Tb) RAID array for the storage, and 3/4 SSDs for work in use and editing? So the workflow would be : keep everything which is done in the RAID array, and keep what you are doing NOW on the SSDs? I mean...is this the "right" way?

If you have any suggestions, please let me know. I know my work well, but I've been badly organized and I'm also very open to learning from whose knows best.

Thank you very much for your time!

Joshua Cadmium
06-16-2014, 01:18 PM
I wouldn't use RAID for long term storage. A lot of professionals use LTO tape, but some people (including Facebook) are currently trying/using Blu-ray discs. The newest ones designed for long term storage are coming in 300GB, 500GB, and 1TB, and they are rated to about 50 years or so.

Pietro Mingotti
06-17-2014, 01:10 AM
Hi Joshua. Thanks for answering. Ok so for long term safe storage, I'd be better to choose another solution.. the point is that most of the "old" work that I've done, I need to use it quite frequently (I am Art Director for an Italian fashion Brand, and many times we need stuff from the past seasons etc etc). But then, if RAID and Hard Drives are not suitable for storage, their best use would be "work in use"? What about read/write speed? I think working with BMC files using SSDs is mandatory. What would be your suggested configuration?

Fergus
06-17-2014, 12:38 PM
Does anyone here use LTO-6 tape backups?

funwithstuff
06-18-2014, 04:08 AM
How about a large RAID (12-20TB) for your primary storage and editing, backed up regularly to your older external drives? For archiving, multiple hard drives that you spin up from time to time is probably the most economical solution. A Thunderbolt RAID0 can hit 600-700MB/s, so an SSD is not necessary.

Pietro Mingotti
06-18-2014, 08:06 AM
Hello funwithstuff, and thank you for your answer (thanks everybody else as well)..

So, what you suggest makes sense. For what concerns backing up on my older drives, you suggest to put them all together in a software controlled RAID (one of them has failed after 4 years of usage, and I'd prefer to have a "safe solution" in case they fail again)? Or just plug them in dividing them by subjcet maybe (scratch disk, photography, video, audio, projects files...). Also, I've been thinking about this for quite a long time: I think Apple's Time Machine is fantastic, since it backups only what changes, avoiding long time consuming backups and filling space unnecessarily.. is there some similar software solution for external hard drives/raids also? (I mean, something that backups your RAID or EHDs onto other EHDs or RAIDs?)

thanks for the advices

funwithstuff
06-18-2014, 07:32 PM
No, I wouldn't use a RAID for long-term backup — back up specific projects on specific drives instead. That way, if any drive fails, the damage is minimal. To be completely safe, you could back up each old project onto two separate drives and store them in different places.

Of course, this isn't as safe for long-term storage as something serious like an LTO solution, but personally, I'm actually very unlikely to need the original source files for a two-year-old job. The final export master is backed up elsewhere, but I'm not going to keep the original source files for every job safe for ever. Other people will have other needs and other strategies.

I do keep my personal video files and any current client files in at least two places at all times — they can't be lost.

If you use FCP X, I've written a guide to backup/archiving over at macProVideo:

http://www.macprovideo.com/hub/final-cut/fcp-x-101-archiving-revisited

Summary:
If you shoot, then copy your media to your working drive and a backup drive, it's in two places. Backup your edit as you go (FCP X will do this automatically, and Premiere will too with the new features announced today) and then your current edit is always in two places as well. When the edit's done, move your final edit to your backup drive next to your media, and you can safely delete everything from your working drive.

mjamerson
06-20-2014, 12:43 PM
66 TB's on the desktop and once I figure out what Thunderbolt enclosure I want I'll expand more. Although the MOBO is expensive it host 22 SATA connections !! Asrock extreme 11

stier
06-21-2014, 01:41 AM
Backup to hard drives, put them away.
Work on hdd raid1 or 1+0 (1.5x or 3x read speeds), or simply on single ssd's.

As soon as those 10Gbit interface cards are affordable, I will setup a little Linux Raidbox. I don't like hardware RAID.

Pietro Mingotti
07-02-2014, 03:18 AM
Hmmm, thinking about this as RAID work storage (in use)

http://www.amazon.it/LaCie-2big-12TB-Thunderbolt-2/dp/B00KW2AYWK/ref=sr_1_21?ie=UTF8&qid=1404289030&sr=8-21&keywords=lacie+2big

I read some very good reviews about it. Did anyone of u use it?

But I'm also very fond of rack mount units... so...

http://www.amazon.it/LaCie-8big-Rack-12TB-Thunderbolt/dp/B00KW2V1AE/ref=sr_1_11?ie=UTF8&qid=1404289132&sr=8-11&keywords=lacie+2big+12