Announcement

Collapse
No announcement yet.

Advice needed on RAID setup

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Advice needed on RAID setup

    Hey BMCusers!
    So I believe the time has come to buy a fast RAID to keep up with my current workflow. I will try to be as detailed as possible with my setup as well as my needs.
    I am currently running a new mac pro with the following:
    3.0GHz 8-core with 25MB of L3 cache
    16GB (4x4GB) of 1866MHz DDR3 ECC
    512GB PCIe-based flash storage
    Dual AMD FirePro D700 GPUs with 6GB of GDDR5 VRAM each

    I also have a Synology DS1513+ NAS as my backup and storage

    I edit on Premiere pro and frequently use After effects for my different VFX needs as well as Resolve for advanced color. For short deadline projects (which seems to be a lot) I use MB Colorista or Looks for times sake.

    All my applications are stored locally on the mac pro. I store all media for each project on my 2TB Passport Pro (thunderbolt) running RAID 0 but my data transfer speeds are only 200MB/s.

    I shoot with a BMCC and BMPCC and up until recently I was really only shooting and editing ProresHQ but have started to venture into RAW to harness the power of these great cameras. However, as I am incorporating more VFX (rotoscoping, animated textbox's/ lower thirds) I am no longer getting real-time playback once I get a few streams playing or add some lower thirds or apply some medium to heavy color grades(within premiere). Sometimes there is 10+ second delay from when I make a change such as moving text to it actually moving! And having to render out a timeline just to see if a change I made works is not an ideal workflow.I am pretty confident it is the slow throughput from the passport pro that is the cause of all this lag.

    What RAID enclosures are popular amongst BMCusers? what kind of read/write speeds are you getting? 4 or 8 disk? RAID 0, 5, 6, 10? I have been reading online for quite a while about RAIDS but there is so much information I keep getting pulled in different directions on what I should buy. From my research I am "narrowed down" to hardware based raids and am also trying to future proof myself when I eventually upgrade my two cameras to either an URSA if HFR are available or an EPIC. I find a lot of my clients wanting slow motion these days. So with that said a lot of people suggest an 8 disk array but those are a little more than I want to spend. I am trying to keep the price south of $2k. But thought-put speed is what I am after so I am willing to go up a bit if the right system shows itself. I have been looking at the OWC thunderbays as well as the Promise Pegasus2's as these are the most talked about on different forums. Also saw Akitio Thunder2 Quad, Areca ARC-8050 Thunderbolt RAID etc. So many options!

    If any of you have suggestions on how to maximize disk speed in my workflow I am open to adapting that. I read that I should store scratch, render files, and cache on a separate disk from the media(passport pro). Will I really gain that much speed by doing so? Does that scratch disk need to be blazing fast as well? Would I get a substancial speed increase upgrading to 32gb of ram...or 64?

    Sorry for the long post, I just want to give you all as much information as possible. I just spent a boatload on my mac pro, now I need the speed in the storage to keep up to that beast so I can get real-time playback.

    Thanks in advance!

  • #2
    I use a Pegasus R6 from Promise Tech.
    http://www.promise.com/storage/raid_...tistic=pegasus

    It's fast and reliable. I have it set-up as RAID5 so if one drive fails I simply replace the drive and it rebuilds the content. Been very happy with it. Easy to setup. Not cheap but you get what you pay for.

    The newer Pegasus2 series now has Thunderbolt 2 so would be a better option for the New Mac Pro.
    http://www.promise.com/promotion_pag...global&rsn=100
    Adam Roberts
    www.adamroberts.net

    Comment


    • #3
      Originally posted by Adam Roberts View Post
      I use a Pegasus R6 from Promise Tech.
      http://www.promise.com/storage/raid_...tistic=pegasus

      It's fast and reliable. I have it set-up as RAID5 so if one drive fails I simply replace the drive and it rebuilds the content. Been very happy with it. Easy to setup. Not cheap but you get what you pay for.

      The newer Pegasus2 series now has Thunderbolt 2 so would be a better option for the New Mac Pro.
      http://www.promise.com/promotion_pag...global&rsn=100
      Thanks for the quick reply Adam. That seemed like one of the most used Raid system amongst professionals. Do you store all render files, cache and media for a project on your R6?

      Comment


      • #4
        Originally posted by cmckinlay View Post
        Thanks for the quick reply Adam. That seemed like one of the most used Raid system amongst professionals. Do you store all render files, cache and media for a project on your R6?
        All my current projects live on the RAID. All media and project files. I then render to a separate, nothing fancy as it's only for the render process. All projects are backed up to external drives. Normally one drive unless it's an important project that has budget for multiple backups. Once a project is done I archive it and the backup so I have 2 copies.
        Adam Roberts
        www.adamroberts.net

        Comment


        • #5
          Originally posted by Adam Roberts View Post
          All my current projects live on the RAID. All media and project files. I then render to a separate, nothing fancy as it's only for the render process. All projects are backed up to external drives. Normally one drive unless it's an important project that has budget for multiple backups. Once a project is done I archive it and the backup so I have 2 copies.
          Just purchased the Pegasus2 R6 12TB Raid on sale at B&H for $1999 and included a Pegasus J2 256GB Drive for free plus free shipping! Can't wait to start editing on my new setup! Thanks very much for the help Adam!

          Comment


          • #6
            Thats a pretty good deal. I just can't bring myself to spending $2k on a drive. But 12TB to last a long time.

            Comment


            • #7
              Originally posted by Adam Roberts View Post
              Not cheap but you get what you pay for. The newer Pegasus2 series now has Thunderbolt 2 so would be a better option for the New Mac Pro.
              http://www.promise.com/promotion_pag...global&rsn=100
              I don't have one of these yet, but when I go this route, this is what I will get. I have experience with the older model that is used to handle a lot of content for a corporate client and theirs is running strong.

              Comment


              • #8
                Once I get it up and running I will post some speed tests!

                Comment


                • #9
                  Just want to point out that it is probably wiser to go RAID 6, rather than 5.
                  The problem is that when a drive fails, it puts a tremendous load on the other drives when backing up.
                  Studies have shown that this increases the likelihood of an immediate second drive failure as all the drives are usually from the same batch.
                  If I wanted my films to look like the real world I'd buy a video camera.

                  Comment


                  • #10
                    Using RAID6 on a six-bay RAID is going to further eat into the available storage you have for your media etc. The next step for some would be RAID6 on an eight-bay system like the Pegasus2 R8. If speed is a paramount concern, then you could go with a simpler RAID, such as 10, to both have the benefit of striping and redundancy.

                    Comment


                    • #11
                      All right, I need to have your latest thoughts on the most appropriate RAID configuration I should use now that I have a Promise Pegasus2 R6 24TB RAID being delivered this coming week. The default configuration is RAID 5 and I think I can live with that but with 6 drives, there are some options. This is a decision that you end up living with forever as you can hardly reconfigure once you have loaded data on the drives. I plan to have media from 4.6K to HD on the system.

                      RAID 0, benefit of striping is very fast media transfers read and write but zero redundancy and I don't have LTO tape backup.

                      RAID 1, benefit of mirroring is an easy recovery if a drive fails but the at the cost of storage only 12TB.

                      RAID 5, benefit of data recovery from a single drive failure at the cost of reducing storage to 21TB.

                      RAID 6, benefit of recovery from a failure of two drives but eats more storage and longer recovery times.

                      RAID 10, fast striping and quick recovery from the mirror at the cost of 12TB storage.

                      I'm leaning towards RAID 10 because a quick recovery and fast data transfers are great, but concerned about the 50% reduction of total storage capacity. I know I should plan on LTO at some point. What would you do?
                      Last edited by rick.lang; 04-09-2016, 03:44 PM.

                      Comment


                      • #12
                        I would rule our RAID 0 and 1. ( you have them backwards in the post just as a side note )

                        The thing about RAID 5 ( and also 6 in this case ) I learned the hard way is its not fool proof.

                        I had a disk in a RAID 5 go bad on me, which as we all know is fine.
                        However into the rebuild process I had another error / drive failure and just like that the whole pool was gone.
                        Learned my lesson:
                        -No cheap consumer drives
                        -No cheap controller

                        Enterprise level drives are advisable in that case given that they will have a lower probability of failure during a possible rebuild.
                        This is based off a drives URE which in most typical consumer drives is 10^14, in enterprise its usually 10^15
                        You can google "RAID 5 URE" lots of discussion about the topic.

                        From what I read people generally advise against RAID 5/6. Lots praise RAID 10.

                        I don't build massive storage systems for a living, so with a grain a salt

                        Take in all the feedback you can. Just my 2 cents...
                        Last edited by Zolac; 04-09-2016, 03:39 PM.

                        Comment


                        • #13
                          Zolac, temporary insanity, corrected!

                          Comment


                          • #14
                            Originally posted by cmckinlay View Post
                            Just purchased the Pegasus2 R6 12TB Raid on sale at B&H for $1999 and included a Pegasus J2 256GB Drive for free plus free shipping! Can't wait to start editing on my new setup! Thanks very much for the help Adam!
                            Am late to the party and am sure the 12TB RAID you have just purchased will come in handy for backups, but that is all you should use it for and nothing else.


                            Firstly you either create a 'scratch' drive from one of these :-

                            http://www.bhphotovideo.com/c/produc...0_pro_m_2.html


                            or this if you can drop it straight onto your motherboard :-

                            http://www.bhphotovideo.com/c/produc...0_pro_m_2.html


                            if that isn't possible, then create a 'working' drive using a couple 256GB cheap arse SSD's configured as RAID 0....after all, that is what the 'i' in RAID stands for...the techs at the time said 'Cheap Arse' but 'Inexpensive' was in the Oxford Dictionary and fit with 'management' so they just went with the flow and we all ended up with another 'i' in our lives. Just like Disney but in a different place and different time...other than that, exactly the same (;->> :-

                            http://www.bhphotovideo.com/c/produc...512gb_2_5.html


                            For both the scratch drive and/or the working drive you will absolutely lose your data should something dumb happen in the few seconds between editing and saving....but they are fastest and fast (respectively) and you will save man-hours of time with the performance benefit whilst editing, compared to any possible risk....including earthquakes.

                            Also am assuming y'all have a thing called an Uninterruptible Power Supply, which at least allows you to go through a planned and normal shutdown...please tell me it is so... as they are stuff all in price and to the power network, make a desktop appear like a laptop :-

                            http://www.bhphotovideo.com/c/produc..._Pro_1500.html


                            UPS's aside and considering the storage stuff is based on silicon and doesn't involve those rubbish spinning things, you could probably not ever backup anything and you would be fine. Once you are on Silicon, the only thing you really need to do is to protect yourself from your own stupidity.

                            Anyway, back to the plan....always create a file by file copy of the scratch and working data to a single standalone spinning drive that supports S.M.A.R.T such as this WD RE (Meant for Servers)....and do a file by file copy as often as you care to ie. hourly, daily etc....when grabbing a break...yep, that is right...just use a single high spec drive on it's own and call it 'prod' or 'production' and then back this up often to your 12TB RAID.

                            Believe it or not a single drive on it's own that supports S.M.A.R.T and a range of other tech is way more useful when considered as part of the overall workflow :-

                            http://www.bhphotovideo.com/c/produc...fsyz_hard.html


                            Only ever use your physical spinning 12TB or RAID whatever or other for backups (using backup software which has error correction) and of course run the normal, Mother, Father, Grandmother, Grandfather for days, weeks, months, years etc.....backup methodology that has stood the test of time since IT began.

                            Then every now and again, pull a 'prod' drive which holds all of your archival data and put it on your bookshelf and never use it again.

                            Basically what will happen is the size of drives you will put on your bookshelf, will increase in disk size and decrease at a price way faster than you could possibly be bothered with.

                            What you don't want to do is waste your editing time using spinning physical things, regardless of whatever the RAID standard....it's just nonsense !!

                            People have confused the whole reason the idea of a 'Redundant Array of Inexpensive Drives' (RAID) was created.

                            This philosophy above all relies on one simply principle...that you throw away the shit footage, regardless of how amazing you think it could be in 100 years time....if it is shit now it will be more shit as each day passes....just delete it, save the disk space and don't torture your friends and family and get on with your life.
                            Last edited by AndrewDeme; 04-10-2016, 05:20 AM.
                            Lotsa Zeon thingos with thousands of cute cores...enough is never enough

                            www.andrewdeme.com
                            http://www.youtube.com/andrewdeme
                            www.facebook.com/andrewdeme
                            http://www.linkedin.com/in/andrewdeme

                            (I reserve the right to edit, modify or delete any content I create anywhere at anytime...it probably wasn't that good anyway)

                            Comment


                            • #15
                              Originally posted by cmckinlay View Post
                              Hey BMCusers!
                              So I believe the time has come to buy a fast RAID to keep up with my current workflow. I will try to be as detailed as possible with my setup as well as my needs.
                              I am currently running a new mac pro with the following:
                              3.0GHz 8-core with 25MB of L3 cache
                              16GB (4x4GB) of 1866MHz DDR3 ECC
                              512GB PCIe-based flash storage
                              Dual AMD FirePro D700 GPUs with 6GB of GDDR5 VRAM each

                              I also have a Synology DS1513+ NAS as my backup and storage

                              I edit on Premiere pro and frequently use After effects for my different VFX needs as well as Resolve for advanced color. For short deadline projects (which seems to be a lot) I use MB Colorista or Looks for times sake.

                              All my applications are stored locally on the mac pro. I store all media for each project on my 2TB Passport Pro (thunderbolt) running RAID 0 but my data transfer speeds are only 200MB/s.

                              I shoot with a BMCC and BMPCC and up until recently I was really only shooting and editing ProresHQ but have started to venture into RAW to harness the power of these great cameras. However, as I am incorporating more VFX (rotoscoping, animated textbox's/ lower thirds) I am no longer getting real-time playback once I get a few streams playing or add some lower thirds or apply some medium to heavy color grades(within premiere). Sometimes there is 10+ second delay from when I make a change such as moving text to it actually moving! And having to render out a timeline just to see if a change I made works is not an ideal workflow.I am pretty confident it is the slow throughput from the passport pro that is the cause of all this lag.

                              What RAID enclosures are popular amongst BMCusers? what kind of read/write speeds are you getting? 4 or 8 disk? RAID 0, 5, 6, 10? I have been reading online for quite a while about RAIDS but there is so much information I keep getting pulled in different directions on what I should buy. From my research I am "narrowed down" to hardware based raids and am also trying to future proof myself when I eventually upgrade my two cameras to either an URSA if HFR are available or an EPIC. I find a lot of my clients wanting slow motion these days. So with that said a lot of people suggest an 8 disk array but those are a little more than I want to spend. I am trying to keep the price south of $2k. But thought-put speed is what I am after so I am willing to go up a bit if the right system shows itself. I have been looking at the OWC thunderbays as well as the Promise Pegasus2's as these are the most talked about on different forums. Also saw Akitio Thunder2 Quad, Areca ARC-8050 Thunderbolt RAID etc. So many options!

                              If any of you have suggestions on how to maximize disk speed in my workflow I am open to adapting that. I read that I should store scratch, render files, and cache on a separate disk from the media(passport pro). Will I really gain that much speed by doing so? Does that scratch disk need to be blazing fast as well? Would I get a substancial speed increase upgrading to 32gb of ram...or 64?

                              Sorry for the long post, I just want to give you all as much information as possible. I just spent a boatload on my mac pro, now I need the speed in the storage to keep up to that beast so I can get real-time playback.

                              Thanks in advance!
                              I use a Thunderbolt 2 20TB Raid 5 external hard drive from macsales.com. It does pretty good for editing 4K. I have a 2015 iMac 5K maxed out with 32GB Ram and 512GB internal flash. Here's the site.

                              http://eshop.macsales.com/shop/Thund...derBay-4-RAID5

                              Comment

                              Working...
                              X