Announcement

Collapse
No announcement yet.

Fastest Way To Offload Footage

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Fastest Way To Offload Footage

    I am need of figuring out a faster way to offload footage from my 256GB cards in the field. What would be a good, fast and portable way to do that in all of your experience? What is a good fast reader?

  • #2
    I use the Lexar Professional Thunderbolt hub because I can download multiple cards at once without slowing down, and I use Hedge (http://hedgeformac.com) to download my cards. The only problem with the Lexar hub is that it requires power, but each card reader can be removed and plugged straight into the computer via USB3 if you're battery only.

    Comment


    • #3
      Well...there's FAST, and there's also RELIABLE.

      (And there's also CHEAP!)

      I picked up a small $275 ACER laptop from B&H. ..small as in an 11-inch screen only. Very lightweight too.
      I put a 1T SSD in it and all the basic apps leave me about 800Gig for daily transfers.

      I like this because I can usa a data-verification app (TeraCopy) for the transfer, and then also immediately VIEW the files too.

      I guess this is not the fastest nor the highest capacity, but it's convenient, useful, EASY, and reliable.

      ... for when you've got to be SURE before leaving the set. (Which is ALWAYS.)
      Cameras: Blackmagic Cinema Camera, Blackmagic Pocket Camera (x2), Panasonic GH2 (x2), Sony RX100 ii, Canon 6D, Canon T2i,
      Mics: Sennheiser, AKG, Shure, Sanken, Audio-Technica, Audix
      Lights: Every Chinese clone you can imagine

      Comment


      • #4
        assuming youre using c fast cards this is what i use. it will be expensive

        https://www.bhphotovideo.com/c/produ...ssd_usb_c.html

        http://www.ebay.com/itm/CFast-Card-R...8AAOSw3KFWeR~9

        Comment


        • #5
          Well, the best thing to do is identify the bottlenecks in your workflow. Ideally you'd have a:

          - Computer with several high speed, independently controlled busses; namely, Thunderbolt 1/2/3 or USB 3.1 Gen1/2.
          - Media reader that can connect to one of those high speed busses.
          - Media management software that can offload to 2+ locations at the same time (Pomfort Silverstack, ShotPut Pro, etc.).
          - At least two destination drives with a high read/write speed.

          That last one trips most people up, because independent content producers are always working on tight budgets. Sure, that portable Seagate 4TB HDD might seem like a great buy at first but when you run into its ~100MB/s throughput you'll start to realize why most good data managers build carts with RAIDs, SSDs, or often both simultaneously! Here's the advice that I give most people when they're managing data:

          1. All hard drives--regardless of the technology in them--will fail, and so it is totally unacceptable and reckless to allow data to exist in only one place. Immediately back up your footage in two places before you lose it. Preferably, find ways to make sure that copies of your data are also stored in different geographic locations to minimize the impact of natural disasters, house fires, etc.

          2. Never fill a drive of any kind past 90% full. Filling drives typically means slower throughput, and it also limits your drives' ability to write around bad sectors/flash memory.

          3. Imagine you say a phrase to one of your friends (e.g. "The quick brown fox jumps over the lazy dog.") and they have to repeat it to someone else from memory. That friend then repeats it to another friend, who repeats it to another friend, who repeats it to another friend. By Friend #24, there's a good chance that the words will have been mixed up to create a different phrase (e.g. "The speedy red cat leapt over the sleepy pup."). Now, imagine instead that when you initially said the phrase to your friend, you also gave them a piece of paper that had the phrase written on it. Along with this phrase, the paper would get passed along the chain too so that each person who heard the phrase could compare it to what was written so that they could be sure what they were being told was accurate. That piece of paper is performing the same basic function that a checksum does. Whenever you offload, always perform a checksum so that you can be sure the data you're storing is the same as the data that came right off the camera. Try to rely on XXHash-64 or CRC32 checksums to ensure the highest throughput during the transfer process.

          4. The three important checks you need to make during offloads are filesize (make sure everything was copied!), checksum (make sure everything was copied right), and visual verification (make sure the camera got what you needed in the first place). Each type of examination is a critical part of the offload process and shouldn't be neglected!
          Last edited by Alex.Mitchell; 03-19-2017, 09:45 PM.

          Comment


          • #6
            Copying to a portable mirrored TB3 RAID with SSDs may be a good solution to combine speed, redundancy, and reliability.

            Comment


            • #7
              Build a super fast and affordable SSD raid which you can later charge half its value per day as a box kit item for DataMgmt. Offload cards 5x faster and have them back on set where they belong.

              Comment


              • #8
                Now that you guys are talking about offloading, I was wondering what kind of transfer speeds you guys normally get when offloading 4k raw files. I often shoot in raw 4:1 with my Ursa Mini 4k, and offload with a Lexar CR1 via USB 3.0 to either an SSD in my desktop, my WD Raid 0 HDD external, or a standard 7200 rpm HDD internal. My transfer speeds are always around 30 MB/s regardless of the storage type I'm using. I've tried all of the different USB 3.0 ports in both my desktop and laptop and the speed didn't change. Also, I'm using Lexar 3400X 128 GB Cfast cards. I know the individual raw files should take longer to transfer than the Prores files, but it's taking on average about 10X longer, which sucks when I'm trying to transfer in the field. I truly appreciate any thoughts and considerations you guys might have.

                Comment


                • #9
                  Originally posted by Joe Grasso View Post
                  Now that you guys are talking about offloading, I was wondering what kind of transfer speeds you guys normally get when offloading 4k raw files. I often shoot in raw 4:1 with my Ursa Mini 4k, and offload with a Lexar CR1 via USB 3.0 to either an SSD in my desktop, my WD Raid 0 HDD external, or a standard 7200 rpm HDD internal. My transfer speeds are always around 30 MB/s regardless of the storage type I'm using. I've tried all of the different USB 3.0 ports in both my desktop and laptop and the speed didn't change. Also, I'm using Lexar 3400X 128 GB Cfast cards. I know the individual raw files should take longer to transfer than the Prores files, but it's taking on average about 10X longer, which sucks when I'm trying to transfer in the field. I truly appreciate any thoughts and considerations you guys might have.
                  The kind of files shouldn't make a difference in a large sequential transfer. To really diagnose though, we need more information. What software are you using to copy the files? Are you positive that your ports aren't defaulting back to USB2 because the cables are either malfunctioning or not inserted all the way? etc.

                  Comment


                  • #10
                    Originally posted by Alex.Mitchell View Post
                    The kind of files shouldn't make a difference in a large sequential transfer. To really diagnose though, we need more information. What software are you using to copy the files? Are you positive that your ports aren't defaulting back to USB2 because the cables are either malfunctioning or not inserted all the way? etc.
                    Thanks for the response, Alex. I'm just using Windows explorer. I open the folders in Explorer, select all of the files (usually a combination of Prores and raw clips). All of the Prores clips copy over as fast as I would expect...in the 250-400 MB/s range, when it gets to the raw files, it goes all the way down to 30 MB/S and even less. I'm using Windows 10.

                    Comment


                    • #11
                      Originally posted by Joe Grasso View Post
                      I'm just using Windows explorer.
                      That's probably the issue right there. As an OSX user I can't speak to the performance penalty of using Windows Explorer--or even writing thousands of small DNG files on NTFS volumes--but I will say that you're much better served by using an application designed for this kind of work rather than the file browser in the OS. Take a look at ShotPut Pro or SlimRAW and see if there are any performance gains.

                      Comment


                      • #12
                        Will do. Thanks for the suggestion.

                        Comment


                        • #13
                          Originally posted by Joe Grasso View Post
                          Now that you guys are talking about offloading, I was wondering what kind of transfer speeds you guys normally get when offloading 4k raw files. I often shoot in raw 4:1 with my Ursa Mini 4k, and offload with a Lexar CR1 via USB 3.0 to either an SSD in my desktop, my WD Raid 0 HDD external, or a standard 7200 rpm HDD internal. My transfer speeds are always around 30 MB/s regardless of the storage type I'm using. I've tried all of the different USB 3.0 ports in both my desktop and laptop and the speed didn't change. Also, I'm using Lexar 3400X 128 GB Cfast cards. I know the individual raw files should take longer to transfer than the Prores files, but it's taking on average about 10X longer, which sucks when I'm trying to transfer in the field. I truly appreciate any thoughts and considerations you guys might have.
                          I don't mean to harp on about Hedge (which won't help you since you're on Windows) but they've published a blog post that sheds some light on your issue: https://medium.hedgeformac.com/backu...117#.nxeptmcpm

                          Comment


                          • #14
                            Originally posted by Alex.Mitchell View Post
                            The kind of files shouldn't make a difference in a large sequential transfer. To really diagnose though, we need more information. What software are you using to copy the files? Are you positive that your ports aren't defaulting back to USB2 because the cables are either malfunctioning or not inserted all the way? etc.

                            Actually it does, especially if you aren't using a multi-threaded transfer system (like using the OS built in copy) so DNG would take long to copy than say prores because it's a bunch of files, it has to find each one process then start the copy and so on for each file. It hads up to a lot in large volumes.

                            Ideally you want to use a mulit-threaded transfer system that also does MD5 comparisons.

                            Comment


                            • #15
                              Originally posted by Jason Finnigan View Post
                              Actually it does, especially if you aren't using a multi-threaded transfer system (like using the OS built in copy) so DNG would take long to copy than say prores because it's a bunch of files, it has to find each one process then start the copy and so on for each file. It hads up to a lot in large volumes.

                              Ideally you want to use a mulit-threaded transfer system that also does MD5 comparisons.
                              My understanding is the same, it takes a lot of time to transfer DNGs because of the large amount of independent files.

                              What kind of software would be able to bypass that? Would SSDs help? I would be interested in hearing how pro DITs work.

                              Comment

                              Working...
                              X