MD5 and xxHash checksums
An MD5 checksum is a 32 digit hexadecimal number that represents the hash of the contents of a file. The calculation of an MD5 is an industry standard so the integrity can be checked on any system. As shown here an MD5 checksum can also be generated using the terminal.
xxHash is a newer checksum that is faster to generate, YoYotta uses the xxHash64be which is a 64bit big endian algorithm. This is the industry standard xxHash and is compatible with other workflow software.
There are other xxHash variants XXH3 and XXH128. YoYotta does not support these and we would not recommend using them to avoid confusion and incompatibilities.
Changes to the contents of the file give a different checksum, so verifying by comparing checksums is a great way to ensure that a file has not been changed and is an exact copy of the original.
During full verification YoYotta reads back every file from the destination, generates an xxHash checksum and compares this against the source checksum. If the two checksums do not match then YoYotta rereads the destination file and generates MD5 and xxHash checksums and compares these against the source checksums. If there are any issues then a warning is logged.
Some companies use MD5 verification workflows, especially when working with LTFS tapes, in this case set the YoYotta verify mode to MD5 + xxHash. Then YoYotta can add the MD5 checksum values to the PDF, MHL or MD5 reports.
When working with camera media or SSD drives and RAID that transfer at speeds faster than 1GB/s then set YoYotta to xxHash verify mode. Then the only limit will be the hardware device or thunderbolt port speed.
Normally always use Full Verify mode, however if running a chain of copy jobs, for example from shuttle drive to RAID and then from RAID to LTFS tape, then the initial copies can be run in Quick Verify mode. YoYotta will still generate xxHash checksums for every file and the full verify can be left to the last copy job, saving time.
The checksums are stored as extended attributes for every copied file. This means that the file can be easily re-checked at any time in the future. YoYotta does this automatically at high speed whenever a file is restored. After verification is complete the checksums can also be stored in MHL, MD5, CSV, ALE and PDF files locally and also on each copy.
Single hash or checksum for entire tapes or drives
YoYotta doesn't create a single hash as we think it is of little value and can easily give false negatives. The reason for this is that when making a hash from individual hashes the order of the files needs to be specified.
Is the sorting alphabetically over the whole tape, or are the folders sorted first, then the contents? What sort type is used, pure alphanumeric or a numeric text-based?
Are extra files like reports included? What about sidecar files like LUTs, are they excluded from this master hash?
So even with all the correct files, combining the hashes in a different order will result in a different result. So verification fails. Also if an extra report or LUT file is written without being used in this hash, then the check fails without giving any information as to what caused the failure.
If additional copies are created on different sized tapes or drives, then what fitted on a single device might need multiple tapes or drives. Also human comparison of hashes for multiple sets of files is not verification.
Instead, an MD5 file (or better an MHL file) gives the names and hashes for each file, so it's easy to tell the name of a missing file, if there are extra files and if any file doesn't match its checksum. Some or all of the files can be reverified and compared against their original checksums.
If the client insists this is needed, then you will need to write a script that reads the YoYotta MD5 or MHL report and hashes together each hash. Then there is a defined order and number of files and the report can be supplied as well, so the process can be replicated.
Of course, that report would give the client everything needed to identify any missing or extra files and verify each file which the single hash wouldn’t. So in summary we feel it's better not to support this and instead rely on verification using industry standard methods.
Double Source Verify
If turned on, then during verification YoYotta reads all the source files a second time to make sure they are consistent. This extra source verify happens in parallel with the destination verify, so if the source read speed is slow then this will increase the verification time.
Use the extra Source verify if you think there is a problem with a camera card or hard drive.
When turned off the source and destination are still completely verified.
When turned on the source is verified twice.
xxHash : Quick Verify
Use Quick Verify when restoring files to skip full verification. Note that YoYotta will check the file size and if there are checksums then whilst restoring the archived files will be fully re-verified.
Quick Verify can also be used when copying files where the original are not being deleted.
Even with Quick Verify YoYotta always calculates and saves the checksums for each file, allowing a full verification to be performed later.
If you are copying material in a chain, for example from camera cards to shuttle drive, then from drives to RAID, then from RAID to LTO tape then you can use Quick Verify for each copy apart from the last one. Each subsequent copy will verify the source against the previous checksum. However you should not erase the source until the copy has been copied.
Do not use Quick Verify when archiving, instead use Full Verify to perform a full verification of each destination.
xxHash : Full Verify
xxHash checksums will be generated whilst copying.
This is the fastest way to run jobs and is completely secure.
During the full verification another checksum will be created for every copied file. Then the two checksums will be compared to ensure that everything has been copied correctly.
MD5 + xxHash : Full Verify
Both MD5 + xxHash checksums will be generated whilst copying.
This will run slower as MD5 checksums take longer to generate.
Use this mode if delivering media to clients who require MD5 reports.
During the full verification another checksum will be created for every copied file. Then the two checksums will be compared to ensure that everything has been copied correctly.
Generate MD5 + xxHash for tape archives
Enabling this will always generate both MD5 + xxHash checksums for all tape archive jobs.
YoYotta can generate MD5 + xxHash checksums faster than tape archiving so there is no disadvantage to keeping this enabled.
Note that changing these settings will only affect new jobs. For existing jobs adjust Verify settings in the Source Browser.
Always use Full Verify when archiving to perform a full verification of each destination.
MHL + MD5 reports
MHL + MD5 are metadata reports that contain file names and checksums.
YoYotta will parse these report files on the source drive and warn if the file index differs from the source index. So if files are missing from the drive then a warning will be shown. Also when YoYotta copies files it will generate MD5 and xxHash checksums for every file and compare these against the MHL checksums.
In the Source Browser Destination tab the generation of MHL + MD5 job reports can be enabled. The reports can also be manually generated for folders in the Source Browser and the Project Browser.
If files have been modified and the checksum files are out of date, turn this option off to avoid seeing lots of warnings.
If possible use MHL files as they are standardised and they support both MD5 and xxHash checksums along with file sizes.
When enabled ARRI ALE metadata files will also be parsed and the durations used to ensure all ARI or ARX frames are present. However these files do not contain checksums.
Verify existing files
Add a drive or drop a folder into the job table. Click the folder above the job table to open the Source Browser. When YoYotta has finished indexing the files click Verify Job.
YoYotta will read back every file and compare each one against the original checksum. This is a full verification.
A quick verification will check to see if the files have ever been verified. If not then they will be fully verified. If they have been verified before, then YoYotta will just check files sizes, modification time and file location. To run a quick verify select xxHash : Quick Verify, then click Verify Job.
Verify running
The verification speed and job time estimate will be shown in the job table.
As files are verified their status will be marked green in the Source Browser file table.
The verify job can be stopped by clicking the main stop/start button.
Create MHL for existing camera rolls
Sometimes drives arrive with footage without checksum reports. To generate an MHL checksum report drop the folder into YoYotta. Open the Source Browser and select the roll folders, then click Create MHL YoYotta will create an MHL inside each roll folder. Here 9 roll folders are selected so 9 MHL files will be generated. To make a single MHL report with all rolls, select the root folder (in this case ALEXA_LF). Then click Create MHL
If a message Verify required appears then some or all of the files do not have checksums. This will be seen in the checksum columns in the table. Copying the footage using YoYotta will automatically generate both MD5 and xxHash. Alternatively if a copy is not needed, then turn on Quick verify and click the Verify Job button to run a verify process. This will create MD5 and xxHash checksums (skipping files that already have checksums). When this is finished click Create MHL again.