An MD5 checksum is a 32 digit hexadecimal number that represents the hash of the contents of a file.
The calculation of an MD5 is an industry standard so the integrity can be checked on any system.
As shown here an MD5 checksum can also be generated using the terminal.
xxHash is a newer checksum that is faster to generate, YoYotta uses the xxHash64be algorithm which is compatible with other workflow software.
Changes to the contents of the file give a different checksum, so verifying by comparing checksums is a great way to ensure that a file has not been changed.
YoYotta generates both MD5 and xxHash checksums, even when Quick Verify is enabled.
So jobs will always be compatible with the delivery specification.
The checksums are stored as extended attributes for every file. This also means that the file can be easily re-checked at any time in the future. YoYotta does this automatically at high speed whilst reading each source file.
Then during the verify YoYotta reads back every file from the destination, generates an xxHash checksum and compares this against the source checksum. If the two checksums do not match then YoYotta rereads the destination file and generates MD5 and xxHash checksums and compares these against the source checksums. If there are any issues then a warning is logged.
If turned on, then during verification YoYotta reads all the source files a second time to make sure they are consistent. This extra source verify happens in parallel with the destination verify, so if the source read speed is slow then this will increase the verification time.
Use Source verify if you think there is a problem with a camera card or hard drive.
When turned off the source and destination are still completely verified.
When turned on the source is verified twice.
Keep Quick Verify turned off when archiving to perform a full verification of each destination.
Quick Verify must be off when spanning archives across multiple tapes
Turn Quick Verify on when restoring files to skip full verification. YoYotta will check the file size and also fully verify the source files.
Even with Quick Verify turned on YoYotta always calculates and saves both MD5 and xxHash checksums for each file, allowing a full verification to be performed later.
If you are copying material in a chain, for example from camera cards to shuttle drive, then from drives to RAID, then from RAID to LTO tape then you can use Quick Verify for each copy apart from the last one. Each subsequent copy will verify the source against the previous checksum. However you should not erase the source until the copy has been copied.
MHL + MD5 are metadata reports that contain file names and checksums.
YoYotta will parse these report files on the source drive and warn if the file index differs from the source index. So if files are missing from the drive then a warning will be shown. Also when YoYotta copies files it will generate MD5 and xxHash checksums for every file and compare these against the MHL checksums.
In the Source Browser Destination tab the generation of MHL + MD5 job reports can be enabled. The reports can also be manually generated for folders in the Source Browser and the Project Browser.
If files have been modified and the checksum files are out of date, turn this option off to avoid seeing lots of warnings.
If possible use MHL files as they are standardised and they support both MD5 and xxHash checksums along with file sizes.
ARRI ALE metadata files can also be parsed and the durations used to ensure all ARI or ARX frames are present. However these files do not contain checksums.
Add a drive or drop a folder into the job table. Click the folder above the job table to open the Source Browser. When YoYotta has finished indexing the files click Verify Job.
YoYotta will read back every file and compare each one against the original checksum. This is a full verification.
A quick verification will check to see if the files have ever been verified. If not then they will be fully verified. If they have been verified before, then YoYotta will just check files sizes, modification time and file location. To run a quick verify turn Quick Verify on, then click Verify Job.
The verification speed and job time estimate will be shown in the job table.
As files are verified their status will be marked green in the Source Browser file table.
The verify job can be stopped by clicking the main stop/start button.
Sometimes drives arrive with footage without checksum reports. To generate an MHL checksum report drop the folder into YoYotta. Open the Source Browser and select the roll folders, then click Create MHL YoYotta will create an MHL inside each roll folder. Here 9 roll folders are selected so 9 MHL files will be generated. To make a single MHL report with all rolls, select the root folder (in this case ALEXA_LF). Then click Create MHL
If a message Verify required appears then some or all of the files do not have checksums. This will be seen in the checksum columns in the table. Copying the footage using YoYotta will automatically generate both MD5 and xxHash. Alternatively if a copy is not needed, then turn on Quick verify and click the Verify Job button to run a verify process. This will create MD5 and xxHash checksums (skipping files that already have checksums). When this is finished click Create MHL again.
© 2021 YoYotta Back to Top