LTO (Linear Tape-Open) is the drive and tape hardware specification
LTFS (Linear Tape File System) is the tape format specification

Archive workflow for post using LTFS

When you rewrite files on tape the space is still used so incremental backups use more and more tapes. Full backups of online storage take some time.
Instead we recommend archiving project by project and also separating original material.
Also the LTFS archive can be used for easy sharing of project assets with other people working on the project in different locations.
Here is a suggested workflow for a post-production environment.

  • Keep your source material separate from project files and rendered material.
  • Wherever possible link camera material into application projects rather than importing or moving it.
  • If possible have two sets of storage, online and near-line. The near-line might be directly attached to the archive Mac computer.

  • Create original footage, project and renders folders for each project.
  • Inside the footage folder create subfolders for each day or each import.
  • When new footage arrives make two copies on LTFS tape.
  • If there is a lot of material, then at the same time transcode it to a proxy format for editorial.
  • After editorial restore the original footage from tape (if you didn't keep it online).
  • Using YoYottaID Conform you can use the editorial timeline to selectively restore from LTFS tapes.
  • When importing into your grading, VFX or online application, link rather than copy material into your project.
  • Create new material (grading, VFX and deliverables) in the render folder. That way you keep the project small in size.
  • Use YoYottaID to backup the projects and new material to near-line storage.
  • The original footage doesn't need to go to near-line, because you have two tape copies.
  • Use YoYottaID LTFS to archive projects and new material from the near-line to tape.
  • On completion remove projects from online. If there is space then keep them on the near-line storage for a bit longer.

  • LTO-5, LTO-6 and LTO-7

    The latest drives support LTO-7 tapes. LTO-7 stores about 5.7TB of data onto each tape cartridge. Each generation reads and writes the previous generation tapes. Also the drive will read two generation back. So an LTO-7 drive will read / write LTO-6 and read LTO-5. LTO-6 stores about 2.35TB and LTO-5 about 1.35TB

    LTO drives and SAS interface TLR support

    There is a SCSI protocol in the T10 SAS Protocol Layer (SPL-4) called TLR that is required for IBM tape drives.
    Note that all LTO-7 drives are manufactured by IBM.
    With LTO-7 the drive will use TLR because of the higher transfer speeds.
    With LTO-6 the drive may use TLR for large file transfers.
    If the SAS interface does not support TLR, then you will get read errors that are not signalled.
    These errors would remain undetected unless the app used compares the data read back against a checksum of the written data.
    HPE, IBM, Quantum, Tandberg and YoYotta only recommend SAS interfaces with TLR support.
    For the Mac the ATTO H680 PCIe card supports TLR, so this is the only interface we recommend.
    To use the H680 with a Thunderbolt Mac put it into a Sonnet or other TB PCIe chassis.

    If you already have an ATTO R680 and SH2068 there is a workaround which limits the link speed, this reduces errors . See this link for more details
    We do not know of any fix for the HighPoint 6328, so this is not suitable for use with LTO-7 and LTFS.

    Tape Compression

    Using compression is recommended, the LTO drive will compress in hardware so the transfer speed will be faster and more files can be stored on the tape.
    With documents and uncompressed files like DPX and ARRIRAW you will see a small space saving. Also the transfer speed will increase. With video files that are already compressed like ProRes or H.264 there won't be any reduction in size, but it's no slower so we suggest leaving it enabled all the time. All compression is lossless. The LTO vendors quote a crazy compressed figure of 15TB on LTO-7 and 6TB on LTO-6. This would only be achieved with text or other files that compress well. We haven't found anyone with that amount of text files !!

    TAR format

    TAR is an older method of storing files as a bundle on a tape. There is no directory and the tapes cannot be mounted on the desktop. This makes it hard to retrieve individual files. YoYottaID LTFS supports reading of most TAR tapes, this allows easy migration to LTFS.

    LTFS format

    LTFS is an open standard, the current version is 2.2.0. Tapes written can be read by LTFS running on Linux, macOS or Windows. So it's a cross platform archive. Older LTFS versions can still be read back.

    LTFS tape labels and format serial code

    LTFS tapes must be formatted with a 6 character code. The code can have letters or numbers, for example 123456, ABCDEF, YO1234, 17376A. The tape barcode label is 8 characters, which is the user selectable 6 characters with either L5, L6, L7 on the end for LTO-5, LTO-6 or LTO-7 tapes. So 123456L5, ABCDEFL6, YO1234L6, 17376AL5. The six digits from the labels should be used for the six digits of the format serial code. Use barcode labels for all tapes, this means that they can be used with YoYottaID Automation and an LTO library.

    After formatting you cannot change the serial code, as it is written into each index on the tape and also written to the chip inside the tape.

    To change the serial you have to reformat the tape.

    Accessing LTFS tapes using the Finder

    When mounted by YoYottaID LTFS the tape will also appear in the Finder (Finder, Go, Computer). You can browse the tape and this will be quick as long as there are only folders on display. When navigating into folders with files then this will get very slow as the Finder tries to create thumbnails and metadata. It is possible to drag and drop folders and files on the tape using the Finder but there is no verification and also the file creation and modified times will be lost. There are also other technical reasons so using the Finder is not recommended.

    Deleting files from tape

    If files or folders are deleted from tape using the Finder or command line they will be removed from the tape directory. Note that the space is not recovered so for this reason there is no delete function in YoYottaID. If a large number of files need to be removed, then it is best to reformat the tape and start again. Files can be moved on tape, but folders cannot. So it is best to organise the structure on the source drive and also setup the destination paths before starting the backup. It is also possible to create an empty folder on the source and link files and folders into it. Then turn on Copy linked files and folders in the Preferences panel.

    Archiving linked files and folders

    YoYottaID LTFS can archive linked files and folders or symlinks. The linked material can be the tape or it can be on another volume. If the link points to another volume then when the tape is restored the link will be recreated.
    However it can be easier to remove the links and copy the linked material instead. This will make the archive larger, but it will be self contained and can be restored anywhere. To do this turn on Copy linked files and folder in the Preference panel.

    LTO tape compression

    We suggest that you format LTO tapes with compression enabled. With documents and uncompressed files like DPX and ARRIRAW the LTO drive will compress in hardware so the transfer speed will be faster and more files can be stored on the tape. With video files that are already compressed like ProRes or H.264 there won't be any further reduction in size, but it's no slower and all compression is lossless.
    Confusingly the LTO vendors quote a crazy compressed figure of 6.25TB for LTO-6 or 15TB for LTO-7. This would only be achieved with text or other files that compress well.
    Also when using LTFS some of the tape capacity is used for indexes, so in reality LTO-6 will hold about 2.4TB and LTO-7 about 5.7TB.

    LTO tape health and errors

    The LTO drive monitors tape read and write errors. The error count is stored in a memory chip inside the tape cartridge. When the tape is loaded YoYottaID LTFS reads the error count.
    When writing sometimes retries are needed and the drive rewrites the data further down the tape. This reduces the capacity of the tape very slightly. These are logged as recovered write errors. A handful of recovered write errors are no problem even with a new tape, however large counts indicate poor tape quality or a drive issue. Recovered read errors indicate that the drive needed to retry the read, again low counts are normal. But high rates in the hundreds or thousands indicate a problem with the tape or drive.
    Reformatting the tape does not clear these errors and is not a solution for a bad tape. If a tape was written and verified without error and then later on when reading back errors occur this indicates that the tape may have been damaged.

    Permanent errors are logged when a read or write to tape fails. Each error may cause a file to be unreadable, however it is possible that the file has been rewritten so the affected area of the tape will no longer be used. We suggest that you do not continue to use a tape with multiple permanent errors. If you have another tape with this data then duplicate and verify this to create another copy. If this is the only copy then restore the data and create another copy.
    If you create another tape and it also has problems, then there may be an issue with the drive. The drive can be tested with either HPE or IBM tape tools.

    © 2017 YoYotta Back to Top