LTO + LTFS FAQ
LTO (Linear Tape-Open) is the drive and tape hardware specification
LTFS (Linear Tape File System) is the tape format specification
Why tape ?
Tape has a proven 30 year+ lifetime. It's a stable way to store multiple copies of your assets.
The LTFS format that is used with LTO tape allows you to share your archives with others as it's supported on Mac, Windows and Linux.
Your archive is not locked into any proprietary software.
There is an initial investment in the LTO drive, similar to buying around 15 hard drives. But after that the tapes are very low cost.
Hard drives are OK for short term backup and file transfer.
Always make multiple copies and keep them in different locations to protect against loss or damage.
YoYotta is ideal for this as it can make multiple copies to drives and tapes in parallel.
Archive workflow using LTFS
LTFS stores files in a folder structure. This means that tapes can be easily read and restored using any LTFS software.
You are not locked into one piece of software, because it's an open archive.
When files are rewritten on tape the old files are left behind, so incremental copies use more and more space.
YoYotta supports spanning of large volumes across multiple tapes, provided they are not changing.
If files are being added and modified on the source volume, then when complete the archive will already be out of date.
Also when the job is run again any new files will not always go onto the same tape as existing related files.
So archiving of large changing volumes where files are being added and modified is not supported.
Also it's not a good idea to continually write to existing tape archives.
Instead we recommend archiving project by project and also separating incoming original material.
Once archived, the material never needs to be archived again. Making each job much smaller.
This also means the LTFS archive can be used for easy sharing of project assets with other people working on the project in different locations.
If there is an existing large volume that cannot be split into projects, then do not do this in one job, instead create smaller jobs of 10 or 20TB.
This way you can ensure that files on shared storage aren't being written by other users during the archive. Each job will complete sooner, but all the jobs will still end up in the same YoYotta metadata project.
Here is a suggested workflow for a post-production environment.
To restore you can use the YoYotta Conform option, the Project Browser or job reports to locate files on a tape. Then the files can be restored by YoYotta or any LTFS system.
This makes the archive much more useful. Whereas a multi-tape incremental archive is much harder to partially restore.
Backups of system disks and applications
System or boot volumes contain lots of small files, links, devices and caches which will not copy to LTFS tape or disk. In addition the backup cannot be restored as a bootable volume.
All your projects, documents and data should be stored on a separate disk volume. This can be backed up to LTFS along with project media.
Use Time Machine or create a disk image for your system disk backup.
This way if you have a problem with the computer or need to change computers it will be very easy to start working quickly as all your creative data will be stored separately.
We recommend an app called SuperDuper to create disk images from a system disk.
Create a sparse bundle using SuperDuper on a local drive or RAID.
This can be restored and then SuperDuper can recreate a bootable system disk.
Alternatively use Time Machine for your system disk backup.
Note that Time Machine backups have a special structure and they cannot be copied to tape.
LTO-8, LTO-7, LTO-6 or LTO-5 Tapes
The latest drives support LTO-8 tapes. Each drive generation reads and writes the current and previous generation tapes.
LTO-7, LTO-6 and LTO-5 drives will read two generations back. So an LTO-7 drive will read / write LTO-6 and read LTO-5.
LTFS creates a separate partition for indexes that uses around 5% of the tape. Also YoYotta reserves about 100GB to ensure the tapes does not over fill.
LTO-8 tape stores up to 11.3 TB and are a lot faster at 250-300MB/s.
LTO-7 tape stores up to 5.6 TB and are a lot faster at 250-300MB/s.
LTO-6 tape stores up to 2.3 TB and are a little faster at 160MB/s.
LTO-5 tape stores up to 1.25 TB and can be read or written at 140MB/s.
Type-M or M8 tapes
Type-M allows new LTO-7 tapes to store up to 9TB instead of 6TB.
Before formatting YoYotta can initialise new LTO-7 tapes loaded into an LTO-8 standalone drive as Type-M. No need to purchase tapes pre initialised.
The tape barcode should have M8 as a suffix. Note that once initialised they cannot be used as normal LTO-7 tapes.
In a library ensure the tape has M8 barcode labels and also check that the library firmware supports Type_M. The library reads the barcode and initialises the tape automatically on loading into the drive.
We don’t recommend using Type-M for long term archive as the tapes are only readable in an LTO-8 drive. They will not be readable in LTO-7 or LTO-9 drives. It was a short term solution introduced because of previous supply issues with LTO-8 tapes.
Instead use LTO-8 which has now dropped well below LTO-7 pricing per TB.
Using tape compression is recommended, internally the LTO drive will compress files that are uncompressed.
Enable compression when formatting in the LTFS panel. The compression is lossless and varies from 1:1 to 2.5:1.
Uncompressed files like DPX and ARRIRAW normally see a 10 to 20% space saving and these files will transfer faster.
For all other video files that are already compressed like ProRes or H.264 there won't be any reduction in size, but it's no slower so we recommend leaving compression enabled when formatting all tapes.
The theoretical ratio 2.5:1 allows the LTO vendors to quote a crazy compressed figure of 30TB on LTO-8, 15TB on LTO-7 and 6TB on LTO-6. This would only be achieved with text or other files that compress easily. We haven't found anyone with that amount of text files !!
The potential compression is unknown until after data has been written so the tape capacity and remaining space will always be the minimum available.
Some studios will ask for tapes with compression turned off, so check the delivery spec before formatting.
Source media speed affecting tape write speed
LTO drives perform speed matching, so they can run the tape slower to work with slower source drives. Theoretically the minimum speed needed is less than 100MB/s, however with a mixture of file types on a NAS with a 1Gb connection, you will probably not get a suitable sustained speed. The same limitation will apply to slower USB drives.
With the higher speed of LTO-7 and LTO-8 media you will need a faster connection to keep the job going. Otherwise the drive will run out of data and the tape will have to keep rewinding and start writing again. You will hear this "shoe shining" behaviour if you listen to the drive. When this happens the average speed will drop drastically.
Also with tape compression enabled, then the source read speed will need to be a little higher with uncompressed files like ARRIRAW and DPX.
In this case use YoYotta to create two sequential jobs. The first copy will be from NAS to fast RAID, then when complete YoYotta will index the new source folder and start a copy from RAID to tape. As described here. Alternatively use faster shared storage or a faster connection to the NAS. If neither of these are possible, you may find using LTO-6 media in an LTO-7 drive to be faster.
Project, file and path naming
For any data workflow, to ensure portability and acceptance by all applications and studios only use the following characters. This means no spaces. To enforce this turn on Strict file and folder naming in Preferences
- Alphabetic Characters [A-Z, a-z]
- Numeric Characters [0-9]
- Underscore [_]
- Hyphen [-]
- Full stop [.]
When this option is turned off then virtually all characters can be used.
Note that on the Mac the : (colon) character cannot be used. Also it's not recommended to use the / as it is not supported by Linux and Windows. If used it will appear as a colon.
As LTFS tapes are an archive and can be used cross platform it is best to avoid all the characters shown in the list below.
- macOS :
- Linux /
- Windows \ / : * ? " < > |
However Linux and LTFS support a shorter maximum of 255 8-bit characters. Unicode characters can be made of multiple 8-bit characters, so some files with valid names on macOS cannot be written to NAS, Linux file systems or LTFS tapes. YoYotta will warn about this.
After indexing open the Source Browser and files with issues will have a red status. Edit the file names or zip the files, then reindex the job.
TAR is an older method of storing files as a bundle on a tape. There is no directory and the tapes cannot be mounted on the desktop. This makes it harder to retrieve individual files. YoYotta LTFS supports reading of most TAR tapes, this allows easy migration to LTFS. More information about restoring TAR tapes.
LTFS is an open standard, the current version is 2.2.0. Tapes written can be read by LTFS running on Linux, macOS or Windows. So it's a cross platform archive. Older LTFS versions can still be read back. Tapes are formatted with a 6 character label, that matches the barcode label.
After formatting this label cannot be changed. To rename you will have to restore, reformat and copy the data again. LTFS stores the tape label in multiple indexes and also inside the tape chip memory.
Do not use / * ? < > " | \ : characters as these are not supported when the LTFS tape is mounted on a Windows system.
Keep file names less than 250 characters and path names less than 1024 characters.
LTFS does not support the use of WORM tapes.
LTFS tape labels and format serial code
LTFS tapes must be formatted with a unique 6 character code. The code can have the letters A-Z and numbers 0-9 in any order, for example 123456, ABCDEF, YO1234, 17376A.
The tape barcode label is 8 characters, which is the user selectable 6 characters with either L5, L6, L7, M8 or L8 on the end for LTO-5, LTO-6, LTO-7, LTO-M8 or LTO-8 tapes. For example 123456L5, ABCDEFL6, YO1234L6, 17376AL7. The six characters from the label must be used for the six digits when formatting.
Do not add any checksum or check digit.
Use barcode labels for all tapes, this means that they can be used with YoYotta Automation and an LTO library. Also they are an easy way to identify tapes.
Always make two copies of all material. One Main and one Safety (or clone) tape.
Do not use the same barcode for copies of tapes
There are three suggested conventions for labelling main / safety tapes, use odd / even numbers, use A-M / N-Z as the 5th character or use a number as 5th character and A/B as the last character.
Odd/Even YT0111, 127634, XTM126 are main tapes and YT0112, 127635, XTM127 are the matching safety tapes.
5th letter HRSMDA, HRSMDB and HRSMDC are main tapes, HRSMQA, HRSMQB and HRSMQC are the matching safety tapes. This method allows 4 letter project codes and 13x26 main and 13x26 safety tapes.
A/B ABC12A, YY100A, H2122A, SHOW1A and 12567A are main tapes, ABC12B, YY100B, H2122B, SHOW1B and 12567B are the matching safety tapes.
After formatting you cannot change this label, as it is written into each index on the tape and also written to the chip inside the tape. To change the label you have to reformat the tape.
In Europe we can supply LTO tapes at great prices with custom YoYotta labels printed on durable plastic. They have a standard barcode, plus you can add unique text labels on the left as shown opposite. So you can clearly label Main and Safety, or add a project ident. Unlike other labels each tape can have a different code, you are not limited to a set sequence.
IBM make all LTO-7 and LTO-8 mechanisms, so we supply IBM Barium Ferrite tapes for perfect compatibility.
Contact firstname.lastname@example.org for a quote.
LTO tape storage
Keep your tapes in the same room as the drive or library. If this is not possible then ensure they are in the room for 24 hours before use. This will allow the temperature of the tape to match the temperature of the drive.
The room temperature should be between 16 to 32°C with 20 to 80% humidity. This is a wide range, however it is important to keep this stable. Rapid changes in temperature or humidity are not good for electronic equipment.
Keep tapes in their cases and ensure that they are protected during shipping.
LTO tape health and errors
Tapes can be re-written hundreds of times, so for typical archive workflow there should not be any issues.
The LTO drive monitors tape read and write errors. The error count is stored in a memory chip inside the tape cartridge. When the tape is loaded YoYotta LTFS reads the error count.
When writing sometimes retries are needed and the drive rewrites the data further down the tape. This reduces the capacity of the tape very slightly. These are logged as recovered write errors. Low recovered write errors are no problem even with a new tape, however large counts can indicate poor tape quality or a drive issue. Recovered read errors indicate that the drive needed to retry the read, again low counts are normal. But high rates in the hundreds or thousands indicate a problem with the tape or drive.
Reformatting the tape does not clear these errors and is not a solution for a bad tape. If a tape was written and verified without error and then later on when reading back errors occur this indicates that the tape may have been damaged or the drive is faulty.
Permanent errors are logged when a read or write to tape fails. This will occur if the drive is disconnected or powered off whilst a tape is loaded. Each error may cause a file to be unreadable, however it is possible that the file has been rewritten so the affected area of the tape will no longer be used. Permanent errors can also be caused by a drive hardware issue or damage to a tape. For example if a tape is dropped the edge can be damaged causing some parts of the tape to become unreadable.
As long as you have at least two verified copies stored in different locations and look after the tapes then the chance of data loss is extremely minimal. Much less chance than storing data on other mediums like SSD or HDD. We suggest that you do not continue to use a tape with multiple permanent errors or hundreds of temporary errors. If you have another tape with this data, replace the original with a copy. If this is the only copy then restore the data and create another copy.
If you create another tape and it also has problems, then there may be an issue with the drive.
There is no technical requirement to reverify tapes that are in storage. Whenever you restore from tape YoYotta will reverify the files. You can also run a full verification at any time.
After about 4 years of daily use, LTO drive heads can wear causing the drive to skip over tape, files are written OK, but the tape capacity is reduced.
Accessing LTFS tapes using the Finder
When mounted by YoYotta LTFS the tape will also appear in the Finder (Finder, Go, Computer). The tape icon will appear on the desktop if Connected Servers is turned on in Finder Preferences.
You can browse the tape and this will be quick as long as there are only folders on display. When navigating into folders with files then this will get very slow as the Finder tries to create thumbnails and metadata. It is possible to drag and drop folders and files on the tape using the Finder but there is no verification and also the file creation and modified times will be lost.
There are also other technical reasons so using the Finder is not recommended.
If required turn on Disable Finder browsing in the LTFS panel, then the tape will not show up in Finder windows.
If you need to access a tape without using YoYotta then you can type the following into a terminal window.
The terminal app is located in /Applications/Utilities
ltfs -o devname=0 /Volumes/LTFS1
The tape will appear in the Finder. There is also an eject button in the Finder.
Moving files and folders on LTFS tape
Files can be moved using the Finder. Files and folders can be renamed. Folders cannot be moved on tape.
However moving or renaming will mean that the reports will no longer be correct for the tape.
So after making the changes store a new index using the Source Browser. See Indexing Tapes
Then afterwards export new reports from the Project Browser.
Deleting files from LTFS tape
If files or folders are deleted from tape using the Finder or command line they will be removed from the tape directory. Note that the space is not recovered, so for this reason there is no delete function in YoYotta. If a large number of files need to be removed, then it is best to reformat the tape and start again. Files can be moved on tape, but folders cannot. So it is best to organise the structure on the source drive and also setup the destination paths before starting the job. It is also possible to create an empty folder on the source and link files and folders into it. Then turn on Copy linked files and folders in the Preferences panel.
Archiving linked files and folders
YoYotta LTFS can archive linked files and folders or symlinks. The linked material can be the tape or it can be on another volume. If the link points to another volume then when the tape is restored the link will be recreated.
However it can be easier to remove the links and copy the linked material instead. This will make the archive larger, but it will be self contained and can be restored anywhere. To do this turn on Copy linked files and folder in the Preferences panel.
Tapes auto mounting?
YoYotta does not automatically mount tapes unless a job is running. It is is best to wait and mount when you need to use the tape. If you want to see what's on a tape see YoYottaIndexing LTFS tapes.
Some other apps have this auto mount "feature". If a tape mounts automatically, (it may appear mounted with ltfs in the name) then you must disable this. Tapes mounted by other apps may not be recognised by YoYotta.
Retrospect can be disabled in System Preferences.
Codex has an Auto Mount Tapes switch in their preferences. Turn this off.
When all YoYotta jobs complete, all tapes will be unmounted, this ensures that the tape indexes are updated and helps protect their contents.
LTO drives and SAS interface TLR support
There is a SCSI protocol in the T10 SAS Protocol Layer (SPL-4) called TLR that is required for IBM tape drives.
Note that all LTO-7 and LTO-8 drives are manufactured by IBM.
With LTO-7 and LTO-8 the drive will use TLR because of the higher transfer speeds.
With LTO-6 the drive may use TLR for large file transfers.
If the SAS interface does not support TLR, then you will get read errors that are not signalled.
These errors would remain undetected unless the app used compares the data read back against a checksum of the written data.
HPE, IBM, Quantum, Tandberg and YoYotta only recommend SAS interfaces with TLR support.
mLogic mTape drives use a controller with TLR support, however some older units did not.
If you have verification issues with an mTape then send us a screenshot of the Apple System Information SAS page and we can advise you as to the type of SAS interface that you have.
If the interface does not support TLR then we recommend it's replacement. Alternatively contact your dealer for assistance.
You can find information on mTape Connection here, this link also shows how to access the Apple System Information pages.
For the Mac the ATTO H680 and H1280 PCIe cards support TLR, so these are the only interfaces we recommend.
To use the ATTO PCIe card with a Thunderbolt Mac put it into a Sonnet or other TB PCIe chassis.
H644 and H608 cards can also be used, but ensure they have the correct SAS connectors for your drive.
See this link for more details on the ATTO.
ATTO Thunderlink SH1068, SH2068 TB-SAS, ThunderStream SC4808 adaptors, R680 or R644 PCIe RAID SAS cards do not support TLR which is a required protocol. So they are not compatible with LTFS running on any operating system.
If you already have an ATTO R680 or Thunderlink there is a workaround for LTO-6 which limits the link speed, which might reduce errors. See this link for more details
However you will definitely need to replace the unit as soon as possible.
Highpoint RocketStor and RocketRAID devices are not compatible with LTFS + LTO drives.
The Highpoint 6328L is a SATA protocol interface. It has Mini-SAS connectors, but does not support SAS protocol. So this does not work with any SAS RAID or tape devices.
The Highpoint 6328 is a SAS protocol interface. But it does not support TLR protocol, so it cannot be used with LTO + LTFS.
We do not know of any fix for the HighPoint 6328, so this is not suitable for use with LTO-7, LTO-8 and LTFS.
© 2020 YoYotta Back to Top