Sponsors
Sponsor Products
Sequential read performance from LTFS
posted by Ken Spickler  on June 8, 2018, 6:35 p.m. (6 months, 3 days ago)
5 Responses     0 Plus One's     0 Comments  
Does anyone have any slick tricks to improve read performance of a DPX sequence from an LTFS tape? I'm trying to restore a large DPX sequence from an LTO7 and it's painfully slow. Doing a "cp" from the tape is getting me about 6- 4K DPX frames per minute. For giggles I also tried "cvcp" but that was no better. The tape is using the default options for LTFS.

TNX,
Ken

Thread Tags:
  discuss-at-studiosysadmins 

Response from Ken Spickler @ June 8, 2018, 10:05 p.m.
Nice. Thanks for the detail. Ill get my fork and knife and dig into this tonight.

Ken SpicklerSent from iPhone. Srry for tpos.
On Jun 8, 2018, at 6:52 PM, Steven Parker <stevenmichaelparker@gmail.com> wrote:

And now that i'm at a computer and have the links handy, here's the links to the tools. We just always used the IBM tools, since it seemed to work on our mixed use tape systems (Quantum, IBM, HP, etc) in standalone and library modes. It just seems to always work!
IBM LTFS Tools v2.4 for Linux, OSX, WindowsIBM LTFS Tools v2.2 for Linux, OSX, Windows
You should match the version of LTFS that wrote the tape, or any higher version. So, if they wrote it with 2.4, you need at least 2.4. But, 2.4 can read 2.2, etc...You're after the ltfscp binary, and the instructions on using it are here. Basically, always give all 3 parameters. To copy all files from a tape, -srcspec=*. For select copies, it can take a partial path from the tape as well, like *.dpx. Batch copy mode is if you know the exact files you want off the tape, you can build a text file and it will copy only those files. It defaults to recreating the directory structure into the dstpath for you, so it should mirror the file structure on the tape for you. No re-assembly required.
We eventually wrote a little python utility that checked for the existence of any "new" tapes (by checking at database), and auto-ingested them to our SAN. Other tools could then perform md5 calculations, compare manifests, extract metadata, etc. It's kind of neat. Shoot me an email off list of you like some more details on any of it!
-Steven
On Fri, Jun 8, 2018 at 6:09 PM, Steven Parker <stevenmichaelparker@gmail.com> wrote:
Yep. ltfscp will read the file order that the files exist on the tape and copy them to a disk. 
As a note, please use ltfscp to write files to a tape as well. As someone who received hundreds of ltfs formatted tapes a week for a large studio, your destination facility (and myself) will be thankful!

The toolset also includes other ltfs tools like ltfsck to check a tape for errors, and can try to recover badly or partially written ltfs tapes as well.
Sent from my iPhone. Sory fer tpyos!-Steven
On Jun 8, 2018, at 3:31 PM, Ken Spickler <ken.spickler@gmail.com> wrote:

Does anyone have any slick tricks to improve read performance of a DPX sequence from an LTFS tape?  I'm trying to restore a large DPX sequence from an LTO7 and it's painfully slow.  Doing a "cp" from the tape is getting me about 6- 4K DPX frames per minute.  For giggles I also tried "cvcp" but that was no better.  The tape is using the default options for LTFS.

TNX,
Ken
To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

0 Plus One's     0 Comments  
   

Response from Anonymous @ June 8, 2018, 9:55 p.m.
And now that i'm at a computer and have the links handy, here's the links to the tools. We just always used the IBM tools, since it seemed to work on our mixed use tape systems (Quantum, IBM, HP, etc) in standalone and library modes. It just seems to always work!
IBM LTFS Tools v2.4 for Linux, OSX, WindowsIBM LTFS Tools v2.2 for Linux, OSX, Windows
You should match the version of LTFS that wrote the tape, or any higher version. So, if they wrote it with 2.4, you need at least 2.4. But, 2.4 can read 2.2, etc...You're after the ltfscp binary, and the instructions on using it are here. Basically, always give all 3 parameters. To copy all files from a tape, -srcspec=*. For select copies, it can take a partial path from the tape as well, like *.dpx. Batch copy mode is if you know the exact files you want off the tape, you can build a text file and it will copy only those files. It defaults to recreating the directory structure into the dstpath for you, so it should mirror the file structure on the tape for you. No re-assembly required.
We eventually wrote a little python utility that checked for the existence of any "new" tapes (by checking at database), and auto-ingested them to our SAN. Other tools could then perform md5 calculations, compare manifests, extract metadata, etc. It's kind of neat.Shoot me an email off list of you like some more details on any of it!
-Steven
On Fri, Jun 8, 2018 at 6:09 PM, Steven Parker <stevenmichaelparker@gmail.com> wrote:
Yep. ltfscp will read the file order that the files exist on the tape and copy them to a disk.
As a note, please use ltfscp to write files to a tape as well. As someone who received hundreds of ltfs formatted tapes a week for a large studio, your destination facility (and myself) will be thankful!

The toolset also includes other ltfs tools like ltfsck to check a tape for errors, and can try to recover badly or partially written ltfs tapes as well.
Sent from my iPhone. Sory fer tpyos!-Steven
On Jun 8, 2018, at 3:31 PM, Ken Spickler <ken.spickler@gmail.com> wrote:

Does anyone have any slick tricks to improve read performance of a DPX sequence from an LTFS tape? I'm trying to restore a large DPX sequence from an LTO7 and it's painfully slow. Doing a "cp" from the tape is getting me about 6- 4K DPX frames per minute. For giggles I also tried "cvcp" but that was no better. The tape is using the default options for LTFS.

TNX,
Ken
To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Anonymous @ June 8, 2018, 9:10 p.m.
Yep. ltfscp will read the file order that the files exist on the tape and copy them to a disk. 
As a note, please use ltfscp to write files to a tape as well. As someone who received hundreds of ltfs formatted tapes a week for a large studio, your destination facility (and myself) will be thankful!

The toolset also includes other ltfs tools like ltfsck to check a tape for errors, and can try to recover badly or partially written ltfs tapes as well. 
Sent from my iPhone. Sory fer tpyos!-Steven
On Jun 8, 2018, at 3:31 PM, Ken Spickler <ken.spickler@gmail.com> wrote:

Does anyone have any slick tricks to improve read performance of a DPX sequence from an LTFS tape?  I'm trying to restore a large DPX sequence from an LTO7 and it's painfully slow.  Doing a "cp" from the tape is getting me about 6- 4K DPX frames per minute.  For giggles I also tried "cvcp" but that was no better.  The tape is using the default options for LTFS.

TNX,
Ken
To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

0 Plus One's     0 Comments  
   

Response from Ken Spickler @ June 8, 2018, 8:55 p.m.
Thanks. Will ltfscp read the files as theyre ordered on tape?  I can coalesce them into sequence with cvcp after theyre restored.

Ken SpicklerSent from iPhone. Srry for tpos.
On Jun 8, 2018, at 5:39 PM, Steven Parker <stevenmichaelparker@gmail.com> wrote:

Use ltfscp. Whoever wrote the tape didnt order files by name, so youre likely shoeshining the tape looking for assets in cps order. ltfscp should have come with your ltfs driver, or can be compiled from IBM ltfs support tools. 

Sent from my iPhone. Sory fer tpyos!- Steven
On Jun 8, 2018, at 3:31 PM, Ken Spickler <ken.spickler@gmail.com> wrote:

Does anyone have any slick tricks to improve read performance of a DPX sequence from an LTFS tape?  I'm trying to restore a large DPX sequence from an LTO7 and it's painfully slow.  Doing a "cp" from the tape is getting me about 6- 4K DPX frames per minute.  For giggles I also tried "cvcp" but that was no better.  The tape is using the default options for LTFS.

TNX,
Ken
To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe
To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

0 Plus One's     0 Comments  
   

Response from Anonymous @ June 8, 2018, 8:40 p.m.
Use ltfscp. Whoever wrote the tape didnt order files by name, so youre likely shoeshining the tape looking for assets in cps order. ltfscp should have come with your ltfs driver, or can be compiled from IBM ltfs support tools. 

Sent from my iPhone. Sory fer tpyos!- Steven
On Jun 8, 2018, at 3:31 PM, Ken Spickler <ken.spickler@gmail.com> wrote:

Does anyone have any slick tricks to improve read performance of a DPX sequence from an LTFS tape?  I'm trying to restore a large DPX sequence from an LTO7 and it's painfully slow.  Doing a "cp" from the tape is getting me about 6- 4K DPX frames per minute.  For giggles I also tried "cvcp" but that was no better.  The tape is using the default options for LTFS.

TNX,
Ken
To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

0 Plus One's     0 Comments