Management here wants confirmation from other studios about deleting files that are accessed on the network. I explained to them that when you delete a file from a network location that it does not go into a trashcan(OS X) or a recycle bin(Windows). Management believes that this is unacceptable, also feels that I may be overlooking something, and requested that I get confirmation from other studios if they deal with the same thing, as well are there are any recommended solutions for this? I'm aware that having a good backup solution in place is a guaranteed suggestion, aside from that are there any other suggestions?

Management is worried about users accidently deleting files from the show location that they had been working on.

So I figured the bigger studios out there could provide me some info and provide some information that I can present to management that will hopefully be acceptable to them.

So can you guys give me some feedback on this and let me know if it is the same for you? And what you do to try and curtail this?

Thanks!!

Curtis

Snapshots save our life. Snapshot technology is available from a variety of data storage vendors.

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Curtis Linstead Sent: Thursday, August 27, 2009 11:08 AM To: discuss@studiosysadmins.com Subject: Question about network deleting.

Management here wants confirmation from other studios about deleting files that are accessed on the network.

I explained to them that when you delete a file from a network location that it does not go into a trashcan(OS X) or a recycle bin(Windows).

Management believes that this is unacceptable, also feels that I may be overlooking something, and requested that I get confirmation from other studios if they deal with the same thing, as well are there are any recommended solutions for this? I'm aware that having a good backup solution in place is a guaranteed suggestion, aside from that are there any other suggestions?

Management is worried about users accidently deleting files from the show location that they had been working on.

So I figured the bigger studios out there could provide me some info and provide some information that I can present to management that will hopefully be acceptable to them.

So can you guys give me some feedback on this and let me know if it is the same for you? And what you do to try and curtail this?

Thanks!!

Curtis

We depend heavily on snapshots on our BlueArc and Isilon systems. Without these features you can easily lose files.

As I'm sure with most studios, your filesystem has to be pretty open so that all the artists can access the files they need. So there is no easy way (if there is a way) to lock files from being overwritten, inadvertently moved, or mistakenly deleted by a user.

We have snapshots running ... every 3 hours, keep them for a day ... every day, keep them for a week ... every week, keep them for 3 weeks

Even then we still have to go to tape sometimes, as some user will move/delete a directory and nobody will notice for some time...

It would be great if we could lock them down so they couldn't be moved/deleted, but then you would need a full time admin doing that on behalf of the users...

-John

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Curtis Linstead Sent: August 27, 2009 2:08 PM To: discuss@studiosysadmins.com Subject: Question about network deleting.

Management here wants confirmation from other studios about deleting files that are accessed on the network. I explained to them that when you delete a file from a network location that it does not go into a trashcan(OS X) or a recycle bin(Windows). Management believes that this is unacceptable, also feels that I may be overlooking something, and requested that I get confirmation from other studios if they deal with the same thing, as well are there are any recommended solutions for this? I'm aware that having a good backup solution in place is a guaranteed suggestion, aside from that are there any other suggestions?

Management is worried about users accidently deleting files from the show location that they had been working on.

So I figured the bigger studios out there could provide me some info and provide some information that I can present to management that will hopefully be acceptable to them.

So can you guys give me some feedback on this and let me know if it is the same for you? And what you do to try and curtail this?

Thanks!!

Curtis

I routinely recover stuff from a variety of types of filers using snapshot/snap backup systems (Isilon, Netapp,BlueArv, etc). One nice thing is that I've noticed that the snapshot structure on some brands of the filers (I'm looking at netapp right now) is actually available from the windows "Previous Versions" shadow copy system, precluding the user getting a data-wrangler involved.

From: Curtis Linstead Sent: Thursday, August 27, 2009 11:08 AM To: discuss@studiosysadmins.com Subject: Question about network deleting.

Management here wants confirmation from other studios about deleting files that are accessed on the network.

I explained to them that when you delete a file from a network location that it does not go into a trashcan(OS X) or a recycle bin(Windows).

Management believes that this is unacceptable, also feels that I may be overlooking something, and requested that I get confirmation from other studios if they deal with the same thing, as well are there are any recommended solutions for this? I'm aware that having a good backup solution in place is a guaranteed suggestion, aside from that are there any other suggestions?

Management is worried about users accidently deleting files from the show location that they had been working on.

So I figured the bigger studios out there could provide me some info and provide some information that I can present to management that will hopefully be acceptable to them.

So can you guys give me some feedback on this and let me know if it is the same for you? And what you do to try and curtail this?

Thanks!!

Curtis

On 2009-08-27, at 11:08 AM, Curtis Linstead wrote:

Management is worried about users accidently deleting files from the
show location that they had been working on.

Unless your filesystem has built-in snapshot technology (e.g. ZFS), or
hardware based snapshot and/or de-duplication, then a regular backup
strategy using rsync to disk, and/or to tape, is recommended. If you
lock every file and folder into place then artists can't use them
easily. One strategy I've used with Xsan is to have a read-only volume
for Elements which need to be accessed and copied but not deleted, and
a read/write main production volume. You can do the same with nfs I'm
sure (make it read-only).

Since I'm paranoid, as most sysadmin/IT are prone to be, I like to
have multiple backup strategies (which should all be tested before you
actually need them). Currently, I setup custom rsync jobs to various
disk and filesystems, including rsync to a ZFS backup volume which has
built-in snapshots. I'm also running incremental tape backup every
night and fulls on the weekend.

Hope that helps,

:)

Mat X

E: matxdotca@gmail.com T: 778-837-1036

Apple Certified Xsan 2 Admin, ACSP (10.5), ACTC (10.5)

Thank you John and BoPo for this info. I have looked into snapshots in the past as well, but current budget does not allow for the hardware that would support this. I would love to have snapshots implemented though.

Thank you and if anyone else has feedback on confirmation of this situation, I would greatly appreciate it as it will provide further insight into how other studios deal with the same situation.

Thanks again!

• CURTIS LINSTEAD

We also depend upon snapshots. Daily for 2 weeks and very rarely do I ever have to go back to tape. User's can do it themselves. Trashcans are a waste of system resources moving data from one location to another. We have locked the trash cans off as people would delete several GB which would end up in their home dir.

Dave ----- Original Message ----- From: "John Hickson" John.Hickson@Starz.com To: discuss@studiosysadmins.com Sent: Thursday, August 27, 2009 11:19:52 AM (GMT-0800) America/Los_Angeles Subject: RE: Question about network deleting.

We depend heavily on snapshots on our BlueArc and Isilon systems. Without these features you can easily lose files.

As I’m sure with most studios, your filesystem has to be pretty open so that all the artists can access the files they need. So there is no easy way (if there is a way) to lock files from being overwritten, inadvertently moved, or mistakenly deleted by a user.

We have snapshots running

… every 3 hours, keep them for a day

… every day, keep them for a week

… every week, keep them for 3 weeks

Even then we still have to go to tape sometimes, as some user will move/delete a directory and nobody will notice for some time…

It would be great if we could lock them down so they couldn’t be moved/deleted, but then you would need a full time admin doing that on behalf of the users…

-John

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Curtis Linstead Sent: August 27, 2009 2:08 PM To: discuss@studiosysadmins.com Subject: Question about network deleting.

Management here wants confirmation from other studios about deleting files that are accessed on the network.

I explained to them that when you delete a file from a network location that it does not go into a trashcan(OS X) or a recycle bin(Windows).

Management believes that this is unacceptable, also feels that I may be overlooking something, and requested that I get confirmation from other studios if they deal with the same thing, as well are there are any recommended solutions for this? I’m aware that having a good backup solution in place is a guaranteed suggestion, aside from that are there any other suggestions?

Management is worried about users accidently deleting files from the show location that they had been working on.

So I figured the bigger studios out there could provide me some info and provide some information that I can present to management that will hopefully be acceptable to them.

So can you guys give me some feedback on this and let me know if it is the same for you? And what you do to try and curtail this?

Thanks!!

Curtis

-- Dave Algar Principal Systems Administrator Rainmaker Entertainment Inc. 500 - 2025 W. Broadway Vancouver, BC Canada, V6J 1Z6

Phone: (604) 714-2628, Fax: (604) 714-2649, E-Mail: dalgar@rainmaker.com

The problem with doing nothing is knowing when you're done.

-- This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.

Once more thanks again all for the info it helped in allowing me to let Management know and understand that our option was implementing a proper backup and/or snapshot.

• CURTIS LINSTEAD

• TECH SUPPORT ¦ SUPPORT TECHNIQUE ———————————————————— T 514.397.9999 x405 c 514.570.4345 w www.rodeofx.com

  [cid:image001.gif@01CA2728.8F0A1C20] [ R O D E O F X ]

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Dave Algar Sent: Thursday, August 27, 2009 2:33 PM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

We also depend upon snapshots. Daily for 2 weeks and very rarely do I ever have to go back to tape. User's can do it themselves. Trashcans are a waste of system resources moving data from one location to another. We have locked the trash cans off as people would delete several GB which would end up in their home dir.

Dave ----- Original Message ----- From: "John Hickson" John.Hickson@Starz.com To: discuss@studiosysadmins.com Sent: Thursday, August 27, 2009 11:19:52 AM (GMT-0800) America/Los_Angeles Subject: RE: Question about network deleting.

We depend heavily on snapshots on our BlueArc and Isilon systems. Without these features you can easily lose files.

As I’m sure with most studios, your filesystem has to be pretty open so that all the artists can access the files they need. So there is no easy way (if there is a way) to lock files from being overwritten, inadvertently moved, or mistakenly deleted by a user.

We have snapshots running … every 3 hours, keep them for a day … every day, keep them for a week … every week, keep them for 3 weeks

Even then we still have to go to tape sometimes, as some user will move/delete a directory and nobody will notice for some time…

It would be great if we could lock them down so they couldn’t be moved/deleted, but then you would need a full time admin doing that on behalf of the users…

-John

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Curtis Linstead Sent: August 27, 2009 2:08 PM To: discuss@studiosysadmins.com Subject: Question about network deleting.

Management here wants confirmation from other studios about deleting files that are accessed on the network. I explained to them that when you delete a file from a network location that it does not go into a trashcan(OS X) or a recycle bin(Windows). Management believes that this is unacceptable, also feels that I may be overlooking something, and requested that I get confirmation from other studios if they deal with the same thing, as well are there are any recommended solutions for this? I’m aware that having a good backup solution in place is a guaranteed suggestion, aside from that are there any other suggestions?

Management is worried about users accidently deleting files from the show location that they had been working on.

So I figured the bigger studios out there could provide me some info and provide some information that I can present to management that will hopefully be acceptable to them.

So can you guys give me some feedback on this and let me know if it is the same for you? And what you do to try and curtail this?

Thanks!!

Curtis

-- Dave Algar Principal Systems Administrator Rainmaker Entertainment Inc. 500 - 2025 W. Broadway Vancouver, BC Canada, V6J 1Z6

Phone: (604) 714-2628, Fax: (604) 714-2649, E-Mail: dalgar@rainmaker.com

The problem with doing nothing is knowing when you're done.

Hi Curtis,

If your users are accessing the share trought Samba, you can create a "network recycle bin". You just need to run a cleanup script every 3 days to clean it.

here is an example : http://ubuntuforums.org/showthread.php?t=155763

Salut ! Dan !

Dan, Hi and thanks for the link! Just a question, this seems like it is for Linux, our Samba server is actually an OSX system, don't think it would work. Good info, I'll check into it.

• CURTIS LINSTEAD

Hi Curtis,

You can try it, it should work. Here is another link from apple: http://developer.apple.com/documentation/Darwin/Reference/ManPages/man8/vfs_recycle.8.html

Try a basic config and test it with testparm before reloading samba.

Salut ! Dan !

I'm not sure what your hardware budget is (if its zero, obviously you can't buy anything) but building a snapshot server is actually pretty ridiculously cheap. We had to build one on an extreme budget and ended up picking up a motherboard with 6xSATA connectors, used Linux software RAID and rsnapshot, and put something together for around 1500 bucks. It is not the fastest or most elegant solution, but it has saved artists' asses on multiple occasions.

(I learned to love snapshots the first week I ever started working in the industry. I was essentially a render monkey on a job and managed to wipe out an entire shot from the file system. Started freaking out, figuring I was going to get fired for gross incompetence, but went and had a quiet word with the IT guy, who restored everything from the snapshot and no one was ever any wiser. Since then, I've always been all about snapshots).

On Thu, Aug 27, 2009 at 2:30 PM, Curtis Linstead curtis@rodeofx.com wrote:

Thank you John and BoPo for this info.

I have looked into snapshots in the past as well, but current budget does not allow for the hardware that would support this.

I would love to have snapshots implemented though.

--- On Thu, 8/27/09, Jesse C crimson.corelio@gmail.com wrote:

rsnapshot

That looks very useful. I wrote something myself (and I'm sure a few other admins on the list did, too) to do exactly the same thing a few years ago (rsync + hard links for space-efficient daily backups). Nice to see it as an open source package with an easy config file and an active maintainer.

Andrew

Yahoo! Canada Toolbar: Search from anywhere on the web, and bookmark your favourite sites. Download it now http://ca.toolbar.yahoo.com.

Jesse, Thanks for the info! I will definitely look further into this as a potential option for us. How large was the pool you were doing the snapshot on and how large was the RAID you built? >From what I'm reading I would need the size of one full backup plus a little more and my current data pool can hit upwards of 16TB. Not sure how easily I could configure that on a montherboard based raid heh, but it definitely interests me. Thanks again!

• CURTIS LINSTEAD

We're currently snapshotting between 1.8TB and 2.3TB of production files with 2.7TB of snapshot disk. We snapshot every 6 hours and keep snapshots for a month by default, but we've got cron scripts that go through and remove older snapshots if we start running out disk space (which can happen if there has been ridiculous amounts of churn in the file system).

We could easily increase the size of our snapshot if we needed to. We've got 5x750GB drives right now in the RAID. If we had to, we could expand out to 6 drives in the RAID and move up to 2TB drives, which would give us close to 10TB. Not quite what you guys need, but there are motherboards out there with 10xSATA ports on them, which when combined with 2TB drives, would get you more into your ballpark.

On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead curtis@rodeofx.com wrote:

Jesse,

Thanks for the info! I will definitely look further into this as a potential option for us.

How large was the pool you were doing the snapshot on and how large was the RAID you built?

From what I?m reading I would need the size of one full backup plus a little more and my current data pool can hit upwards of 16TB.

Not sure how easily I could configure that on a montherboard based raid heh, but it definitely interests me.

Thanks again!

Just followed this thread and wanted to give my two cents on this.
Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then setup a
cheap RAIDed solution as Jesse C suggests, and do an rsync every night
from 12 midnight to 6am on the folders you need rsynced. Its really
quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up on
Linux for under $2000 and with a more stable and secure MAC OSX server
and XRAID for under $5000. The RAIDed linux software solution would
yield about 4TB of usable space, and the MAC Server XRAID solution
would yield around 6TB of usable space. Keep in mind that Apple only
delivered IDE XRAIDs with maximum 500GB IDE disks. I bought the 750GB
disks from CDW (OEM) and they work just fine under non-warranty
conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of low
to minimal production. And make this KNOWN to the artists and folks
that there are only 24 hour back ups as there is no money for a more
expensive solution.

The more expensive hardware like Isolons and NetApps and especially
BlueArcs have more elegant solutions and they will actually give you
scripts. In Blue Arcs case, they actually have prewritten stuff as
part of the deal. SUN MIcroSystems is a class in itself and are
excellent in this type of support. If you pay $70,000 for a small 9TB
usable volume (16TB raw), you might expect them to bend over backwards
to help you out. And in the three companies I have recently worked for
since SONY, they did (Jim Henson Co, YuCo, and StudioGPU). I rewrote
some stuff elegantly and the snap shots approached runtime and were
not ~quasi~ snapshots or nightly backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote:

We're currently snapshotting between 1.8TB and 2.3TB of production
files with 2.7TB of snapshot disk. We snapshot every 6 hours and
keep snapshots for a month by default, but we've got cron scripts
that go through and remove older snapshots if we start running out
disk space (which can happen if there has been ridiculous amounts of
churn in the file system).

We could easily increase the size of our snapshot if we needed to.
We've got 5x750GB drives right now in the RAID. If we had to, we
could expand out to 6 drives in the RAID and move up to 2TB drives,
which would give us close to 10TB. Not quite what you guys need,
but there are motherboards out there with 10xSATA ports on them,
which when combined with 2TB drives, would get you more into your
ballpark.

On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead
curtis@rodeofx.com wrote: Jesse,

Thanks for the info! I will definitely look further into this as a potential option for us.

How large was the pool you were doing the snapshot on and how large
was the RAID you built?

From what I?m reading I would need the size of one full backup plus
a little more and my current data pool can hit upwards of 16TB.

Not sure how easily I could configure that on a montherboard based
raid heh, but it definitely interests me.

Thanks again!

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Jorg-Ulrich Mohnen, M.Sc. MBA WavGen / Terratracer Incorporated 901 North Curson Avenue West Hollywood, CA 90046 Tel 888-654-5615 Email jorg.mohnen@wavgen.com

For the sake of accuracy, Apple shipped the Xserve RAID (the XRAID is
entirely a different product - http://www.getactivestorage.com) with
750GB drives. The 750GB drive modules were backwards compatible to the
first gen Xserve RAID (SFP) from 2004.

http://support.apple.com/kb/HT1219

~Ian

On Aug 28, 2009, at 12:20 PM, Jorg-Ulrich Mohnen, M.Sc., MBA wrote:

Just followed this thread and wanted to give my two cents on this.
Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then setup
a cheap RAIDed solution as Jesse C suggests, and do an rsync every
night from 12 midnight to 6am on the folders you need rsynced. Its
really quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up on
Linux for under $2000 and with a more stable and secure MAC OSX
server and XRAID for under $5000. The RAIDed linux software solution
would yield about 4TB of usable space, and the MAC Server XRAID
solution would yield around 6TB of usable space. Keep in mind that
Apple only delivered IDE XRAIDs with maximum 500GB IDE disks. I
bought the 750GB disks from CDW (OEM) and they work just fine under
non-warranty conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of low
to minimal production. And make this KNOWN to the artists and folks
that there are only 24 hour back ups as there is no money for a more
expensive solution.

The more expensive hardware like Isolons and NetApps and especially
BlueArcs have more elegant solutions and they will actually give you
scripts. In Blue Arcs case, they actually have prewritten stuff as
part of the deal. SUN MIcroSystems is a class in itself and are
excellent in this type of support. If you pay $70,000 for a small
9TB usable volume (16TB raw), you might expect them to bend over
backwards to help you out. And in the three companies I have
recently worked for since SONY, they did (Jim Henson Co, YuCo, and
StudioGPU). I rewrote some stuff elegantly and the snap shots
approached runtime and were not ~quasi~ snapshots or nightly
backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote:

We're currently snapshotting between 1.8TB and 2.3TB of production
files with 2.7TB of snapshot disk. We snapshot every 6 hours and
keep snapshots for a month by default, but we've got cron scripts
that go through and remove older snapshots if we start running out
disk space (which can happen if there has been ridiculous amounts
of churn in the file system).

We could easily increase the size of our snapshot if we needed to.
We've got 5x750GB drives right now in the RAID. If we had to, we
could expand out to 6 drives in the RAID and move up to 2TB drives,
which would give us close to 10TB. Not quite what you guys need,
but there are motherboards out there with 10xSATA ports on them,
which when combined with 2TB drives, would get you more into your
ballpark.

On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead
curtis@rodeofx.com wrote: Jesse,

Thanks for the info! I will definitely look further into this as a potential option for
us.

How large was the pool you were doing the snapshot on and how large
was the RAID you built?

From what I?m reading I would need the size of one full backup plus
a little more and my current data pool can hit upwards of 16TB.

Not sure how easily I could configure that on a montherboard based
raid heh, but it definitely interests me.

Thanks again!

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Jorg-Ulrich Mohnen, M.Sc. MBA WavGen / Terratracer Incorporated 901 North Curson Avenue West Hollywood, CA 90046 Tel 888-654-5615 Email jorg.mohnen@wavgen.com

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Did you opt for using the built-in SATA connectors for price or performance? There's quite a few cards in the $500 range that offload the workload from the CPU but from what I've been seeing, the on-board RAID on some motherboards seems to outperform them.

On Thu, Aug 27, 2009 at 6:08 PM, Jesse C crimson.corelio@gmail.com wrote:

I'm not sure what your hardware budget is (if its zero, obviously you can't buy anything) but building a snapshot server is actually pretty ridiculously cheap. We had to build one on an extreme budget and ended up picking up a motherboard with 6xSATA connectors, used Linux software RAID and rsnapshot, and put something together for around 1500 bucks. It is not the fastest or most elegant solution, but it has saved artists' asses on multiple occasions.

(I learned to love snapshots the first week I ever started working in the industry. I was essentially a render monkey on a job and managed to wipe out an entire shot from the file system. Started freaking out, figuring I was going to get fired for gross incompetence, but went and had a quiet word with the IT guy, who restored everything from the snapshot and no one was ever any wiser. Since then, I've always been all about snapshots).

On Thu, Aug 27, 2009 at 2:30 PM, Curtis Linstead curtis@rodeofx.comwrote:

Thank you John and BoPo for this info.

I have looked into snapshots in the past as well, but current budget does not allow for the hardware that would support this.

I would love to have snapshots implemented though.

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Since all the machine is doing is the snapshotting, it didn't really seem all that necessary to offload CPU workload. I considered getting one, but decided against it to keep costs down.

As a warning note, we discovered that some motherboard raids are not supported in Linux. The IP35 chipset (which we're using) does not have linux driver support for the motherboard raid. All the drives will show up and we've using linux's software raid, but we're obviously losing a good deal of performance that way. Make sure the chipset has driver support if you're going with a motherboard raid.

jesse

On Sat, Aug 29, 2009 at 12:53 AM, Daniel Roizman roizman@kolektiv.comwrote:

Did you opt for using the built-in SATA connectors for price or performance? There's quite a few cards in the $500 range that offload the workload from the CPU but from what I've been seeing, the on-board RAID on some motherboards seems to outperform them.

--- On Fri, 8/28/09, Jesse C crimson.corelio@gmail.com wrote:

All the drives will show up and we've using linux's software raid, but we're obviously losing a good deal of performance that way.

Are you sure?

Andrew

Get a sneak peak at messages with a handy reading pane with All new Yahoo! Mail: http://ca.promos.yahoo.com/newmail/overview2/

Another free non rsync approach is a 'copy on write' file system, such as ZFS. I took a look at a snapshot solution on linux using XFS at a previous job which worked as expected. I'm sure there are other filesystems out there that support it. I don't think you have to spend a lot of money to get file level protection, its more a matter of available capacity.

-g

On 8/28/09 12:20 PM, "Jorg-Ulrich Mohnen, M.Sc., MBA" jorg.mohnen@wavgen.com wrote:

Just followed this thread and wanted to give my two cents on this. Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then setup a cheap RAIDed solution as Jesse C suggests, and do an rsync every night from 12 midnight to 6am on the folders you need rsynced. Its really quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up on Linux for under $2000 and with a more stable and secure MAC OSX server and XRAID for under $5000. The RAIDed linux software solution would yield about 4TB of usable space, and the MAC Server XRAID solution would yield around 6TB of usable space. Keep in mind that Apple only delivered IDE XRAIDs with maximum 500GB IDE disks. I bought the 750GB disks from CDW (OEM) and they work just fine under non-warranty conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of low to minimal production. And make this KNOWN to the artists and folks that there are only 24 hour back ups as there is no money for a more expensive solution.

The more expensive hardware like Isolons and NetApps and especially BlueArcs have more elegant solutions and they will actually give you scripts. In Blue Arcs case, they actually have prewritten stuff as part of the deal. SUN MIcroSystems is a class in itself and are excellent in this type of support. If you pay $70,000 for a small 9TB usable volume (16TB raw), you might expect them to bend over backwards to help you out. And in the three companies I have recently worked for since SONY, they did (Jim Henson Co, YuCo, and StudioGPU). I rewrote some stuff elegantly and the snap shots approached runtime and were not ~quasi~ snapshots or nightly backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote:

We're currently snapshotting between 1.8TB and 2.3TB of production files with 2.7TB of snapshot disk. We snapshot every 6 hours and keep snapshots for a month by default, but we've got cron scripts that go through and remove older snapshots if we start running out disk space (which can happen if there has been ridiculous amounts of churn in the file system).

We could easily increase the size of our snapshot if we needed to. We've got 5x750GB drives right now in the RAID. If we had to, we could expand out to 6 drives in the RAID and move up to 2TB drives, which would give us close to 10TB. Not quite what you guys need, but there are motherboards out there with 10xSATA ports on them, which when combined with 2TB drives, would get you more into your ballpark.

On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead curtis@rodeofx.com wrote:

Jesse,

Thanks for the info! I will definitely look further into this as a potential option for us.

How large was the pool you were doing the snapshot on and how large was the RAID you built?

From what I'm reading I would need the size of one full backup plus a little more and my current data pool can hit upwards of 16TB.

Not sure how easily I could configure that on a montherboard based raid heh, but it definitely interests me.

Thanks again!

Jorg-Ulrich Mohnen, M.Sc. MBA WavGen / Terratracer Incorporated 901 North Curson Avenue West Hollywood, CA 90046 Tel 888-654-5615 Email jorg.mohnen@wavgen.com

well, not 100% sure since I can't run it in hardware raid mode to test the difference. I would suspect forcing all the RAID computations onto the main CPU rather than the specialized RAID controller would result in a performance hit. Is that a bad assumption?

jesse

On Mon, Aug 31, 2009 at 3:29 PM, Andrew Klaassen clawsoon@yahoo.com wrote:

--- On Fri, 8/28/09, Jesse C crimson.corelio@gmail.com wrote:

All the drives will show up and we've using linux's software raid, but we're obviously losing a good deal of performance that way.

Are you sure?

Andrew

Jesse C wrote: > well, not 100% sure since I can't run it in hardware raid mode to test > the difference. I would suspect forcing all the RAID computations onto > the main CPU rather than the specialized RAID controller would result in > a performance hit. Is that a bad assumption?

I'd think there'd only be a performance hit if you're hitting
actual bottlenecks with the cpu raid software.I've seen software based raids perform quite well and not get
cpu loaded during heavy use. Its activity was almost invisible.Usually other things caused the bottle necks;
network i/o, nfs crapouts, and a biggie in linux
(some years ago anyway) was file system throughput,
where ext2 was freaking out, causing weird load issues
which we could replicate with and without the raid in place.
I think Bill Shortell debugged this quite a bit too
around the same time I did. I love linux, but esp at that time
I recommended against it for use as a file server for render farms
because of its instability under load.There were problems in NFS too, esp. with sgi clients
causing files on the linux server to appear and disappear
in ls(1) listings, which IIRC was due to the 32bit linux
server's handling of SGI's 64bit nfs clients. There were
kernel patches for that problem, but even when applied,
problems would still pop up.

Raid has been around for a very long time. The actual math the processor is drop dead simple. High end raid cards of yester year we using "simple" processors, z80 class-ish. That being said, today's cpus are basically untouched when calculating parity and other raid functions for the smallish disk sets they are servicing.

Bottlenecks would be seen on the (a) bus when spindle count gets high I'd think.

Hardware raid brings to the table things like battery backed up cache, hot swap, array building tools, etc. In my opinion hw raid doesn't out perform sw raid (implemented on modern server class hardware) by any great amount until certain thresholds are passed.

G

well, not 100% sure since I can't run it in hardware raid mode to test the difference. I would suspect forcing all the RAID computations onto the main CPU rather than the specialized RAID controller would result in a performance hit. Is that a bad assumption?

jesse

On Mon, Aug 31, 2009 at 3:29 PM, Andrew Klaassen <clawsoon@yahoo.comclawsoon@yahoo.com> wrote: --- On Fri, 8/28/09, Jesse C <crimson.corelio@gmail.comcrimson.corelio@gmail.com> wrote:

All the drives will show up and we've using linux's software raid, but we're obviously losing a good deal of performance that way.

Are you sure?

Andrew

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-chea p-cloud-storage/

Klaus

On 8/31/09 12:43 PM, "Greg Whynott" Greg.Whynott@oicr.on.ca etched on stone tablets:

Another free non rsync approach is a ?copy on write? file system, such as ZFS. I took a look at a snapshot solution on linux using XFS at a previous job which worked as expected. I?m sure there are other filesystems out there that support it. I don?t think you have to spend a lot of money to get file level protection, its more a matter of available capacity.

-g

On 8/28/09 12:20 PM, "Jorg-Ulrich Mohnen, M.Sc., MBA" jorg.mohnen@wavgen.com wrote:

Just followed this thread and wanted to give my two cents on this. Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then setup a cheap RAIDed solution as Jesse C suggests, and do an rsync every night from 12 midnight to 6am on the folders you need rsynced. Its really quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up on Linux for under $2000 and with a more stable and secure MAC OSX server and XRAID for under $5000. The RAIDed linux software solution would yield about 4TB of usable space, and the MAC Server XRAID solution would yield around 6TB of usable space. Keep in mind that Apple only delivered IDE XRAIDs with maximum 500GB IDE disks. I bought the 750GB disks from CDW (OEM) and they work just fine under non-warranty conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of low to minimal production. And make this KNOWN to the artists and folks that there are only 24 hour back ups as there is no money for a more expensive solution.

The more expensive hardware like Isolons and NetApps and especially BlueArcs have more elegant solutions and they will actually give you scripts. In Blue Arcs case, they actually have prewritten stuff as part of the deal. SUN MIcroSystems is a class in itself and are excellent in this type of support. If you pay $70,000 for a small 9TB usable volume (16TB raw), you might expect them to bend over backwards to help you out. And in the three companies I have recently worked for since SONY, they did (Jim Henson Co, YuCo, and StudioGPU). I rewrote some stuff elegantly and the snap shots approached runtime and were not ~quasi~ snapshots or nightly backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote:

We're currently snapshotting between 1.8TB and 2.3TB of production files with 2.7TB of snapshot disk. We snapshot every 6 hours and keep snapshots for a month by default, but we've got cron scripts that go through and remove older snapshots if we start running out disk space (which can happen if there has been ridiculous amounts of churn in the file system).
We could easily increase the size of our snapshot if we needed to. We've got 5x750GB drives right now in the RAID. If we had to, we could expand out to 6 drives in the RAID and move up to 2TB drives, which would give us close to 10TB. Not quite what you guys need, but there are motherboards out there with 10xSATA ports on them, which when combined with 2TB drives, would get you more into your ballpark.
On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead curtis@rodeofx.com wrote: >
>
> Jesse, > > Thanks for the info! > I will definitely look further into this as a potential option for us. > > How large was the pool you were doing the snapshot on and how large was the > RAID you built? > >> >From what I?m reading I would need the size of one full backup plus a >> little more and my current data pool can hit upwards of 16TB. > > Not sure how easily I could configure that on a montherboard based raid > heh, but it definitely interests me. > > Thanks again! > >

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Jorg-Ulrich Mohnen, M.Sc. MBA WavGen / Terratracer Incorporated 901 North Curson Avenue West Hollywood, CA 90046 Tel 888-654-5615 Email jorg.mohnen@wavgen.com

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Would love to see some performance specs on that thing. Especially with a 10Gig Ethernet connection. WOW. Article brings up another good point about the speed of http for file transfer. Anyone using http/https instead of nfs/smb/etc..? I suppose it would take quite a bit of work to setup a web-based facility but the scalability options (especially in regards to an increasing presence of offsite freelancers and a decentralization of the studio) are definitely attractive.

Michael Oliver

Look Effects, Inc.

moliver@lookfx.com

wk: 323.469.4230 x111

cell: 858.336.1438

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Klaus Steden Sent: Tuesday, September 01, 2009 5:06 PM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-chea p-cloud-storage/

Klaus

On 8/31/09 12:43 PM, "Greg Whynott" Greg.Whynott@oicr.on.ca etched on stone tablets:

Another free non rsync approach is a 'copy on write' file system, such as ZFS. I took a look at a snapshot solution on linux using XFS at a previous job which worked as expected. I'm sure there are other filesystems out there that support it. I don't think you have to spend a lot of money to get file level protection, its more a matter of available capacity.

-g

On 8/28/09 12:20 PM, "Jorg-Ulrich Mohnen, M.Sc., MBA" jorg.mohnen@wavgen.com wrote:

Just followed this thread and wanted to give my two cents on this. Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then setup a cheap RAIDed solution as Jesse C suggests, and do an rsync every night from 12 midnight to 6am on the folders you need rsynced. Its really quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up on Linux for under $2000 and with a more stable and secure MAC OSX server and XRAID for under $5000. The RAIDed linux software solution would yield about 4TB of usable space, and the MAC Server XRAID solution would yield around 6TB of usable space. Keep in mind that Apple only delivered IDE XRAIDs with maximum 500GB IDE disks. I bought the 750GB disks from CDW (OEM) and they work just fine under non-warranty conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of low to minimal production. And make this KNOWN to the artists and folks that there are only 24 hour back ups as there is no money for a more expensive solution.

The more expensive hardware like Isolons and NetApps and especially BlueArcs have more elegant solutions and they will actually give you scripts. In Blue Arcs case, they actually have prewritten stuff as part of the deal. SUN MIcroSystems is a class in itself and are excellent in this type of support. If you pay $70,000 for a small 9TB usable volume (16TB raw), you might expect them to bend over backwards to help you out. And in the three companies I have recently worked for since SONY, they did (Jim Henson Co, YuCo, and StudioGPU). I rewrote some stuff elegantly and the snap shots approached runtime and were not ~quasi~ snapshots or nightly backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote:

We're currently snapshotting between 1.8TB and 2.3TB of production files with 2.7TB of snapshot disk. We snapshot every 6 hours and keep snapshots for a month by default, but we've got cron scripts that go through and remove older snapshots if we start running out disk space (which can happen if there has been ridiculous amounts of churn in the file system).

We could easily increase the size of our snapshot if we needed to. We've got 5x750GB drives right now in the RAID. If we had to, we could expand out to 6 drives in the RAID and move up to 2TB drives, which would give us close to 10TB. Not quite what you guys need, but there are motherboards out there with 10xSATA ports on them, which when combined with 2TB drives, would get you more into your ballpark.

On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead curtis@rodeofx.com wrote:

Jesse,

Thanks for the info! I will definitely look further into this as a potential option for us.

How large was the pool you were doing the snapshot on and how large was the RAID you built?

From what I'm reading I would need the size of one full backup plus a

little more and my current data pool can hit upwards of 16TB.

Not sure how easily I could configure that on a montherboard based raid heh, but it definitely interests me.

Thanks again!

Jorg-Ulrich Mohnen, M.Sc. MBA WavGen / Terratracer Incorporated 901 North Curson Avenue West Hollywood, CA 90046 Tel 888-654-5615 Email jorg.mohnen@wavgen.com

No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.409 / Virus Database: 270.13.74/2339 - Release Date: 09/01/09 06:52:00

wow, great that they are sharing their design. one thing I wonder about is PSU failure. from the way he outlined the connections, the raid is spread across PSU's so if one PSU goes down, more than 2 drives will fail which would potentially corrupt the raid, no?

with google sharing some of their server specs and blackblaze sharing their storage, i wonder how long it'll be before we start seeing some opensource hardware configurations tightly tuned with linux distros.

On Tue, Sep 1, 2009 at 5:06 PM, Klaus Steden klaus-s@moving-picture.comwrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

Klaus

On 8/31/09 12:43 PM, "Greg Whynott" Greg.Whynott@oicr.on.ca etched on stone tablets:

Another free non rsync approach is a ?copy on write? file system, such as ZFS. I took a look at a snapshot solution on linux using XFS at a previous job which worked as expected. I?m sure there are other filesystems out there that support it. I don?t think you have to spend a lot of money to get file level protection, its more a matter of available capacity.

-g

On 8/28/09 12:20 PM, "Jorg-Ulrich Mohnen, M.Sc., MBA" jorg.mohnen@wavgen.com wrote:

Just followed this thread and wanted to give my two cents on this. Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then setup a cheap RAIDed solution as Jesse C suggests, and do an rsync every night from 12 midnight to 6am on the folders you need rsynced. Its really quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up on Linux for under $2000 and with a more stable and secure MAC OSX server and XRAID for under $5000. The RAIDed linux software solution would yield about 4TB of usable space, and the MAC Server XRAID solution would yield around 6TB of usable space. Keep in mind that Apple only delivered IDE XRAIDs with maximum 500GB IDE disks. I bought the 750GB disks from CDW (OEM) and they work just fine under non-warranty conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of low to minimal production. And make this KNOWN to the artists and folks that there are only 24 hour back ups as there is no money for a more expensive solution.

The more expensive hardware like Isolons and NetApps and especially BlueArcs have more elegant solutions and they will actually give you scripts. In Blue Arcs case, they actually have prewritten stuff as part of the deal. SUN MIcroSystems is a class in itself and are excellent in this type of support. If you pay $70,000 for a small 9TB usable volume (16TB raw), you might expect them to bend over backwards to help you out. And in the three companies I have recently worked for since SONY, they did (Jim Henson Co, YuCo, and StudioGPU). I rewrote some stuff elegantly and the snap shots approached runtime and were not ~quasi~ snapshots or nightly backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote:

We're currently snapshotting between 1.8TB and 2.3TB of production files with 2.7TB of snapshot disk. We snapshot every 6 hours and keep snapshots for a month by default, but we've got cron scripts that go through and remove older snapshots if we start running out disk space (which can happen if there has been ridiculous amounts of churn in the file system).

We could easily increase the size of our snapshot if we needed to. We've got 5x750GB drives right now in the RAID. If we had to, we could expand out to 6 drives in the RAID and move up to 2TB drives, which would give us close to 10TB. Not quite what you guys need, but there are motherboards out there with 10xSATA ports on them, which when combined with 2TB drives, would get you more into your ballpark.

On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead curtis@rodeofx.com wrote:

Jesse,

Thanks for the info! I will definitely look further into this as a potential option for us.

How large was the pool you were doing the snapshot on and how large was the RAID you built?

From what I?m reading I would need the size of one full backup plus a

little more and my current data pool can hit upwards of 16TB.

Not sure how easily I could configure that on a montherboard based raid heh, but it definitely interests me.

Thanks again!

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Jorg-Ulrich Mohnen, M.Sc. MBA WavGen / Terratracer Incorporated 901 North Curson Avenue West Hollywood, CA 90046 Tel 888-654-5615 Email jorg.mohnen@wavgen.com

------------------------------ _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

One thing that struck me was that they're using cheap PSUs too, so they're more likely to fail but more to the point they're comparatively inefficient. The post mentions that electricity costs more than bandwidth so wouldn't it make more sense to use PSUs that waste less electricity and generate less heat?

On Tue, 1 Sep 2009, Daniel Roizman wrote:

wow, great that they are sharing their design. one thing I wonder about is PSU failure. from the way he outlined the connections, the raid is spread across PSU's so if one PSU goes down, more than 2 drives will fail which would potentially corrupt the raid, no?

with google sharing some of their server specs and blackblaze sharing their storage, i wonder how long it'll be before we start seeing some opensource hardware configurations tightly tuned with linux distros.

[snip]

-Jason

Hm.

I'm wondering if a collection of backblaze systems could, if organized into a Lustre file system and front-ended by an adequate Linux Samba/NFS file server, provide a useable and relatively high-density, low-cost solution for multi-hundred TB online archives.

Has anyone on the list been working with Lustre?

• Sean

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Daniel Roizman Sent: Tuesday, September 01, 2009 8:07 PM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

wow, great that they are sharing their design. one thing I wonder about is PSU failure. from the way he outlined the connections, the raid is spread across PSU's so if one PSU goes down, more than 2 drives will fail which would potentially corrupt the raid, no?

with google sharing some of their server specs and blackblaze sharing their storage, i wonder how long it'll be before we start seeing some opensource hardware configurations tightly tuned with linux distros.

On Tue, Sep 1, 2009 at 5:06 PM, Klaus Steden <klaus-s@moving-picture.comklaus-s@moving-picture.com> wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

Klaus

On 8/31/09 12:43 PM, "Greg Whynott" <Greg.Whynott@oicr.on.caGreg.Whynott@oicr.on.ca> etched on stone tablets: Another free non rsync approach is a 'copy on write' file system, such as ZFS. I took a look at a snapshot solution on linux using XFS at a previous job which worked as expected. I'm sure there are other filesystems out there that support it. I don't think you have to spend a lot of money to get file level protection, its more a matter of available capacity.

-g

On 8/28/09 12:20 PM, "Jorg-Ulrich Mohnen, M.Sc., MBA" <jorg.mohnen@wavgen.comjorg.mohnen@wavgen.com> wrote: Just followed this thread and wanted to give my two cents on this. Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then setup a cheap RAIDed solution as Jesse C suggests, and do an rsync every night from 12 midnight to 6am on the folders you need rsynced. Its really quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up on Linux for under $2000 and with a more stable and secure MAC OSX server and XRAID for under $5000. The RAIDed linux software solution would yield about 4TB of usable space, and the MAC Server XRAID solution would yield around 6TB of usable space. Keep in mind that Apple only delivered IDE XRAIDs with maximum 500GB IDE disks. I bought the 750GB disks from CDW (OEM) and they work just fine under non-warranty conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of low to minimal production. And make this KNOWN to the artists and folks that there are only 24 hour back ups as there is no money for a more expensive solution.

The more expensive hardware like Isolons and NetApps and especially BlueArcs have more elegant solutions and they will actually give you scripts. In Blue Arcs case, they actually have prewritten stuff as part of the deal. SUN MIcroSystems is a class in itself and are excellent in this type of support. If you pay $70,000 for a small 9TB usable volume (16TB raw), you might expect them to bend over backwards to help you out. And in the three companies I have recently worked for since SONY, they did (Jim Henson Co, YuCo, and StudioGPU). I rewrote some stuff elegantly and the snap shots approached runtime and were not ~quasi~ snapshots or nightly backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote: We're currently snapshotting between 1.8TB and 2.3TB of production files with 2.7TB of snapshot disk. We snapshot every 6 hours and keep snapshots for a month by default, but we've got cron scripts that go through and remove older snapshots if we start running out disk space (which can happen if there has been ridiculous amounts of churn in the file system).

We could easily increase the size of our snapshot if we needed to. We've got 5x750GB drives right now in the RAID. If we had to, we could expand out to 6 drives in the RAID and move up to 2TB drives, which would give us close to 10TB. Not quite what you guys need, but there are motherboards out there with 10xSATA ports on them, which when combined with 2TB drives, would get you more into your ballpark.

On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead <curtis@rodeofx.comcurtis@rodeofx.com> wrote:

Jesse,

Thanks for the info! I will definitely look further into this as a potential option for us.

How large was the pool you were doing the snapshot on and how large was the RAID you built?

From what I'm reading I would need the size of one full backup plus a little more and my current data pool can hit upwards of 16TB.

Not sure how easily I could configure that on a montherboard based raid heh, but it definitely interests me.

Thanks again!

Jorg-Ulrich Mohnen, M.Sc. MBA WavGen / Terratracer Incorporated 901 North Curson Avenue West Hollywood, CA 90046 Tel 888-654-5615 Email jorg.mohnen@wavgen.comjorg.mohnen@wavgen.com

Lustre works with standard FS, no more stone FS, so in theory this
could work as long as the OS can see it, IIRC.. I'll try and get a
look a little later this week.

On 1-Sep-09, at 8:34 PM, Sean Laverty wrote:

Hm.

I?m wondering if a collection of backblaze systems could, if
organized into a Lustre file system and front-ended by an adequate
Linux Samba/NFS file server, provide a useable and relatively high- density, low-cost solution for multi-hundred TB online archives.

Has anyone on the list been working with Lustre?

• Sean

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com ]On Behalf Of Daniel Roizman Sent: Tuesday, September 01, 2009 8:07 PM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

wow, great that they are sharing their design. one thing I wonder
about is PSU failure. from the way he outlined the connections, the
raid is spread across PSU's so if one PSU goes down, more than 2
drives will fail which would potentially corrupt the raid, no?

with google sharing some of their server specs and blackblaze
sharing their storage, i wonder how long it'll be before we start
seeing some opensource hardware configurations tightly tuned with
linux distros.

On Tue, Sep 1, 2009 at 5:06 PM, Klaus Steden wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

Klaus

On 8/31/09 12:43 PM, "Greg Whynott" Greg.Whynott@oicr.on.ca etched
on stone tablets:

Another free non rsync approach is a ?copy on write? file system,
such as ZFS. I took a look at a snapshot solution on linux
using XFS at a previous job which worked as expected. I?m sure
there are other filesystems out there that support it. I don?t
think you have to spend a lot of money to get file level
protection, its more a matter of available capacity.

-g

On 8/28/09 12:20 PM, "Jorg-Ulrich Mohnen, M.Sc., MBA" wrote:

Just followed this thread and wanted to give my two cents on this.
Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then setup
a cheap RAIDed solution as Jesse C suggests, and do an rsync every
night from 12 midnight to 6am on the folders you need rsynced. Its
really quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up on
Linux for under $2000 and with a more stable and secure MAC OSX
server and XRAID for under $5000. The RAIDed linux software solution
would yield about 4TB of usable space, and the MAC Server XRAID
solution would yield around 6TB of usable space. Keep in mind that
Apple only delivered IDE XRAIDs with maximum 500GB IDE disks. I
bought the 750GB disks from CDW (OEM) and they work just fine under
non-warranty conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of low
to minimal production. And make this KNOWN to the artists and folks
that there are only 24 hour back ups as there is no money for a more
expensive solution.

The more expensive hardware like Isolons and NetApps and especially
BlueArcs have more elegant solutions and they will actually give you
scripts. In Blue Arcs case, they actually have prewritten stuff as
part of the deal. SUN MIcroSystems is a class in itself and are
excellent in this type of support. If you pay $70,000 for a small
9TB usable volume (16TB raw), you might expect them to bend over
backwards to help you out. And in the three companies I have
recently worked for since SONY, they did (Jim Henson Co, YuCo, and
StudioGPU). I rewrote some stuff elegantly and the snap shots
approached runtime and were not ~quasi~ snapshots or nightly
backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote:

We're currently snapshotting between 1.8TB and 2.3TB of production
files with 2.7TB of snapshot disk. We snapshot every 6 hours and
keep snapshots for a month by default, but we've got cron scripts
that go through and remove older snapshots if we start running out
disk space (which can happen if there has been ridiculous amounts of
churn in the file system).

We could easily increase the size of our snapshot if we needed to.
We've got 5x750GB drives right now in the RAID. If we had to, we
could expand out to 6 drives in the RAID and move up to 2TB drives,
which would give us close to 10TB. Not quite what you guys need,
but there are motherboards out there with 10xSATA ports on them,
which when combined with 2TB drives, would get you more into your
ballpark.

On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead
curtis@rodeofx.com wrote:

Jesse,

Thanks for the info! I will definitely look further into this as a potential option for
us.

How large was the pool you were doing the snapshot on and how large
was the RAID you built?

From what I?m reading I would need the size of one full backup plus

a little more and my current data pool can hit upwards of 16TB.

Not sure how easily I could configure that on a montherboard based
raid heh, but it definitely interests me.

Thanks again!

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Jorg-Ulrich Mohnen, M.Sc. MBA WavGen / Terratracer Incorporated 901 North Curson Avenue West Hollywood, CA 90046 Tel 888-654-5615 Email jorg.mohnen@wavgen.com

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

I did a fair bit of work with Lustre over 2007-2008. It?s pretty sweet technology, but whether or not something like this backblaze would work with it would depend a great deal on how the host OS presents storage. Lustre is just another Linux file system, so it looks for LUNs upon which to construct Lustre file system objects (metadata storage, object storage, etc.), so if these things act as NAS devices, it would be a no-go. The one thing I really like about Lustre is that it supports a variety of network interconnects, so you can build a clustered SAN using GigE, IB, 10 GigE, etc. and it?s all transparent to the client. It?s fast as hell, too. I don?t get too jazzed about file systems and storage, but Lustre is definitely pretty interesting technology.

Klaus

On 9/1/09 8:34 PM, "Sean Laverty" SLaverty@blizzard.com etched on stone tablets:

Hm.
I?m wondering if a collection of backblaze systems could, if organized into a Lustre file system and front-ended by an adequate Linux Samba/NFS file server, provide a useable and relatively high-density, low-cost solution for multi-hundred TB online archives.
Has anyone on the list been working with Lustre?
- Sean

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Daniel Roizman Sent: Tuesday, September 01, 2009 8:07 PM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.
wow, great that they are sharing their design. one thing I wonder about is PSU failure. from the way he outlined the connections, the raid is spread across PSU's so if one PSU goes down, more than 2 drives will fail which would potentially corrupt the raid, no?

with google sharing some of their server specs and blackblaze sharing their storage, i wonder how long it'll be before we start seeing some opensource hardware configurations tightly tuned with linux distros.

On Tue, Sep 1, 2009 at 5:06 PM, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap- cloud-storage/

Klaus

On 8/31/09 12:43 PM, "Greg Whynott" Greg.Whynott@oicr.on.ca etched on stone tablets: Another free non rsync approach is a ?copy on write? file system, such as ZFS. I took a look at a snapshot solution on linux using XFS at a previous job which worked as expected. I?m sure there are other filesystems out there that support it. I don?t think you have to spend a lot of money to get file level protection, its more a matter of available capacity.

-g

On 8/28/09 12:20 PM, "Jorg-Ulrich Mohnen, M.Sc., MBA" jorg.mohnen@wavgen.com wrote: Just followed this thread and wanted to give my two cents on this. Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then setup a cheap RAIDed solution as Jesse C suggests, and do an rsync every night from 12 midnight to 6am on the folders you need rsynced. Its really quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up on Linux for under $2000 and with a more stable and secure MAC OSX server and XRAID for under $5000. The RAIDed linux software solution would yield about 4TB of usable space, and the MAC Server XRAID solution would yield around 6TB of usable space. Keep in mind that Apple only delivered IDE XRAIDs with maximum 500GB IDE disks. I bought the 750GB disks from CDW (OEM) and they work just fine under non-warranty conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of low to minimal production. And make this KNOWN to the artists and folks that there are only 24 hour back ups as there is no money for a more expensive solution.

The more expensive hardware like Isolons and NetApps and especially BlueArcs have more elegant solutions and they will actually give you scripts. In Blue Arcs case, they actually have prewritten stuff as part of the deal. SUN MIcroSystems is a class in itself and are excellent in this type of support. If you pay $70,000 for a small 9TB usable volume (16TB raw), you might expect them to bend over backwards to help you out. And in the three companies I have recently worked for since SONY, they did (Jim Henson Co, YuCo, and StudioGPU). I rewrote some stuff elegantly and the snap shots approached runtime and were not ~quasi~ snapshots or nightly backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote: We're currently snapshotting between 1.8TB and 2.3TB of production files with 2.7TB of snapshot disk. We snapshot every 6 hours and keep snapshots for a month by default, but we've got cron scripts that go through and remove older snapshots if we start running out disk space (which can happen if there has been ridiculous amounts of churn in the file system).
We could easily increase the size of our snapshot if we needed to. We've got 5x750GB drives right now in the RAID. If we had to, we could expand out to 6 drives in the RAID and move up to 2TB drives, which would give us close to 10TB. Not quite what you guys need, but there are motherboards out there with 10xSATA ports on them, which when combined with 2TB drives, would get you more into your ballpark.
On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead curtis@rodeofx.com wrote:

Jesse,

Thanks for the info! I will definitely look further into this as a potential option for us.

How large was the pool you were doing the snapshot on and how large was the RAID you built?

From what I?m reading I would need the size of one full backup plus a little

more and my current data pool can hit upwards of 16TB.

Not sure how easily I could configure that on a montherboard based raid heh, but it definitely interests me.

Thanks again!

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Jorg-Ulrich Mohnen, M.Sc. MBA WavGen / Terratracer Incorporated 901 North Curson Avenue West Hollywood, CA 90046 Tel 888-654-5615 Email jorg.mohnen@wavgen.com

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Yup, all that plus can handle Red footage in real time with the latest
hardware - I might be mistaken on that, talk to a sales droid - but
i'm pretty sure of that. I'll be talking to Autodesk support (my
preference instead of sales guys) later this week and will confirm and
update the list if I'm wrong on that..

On 1-Sep-09, at 10:57 PM, Klaus Steden wrote:

I did a fair bit of work with Lustre over 2007-2008. It?s pretty
sweet technology, but whether or not something like this backblaze
would work with it would depend a great deal on how the host OS
presents storage. Lustre is just another Linux file system, so it
looks for LUNs upon which to construct Lustre file system objects
(metadata storage, object storage, etc.), so if these things act as
NAS devices, it would be a no-go. The one thing I really like about
Lustre is that it supports a variety of network interconnects, so
you can build a clustered SAN using GigE, IB, 10 GigE, etc. and it?s
all transparent to the client. It?s fast as hell, too. I don?t get
too jazzed about file systems and storage, but Lustre is definitely
pretty interesting technology.

Klaus

On 9/1/09 8:34 PM, "Sean Laverty" SLaverty@blizzard.com etched on
stone tablets:

Hm.

I?m wondering if a collection of backblaze systems could, if
organized into a Lustre file system and front-ended by an adequate
Linux Samba/NFS file server, provide a useable and relatively high- density, low-cost solution for multi-hundred TB online archives.

Has anyone on the list been working with Lustre?

• Sean

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com ] On Behalf Of Daniel Roizman Sent: Tuesday, September 01, 2009 8:07 PM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

wow, great that they are sharing their design. one thing I wonder
about is PSU failure. from the way he outlined the connections,
the raid is spread across PSU's so if one PSU goes down, more than
2 drives will fail which would potentially corrupt the raid, no?

with google sharing some of their server specs and blackblaze
sharing their storage, i wonder how long it'll be before we start
seeing some opensource hardware configurations tightly tuned with
linux distros.

On Tue, Sep 1, 2009 at 5:06 PM, Klaus Steden wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

Klaus

On 8/31/09 12:43 PM, "Greg Whynott" Greg.Whynott@oicr.on.ca
etched on stone tablets: Another free non rsync approach is a ?copy on write? file system,
such as ZFS. I took a look at a snapshot solution on linux
using XFS at a previous job which worked as expected. I?m sure
there are other filesystems out there that support it. I don?t
think you have to spend a lot of money to get file level
protection, its more a matter of available capacity.

-g

On 8/28/09 12:20 PM, "Jorg-Ulrich Mohnen, M.Sc., MBA" wrote: Just followed this thread and wanted to give my two cents on this.
Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then
setup a cheap RAIDed solution as Jesse C suggests, and do an rsync
every night from 12 midnight to 6am on the folders you need
rsynced. Its really quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up on
Linux for under $2000 and with a more stable and secure MAC OSX
server and XRAID for under $5000. The RAIDed linux software
solution would yield about 4TB of usable space, and the MAC Server
XRAID solution would yield around 6TB of usable space. Keep in mind
that Apple only delivered IDE XRAIDs with maximum 500GB IDE disks.
I bought the 750GB disks from CDW (OEM) and they work just fine
under non-warranty conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of
low to minimal production. And make this KNOWN to the artists and
folks that there are only 24 hour back ups as there is no money for
a more expensive solution.

The more expensive hardware like Isolons and NetApps and especially
BlueArcs have more elegant solutions and they will actually give
you scripts. In Blue Arcs case, they actually have prewritten stuff
as part of the deal. SUN MIcroSystems is a class in itself and are
excellent in this type of support. If you pay $70,000 for a small
9TB usable volume (16TB raw), you might expect them to bend over
backwards to help you out. And in the three companies I have
recently worked for since SONY, they did (Jim Henson Co, YuCo, and
StudioGPU). I rewrote some stuff elegantly and the snap shots
approached runtime and were not ~quasi~ snapshots or nightly
backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote: We're currently snapshotting between 1.8TB and 2.3TB of production
files with 2.7TB of snapshot disk. We snapshot every 6 hours and
keep snapshots for a month by default, but we've got cron scripts
that go through and remove older snapshots if we start running out
disk space (which can happen if there has been ridiculous amounts
of churn in the file system).

We could easily increase the size of our snapshot if we needed to.
We've got 5x750GB drives right now in the RAID. If we had to, we
could expand out to 6 drives in the RAID and move up to 2TB drives,
which would give us close to 10TB. Not quite what you guys need,
but there are motherboards out there with 10xSATA ports on them,
which when combined with 2TB drives, would get you more into your
ballpark.

On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead
curtis@rodeofx.com wrote:

Jesse,

Thanks for the info! I will definitely look further into this as a potential option for
us.

How large was the pool you were doing the snapshot on and how large
was the RAID you built?

From what I?m reading I would need the size of one full backup

plus a little more and my current data pool can hit upwards of 16TB.

Not sure how easily I could configure that on a montherboard based
raid heh, but it definitely interests me.

Thanks again!

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Jorg-Ulrich Mohnen, M.Sc. MBA WavGen / Terratracer Incorporated 901 North Curson Avenue West Hollywood, CA 90046 Tel 888-654-5615 Email jorg.mohnen@wavgen.com

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Hi Derrick,

I think we're actually talking about different products which have the same name and float around in the same HPC soup ... there's Lustre the grading product from Autodesk, and Lustre the distributed parallel file system from Sun now Oracle a/k/a CFS. Both are good, but very different. :-)

My posting was about CFS, the distributed parallel file system. If anyone's interested ... http://en.wikipedia.org/wiki/Lustre_(file_system)

cheers, Klaus

On 9/1/09 11:01 PM, "Derrick MacPherson" derrick@packetsafe.net etched on stone tablets:

Yup, all that plus can handle Red footage in real time with the latest hardware - I might be mistaken on that, talk to a sales droid - but i'm pretty sure of that. I'll be talking to Autodesk support (my preference instead of sales guys) later this week and will confirm and update the list if I'm wrong on that..

On 1-Sep-09, at 10:57 PM, Klaus Steden wrote:

I did a fair bit of work with Lustre over 2007-2008. It?s pretty sweet technology, but whether or not something like this backblaze would work with it would depend a great deal on how the host OS presents storage. Lustre is just another Linux file system, so it looks for LUNs upon which to construct Lustre file system objects (metadata storage, object storage, etc.), so if these things act as NAS devices, it would be a no-go. The one thing I really like about Lustre is that it supports a variety of network interconnects, so you can build a clustered SAN using GigE, IB, 10 GigE, etc. and it?s all transparent to the client. It?s fast as hell, too. I don?t get too jazzed about file systems and storage, but Lustre is definitely pretty interesting technology.

Klaus

On 9/1/09 8:34 PM, "Sean Laverty" SLaverty@blizzard.com etched on stone tablets:

Hm.

I?m wondering if a collection of backblaze systems could, if organized into a Lustre file system and front-ended by an adequate Linux Samba/NFS file server, provide a useable and relatively high- density, low-cost solution for multi-hundred TB online archives.

Has anyone on the list been working with Lustre?

• Sean

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com ] On Behalf Of Daniel Roizman Sent: Tuesday, September 01, 2009 8:07 PM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

wow, great that they are sharing their design. one thing I wonder about is PSU failure. from the way he outlined the connections, the raid is spread across PSU's so if one PSU goes down, more than 2 drives will fail which would potentially corrupt the raid, no?

with google sharing some of their server specs and blackblaze sharing their storage, i wonder how long it'll be before we start seeing some opensource hardware configurations tightly tuned with linux distros.

On Tue, Sep 1, 2009 at 5:06 PM, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-chea p-cloud-storage/

Klaus

On 8/31/09 12:43 PM, "Greg Whynott" Greg.Whynott@oicr.on.ca etched on stone tablets: Another free non rsync approach is a ?copy on write? file system, such as ZFS. I took a look at a snapshot solution on linux using XFS at a previous job which worked as expected. I?m sure there are other filesystems out there that support it. I don?t think you have to spend a lot of money to get file level protection, its more a matter of available capacity.

-g

On 8/28/09 12:20 PM, "Jorg-Ulrich Mohnen, M.Sc., MBA" jorg.mohnen@wavgen.com wrote: Just followed this thread and wanted to give my two cents on this. Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then setup a cheap RAIDed solution as Jesse C suggests, and do an rsync every night from 12 midnight to 6am on the folders you need rsynced. Its really quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up on Linux for under $2000 and with a more stable and secure MAC OSX server and XRAID for under $5000. The RAIDed linux software solution would yield about 4TB of usable space, and the MAC Server XRAID solution would yield around 6TB of usable space. Keep in mind that Apple only delivered IDE XRAIDs with maximum 500GB IDE disks. I bought the 750GB disks from CDW (OEM) and they work just fine under non-warranty conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of low to minimal production. And make this KNOWN to the artists and folks that there are only 24 hour back ups as there is no money for a more expensive solution.

The more expensive hardware like Isolons and NetApps and especially BlueArcs have more elegant solutions and they will actually give you scripts. In Blue Arcs case, they actually have prewritten stuff as part of the deal. SUN MIcroSystems is a class in itself and are excellent in this type of support. If you pay $70,000 for a small 9TB usable volume (16TB raw), you might expect them to bend over backwards to help you out. And in the three companies I have recently worked for since SONY, they did (Jim Henson Co, YuCo, and StudioGPU). I rewrote some stuff elegantly and the snap shots approached runtime and were not ~quasi~ snapshots or nightly backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote: We're currently snapshotting between 1.8TB and 2.3TB of production files with 2.7TB of snapshot disk. We snapshot every 6 hours and keep snapshots for a month by default, but we've got cron scripts that go through and remove older snapshots if we start running out disk space (which can happen if there has been ridiculous amounts of churn in the file system).

We could easily increase the size of our snapshot if we needed to. We've got 5x750GB drives right now in the RAID. If we had to, we could expand out to 6 drives in the RAID and move up to 2TB drives, which would give us close to 10TB. Not quite what you guys need, but there are motherboards out there with 10xSATA ports on them, which when combined with 2TB drives, would get you more into your ballpark.

On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead curtis@rodeofx.com wrote:

Jesse,

Thanks for the info! I will definitely look further into this as a potential option for us.

How large was the pool you were doing the snapshot on and how large was the RAID you built?

From what I?m reading I would need the size of one full backup

plus a little more and my current data pool can hit upwards of 16TB.

Not sure how easily I could configure that on a montherboard based raid heh, but it definitely interests me.

Thanks again!

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Jorg-Ulrich Mohnen, M.Sc. MBA WavGen / Terratracer Incorporated 901 North Curson Avenue West Hollywood, CA 90046 Tel 888-654-5615 Email jorg.mohnen@wavgen.com

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

lol.. I wondered that and had emailed Sean off list to ask but hadn't
heard back... i should've waited so i was sure i was on topic.

is anyone out there using a distributed file system at all?

On 1-Sep-09, at 11:36 PM, Klaus Steden wrote:

Hi Derrick,

I think we're actually talking about different products which have
the same name and float around in the same HPC soup ... there's Lustre the
grading product from Autodesk, and Lustre the distributed parallel file
system from Sun now Oracle a/k/a CFS. Both are good, but very different. :-)

My posting was about CFS, the distributed parallel file system. If
anyone's interested ... http://en.wikipedia.org/wiki/Lustre_(file_system)

cheers, Klaus

On 9/1/09 11:01 PM, "Derrick MacPherson" derrick@packetsafe.net
etched on stone tablets:

Yup, all that plus can handle Red footage in real time with the
latest hardware - I might be mistaken on that, talk to a sales droid - but i'm pretty sure of that. I'll be talking to Autodesk support (my preference instead of sales guys) later this week and will confirm
and update the list if I'm wrong on that..

On 1-Sep-09, at 10:57 PM, Klaus Steden wrote:

I did a fair bit of work with Lustre over 2007-2008. It?s pretty sweet technology, but whether or not something like this backblaze would work with it would depend a great deal on how the host OS presents storage. Lustre is just another Linux file system, so it looks for LUNs upon which to construct Lustre file system objects (metadata storage, object storage, etc.), so if these things act as NAS devices, it would be a no-go. The one thing I really like about Lustre is that it supports a variety of network interconnects, so you can build a clustered SAN using GigE, IB, 10 GigE, etc. and it?s all transparent to the client. It?s fast as hell, too. I don?t get too jazzed about file systems and storage, but Lustre is definitely pretty interesting technology.

Klaus

On 9/1/09 8:34 PM, "Sean Laverty" SLaverty@blizzard.com etched on stone tablets:

Hm.

I?m wondering if a collection of backblaze systems could, if organized into a Lustre file system and front-ended by an adequate Linux Samba/NFS file server, provide a useable and relatively high- density, low-cost solution for multi-hundred TB online archives.

Has anyone on the list been working with Lustre?

• Sean

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com ] On Behalf Of Daniel Roizman Sent: Tuesday, September 01, 2009 8:07 PM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

wow, great that they are sharing their design. one thing I wonder about is PSU failure. from the way he outlined the connections, the raid is spread across PSU's so if one PSU goes down, more than 2 drives will fail which would potentially corrupt the raid, no?

with google sharing some of their server specs and blackblaze sharing their storage, i wonder how long it'll be before we start seeing some opensource hardware configurations tightly tuned with linux distros.

On Tue, Sep 1, 2009 at 5:06 PM, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-chea p-cloud-storage/

Klaus

On 8/31/09 12:43 PM, "Greg Whynott" Greg.Whynott@oicr.on.ca etched on stone tablets: Another free non rsync approach is a ?copy on write? file system, such as ZFS. I took a look at a snapshot solution on linux using XFS at a previous job which worked as expected. I?m sure there are other filesystems out there that support it. I don?t think you have to spend a lot of money to get file level protection, its more a matter of available capacity.

-g

On 8/28/09 12:20 PM, "Jorg-Ulrich Mohnen, M.Sc., MBA" jorg.mohnen@wavgen.com wrote: Just followed this thread and wanted to give my two cents on this. Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then setup a cheap RAIDed solution as Jesse C suggests, and do an rsync every night from 12 midnight to 6am on the folders you need rsynced. Its really quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up on Linux for under $2000 and with a more stable and secure MAC OSX server and XRAID for under $5000. The RAIDed linux software solution would yield about 4TB of usable space, and the MAC Server XRAID solution would yield around 6TB of usable space. Keep in mind that Apple only delivered IDE XRAIDs with maximum 500GB IDE disks. I bought the 750GB disks from CDW (OEM) and they work just fine under non-warranty conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of low to minimal production. And make this KNOWN to the artists and folks that there are only 24 hour back ups as there is no money for a more expensive solution.

The more expensive hardware like Isolons and NetApps and especially BlueArcs have more elegant solutions and they will actually give you scripts. In Blue Arcs case, they actually have prewritten stuff as part of the deal. SUN MIcroSystems is a class in itself and are excellent in this type of support. If you pay $70,000 for a small 9TB usable volume (16TB raw), you might expect them to bend over backwards to help you out. And in the three companies I have recently worked for since SONY, they did (Jim Henson Co, YuCo, and StudioGPU). I rewrote some stuff elegantly and the snap shots approached runtime and were not ~quasi~ snapshots or nightly backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote: We're currently snapshotting between 1.8TB and 2.3TB of production files with 2.7TB of snapshot disk. We snapshot every 6 hours and keep snapshots for a month by default, but we've got cron scripts that go through and remove older snapshots if we start running out disk space (which can happen if there has been ridiculous amounts of churn in the file system).

We could easily increase the size of our snapshot if we needed to. We've got 5x750GB drives right now in the RAID. If we had to, we could expand out to 6 drives in the RAID and move up to 2TB drives, which would give us close to 10TB. Not quite what you guys need, but there are motherboards out there with 10xSATA ports on them, which when combined with 2TB drives, would get you more into your ballpark.

On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead curtis@rodeofx.com wrote:

Jesse,

Thanks for the info! I will definitely look further into this as a potential option for us.

How large was the pool you were doing the snapshot on and how large was the RAID you built?

From what I?m reading I would need the size of one full backup

plus a little more and my current data pool can hit upwards of
16TB.

Not sure how easily I could configure that on a montherboard based raid heh, but it definitely interests me.

Thanks again!

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Jorg-Ulrich Mohnen, M.Sc. MBA WavGen / Terratracer Incorporated 901 North Curson Avenue West Hollywood, CA 90046 Tel 888-654-5615 Email jorg.mohnen@wavgen.com

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

I know that Framestore-CFC adopted Lustre/CFS way back in the v1.4 days (about 4-5 years ago, I think), and published a press release about it. Isilon should probably qualify as a distributed file system for the purposes of this discussion, as would Panassas I believe, but most of the other parallel file systems (CFS, PVFS, GFS, GPFS, etc.) are still largely the domain of HPC research nerds working science and high-tech, I don't know that they're that common in production houses (but they probably should be!)

Klaus

On 9/1/09 11:44 PM, "Derrick MacPherson" derrick@packetsafe.net etched on stone tablets:

lol.. I wondered that and had emailed Sean off list to ask but hadn't heard back... i should've waited so i was sure i was on topic.

is anyone out there using a distributed file system at all?

On 1-Sep-09, at 11:36 PM, Klaus Steden wrote:

Hi Derrick,

I think we're actually talking about different products which have the same name and float around in the same HPC soup ... there's Lustre the grading product from Autodesk, and Lustre the distributed parallel file system from Sun now Oracle a/k/a CFS. Both are good, but very different. :-)

My posting was about CFS, the distributed parallel file system. If anyone's interested ... http://en.wikipedia.org/wiki/Lustre_(file_system)

cheers, Klaus

On 9/1/09 11:01 PM, "Derrick MacPherson" derrick@packetsafe.net etched on stone tablets:

Yup, all that plus can handle Red footage in real time with the latest hardware - I might be mistaken on that, talk to a sales droid - but i'm pretty sure of that. I'll be talking to Autodesk support (my preference instead of sales guys) later this week and will confirm and update the list if I'm wrong on that..

On 1-Sep-09, at 10:57 PM, Klaus Steden wrote:

I did a fair bit of work with Lustre over 2007-2008. It?s pretty sweet technology, but whether or not something like this backblaze would work with it would depend a great deal on how the host OS presents storage. Lustre is just another Linux file system, so it looks for LUNs upon which to construct Lustre file system objects (metadata storage, object storage, etc.), so if these things act as NAS devices, it would be a no-go. The one thing I really like about Lustre is that it supports a variety of network interconnects, so you can build a clustered SAN using GigE, IB, 10 GigE, etc. and it?s all transparent to the client. It?s fast as hell, too. I don?t get too jazzed about file systems and storage, but Lustre is definitely pretty interesting technology.

Klaus

On 9/1/09 8:34 PM, "Sean Laverty" SLaverty@blizzard.com etched on stone tablets:

Hm.

I?m wondering if a collection of backblaze systems could, if organized into a Lustre file system and front-ended by an adequate Linux Samba/NFS file server, provide a useable and relatively high- density, low-cost solution for multi-hundred TB online archives.

Has anyone on the list been working with Lustre?

• Sean

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com ] On Behalf Of Daniel Roizman Sent: Tuesday, September 01, 2009 8:07 PM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

wow, great that they are sharing their design. one thing I wonder about is PSU failure. from the way he outlined the connections, the raid is spread across PSU's so if one PSU goes down, more than 2 drives will fail which would potentially corrupt the raid, no?

with google sharing some of their server specs and blackblaze sharing their storage, i wonder how long it'll be before we start seeing some opensource hardware configurations tightly tuned with linux distros.

On Tue, Sep 1, 2009 at 5:06 PM, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-ch ea p-cloud-storage/

Klaus

On 8/31/09 12:43 PM, "Greg Whynott" Greg.Whynott@oicr.on.ca etched on stone tablets: Another free non rsync approach is a ?copy on write? file system, such as ZFS. I took a look at a snapshot solution on linux using XFS at a previous job which worked as expected. I?m sure there are other filesystems out there that support it. I don?t think you have to spend a lot of money to get file level protection, its more a matter of available capacity.

-g

On 8/28/09 12:20 PM, "Jorg-Ulrich Mohnen, M.Sc., MBA" jorg.mohnen@wavgen.com wrote: Just followed this thread and wanted to give my two cents on this. Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then setup a cheap RAIDed solution as Jesse C suggests, and do an rsync every night from 12 midnight to 6am on the folders you need rsynced. Its really quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up on Linux for under $2000 and with a more stable and secure MAC OSX server and XRAID for under $5000. The RAIDed linux software solution would yield about 4TB of usable space, and the MAC Server XRAID solution would yield around 6TB of usable space. Keep in mind that Apple only delivered IDE XRAIDs with maximum 500GB IDE disks. I bought the 750GB disks from CDW (OEM) and they work just fine under non-warranty conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of low to minimal production. And make this KNOWN to the artists and folks that there are only 24 hour back ups as there is no money for a more expensive solution.

The more expensive hardware like Isolons and NetApps and especially BlueArcs have more elegant solutions and they will actually give you scripts. In Blue Arcs case, they actually have prewritten stuff as part of the deal. SUN MIcroSystems is a class in itself and are excellent in this type of support. If you pay $70,000 for a small 9TB usable volume (16TB raw), you might expect them to bend over backwards to help you out. And in the three companies I have recently worked for since SONY, they did (Jim Henson Co, YuCo, and StudioGPU). I rewrote some stuff elegantly and the snap shots approached runtime and were not ~quasi~ snapshots or nightly backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote: We're currently snapshotting between 1.8TB and 2.3TB of production files with 2.7TB of snapshot disk. We snapshot every 6 hours and keep snapshots for a month by default, but we've got cron scripts that go through and remove older snapshots if we start running out disk space (which can happen if there has been ridiculous amounts of churn in the file system).

We could easily increase the size of our snapshot if we needed to. We've got 5x750GB drives right now in the RAID. If we had to, we could expand out to 6 drives in the RAID and move up to 2TB drives, which would give us close to 10TB. Not quite what you guys need, but there are motherboards out there with 10xSATA ports on them, which when combined with 2TB drives, would get you more into your ballpark.

On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead curtis@rodeofx.com wrote:

Jesse,

Thanks for the info! I will definitely look further into this as a potential option for us.

How large was the pool you were doing the snapshot on and how large was the RAID you built?

From what I?m reading I would need the size of one full backup

plus a little more and my current data pool can hit upwards of 16TB.

Not sure how easily I could configure that on a montherboard based raid heh, but it definitely interests me.

Thanks again!

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com

http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discus>>>>> s >>>>> >>>>> >>>>> Jorg-Ulrich Mohnen, M.Sc. MBA >>>>> WavGen / Terratracer Incorporated >>>>> 901 North Curson Avenue >>>>> West Hollywood, CA 90046 >>>>> Tel 888-654-5615 >>>>> Email jorg.mohnen@wavgen.com >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _ >>>>> StudioSysAdmins-Discuss mailing list >>>>> StudioSysAdmins-Discuss@studiosysadmins.com >>>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discus>>>>> s >>>>> >>>>> >>>>> __ >>>>> StudioSysAdmins-Discuss mailing list >>>>> StudioSysAdmins-Discuss@studiosysadmins.com >>>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discus>>>>> s >>>>> >>>>> >>>>> __ >>>>> StudioSysAdmins-Discuss mailing list >>>>> StudioSysAdmins-Discuss@studiosysadmins.com >>>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discus>>>>> s >>>> >>>> __ >>>> StudioSysAdmins-Discuss mailing list >>>> StudioSysAdmins-Discuss@studiosysadmins.com >>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >>> >>> __ >>> StudioSysAdmins-Discuss mailing list >>> StudioSysAdmins-Discuss@studiosysadmins.com >>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >> >> __ >> StudioSysAdmins-Discuss mailing list >> StudioSysAdmins-Discuss@studiosysadmins.com >> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss > > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

What would some factors be that have kept it out of the production
industry? There's a lurker on this list that I've had a few
discussions about this and I'd love to learn more about why we don't
see it

On 1-Sep-09, at 11:49 PM, Klaus Steden wrote:

I know that Framestore-CFC adopted Lustre/CFS way back in the v1.4
days (about 4-5 years ago, I think), and published a press release about
it. Isilon should probably qualify as a distributed file system for the
purposes of this discussion, as would Panassas I believe, but most of the other parallel file systems (CFS, PVFS, GFS, GPFS, etc.) are still largely
the domain of HPC research nerds working science and high-tech, I don't
know that they're that common in production houses (but they probably
should be!)

Klaus

On 9/1/09 11:44 PM, "Derrick MacPherson" derrick@packetsafe.net
etched on stone tablets:

lol.. I wondered that and had emailed Sean off list to ask but hadn't heard back... i should've waited so i was sure i was on topic.

is anyone out there using a distributed file system at all?

On 1-Sep-09, at 11:36 PM, Klaus Steden wrote:

Hi Derrick,

I think we're actually talking about different products which have the same name and float around in the same HPC soup ... there's Lustre the grading product from Autodesk, and Lustre the distributed parallel file system from Sun now Oracle a/k/a CFS. Both are good, but very different. :-)

My posting was about CFS, the distributed parallel file system. If anyone's interested ... http://en.wikipedia.org/wiki/Lustre_(file_system)

cheers, Klaus

On 9/1/09 11:01 PM, "Derrick MacPherson" derrick@packetsafe.net etched on stone tablets:

Yup, all that plus can handle Red footage in real time with the latest hardware - I might be mistaken on that, talk to a sales droid - but i'm pretty sure of that. I'll be talking to Autodesk support (my preference instead of sales guys) later this week and will confirm and update the list if I'm wrong on that..

On 1-Sep-09, at 10:57 PM, Klaus Steden wrote:

I did a fair bit of work with Lustre over 2007-2008. It?s pretty sweet technology, but whether or not something like this backblaze would work with it would depend a great deal on how the host OS presents storage. Lustre is just another Linux file system, so it looks for LUNs upon which to construct Lustre file system objects (metadata storage, object storage, etc.), so if these things act
as NAS devices, it would be a no-go. The one thing I really like
about Lustre is that it supports a variety of network interconnects, so you can build a clustered SAN using GigE, IB, 10 GigE, etc. and
it?s all transparent to the client. It?s fast as hell, too. I don?t get too jazzed about file systems and storage, but Lustre is
definitely pretty interesting technology.

Klaus

On 9/1/09 8:34 PM, "Sean Laverty" SLaverty@blizzard.com etched
on stone tablets:

Hm.

I?m wondering if a collection of backblaze systems could, if organized into a Lustre file system and front-ended by an
adequate Linux Samba/NFS file server, provide a useable and relatively
high- density, low-cost solution for multi-hundred TB online archives.

Has anyone on the list been working with Lustre?

• Sean

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com ] On Behalf Of Daniel Roizman Sent: Tuesday, September 01, 2009 8:07 PM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

wow, great that they are sharing their design. one thing I
wonder about is PSU failure. from the way he outlined the connections, the raid is spread across PSU's so if one PSU goes down, more
than 2 drives will fail which would potentially corrupt the raid, no?

with google sharing some of their server specs and blackblaze sharing their storage, i wonder how long it'll be before we start seeing some opensource hardware configurations tightly tuned with linux distros.

On Tue, Sep 1, 2009 at 5:06 PM, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-ch ea p-cloud-storage/

Klaus

On 8/31/09 12:43 PM, "Greg Whynott" Greg.Whynott@oicr.on.ca etched on stone tablets: Another free non rsync approach is a ?copy on write? file system, such as ZFS. I took a look at a snapshot solution on linux using XFS at a previous job which worked as expected. I?m sure there are other filesystems out there that support it. I don?t think you have to spend a lot of money to get file level protection, its more a matter of available capacity.

-g

On 8/28/09 12:20 PM, "Jorg-Ulrich Mohnen, M.Sc., MBA" jorg.mohnen@wavgen.com wrote: Just followed this thread and wanted to give my two cents on
this. Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then setup a cheap RAIDed solution as Jesse C suggests, and do an
rsync every night from 12 midnight to 6am on the folders you need rsynced. Its really quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up
on Linux for under $2000 and with a more stable and secure MAC OSX server and XRAID for under $5000. The RAIDed linux software solution would yield about 4TB of usable space, and the MAC
Server XRAID solution would yield around 6TB of usable space. Keep in
mind that Apple only delivered IDE XRAIDs with maximum 500GB IDE
disks. I bought the 750GB disks from CDW (OEM) and they work just fine under non-warranty conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of low to minimal production. And make this KNOWN to the artists and folks that there are only 24 hour back ups as there is no money
for a more expensive solution.

The more expensive hardware like Isolons and NetApps and
especially BlueArcs have more elegant solutions and they will actually give you scripts. In Blue Arcs case, they actually have prewritten
stuff as part of the deal. SUN MIcroSystems is a class in itself and
are excellent in this type of support. If you pay $70,000 for a small 9TB usable volume (16TB raw), you might expect them to bend over backwards to help you out. And in the three companies I have recently worked for since SONY, they did (Jim Henson Co, YuCo,
and StudioGPU). I rewrote some stuff elegantly and the snap shots approached runtime and were not ~quasi~ snapshots or nightly backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote: We're currently snapshotting between 1.8TB and 2.3TB of
production files with 2.7TB of snapshot disk. We snapshot every 6 hours and keep snapshots for a month by default, but we've got cron scripts that go through and remove older snapshots if we start running
out disk space (which can happen if there has been ridiculous amounts of churn in the file system).

We could easily increase the size of our snapshot if we needed
to. We've got 5x750GB drives right now in the RAID. If we had to, we could expand out to 6 drives in the RAID and move up to 2TB
drives, which would give us close to 10TB. Not quite what you guys need, but there are motherboards out there with 10xSATA ports on them, which when combined with 2TB drives, would get you more into your ballpark.

On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead curtis@rodeofx.com wrote:

Jesse,

Thanks for the info! I will definitely look further into this as a potential option
for us.

How large was the pool you were doing the snapshot on and how
large was the RAID you built?

From what I?m reading I would need the size of one full backup

plus a little more and my current data pool can hit upwards of 16TB.

Not sure how easily I could configure that on a montherboard
based raid heh, but it definitely interests me.

Thanks again!

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com

http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discus >>>>> s >>>>> >>>>> >>>>> Jorg-Ulrich Mohnen, M.Sc. MBA >>>>> WavGen / Terratracer Incorporated >>>>> 901 North Curson Avenue >>>>> West Hollywood, CA 90046 >>>>> Tel 888-654-5615 >>>>> Email jorg.mohnen@wavgen.com >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _ >>>>> StudioSysAdmins-Discuss mailing list >>>>> StudioSysAdmins-Discuss@studiosysadmins.com >>>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discus >>>>> s >>>>> >>>>> >>>>> __ >>>>> StudioSysAdmins-Discuss mailing list >>>>> StudioSysAdmins-Discuss@studiosysadmins.com >>>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discus >>>>> s >>>>> >>>>> >>>>> __ >>>>> StudioSysAdmins-Discuss mailing list >>>>> StudioSysAdmins-Discuss@studiosysadmins.com >>>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discus >>>>> s >>>> >>>> __ >>>> StudioSysAdmins-Discuss mailing list >>>> StudioSysAdmins-Discuss@studiosysadmins.com >>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >>> >>> __ >>> StudioSysAdmins-Discuss mailing list >>> StudioSysAdmins-Discuss@studiosysadmins.com >>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >> >> __ >> StudioSysAdmins-Discuss mailing list >> StudioSysAdmins-Discuss@studiosysadmins.com >> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss > > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

See, this is the really odd thing, I think ... we're all very comfortable with a whole ton of open source software, even to the point of accepting a lot of it as the gold standard for various protocols, services, and methods (NFS, Samba, Apache, PHP, Perl, Python, MySQL, Postgres, Mailman, sendmail, Postfix, gcc/g++, Linux, etc.) -- but the idea of using an open source storage solution still makes some people nervous.

At its heart, that's really all CFS is -- to build a decent cluster, you shoot your bolt on good hardware -- fast switches, RDMA-capable HBAs/HCAs, remotely manageable power, and beefy disks -- and all the software that you need to make it kick a lot of ass is free -- CFS/Lustre, Linux-HA, ROCKS -- none of it costs a dime. It's all quite well-suited to the types of HPC environments that are the bread and butter of us people in entertainment -- but the only ones bold enough to use the stuff work in academia/government research -- and produce results to complex computing problems (seismic analysis, climate modelling, cryptanalysis) with smaller budgets and less technical people on staff.

I think that's probably what it comes down to -- money. There are lots of players in the market who have good things to offer us, there's a considerable amount of horse-trading involved between vendor and consumer, and the process tends to be a bit more hands-on, i.e. "we made a deal with vendor XYZ, implement their technology for us k?" -- where researchers tend to be left their own devices when it comes to solutions.

I know that HP for a while (they may still) offered a bundled Lustre solution more like a turnkey, but that was built around v1.4 which is by now pretty long in the tooth and very significantly outmatched by v1.6. What I'm still secretly holding out hope for is vendor support for client-side clustered file system drivers of ANY kind (GFS, GPFS, CFS, PVFS, etc.) on turnkey systems -- black boxes that you can only customize so much before the vendor starts refusing to support them.

Something like CFS could really be used to do amazing things in environments like ours, for a couple of reasons:

• performance scales more or less linearly upwards as you add hardware, something even the best vendor solutions can have trouble with

• other than the software used to manage and operate the Lustre file system, these machines are plain as toast Linux machines: no exotic hardware, no tricked-out firmware, no black boxes, nothing proprietary

• it's network-interface agnostic -- you can build a storage environment that incorporates IB, 10 GigE, GigE, etc. and it will function within the same namespace, allowing you to build an environment that can support high-end realtime systems alongside hundreds of workstations and render nodes -- try doing that with a SAN!

• it supports MAN/WAN topologies, allowing you to build storage environments that can span large distances between multiple physical locations transparently to end users -- this is a challenge with even the best commercial solutions

... I think I may be ranting by now, but yeah, I find it a little frustrating to see that this stuff has next to no visibility in an industry that's so tech-heavy as entertainment has become, where it could be leveraged to great advantage for very little cost -- so take this with a grain of salt, it's my two cents and it's late at night here. :-)

cheers, Klaus

On 9/1/09 11:52 PM, "Derrick MacPherson" derrick@packetsafe.net etched on stone tablets:

What would some factors be that have kept it out of the production industry? There's a lurker on this list that I've had a few discussions about this and I'd love to learn more about why we don't see it

--- On Wed, 9/2/09, Klaus Steden klaus-s@moving-picture.com wrote:

See, this is the really odd thing, I think ... we're all very comfortable with a whole ton of open source software, even to the point of accepting a lot of it as the gold standard for various protocols, services, and methods (NFS, Samba, Apache, PHP, Perl, Python, MySQL, Postgres, Mailman, sendmail, Postfix, gcc/g++, Linux, etc.) -- but the idea of using an open source storage solution still makes some people nervous.

At its heart, that's really all CFS is -- to build a decent cluster, you shoot your bolt on good hardware -- fast switches, RDMA-capable HBAs/HCAs, remotely manageable power, and beefy disks -- and all the software that you need to make it kick a lot of ass is free -- CFS/Lustre, Linux-HA, ROCKS -- none of it costs a dime.

I've built a handful of open-source on-the-cheap servers for production houses. One advantage that the vendors have is that they can test different gear before putting a unified product together.

Me? I'm stuck reading reviews on websites and pinging mailing lists for info and crossing my fingers that the gear I order - the combination of gear I order - will stand up under load, will scale well, won't have any driver issues, etc. My time (and money) for testing is always limited. That, I think, is the primary attraction of a (presumably) well-tested turnkey storage product from a vendor.

Plus, when you put the cost of really good gear together - the stuff you're talking about, not the stuff I'm typically forced to use - the cost often isn't much less than the price from an arm-twisted vendor.

It's all quite well-suited to the types of HPC environments that are the bread and butter of us people in entertainment -- but the only ones bold enough to use the stuff work in academia/government research -- and produce results to complex computing problems (seismic analysis, climate modelling, cryptanalysis) with smaller budgets and less technical people on staff.

I bet they a) have less strict downtime requirements and b) share a lot more information amongst themselves. If we shared homebuilt config ideas and benchmarks, we could probably come up with known-stable, high-performance configs for great prices.

BTW, if anybody in Toronto needs lots of high-performance storage for cheap, this would be a good place to start:

http://cgi.ebay.ca/ws/eBayISAPI.dll?ViewItem&item=200379234213

Andrew

Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/

--- On Tue, 9/1/09, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

I've been thinking about building something like that, but I've had a hard time finding good info about port-multiplier backplanes. The good news is that Backblaze seems to have a working, well-tested config using port multipliers; the bad news is that they thank "James Lee at Chyang Fun Industries in Taiwan for customizing SATA boards to simplify our design". I probably don't have the buying power clout to get James Lee - as nice a guy as he might be - to do that for me. (Although... if they have a few extras left over from the Backblaze build...)

Andrew

Make your browsing faster, safer, and easier with the new Internet Explorer? 8. Optimized for Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/

Oh, and they also thank a couple of engineers at Silicon Image "who helped us debug SATA protocol problems and loaned us 10 different SATA cards to test against".

That's the kind of testing I'm talking about.

What's the chance they're running custom SATA drivers, what with all that SATA protocol debugging? That's the kind of thing that it's hard for a small studio (or even a large one) to do. What's the chance that those drivers are going to make it back into the mainstream kernel? (One can only hope.)

Andrew

--- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote:

From: Andrew Klaassen clawsoon@yahoo.com Subject: Re: Question about network deleting. To: discuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:19 AM --- On Tue, 9/1/09, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

I've been thinking about building something like that, but I've had a hard time finding good info about port-multiplier backplanes.? The good news is that Backblaze seems to have a working, well-tested config using port multipliers; the bad news is that they thank "James Lee at Chyang Fun Industries in Taiwan for customizing SATA boards to simplify our design".? I probably don't have the buying power clout to get James Lee - as nice a guy as he might be - to do that for me.? (Although... if they have a few extras left over from the Backblaze build...)

Andrew

? ? ? _ Make your browsing faster, safer, and easier with the new Internet Explorer? 8. Optimized for Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/ __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/

If you put 5 Seagate ST31500341AS drives, (which will max out @ 120 MB/sec) on a port-multiplier backplane, won't that oversubscribe the SATA II port connected to the backplane?

Lessee... Sata II provides 3 Gbits/sec = 3000 Mbits/sec divide by 8 for Bytes = 375 MBytes/sec

5 drives maxed out (120 MBytes/sec * 5) = 600 MBytes/sec

So for max port multiplier utilization you would only want 3 drives per SATA II port, for a max of 360 MB/sec, which will fit under your 375 MB/s SATA II port throughput.

Does a 1.5 TB Seagate AS series drive really deliver 120MB/sec sustained? Maybe you could get away with 4 drives per port-multiplier backplane, but 5?

I suppose if you paid attention to which spindles were attached to which RAID you could make sure that all 5 drives per backplane weren't reading or writing at the same time.

Wouldn't LVM have more flexibility in this regard than mdadm?

.../Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Andrew Klaassen Sent: Wednesday, September 02, 2009 10:19 AM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

--- On Tue, 9/1/09, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build- cheap-cloud-storage/

I've been thinking about building something like that, but I've had a hard time finding good info about port-multiplier backplanes. The good news is that Backblaze seems to have a working, well-tested config using port multipliers; the bad news is that they thank "James Lee at Chyang Fun Industries in Taiwan for customizing SATA boards to simplify our design". I probably don't have the buying power clout to get James Lee - as nice a guy as he might be - to do that for me. (Although... if they have a few extras left over from the Backblaze build...)

Andrew

Make your browsing faster, safer, and easier with the new Internet Explorer(r) 8. Optimized for Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/

These numbers are for sequentials read write. Unfortunatly random
requests and fragmentation will make the numbers really lower and
this is where number of spindle might really help.

serge

On 2-Sep-09, at 10:48 AM, Burns Tom wrote:

If you put 5 Seagate ST31500341AS drives, (which will max out @ 120 MB/sec) on a port-multiplier backplane, won't that oversubscribe the SATA II port connected to the backplane?

Lessee... Sata II provides 3 Gbits/sec = 3000 Mbits/sec divide by 8 for Bytes = 375 MBytes/sec

5 drives maxed out (120 MBytes/sec * 5) = 600 MBytes/sec

So for max port multiplier utilization you would only want 3 drives per SATA II port, for a max of 360 MB/sec, which will fit under your 375 MB/s SATA II port throughput.

Does a 1.5 TB Seagate AS series drive really deliver 120MB/sec sustained? Maybe you could get away with 4 drives per port-multiplier backplane, but 5?

I suppose if you paid attention to which spindles were attached to which RAID you could make sure that all 5 drives per backplane weren't reading or writing at the same time.

Wouldn't LVM have more flexibility in this regard than mdadm?

.../Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Andrew Klaassen Sent: Wednesday, September 02, 2009 10:19 AM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

--- On Tue, 9/1/09, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build- cheap-cloud-storage/

I've been thinking about building something like that, but I've had a hard time finding good info about port-multiplier backplanes. The good news is that Backblaze seems to have a working, well-tested config using port multipliers; the bad news is that they thank "James Lee at Chyang Fun Industries in Taiwan for customizing SATA boards to simplify our design". I probably don't have the buying power clout to get James Lee - as nice a guy as he might be - to do that for me. (Although... if they have a few extras left over from the Backblaze build...)

Andrew

Make your browsing faster, safer, and easier with the new Internet Explorer(r) 8. Optimized for Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/

Aye... in theory, a single hard disk could max out a GigE connection. In practice, for a server with multiple clients, no such luck.

Cheap, reliable SSDs will make storage server design so much easier...

Andrew

--- On Wed, 9/2/09, Serge Dumoulin serge@ordigraphe.com wrote:

From: Serge Dumoulin serge@ordigraphe.com Subject: Re: Question about network deleting. To: discuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:59 AM These numbers are for sequentials read write. Unfortunatly? random requests? and fragmentation will make the numbers really lower? and this is where number of spindle might really help.

serge

On 2-Sep-09, at 10:48 AM, Burns Tom wrote:

If you put 5 Seagate ST31500341AS drives, (which will max out @ 120 MB/sec) on a port-multiplier backplane, won't that oversubscribe the SATA II port connected to the backplane?

Lessee...? Sata II provides 3 Gbits/sec = 3000 Mbits/sec divide by 8 for Bytes = 375 MBytes/sec

5 drives maxed out (120 MBytes/sec * 5) = 600 MBytes/sec

So for max port multiplier utilization you would only want 3 drives per SATA II port, for a max of 360 MB/sec, which will fit under your 375 MB/s SATA II port throughput.

Does a 1.5 TB Seagate AS series drive really deliver 120MB/sec sustained? Maybe you could get away with 4 drives per port-multiplier backplane, but 5?

I suppose if you paid attention to which spindles were attached to which RAID you could make sure that all 5 drives per backplane weren't reading or writing at the same time.

Wouldn't LVM have more flexibility in this regard than mdadm?

.../Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Andrew Klaassen Sent: Wednesday, September 02, 2009 10:19 AM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

--- On Tue, 9/1/09, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build- cheap-cloud-storage/

I've been thinking about building something like that, but I've had a hard time finding good info about port-multiplier backplanes.? The good news is that Backblaze seems to have a working, well-tested config using port multipliers; the bad news is that they thank "James Lee at Chyang Fun Industries in Taiwan for customizing SATA boards to simplify our design".? I probably don't have the buying power clout to get James Lee - as nice a guy as he might be - to do that for me.? (Although... if they have a few extras left over from the Backblaze build...)

Andrew

? ???___ Make your browsing faster, safer, and easier with the new Internet Explorer(r) 8. Optimized for Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/ __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-disc uss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Ask a question on any topic and get answers from real people. Go to Yahoo! Answers and share what you know at http://ca.answers.yahoo.com

I Wuz Wrong. I emailed the Backblaze guys, and they're actually using a stock kernel and stock firmware. Here's the response:

"Regarding your questions below, we run the Debian backport of 2.6.26, use stock firmware from Silicon Image, and the back planes we actually get from a company called CFY in Taiwan (not Silicon Image.)

"We actually do plan to publish at least a few more blog posts with more data & learnings about the pods. (Follow us on Twitter if you want to be notified when we do...)

"Also, we've received so many requests to buy the storage pods in the last 24 hours despite providing all the specs that we're contemplating where to do this somehow. If people are interested in possibly doing this, they can email salescontact@backblaze.com and we'll keep them abreast of whether we'll do that."

I wonder if you could make a Lustre cluster of these guys, or if there'd be problems with redundancy...

Andrew

--- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote:

From: Andrew Klaassen clawsoon@yahoo.com Subject: Re: Question about network deleting. To: discuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:24 AM Oh, and they also thank a couple of engineers at Silicon Image "who helped us debug SATA protocol problems and loaned us 10 different SATA cards to test against".

That's the kind of testing I'm talking about.

What's the chance they're running custom SATA drivers, what with all that SATA protocol debugging?? That's the kind of thing that it's hard for a small studio (or even a large one) to do.? What's the chance that those drivers are going to make it back into the mainstream kernel?? (One can only hope.)

Andrew

--- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote:

From: Andrew Klaassen clawsoon@yahoo.com Subject: Re: Question about network deleting. To: discuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:19 AM --- On Tue, 9/1/09, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

I've been thinking about building something like that,

but > I've had a hard time finding good info about port-multiplier > backplanes.? The good news is that Backblaze seems to > have a working, well-tested config using port multipliers; > the bad news is that they thank "James Lee at Chyang Fun > Industries in Taiwan for customizing SATA boards to simplify > our design".? I probably don't have the buying power > clout to get James Lee - as nice a guy as he might be - to > do that for me.? (Although... if they have a few extras > left over from the Backblaze build...) > > Andrew > > > > > ? ? ? > _ > Make your browsing faster, safer, and easier with the new > Internet Explorer? 8. Optimized for Yahoo! Get it Now for > Free! at http://downloads.yahoo.com/ca/internetexplorer/ > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >

? ? ? Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/ _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

The new Internet Explorer? 8 - Faster, safer, easier. Optimized for Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/

you can use baloney sandwiches and make a luster cluster, buster!

8)

-g

On Sep 4, 2009, at 10:33 AM, Andrew Klaassen wrote:

I Wuz Wrong. I emailed the Backblaze guys, and they're actually
using a stock kernel and stock firmware. Here's the response:

"Regarding your questions below, we run the Debian backport of
2.6.26, use stock firmware from Silicon Image, and the back planes
we actually get from a company called CFY in Taiwan (not Silicon
Image.)

"We actually do plan to publish at least a few more blog posts with
more data & learnings about the pods. (Follow us on Twitter if you
want to be notified when we do...)

"Also, we've received so many requests to buy the storage pods in
the last 24 hours despite providing all the specs that we're
contemplating where to do this somehow. If people are interested in
possibly doing this, they can email salescontact@backblaze.com and
we'll keep them abreast of whether we'll do that."

I wonder if you could make a Lustre cluster of these guys, or if
there'd be problems with redundancy...

Andrew

--- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote:

From: Andrew Klaassen clawsoon@yahoo.com Subject: Re: Question about network deleting. To: discuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:24 AM Oh, and they also thank a couple of engineers at Silicon Image "who helped us debug SATA protocol problems and loaned us 10 different SATA cards to test against".

That's the kind of testing I'm talking about.

What's the chance they're running custom SATA drivers, what with all that SATA protocol debugging? That's the kind of thing that it's hard for a small studio (or even a large one) to do. What's the chance that those drivers are going to make it back into the mainstream kernel? (One can only hope.)

Andrew

--- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote:

From: Andrew Klaassen clawsoon@yahoo.com Subject: Re: Question about network deleting. To: discuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:19 AM --- On Tue, 9/1/09, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

I've been thinking about building something like that,

but > I've had a hard time finding good info about port-multiplier > backplanes. The good news is that Backblaze seems to > have a working, well-tested config using port multipliers; > the bad news is that they thank "James Lee at Chyang Fun > Industries in Taiwan for customizing SATA boards to simplify > our design". I probably don't have the buying power > clout to get James Lee - as nice a guy as he might be - to > do that for me. (Although... if they have a few extras > left over from the Backblaze build...) > > Andrew > > > > > > _ > Make your browsing faster, safer, and easier with the new > Internet Explorer? 8. Optimized for Yahoo! Get it Now for > Free! at http://downloads.yahoo.com/ca/internetexplorer/ > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >

Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/ _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

  <br />

The new Internet Explorer? 8 - Faster, safer, easier. Optimized for
Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/ _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

But what happens if someone eats one of the sandwiches? What happens to your data?

Andrew

--- On Fri, 9/4/09, Greg Whynott Greg.Whynott@oicr.on.ca wrote:

you can use baloney sandwiches and make a luster cluster,? buster!

8)

-g

On Sep 4, 2009, at 10:33 AM, Andrew Klaassen wrote:

I Wuz Wrong.? I emailed the Backblaze guys, and

they're actually? > using a stock kernel and stock firmware.? Here's the response: > > "Regarding your questions below, we run the Debian backport of? > 2.6.26, use stock firmware from Silicon Image, and the back planes? > we actually get from a company called CFY in Taiwan (not Silicon? > Image.) > > "We actually do plan to publish at least a few more blog posts with? > more data & learnings about the pods. (Follow us on Twitter if you? > want to be notified when we do...) > > "Also, we've received so many requests to buy the storage pods in? > the last 24 hours despite providing all the specs that we're? > contemplating where to do this somehow. If people are interested in? > possibly doing this, they can email salescontact@backblaze.com and? > we'll keep them abreast of whether we'll do that." > > I wonder if you could make a Lustre cluster of these guys, or if? > there'd be problems with redundancy... > > Andrew > > > --- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote: > >> From: Andrew Klaassen clawsoon@yahoo.com >> Subject: Re: Question about network deleting. >> To: discuss@studiosysadmins.com >> Received: Wednesday, September 2, 2009, 7:24 AM >> Oh, and they also thank a couple of >> engineers at Silicon Image "who helped us debug SATA >> protocol problems and loaned us 10 different SATA cards to >> test against". >> >> That's the kind of testing I'm talking about. >> >> What's the chance they're running custom SATA drivers, what >> with all that SATA protocol debugging?? That's the kind >> of thing that it's hard for a small studio (or even a large >> one) to do.? What's the chance that those drivers are >> going to make it back into the mainstream kernel?? (One >> can only hope.) >> >> Andrew >> >> >> --- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com >> wrote: >> >>> From: Andrew Klaassen clawsoon@yahoo.com >>> Subject: Re: Question about network deleting. >>> To: discuss@studiosysadmins.com >>> Received: Wednesday, September 2, 2009, 7:19 AM >>> --- On Tue, 9/1/09, Klaus Steden >>> klaus-s@moving-picture.com >>> wrote: >>> >>>> You could try one of these ... >>>> >>>> >>>> >>>> http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ >>> >>> I've been thinking about building something like that, >> but >>> I've had a hard time finding good info about >> port-multiplier >>> backplanes.? The good news is that Backblaze seems >> to >>> have a working, well-tested config using port >> multipliers; >>> the bad news is that they thank "James Lee at Chyang >> Fun >>> Industries in Taiwan for customizing SATA boards to >> simplify >>> our design".? I probably don't have the buying power >>> clout to get James Lee - as nice a guy as he might be >> - to >>> do that for me.? (Although... if they have a few >> extras >>> left over from the Backblaze build...) >>> >>> Andrew >>> >>> >>> >>> >>> >>> >> _ >>> Make your browsing faster, safer, and easier with the >> new >>> Internet Explorer? 8. Optimized for Yahoo! Get it Now >> for >>> Free! at http://downloads.yahoo.com/ca/internetexplorer/ >>> __ >>> StudioSysAdmins-Discuss mailing list >>> StudioSysAdmins-Discuss@studiosysadmins.com >>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >>> >> >> >> >> >> Looking for the perfect gift? Give the gift of Flickr! >> >> http://www.flickr.com/gift/ >> __ >> StudioSysAdmins-Discuss mailing list >> StudioSysAdmins-Discuss@studiosysadmins.com >> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >> > > >? ? ??? > > The new Internet Explorer? 8 - Faster, safer, easier.? Optimized for? > Yahoo!? Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/ > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/

you'll be able to recover it from snapshots later on... ( about an 8
hour delay)

8)

-g

On Sep 4, 2009, at 11:04 AM, Andrew Klaassen wrote:

But what happens if someone eats one of the sandwiches? What
happens to your data?

Andrew

--- On Fri, 9/4/09, Greg Whynott Greg.Whynott@oicr.on.ca wrote:

you can use baloney sandwiches and make a luster cluster, buster!

8)

-g

On Sep 4, 2009, at 10:33 AM, Andrew Klaassen wrote:

I Wuz Wrong. I emailed the Backblaze guys, and

they're actually > using a stock kernel and stock firmware. Here's the response: > > "Regarding your questions below, we run the Debian backport of > 2.6.26, use stock firmware from Silicon Image, and the back planes > we actually get from a company called CFY in Taiwan (not Silicon > Image.) > > "We actually do plan to publish at least a few more blog posts with > more data & learnings about the pods. (Follow us on Twitter if you > want to be notified when we do...) > > "Also, we've received so many requests to buy the storage pods in > the last 24 hours despite providing all the specs that we're > contemplating where to do this somehow. If people are interested in > possibly doing this, they can email salescontact@backblaze.com and > we'll keep them abreast of whether we'll do that." > > I wonder if you could make a Lustre cluster of these guys, or if > there'd be problems with redundancy... > > Andrew > > > --- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote: > >> From: Andrew Klaassen clawsoon@yahoo.com >> Subject: Re: Question about network deleting. >> To: discuss@studiosysadmins.com >> Received: Wednesday, September 2, 2009, 7:24 AM >> Oh, and they also thank a couple of >> engineers at Silicon Image "who helped us debug SATA >> protocol problems and loaned us 10 different SATA cards to >> test against". >> >> That's the kind of testing I'm talking about. >> >> What's the chance they're running custom SATA drivers, what >> with all that SATA protocol debugging? That's the kind >> of thing that it's hard for a small studio (or even a large >> one) to do. What's the chance that those drivers are >> going to make it back into the mainstream kernel? (One >> can only hope.) >> >> Andrew >> >> >> --- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com >> wrote: >> >>> From: Andrew Klaassen clawsoon@yahoo.com >>> Subject: Re: Question about network deleting. >>> To: discuss@studiosysadmins.com >>> Received: Wednesday, September 2, 2009, 7:19 AM >>> --- On Tue, 9/1/09, Klaus Steden >>> klaus-s@moving-picture.com >>> wrote: >>> >>>> You could try one of these ... >>>> >>>> >>>> >>>> http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ >>> >>> I've been thinking about building something like that, >> but >>> I've had a hard time finding good info about >> port-multiplier >>> backplanes. The good news is that Backblaze seems >> to >>> have a working, well-tested config using port >> multipliers; >>> the bad news is that they thank "James Lee at Chyang >> Fun >>> Industries in Taiwan for customizing SATA boards to >> simplify >>> our design". I probably don't have the buying power >>> clout to get James Lee - as nice a guy as he might be >> - to >>> do that for me. (Although... if they have a few >> extras >>> left over from the Backblaze build...) >>> >>> Andrew >>> >>> >>> >>> >>> >>> >> _ >>> Make your browsing faster, safer, and easier with the >> new >>> Internet Explorer? 8. Optimized for Yahoo! Get it Now >> for >>> Free! at http://downloads.yahoo.com/ca/internetexplorer/ >>> __ >>> StudioSysAdmins-Discuss mailing list >>> StudioSysAdmins-Discuss@studiosysadmins.com >>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >>> >> >> >> >> >> Looking for the perfect gift? Give the gift of Flickr! >> >> http://www.flickr.com/gift/ >> __ >> StudioSysAdmins-Discuss mailing list >> StudioSysAdmins-Discuss@studiosysadmins.com >> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >> > > > > > The new Internet Explorer? 8 - Faster, safer, easier. Optimized for > Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/ > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

  <br />

Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/ _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

I can't speak for sandwhiches though I prefer montreal smoked...

I contacted the case vendor for Blackblaze and they are selling the cases:

Quantity

Price

1 ? 4

$872

5 ? 9

$812

10 ? 19

$782

20+

$758

-Daniel

Is that the CFY Vendor from Taiwan that was mentioned earlier? :) And if not (or if so, even!), do you have any contact information? :)

I totally wanna build one of these, and getting the backplane and chassis is the only thing holding me back. :D

-- David Fix Senior Systems Administrator Mr. X Inc. 35 McCaul Street, Ste. #100 Toronto, ON M5T 1V7 T: (416) 595-6222, x 241 F: (416) 595-9122 E: davidf@mrxfx.com

----- Original Message ----- From: "Daniel Roizman" roizman@kolektiv.com To: discuss@studiosysadmins.com Sent: Friday, September 4, 2009 11:45:41 AM GMT -05:00 US/Canada Eastern Subject: Re: Question about network deleting.

I can't speak for sandwhiches though I prefer montreal smoked...

I contacted the case vendor for Blackblaze and they are selling the cases:

Quantity

Price

1 – 4

$872

5 – 9

$812

10 – 19

$782

20+

$758

-Daniel

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

That must be the data protection scheme. "Yeah, we're gonna give Doug a tour of our facilities today, so make sure you put some mustard and mayo on the baloney. Don't want to lose any data. Tomorrow Andrew's coming by, so stock up on some hot sauce."

Andrew

--- On Fri, 9/4/09, Douglas C. Atkinson Douglas@sevengroup.ca wrote:

And what if I don't like mustard or Mayo?? Am I suppose to eat the sandwich also??

---

Douglas C. Atkinson 101-3738 North Fraser Way? ? Burnaby, B.C. V5J 5G7? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 604.419.8585? ? ? ???Direct? ? ? ? ? ? ? ? ? ? ? ? 604.434.2035? ? ? ? ? Fax? ? ? ? ? ? ? ? ? ? ? ? ? ??? 604.619.9430? ? ? ? ? Cell

Seven Group - Calgary 106-919 Centre Street N.W? ? Calgary, Alberta? ? ? ? ? ? ? ? ? ? ??? T2A 2P6? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

1-877-462-1777 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? "Data Management with Integrity"? ? ? ???

----- Original Message ----- From: studiosysadmins-discuss-bounces@studiosysadmins.com studiosysadmins-discuss-bounces@studiosysadmins.com To: discuss@studiosysadmins.com discuss@studiosysadmins.com Sent: Fri Sep 04 08:04:34 2009 Subject: Re: Question about network deleting.

But what happens if someone eats one of the sandwiches?? What happens to your data?

Andrew

--- On Fri, 9/4/09, Greg Whynott Greg.Whynott@oicr.on.ca wrote:

you can use baloney sandwiches and make a luster cluster,? buster!

8)

-g

On Sep 4, 2009, at 10:33 AM, Andrew Klaassen wrote:

I Wuz Wrong.? I emailed the Backblaze guys, and

they're actually? > using a stock kernel and stock firmware.?

Here's > the response: > > > > "Regarding your questions below, we run the Debian > backport of? > > 2.6.26, use stock firmware from Silicon Image, and the > back planes? > > we actually get from a company called CFY in Taiwan > (not Silicon? > > Image.) > > > > "We actually do plan to publish at least a few more > blog posts with? > > more data & learnings about the pods. (Follow us > on Twitter if you? > > want to be notified when we do...) > > > > "Also, we've received so many requests to buy the > storage pods in? > > the last 24 hours despite providing all the specs that > we're? > > contemplating where to do this somehow. If people are > interested in? > > possibly doing this, they can email salescontact@backblaze.com > and? > > we'll keep them abreast of whether we'll do that." > > > > I wonder if you could make a Lustre cluster of these > guys, or if? > > there'd be problems with redundancy... > > > > Andrew > > > > > > --- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com > wrote: > > > >> From: Andrew Klaassen clawsoon@yahoo.com > >> Subject: Re: Question about network deleting. > >> To: discuss@studiosysadmins.com > >> Received: Wednesday, September 2, 2009, 7:24 AM > >> Oh, and they also thank a couple of > >> engineers at Silicon Image "who helped us debug > SATA > >> protocol problems and loaned us 10 different SATA > cards to > >> test against". > >> > >> That's the kind of testing I'm talking about. > >> > >> What's the chance they're running custom SATA > drivers, what > >> with all that SATA protocol debugging?? > That's the kind > >> of thing that it's hard for a small studio (or > even a large > >> one) to do.? What's the chance that those > drivers are > >> going to make it back into the mainstream > kernel?? (One > >> can only hope.) > >> > >> Andrew > >> > >> > >> --- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com > >> wrote: > >> > >>> From: Andrew Klaassen clawsoon@yahoo.com > >>> Subject: Re: Question about network deleting. > >>> To: discuss@studiosysadmins.com > >>> Received: Wednesday, September 2, 2009, 7:19 > AM > >>> --- On Tue, 9/1/09, Klaus Steden > >>> klaus-s@moving-picture.com > >>> wrote: > >>> > >>>> You could try one of these ... > >>>> > >>>> > >>>> > >>>> http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

I've been thinking about building

something > like that, > >> but > >>> I've had a hard time finding good info about > >> port-multiplier > >>> backplanes.? The good news is that > Backblaze seems > >> to > >>> have a working, well-tested config using port > >> multipliers; > >>> the bad news is that they thank "James Lee at > Chyang > >> Fun > >>> Industries in Taiwan for customizing SATA > boards to > >> simplify > >>> our design".? I probably don't have the > buying power > >>> clout to get James Lee - as nice a guy as he > might be > >> - to > >>> do that for me.? (Although... if they > have a few > >> extras > >>> left over from the Backblaze build...) > >>> > >>> Andrew > >>> > >>> > >>> > >>> > >>> > >>> > >> > > >>> Make your browsing faster, safer, and easier > with the > >> new > >>> Internet Explorer? 8. Optimized for Yahoo! > Get it Now > >> for > >>> Free! at http://downloads.yahoo.com/ca/internetexplorer/

_ >>> StudioSysAdmins-Discuss mailing list >>> StudioSysAdmins-Discuss@studiosysadmins.com >>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

> >> Looking for the perfect gift? Give the gift of > Flickr! > >> > >> http://www.flickr.com/gift/

_ > >> StudioSysAdmins-Discuss mailing list > >> StudioSysAdmins-Discuss@studiosysadmins.com > >> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

? ? ???

> > The new Internet Explorer? 8 - Faster, safer, > easier.? Optimized for? > > Yahoo!? Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

? ? ? Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/

Ditto... would definitely like any information you've got.

Andrew

--- On Fri, 9/4/09, David Fix davidf@mrxfx.com wrote:

Is that the CFY Vendor from Taiwan that was mentioned earlier?  :)  And if not (or if so, even!), do you have any contact information?  :)

I totally wanna build one of these, and getting the backplane and chassis is the only thing holding me back.  :D

-- David Fix Senior Systems Administrator Mr. X Inc. 35 McCaul Street, Ste. #100 Toronto, ON  M5T 1V7 T:  (416) 595-6222, x 241 F:  (416) 595-9122 E:  davidf@mrxfx.com

----- Original Message ----- From: "Daniel Roizman" roizman@kolektiv.com To: discuss@studiosysadmins.com Sent: Friday, September 4, 2009 11:45:41 AM GMT -05:00 US/Canada Eastern Subject: Re: Question about network deleting.

I can't speak for sandwhiches though I prefer montreal smoked...

I contacted the case vendor for Blackblaze and they are selling the cases:

Quantity

Price



1 – 4

$872



5 – 9

$812



10 – 19

$782



20+

$758

-Daniel

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

-----Inline Attachment Follows-----

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Reclaim your name @ymail.com or @rocketmail.com. Get your new email address now! Go to http://ca.promos.yahoo.com/jacko/

Only if they're presented as LUNs ... /dev/baloney0, /dev/baloney1, etc.

-----Original Message----- From: Greg Whynott [mailto:Greg.Whynott@oicr.on.ca] Sent: Fri 9/4/2009 7:47 AM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

you can use baloney sandwiches and make a luster cluster, buster!

8)

-g

On Sep 4, 2009, at 10:33 AM, Andrew Klaassen wrote:

I Wuz Wrong. I emailed the Backblaze guys, and they're actually
using a stock kernel and stock firmware. Here's the response:

"Regarding your questions below, we run the Debian backport of
2.6.26, use stock firmware from Silicon Image, and the back planes
we actually get from a company called CFY in Taiwan (not Silicon
Image.)

"We actually do plan to publish at least a few more blog posts with
more data & learnings about the pods. (Follow us on Twitter if you
want to be notified when we do...)

"Also, we've received so many requests to buy the storage pods in
the last 24 hours despite providing all the specs that we're
contemplating where to do this somehow. If people are interested in
possibly doing this, they can email salescontact@backblaze.com and
we'll keep them abreast of whether we'll do that."

I wonder if you could make a Lustre cluster of these guys, or if
there'd be problems with redundancy...

Andrew

--- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote:

From: Andrew Klaassen clawsoon@yahoo.com Subject: Re: Question about network deleting. To: discuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:24 AM Oh, and they also thank a couple of engineers at Silicon Image "who helped us debug SATA protocol problems and loaned us 10 different SATA cards to test against".

That's the kind of testing I'm talking about.

What's the chance they're running custom SATA drivers, what with all that SATA protocol debugging? That's the kind of thing that it's hard for a small studio (or even a large one) to do. What's the chance that those drivers are going to make it back into the mainstream kernel? (One can only hope.)

Andrew

--- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote:

From: Andrew Klaassen clawsoon@yahoo.com Subject: Re: Question about network deleting. To: discuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:19 AM --- On Tue, 9/1/09, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

I've been thinking about building something like that,

but > I've had a hard time finding good info about port-multiplier > backplanes. The good news is that Backblaze seems to > have a working, well-tested config using port multipliers; > the bad news is that they thank "James Lee at Chyang Fun > Industries in Taiwan for customizing SATA boards to simplify > our design". I probably don't have the buying power > clout to get James Lee - as nice a guy as he might be - to > do that for me. (Although... if they have a few extras > left over from the Backblaze build...) > > Andrew > > > > > > _ > Make your browsing faster, safer, and easier with the new > Internet Explorer? 8. Optimized for Yahoo! Get it Now for > Free! at http://downloads.yahoo.com/ca/internetexplorer/ > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >

Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/ _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

  <br />

The new Internet Explorer? 8 - Faster, safer, easier. Optimized for
Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/ _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Aha, I emailed Backblaze directly as well, and got this:

You're right! I feel like everyone wants a storage pod! Well, the good news is that the case is manufactured in your part of the world by a company called Protocase. They are selling the cases directly on their website after receiving a few hundred inquiries... Check out http://www.protocase.com

As for the backplanes, we originally discovered them by disassembling a Sans Digital TowerRAID TR5M1... http://www.sansdigital.com/towerraid/tr5m1.html

Buying the enclosure just for the backplane is a bit expensive so you can order them directly from the CFI Group in Taiwan ( http://www.chyangfun.com/ ) for $42/each, in quantity. Not sure how much they would charge for a smaller order.

Good luck! ---

-- David Fix Senior Systems Administrator Mr. X Inc. 35 McCaul Street, Ste. #100 Toronto, ON M5T 1V7 T: (416) 595-6222, x 241 F: (416) 595-9122 E: davidf@mrxfx.com

----- Original Message ----- From: "Andrew Klaassen" clawsoon@yahoo.com To: discuss@studiosysadmins.com Sent: Friday, September 4, 2009 12:16:49 PM GMT -05:00 US/Canada Eastern Subject: Re: Question about network deleting.

Ditto... would definitely like any information you've got.

Andrew

--- On Fri, 9/4/09, David Fix davidf@mrxfx.com wrote:

Is that the CFY Vendor from Taiwan that was mentioned earlier? :) And if not (or if so, even!), do you have any contact information? :)

I totally wanna build one of these, and getting the backplane and chassis is the only thing holding me back. :D

-- David Fix Senior Systems Administrator Mr. X Inc. 35 McCaul Street, Ste. #100 Toronto, ON M5T 1V7 T: (416) 595-6222, x 241 F: (416) 595-9122 E: davidf@mrxfx.com

----- Original Message ----- From: "Daniel Roizman" roizman@kolektiv.com To: discuss@studiosysadmins.com Sent: Friday, September 4, 2009 11:45:41 AM GMT -05:00 US/Canada Eastern Subject: Re: Question about network deleting.

I can't speak for sandwhiches though I prefer montreal smoked...

I contacted the case vendor for Blackblaze and they are selling the cases:

Quantity

Price

1 – 4

$872

5 – 9

$812

10 – 19

$782

20+

$758

-Daniel

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

-----Inline Attachment Follows-----

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Shouldn't it be:

/dev/bread0

/dev/baloney0

/dev/bread1

;)

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Klaus Steden Sent: Friday, September 04, 2009 7:01 AM To: discuss@studiosysadmins.com; discuss@studiosysadmins.com Subject: RE: Question about network deleting.

Only if they're presented as LUNs ... /dev/baloney0, /dev/baloney1, etc.

-----Original Message----- From: Greg Whynott [mailto:Greg.Whynott@oicr.on.ca] Sent: Fri 9/4/2009 7:47 AM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

you can use baloney sandwiches and make a luster cluster, buster!

8)

-g

On Sep 4, 2009, at 10:33 AM, Andrew Klaassen wrote:

I Wuz Wrong. I emailed the Backblaze guys, and they're actually using a stock kernel and stock firmware. Here's the response:

"Regarding your questions below, we run the Debian backport of 2.6.26, use stock firmware from Silicon Image, and the back planes we actually get from a company called CFY in Taiwan (not Silicon Image.)

"We actually do plan to publish at least a few more blog posts with more data & learnings about the pods. (Follow us on Twitter if you want to be notified when we do...)

"Also, we've received so many requests to buy the storage pods in the last 24 hours despite providing all the specs that we're contemplating where to do this somehow. If people are interested in possibly doing this, they can email salescontact@backblaze.com and we'll keep them abreast of whether we'll do that."

I wonder if you could make a Lustre cluster of these guys, or if there'd be problems with redundancy...

Andrew

--- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote:

From: Andrew Klaassen clawsoon@yahoo.com Subject: Re: Question about network deleting. To: discuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:24 AM Oh, and they also thank a couple of engineers at Silicon Image "who helped us debug SATA protocol problems and loaned us 10 different SATA cards to test against".

That's the kind of testing I'm talking about.

What's the chance they're running custom SATA drivers, what with all that SATA protocol debugging? That's the kind of thing that it's hard for a small studio (or even a large one) to do. What's the chance that those drivers are going to make it back into the mainstream kernel? (One can only hope.)

Andrew

--- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote:

From: Andrew Klaassen clawsoon@yahoo.com Subject: Re: Question about network deleting. To: discuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:19 AM --- On Tue, 9/1/09, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-chea p-cloud-storage/ >>> >>> I've been thinking about building something like that, >> but >>> I've had a hard time finding good info about >> port-multiplier >>> backplanes. The good news is that Backblaze seems >> to >>> have a working, well-tested config using port >> multipliers; >>> the bad news is that they thank "James Lee at Chyang >> Fun >>> Industries in Taiwan for customizing SATA boards to >> simplify >>> our design". I probably don't have the buying power >>> clout to get James Lee - as nice a guy as he might be >> - to >>> do that for me. (Although... if they have a few >> extras >>> left over from the Backblaze build...) >>> >>> Andrew >>> >>> >>> >>> >>> >>> >> >>> Make your browsing faster, safer, and easier with the >> new >>> Internet ExplorerR 8. Optimized for Yahoo! Get it Now >> for >>> Free! at http://downloads.yahoo.com/ca/internetexplorer/ >>> __ >>> StudioSysAdmins-Discuss mailing list >>> StudioSysAdmins-Discuss@studiosysadmins.com >>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >>> >> >> >> >> >> Looking for the perfect gift? Give the gift of Flickr! >> >> http://www.flickr.com/gift/ >> __ >> StudioSysAdmins-Discuss mailing list >> StudioSysAdmins-Discuss@studiosysadmins.com >> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >> > > >
> _ > The new Internet ExplorerR 8 - Faster, safer, easier. Optimized for > Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/ > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Shouldn't it be:

/dev/bread0

/dev/baloney0

/dev/bread1

;)

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Klaus Steden Sent: Friday, September 04, 2009 7:01 AM To: discuss@studiosysadmins.com; discuss@studiosysadmins.com Subject: RE: Question about network deleting.

Only if they're presented as LUNs ... /dev/baloney0, /dev/baloney1, etc.

-----Original Message----- From: Greg Whynott [mailto:Greg.Whynott@oicr.on.ca] Sent: Fri 9/4/2009 7:47 AM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

you can use baloney sandwiches and make a luster cluster, buster!

8)

-g

On Sep 4, 2009, at 10:33 AM, Andrew Klaassen wrote:

I Wuz Wrong. I emailed the Backblaze guys, and they're actually using a stock kernel and stock firmware. Here's the response:

"Regarding your questions below, we run the Debian backport of 2.6.26, use stock firmware from Silicon Image, and the back planes we actually get from a company called CFY in Taiwan (not Silicon Image.)

"We actually do plan to publish at least a few more blog posts with more data & learnings about the pods. (Follow us on Twitter if you want to be notified when we do...)

"Also, we've received so many requests to buy the storage pods in the last 24 hours despite providing all the specs that we're contemplating where to do this somehow. If people are interested in possibly doing this, they can email salescontact@backblaze.com and we'll keep them abreast of whether we'll do that."

I wonder if you could make a Lustre cluster of these guys, or if there'd be problems with redundancy...

Andrew

--- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote:

From: Andrew Klaassen clawsoon@yahoo.com Subject: Re: Question about network deleting. To: discuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:24 AM Oh, and they also thank a couple of engineers at Silicon Image "who helped us debug SATA protocol problems and loaned us 10 different SATA cards to test against".

That's the kind of testing I'm talking about.

What's the chance they're running custom SATA drivers, what with all that SATA protocol debugging? That's the kind of thing that it's hard for a small studio (or even a large one) to do. What's the chance that those drivers are going to make it back into the mainstream kernel? (One can only hope.)

Andrew

--- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote:

From: Andrew Klaassen clawsoon@yahoo.com Subject: Re: Question about network deleting. To: discuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:19 AM --- On Tue, 9/1/09, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-chea p-cloud-storage/ >>> >>> I've been thinking about building something like that, >> but >>> I've had a hard time finding good info about >> port-multiplier >>> backplanes. The good news is that Backblaze seems >> to >>> have a working, well-tested config using port >> multipliers; >>> the bad news is that they thank "James Lee at Chyang >> Fun >>> Industries in Taiwan for customizing SATA boards to >> simplify >>> our design". I probably don't have the buying power >>> clout to get James Lee - as nice a guy as he might be >> - to >>> do that for me. (Although... if they have a few >> extras >>> left over from the Backblaze build...) >>> >>> Andrew >>> >>> >>> >>> >>> >>> >> >>> Make your browsing faster, safer, and easier with the >> new >>> Internet ExplorerR 8. Optimized for Yahoo! Get it Now >> for >>> Free! at http://downloads.yahoo.com/ca/internetexplorer/ >>> __ >>> StudioSysAdmins-Discuss mailing list >>> StudioSysAdmins-Discuss@studiosysadmins.com >>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >>> >> >> >> >> >> Looking for the perfect gift? Give the gift of Flickr! >> >> http://www.flickr.com/gift/ >> __ >> StudioSysAdmins-Discuss mailing list >> StudioSysAdmins-Discuss@studiosysadmins.com >> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >> > > >
> _ > The new Internet ExplorerR 8 - Faster, safer, easier. Optimized for > Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/ > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

/dev/nerds

On Fri, Sep 4, 2009 at 1:57 PM, Jeff Bell admin@twitchmedia.com wrote:

Shouldn?t it be:

/dev/bread0

/dev/baloney0

/dev/bread1

;)

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto: studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Klaus Steden Sent: Friday, September 04, 2009 7:01 AM To: discuss@studiosysadmins.com; discuss@studiosysadmins.com Subject: RE: Question about network deleting.

Only if they're presented as LUNs ... /dev/baloney0, /dev/baloney1, etc.

-----Original Message----- From: Greg Whynott [mailto:Greg.Whynott@oicr.on.caGreg.Whynott@oicr.on.ca ] Sent: Fri 9/4/2009 7:47 AM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

you can use baloney sandwiches and make a luster cluster, buster!

8)

-g

On Sep 4, 2009, at 10:33 AM, Andrew Klaassen wrote:

I Wuz Wrong. I emailed the Backblaze guys, and they're actually using a stock kernel and stock firmware. Here's the response:

"Regarding your questions below, we run the Debian backport of 2.6.26, use stock firmware from Silicon Image, and the back planes we actually get from a company called CFY in Taiwan (not Silicon Image.)

"We actually do plan to publish at least a few more blog posts with more data & learnings about the pods. (Follow us on Twitter if you want to be notified when we do...)

"Also, we've received so many requests to buy the storage pods in the last 24 hours despite providing all the specs that we're contemplating where to do this somehow. If people are interested in possibly doing this, they can email salescontact@backblaze.com and we'll keep them abreast of whether we'll do that."

I wonder if you could make a Lustre cluster of these guys, or if there'd be problems with redundancy...

Andrew

--- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote:

From: Andrew Klaassen clawsoon@yahoo.com Subject: Re: Question about network deleting. To: discuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:24 AM Oh, and they also thank a couple of engineers at Silicon Image "who helped us debug SATA protocol problems and loaned us 10 different SATA cards to test against".

That's the kind of testing I'm talking about.

What's the chance they're running custom SATA drivers, what with all that SATA protocol debugging? That's the kind of thing that it's hard for a small studio (or even a large one) to do. What's the chance that those drivers are going to make it back into the mainstream kernel? (One can only hope.)

Andrew

--- On Wed, 9/2/09, Andrew Klaassen clawsoon@yahoo.com wrote:

From: Andrew Klaassen clawsoon@yahoo.com Subject: Re: Question about network deleting. To: discuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:19 AM --- On Tue, 9/1/09, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ >>> >>> I've been thinking about building something like that, >> but >>> I've had a hard time finding good info about >> port-multiplier >>> backplanes. The good news is that Backblaze seems >> to >>> have a working, well-tested config using port >> multipliers; >>> the bad news is that they thank "James Lee at Chyang >> Fun >>> Industries in Taiwan for customizing SATA boards to >> simplify >>> our design". I probably don't have the buying power >>> clout to get James Lee - as nice a guy as he might be >> - to >>> do that for me. (Although... if they have a few >> extras >>> left over from the Backblaze build...) >>> >>> Andrew >>> >>> >>> >>> >>> >>> >> _ >>> Make your browsing faster, safer, and easier with the >> new >>> Internet Explorer? 8. Optimized for Yahoo! Get it Now >> for >>> Free! at http://downloads.yahoo.com/ca/internetexplorer/ >>> __ >>> StudioSysAdmins-Discuss mailing list >>> StudioSysAdmins-Discuss@studiosysadmins.com >>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >>> >> >> >> >> >> Looking for the perfect gift? Give the gift of Flickr! >> >> http://www.flickr.com/gift/ >> __ >> StudioSysAdmins-Discuss mailing list >> StudioSysAdmins-Discuss@studiosysadmins.com >> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >> > > > > > The new Internet Explorer? 8 - Faster, safer, easier. Optimized for > Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/ > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

I didn't get that when I did pwd :)

or do I do. Rm -r * , killall * (Yep good old sgi)

That will do it

/O Sent from my BlackBerry device on the Rogers Wireless Network

-----Original Message----- From: Jeffrey Klug jeff@intelligentcreatures.com

Date: Fri, 4 Sep 2009 14:53:55 To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

/dev/b0b1b1, if you want to pass off the entire device as a snack. 8)

-g

On Sep 4, 2009, at 1:57 PM, Jeff Bell wrote:

Shouldn?t it be:

/dev/bread0 /dev/baloney0 /dev/bread1

;)

From: studiosysadmins-discuss-bounces@studiosysadmins.comstudiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Klaus Steden Sent: Friday, September 04, 2009 7:01 AM To: discuss@studiosysadmins.comdiscuss@studiosysadmins.com; discuss@studiosysadmins.comdiscuss@studiosysadmins.com Subject: RE: Question about network deleting.

Only if they're presented as LUNs ... /dev/baloney0, /dev/baloney1, etc.

-----Original Message----- From: Greg Whynott [mailto:Greg.Whynott@oicr.on.ca] Sent: Fri 9/4/2009 7:47 AM To: discuss@studiosysadmins.comdiscuss@studiosysadmins.com Subject: Re: Question about network deleting.

you can use baloney sandwiches and make a luster cluster, buster!

8)

-g

On Sep 4, 2009, at 10:33 AM, Andrew Klaassen wrote:

I Wuz Wrong. I emailed the Backblaze guys, and they're actually using a stock kernel and stock firmware. Here's the response:

"Regarding your questions below, we run the Debian backport of 2.6.26, use stock firmware from Silicon Image, and the back planes we actually get from a company called CFY in Taiwan (not Silicon Image.)

"We actually do plan to publish at least a few more blog posts with more data & learnings about the pods. (Follow us on Twitter if you want to be notified when we do...)

"Also, we've received so many requests to buy the storage pods in the last 24 hours despite providing all the specs that we're contemplating where to do this somehow. If people are interested in possibly doing this, they can email salescontact@backblaze.comsalescontact@backblaze.com and we'll keep them abreast of whether we'll do that."

I wonder if you could make a Lustre cluster of these guys, or if there'd be problems with redundancy...

Andrew

--- On Wed, 9/2/09, Andrew Klaassen <clawsoon@yahoo.comclawsoon@yahoo.com> wrote:

From: Andrew Klaassen <clawsoon@yahoo.comclawsoon@yahoo.com> Subject: Re: Question about network deleting. To: discuss@studiosysadmins.comdiscuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:24 AM Oh, and they also thank a couple of engineers at Silicon Image "who helped us debug SATA protocol problems and loaned us 10 different SATA cards to test against".

That's the kind of testing I'm talking about.

What's the chance they're running custom SATA drivers, what with all that SATA protocol debugging? That's the kind of thing that it's hard for a small studio (or even a large one) to do. What's the chance that those drivers are going to make it back into the mainstream kernel? (One can only hope.)

Andrew

--- On Wed, 9/2/09, Andrew Klaassen <clawsoon@yahoo.comclawsoon@yahoo.com> wrote:

From: Andrew Klaassen <clawsoon@yahoo.comclawsoon@yahoo.com> Subject: Re: Question about network deleting. To: discuss@studiosysadmins.comdiscuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:19 AM --- On Tue, 9/1/09, Klaus Steden <klaus-s@moving-picture.comklaus-s@moving-picture.com> wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

I've been thinking about building something like that,

but > I've had a hard time finding good info about port-multiplier > backplanes. The good news is that Backblaze seems to > have a working, well-tested config using port multipliers; > the bad news is that they thank "James Lee at Chyang Fun > Industries in Taiwan for customizing SATA boards to simplify > our design". I probably don't have the buying power > clout to get James Lee - as nice a guy as he might be - to > do that for me. (Although... if they have a few extras > left over from the Backblaze build...) > > Andrew > > > > > > _ > Make your browsing faster, safer, and easier with the new > Internet Explorer? 8. Optimized for Yahoo! Get it Now for > Free! at http://downloads.yahoo.com/ca/internetexplorer/ > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@studiosysadmins.comStudioSysAdmins-Discuss@studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >

Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/ _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.comStudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ The new Internet Explorer? 8 - Faster, safer, easier. Optimized for Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/ __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.comStudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

OK guys... I know it's Friday and a long weekend... But not everyone needs to read about your lunch :) A little off topic...

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Greg Whynott Sent: September 4, 2009 3:16 PM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

/dev/b0b1b1, if you want to pass off the entire device as a snack. 8)

-g

On Sep 4, 2009, at 1:57 PM, Jeff Bell wrote:

Shouldn't it be:

/dev/bread0 /dev/baloney0 /dev/bread1

;)

From: studiosysadmins-discuss-bounces@studiosysadmins.comstudiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of Klaus Steden Sent: Friday, September 04, 2009 7:01 AM To: discuss@studiosysadmins.comdiscuss@studiosysadmins.com; discuss@studiosysadmins.comdiscuss@studiosysadmins.com Subject: RE: Question about network deleting.

Only if they're presented as LUNs ... /dev/baloney0, /dev/baloney1, etc.

-----Original Message----- From: Greg Whynott [mailto:Greg.Whynott@oicr.on.ca] Sent: Fri 9/4/2009 7:47 AM To: discuss@studiosysadmins.comdiscuss@studiosysadmins.com Subject: Re: Question about network deleting.

you can use baloney sandwiches and make a luster cluster, buster!

8)

-g

On Sep 4, 2009, at 10:33 AM, Andrew Klaassen wrote:

I Wuz Wrong. I emailed the Backblaze guys, and they're actually using a stock kernel and stock firmware. Here's the response:

"Regarding your questions below, we run the Debian backport of 2.6.26, use stock firmware from Silicon Image, and the back planes we actually get from a company called CFY in Taiwan (not Silicon Image.)

"We actually do plan to publish at least a few more blog posts with more data & learnings about the pods. (Follow us on Twitter if you want to be notified when we do...)

"Also, we've received so many requests to buy the storage pods in the last 24 hours despite providing all the specs that we're contemplating where to do this somehow. If people are interested in possibly doing this, they can email salescontact@backblaze.comsalescontact@backblaze.com and we'll keep them abreast of whether we'll do that."

I wonder if you could make a Lustre cluster of these guys, or if there'd be problems with redundancy...

Andrew

--- On Wed, 9/2/09, Andrew Klaassen <clawsoon@yahoo.comclawsoon@yahoo.com> wrote:

From: Andrew Klaassen <clawsoon@yahoo.comclawsoon@yahoo.com> Subject: Re: Question about network deleting. To: discuss@studiosysadmins.comdiscuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:24 AM Oh, and they also thank a couple of engineers at Silicon Image "who helped us debug SATA protocol problems and loaned us 10 different SATA cards to test against".

That's the kind of testing I'm talking about.

What's the chance they're running custom SATA drivers, what with all that SATA protocol debugging? That's the kind of thing that it's hard for a small studio (or even a large one) to do. What's the chance that those drivers are going to make it back into the mainstream kernel? (One can only hope.)

Andrew

--- On Wed, 9/2/09, Andrew Klaassen <clawsoon@yahoo.comclawsoon@yahoo.com> wrote:

From: Andrew Klaassen <clawsoon@yahoo.comclawsoon@yahoo.com> Subject: Re: Question about network deleting. To: discuss@studiosysadmins.comdiscuss@studiosysadmins.com Received: Wednesday, September 2, 2009, 7:19 AM --- On Tue, 9/1/09, Klaus Steden <klaus-s@moving-picture.comklaus-s@moving-picture.com> wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

I've been thinking about building something like that,

but > I've had a hard time finding good info about port-multiplier > backplanes. The good news is that Backblaze seems to > have a working, well-tested config using port multipliers; > the bad news is that they thank "James Lee at Chyang Fun > Industries in Taiwan for customizing SATA boards to simplify > our design". I probably don't have the buying power > clout to get James Lee - as nice a guy as he might be - to > do that for me. (Although... if they have a few extras > left over from the Backblaze build...) > > Andrew > > > > > > _ > Make your browsing faster, safer, and easier with the new > Internet Explorer(r) 8. Optimized for Yahoo! Get it Now for > Free! at http://downloads.yahoo.com/ca/internetexplorer/ > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@studiosysadmins.comStudioSysAdmins-Discuss@studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >

Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/ _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.comStudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ The new Internet Explorer(r) 8 - Faster, safer, easier. Optimized for Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/ __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.comStudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss