On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with their > sales-pitch.? Can anyone share their experiences with Isilon products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. >From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry

I graph our bluearcs, disk free, system iops, fs iops, nfs, threwput. With cacti. No issues getting the data.

G

Hi Dave,

What were all the small files for?

Andrew

--- On Mon, 3/8/10, Dave Algar dalgar@rainmaker.com wrote:

Hi Ian,

 

We have both Isilon and Bluearc.  I have heard that a lot of people are quite happy with their Isilon system but our experience didn’t go that well.  I think a lot has to do with your file size.  At the time we had an enormous amount of small files and the Isilon didn’t fare too well.  We had 24 nodes at the time with about 250 to 300 artists.  At the time they did not have a backup node so all we had were 1Gb connections available.  There were no 10Gb nodes at the time so balancing user’s across nodes was another issue.  Troubleshooting problems with a lot of nodes wasn’t a lot of fun.  They do have some nice features like quota’s on a directory or snapshots but it just didn’t handle our load.  We had IQ3000 nodes.

 

Dave

 

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Ian Haskin

Sent: Monday, March 08, 2010 2:01 PM

To: discuss@studiosysadmins.com

Subject: Researching: Isilon and alternatives

 

Hey everyone,

 

I've been looking into Isilon storage and I'm quite impressed with their sales-pitch.  Can anyone share their experiences with Isilon products or any other products that compete/compare with them?

 

 

Thanks,

 

Ian Haskin

TOPIX

This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.

-----Inline Attachment Follows-----

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/

What size is small? Just wondering as from a fs level a meg is considered large usually.
G

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with their > sales-pitch.? Can anyone share their experiences with Isilon products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. >From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with their > sales-pitch. Can anyone share their experiences with Isilon products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

it mostly comes down to budget. at the top end a pair of BA Titans can cost you the price of a condo in downtown toronto. I think to get into the NAS arena with the big names, the price of admission will be at least 50k, it it'll climb rapidly.

-g

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with their > sales-pitch. Can anyone share their experiences with Isilon products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. >From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry

only NetApp of the ones you mentioned. BlueArc is all NAS, as is Isilon.

-g

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with their > sales-pitch. Can anyone share their experiences with Isilon products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

just to tocuh on something you said, you can back up BlueArc over FC via NDMP. I had my back end BlueArc FC switch connected directly to our tape libs and was doing backups via NDMP, totally avoiding the ethernet.

-g

only NetApp of the ones you mentioned. BlueArc is all NAS, as is Isilon.

-g

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with their > sales-pitch. Can anyone share their experiences with Isilon products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

I could trade in my condo and become a homeless guy with a NAS.

Andrew

--- On Mon, 3/8/10, Greg Whynott Greg.Whynott@oicr.on.ca wrote:

it mostly comes down to budget.? at the top end a pair of BA Titans can cost you the price of a condo in downtown toronto.? ? I think to get into the NAS arena with the big names,? the price of admission will be at least 50k,? it it'll climb rapidly.

-g

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Ian Haskin [ian@topixfx.com] Sent: Monday, March 08, 2010 6:18 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS.? I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with their > sales-pitch.? Can anyone share their experiences with Isilon products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

__ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/

at least you would be warm. 8)

-g

I could trade in my condo and become a homeless guy with a NAS.

Andrew

--- On Mon, 3/8/10, Greg Whynott Greg.Whynott@oicr.on.ca wrote:

it mostly comes down to budget. at the top end a pair of BA Titans can cost you the price of a condo in downtown toronto. I think to get into the NAS arena with the big names, the price of admission will be at least 50k, it it'll climb rapidly.

-g

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Ian Haskin [ian@topixfx.com] Sent: Monday, March 08, 2010 6:18 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with their > sales-pitch. Can anyone share their experiences with Isilon products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

__ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/

Right, that's what we are doing now. My big complaint about the setup is there is no way to do a true incremental. Which means you either have do to a full every 7 days to keep the dump sizes down (which for a single LTO4 is impossible to do a full levle 0 dump of a 34 TB volume in less then a week), or after a while the level 1 dump size gets to be so big it takes days to do it anyways.. I am not a big fan of the whole NDMP protocol's dump level design. It's very coarse and not at all tunable. I actually miss the days when I could do an incremental to tape every 4 hours. With NDMP, that's almost completely impossible. Unless I'm missing something in the protocol. If anyone knows of a way to get a true incremental out of NDMP, I would become your biggest fan...

J^2

On Mar 8, 2010, at 3:21 PM, Greg Whynott wrote:

just to tocuh on something you said, you can back up BlueArc over FC via NDMP. I had my back end BlueArc FC switch connected directly to our tape libs and was doing backups via NDMP, totally avoiding the ethernet.

-g

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Greg Whynott [Greg.Whynott@oicr.on.ca] Sent: Monday, March 08, 2010 6:19 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

only NetApp of the ones you mentioned. BlueArc is all NAS, as is Isilon.

-g

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of J. J. Franzen [jjfranzen@mac.com] Sent: Monday, March 08, 2010 6:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with their > sales-pitch. Can anyone share their experiences with Isilon products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

We're a smaller shop with ~50 artists, a 4 node IQ6000 isilon cluster, and have had nothing but good experiences. I found BA to be a little out of our pricerange.

Yes so far we're in the same boat with similar setup and it's been the best server for the price we could afford at the time (2-3 years ago), can't say what we'll do next, depends on the next big show and it's requirements.

I'm interested in the high-end tent. ;D Seriously though, as long as I'm also buying reliability as well as speed and size we have some budget to play with. I'm looking near the entry-level end to be sure. We aren't big enough to justify the higher-tier gear.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Greg Whynott Sent: March-08-10 6:17 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

it mostly comes down to budget. at the top end a pair of BA Titans can cost you the price of a condo in downtown toronto. I think to get into the NAS arena with the big names, the price of admission will be at least 50k, it it'll climb rapidly.

-g

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with their > sales-pitch. Can anyone share their experiences with Isilon products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. >From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry

I'm also very interested in the Mercury 100 as our next possible server in the next year or so.... any smaller VFX shops with this server yet?

We have a pair of them, but they are not part of the pipeline yet so I can't comment on the performance yet.

G

We trialled it late last year though our trial was broader than just NAS. We were trying to figure out if NAS would even do what we wanted it to in our environment. (which it didn't) Unfortunately BA wasn't even in Australia at the time so we didn't get them in on the trial. Incidentally we also couldn't get 10Gb EN to work on Isilon at all. At any rate we settled for Quantum StorNext which is a SAN not NAS. That said, its worth looking at. It gives us huge flexibility over hardware, performance, connection and backup. We've got 4Gb and 8Gb FC clients as well as 1Gb and 10Gb EN clients. Weirdly, EN clients are actually cheaper than SAN ones which means 10Gb is more cost effective. This is spreas across Win, Linux and Mac platforms. It sounds like you want high performance but after looking at the BA and Isilon price structure you're a bit overawed. So were we. Isilon actually HALVED their price for us because they wanted in on the market and it still worked out to be US$70K per 7.5TB of usable space. Farrk! StorNext allowed us to by appropriate performance (and HA reliability) storage for different tasks and has a built in near-line tape and off-line tape system governed by policies. It may not work for you but it certainly worked for us.

Julian

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Ian Haskin Sent: Tuesday, 9 March 2010 9:01 AM To: discuss@studiosysadmins.com Subject: Researching: Isilon and alternatives

Hey everyone,

I've been looking into Isilon storage and I'm quite impressed with their sales-pitch. Can anyone share their experiences with Isilon products or any other products that compete/compare with them?

Thanks,

Ian Haskin

TOPIX

Trial what you can (as others have suggested). NAS (like Isilon) may not be the best way to go. Look at SAN solutions too. SGI CXFS (if its still supported), Quantum StorNext, Compellent, BlueArc...

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Ian Haskin Sent: Tuesday, 9 March 2010 10:19 AM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with their > sales-pitch.? Can anyone share their experiences with Isilon products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins- discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins- discuss

Ian,

For anyone looking at BlueArc or Isilon definitely take a look at Panasas ( http://www.panasas.com ), a parallel, object based, scalable storage system. You can start small and linearly scale as big and fast as you want using a building block approach, all in a true single namespace.

Disclaimer: I liked the technology enough to join the company, but it IS a good solution.

-RW

On Mar 8, 2010, at 4:36 PM, Julian Firminger wrote:

Trial what you can (as others have suggested). NAS (like Isilon) may not be the best way to go. Look at SAN solutions too. SGI CXFS (if its still supported), Quantum StorNext, Compellent, BlueArc...

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Ian Haskin Sent: Tuesday, 9 March 2010 10:19 AM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with their > sales-pitch. Can anyone share their experiences with Isilon products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins- discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins- discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Those are small. 8)

Wow didn't think anything could be that small in vfx/3d.

Thanks for the reply,
G

I think any of the vendors we're talking about here would struggle with a workload like that. It's not because they sell bad servers; it's the nature of platter-based hard drives. Let's do some math with the Fastest Hard Drive In The World (according to the press release), a 15K SCSI drive with a 3.2ms seek time:

1 second / 3.2 milliseconds = 312.5 seeks per second

312.5 seeks per second * 2KB per seek = 625KB/second per hard drive

Damn, that's low. Let's try with 4KB files:

312.5 seeks per second * 4KB per seek = 1250KB/second per hard drive

If you're constantly seeking for 4KB files, that's as fast as a single hard drive (a damn fast hard drive) is going to be able to pump out the data. Now, let's say you want to fill up a 1 GigE network pipe coming out of that server:

1Gb/s = 128MB/s = 131072KB/s

131072 / 1250 = 104.9 hard drives needed

With files that small, you'd need over one hundred 15K hard drives just to fill a single GigE network connection. I've never used Isilon, so I don't know if it's great or if it's shit, but in this case Isilon definitely can't be blamed for poor performance. Any vendor would've struggled with a workload like that.

Andrew

--- On Tue, 3/9/10, Dave Algar dalgar@rainmaker.com wrote:

The entire project used a little over 11TB and had 53 million files for an average of just over 200KB for file size.  I don’t have specifics on how small the files actually got but I do recall seeing a lot of files in the 2 to 4k range.  At the time the decision was made by the pipeline developer to use a lot of small files and bake out a lot of things for quicker load times.  Maybe it worked for him but it didn’t for me.

 

Dave

 

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Greg Whynott

Sent: Monday, March 08, 2010 2:50 PM

To: ' discuss@studiosysadmins.com '

Subject: Re: Researching: Isilon and alternatives

 

What size is small? Just wondering as from a fs level a meg is considered large usually.

G

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com studiosysadmins-discuss-bounces@mailman.studiosysadmins.com

To: discuss@studiosysadmins.com < discuss@studiosysadmins.com >

Sent: Mon Mar 08 17:46:32 2010

Subject: RE: Researching: Isilon and alternatives

Hi Ian,

 

We have both Isilon and Bluearc.  I have heard that a lot of people are quite happy with their Isilon system but our experience didn’t go that well.  I think a lot has to do with your file size.  At the time we had an enormous amount of small files and the Isilon didn’t fare too well.  We had 24 nodes at the time with about 250 to 300 artists.  At the time they did not have a backup node so all we had were 1Gb connections available.  There were no 10Gb nodes at the time so balancing user’s across nodes was another issue.  Troubleshooting problems with a lot of nodes wasn’t a lot of fun.  They do have some nice features like quota’s on a directory or snapshots but it just didn’t handle our load.  We had IQ3000 nodes.

 

Dave

 

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Ian Haskin

Sent: Monday, March 08, 2010 2:01 PM

To: discuss@studiosysadmins.com

Subject: Researching: Isilon and alternatives

 

Hey everyone,

 

I've been looking into Isilon storage and I'm quite impressed with their sales-pitch.  Can anyone share their experiences with Isilon products or any other products that compete/compare with them?

 

 

Thanks,

 

Ian Haskin

TOPIX

 

This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.

 

This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.

-----Inline Attachment Follows-----

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Ask a question on any topic and get answers from real people. Go to Yahoo! Answers and share what you know at http://ca.answers.yahoo.com

While it's true any vendor would struggle a bit with this Isilon only offered sata disks "at that time". Also I had to use nfs for backups as my Netbackup 5.1 was not certified by Isilon for ndmp and indeed it does not work very well. I use it but there are definite problems. I'm going to upgrade but not that I use BA as primary I can do so at my leisure. BA had FC disk, FC connectivity for ndmp and handles small files much better. I like my titan heads and their performance. Our pipeline has changed (thank the gods above) and things are much smoother now but I'm a real fan of ndmp backups for daily backups because of their simplicity and speed.

Dave

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Andrew Klaassen Sent: Tuesday, March 09, 2010 9:49 AM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

I think any of the vendors we're talking about here would struggle with a workload like that. It's not because they sell bad servers; it's the nature of platter-based hard drives. Let's do some math with the Fastest Hard Drive In The World (according to the press release), a 15K SCSI drive with a 3.2ms seek time:

1 second / 3.2 milliseconds = 312.5 seeks per second

312.5 seeks per second * 2KB per seek = 625KB/second per hard drive

Damn, that's low. Let's try with 4KB files:

312.5 seeks per second * 4KB per seek = 1250KB/second per hard drive

If you're constantly seeking for 4KB files, that's as fast as a single hard drive (a damn fast hard drive) is going to be able to pump out the data. Now, let's say you want to fill up a 1 GigE network pipe coming out of that server:

1Gb/s = 128MB/s = 131072KB/s

131072 / 1250 = 104.9 hard drives needed

With files that small, you'd need over one hundred 15K hard drives just to fill a single GigE network connection. I've never used Isilon, so I don't know if it's great or if it's shit, but in this case Isilon definitely can't be blamed for poor performance. Any vendor would've struggled with a workload like that.

Andrew

--- On Tue, 3/9/10, Dave Algar dalgar@rainmaker.com wrote:

The entire project used a little over 11TB and had 53 million files for an average of just over 200KB for file size.? I don't have specifics on how small the files actually got but I do recall seeing a lot of files in the 2 to 4k range.? At the time the decision was made by the pipeline developer to use a lot of small files and bake out a lot of things for quicker load times.? Maybe it worked for him but it didn't for me.

?

Dave

?

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Greg Whynott

Sent: Monday, March 08, 2010 2:50 PM

To: ' discuss@studiosysadmins.com '

Subject: Re: Researching: Isilon and alternatives

?

What size is small? Just wondering as from a fs level a meg is considered large usually.

G

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com studiosysadmins-discuss-bounces@mailman.studiosysadmins.com

To: discuss@studiosysadmins.com < discuss@studiosysadmins.com >

Sent: Mon Mar 08 17:46:32 2010

Subject: RE: Researching: Isilon and alternatives

Hi Ian,

?

We have both Isilon and Bluearc.? I have heard that a lot of people are quite happy with their Isilon system but our experience didn't go that well.? I think a lot has to do with your file size.? At the time we had an enormous amount of small files and the Isilon didn't fare too well.? We had 24 nodes at the time with about 250 to 300 artists.? At the time they did not have a backup node so all we had were 1Gb connections available.? There were no 10Gb nodes at the time so balancing user's across nodes was another issue.? Troubleshooting problems with a lot of nodes wasn't a lot of fun.? They do have some nice features like quota's on a directory or snapshots but it just didn't handle our load.? We had IQ3000 nodes.

?

Dave

?

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Ian Haskin

Sent: Monday, March 08, 2010 2:01 PM

To: discuss@studiosysadmins.com

Subject: Researching: Isilon and alternatives

?

Hey everyone,

?

I've been looking into Isilon storage and I'm quite impressed with their sales-pitch.? Can anyone share their experiences with Isilon products or any other products that compete/compare with them?

?

?

Thanks,

?

Ian Haskin

TOPIX

?

This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.

?

This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.

-----Inline Attachment Follows-----

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Ask a question on any topic and get answers from real people. Go to Yahoo! Answers and share what you know at http://ca.answers.yahoo.com

-- This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.

I bet if you threw your old workflow at the Bluearc it'd struggle mightily, too... :-)

Andrew

--- On Tue, 3/9/10, Dave Algar dalgar@rainmaker.com wrote:

While it's true any vendor would struggle a bit with this Isilon only offered sata disks "at that time".? Also I had to use nfs for backups as my Netbackup 5.1 was not certified by Isilon for ndmp and indeed it does not work very well.? I use it but there are definite problems.? I'm going to upgrade but not that I use BA as primary I can do so at my leisure.? BA had FC disk, FC connectivity for ndmp and handles small files much better.? I like my titan heads and their performance.? Our pipeline has changed (thank the gods above) and things are much smoother now but I'm a real fan of ndmp backups for daily backups because of their simplicity and speed.?

Dave

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Andrew Klaassen Sent: Tuesday, March 09, 2010 9:49 AM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

I think any of the vendors we're talking about here would struggle with a workload like that.? It's not because they sell bad servers; it's the nature of platter-based hard drives.? Let's do some math with the Fastest Hard Drive In The World (according to the press release), a 15K SCSI drive with a 3.2ms seek time:

1 second / 3.2 milliseconds = 312.5 seeks per second

312.5 seeks per second * 2KB per seek = 625KB/second per hard drive

Damn, that's low.? Let's try with 4KB files:

312.5 seeks per second * 4KB per seek = 1250KB/second per hard drive

If you're constantly seeking for 4KB files, that's as fast as a single hard drive (a damn fast hard drive) is going to be able to pump out the data.? Now, let's say you want to fill up a 1 GigE network pipe coming out of that server:

1Gb/s = 128MB/s = 131072KB/s

131072 / 1250 = 104.9 hard drives needed

With files that small, you'd need over one hundred 15K hard drives just to fill a single GigE network connection.? I've never used Isilon, so I don't know if it's great or if it's shit, but in this case Isilon definitely can't be blamed for poor performance.? Any vendor would've struggled with a workload like that.

Andrew

--- On Tue, 3/9/10, Dave Algar dalgar@rainmaker.com wrote:

The entire project used a little over 11TB and had 53 million files for an average of just over

200KB > for file size.? > I don't have specifics on how small the files > actually got but I do > recall seeing a lot of files in the 2 to 4k range.? At > the time the > decision was made by the pipeline developer to use a lot of > small files and > bake out a lot of things for quicker load times.? > Maybe it worked for him > but it didn't for me.? > > > ? > > Dave > > > > ? > > > > > > > > > > From: > studiosysadmins-discuss-bounces@mailman.studiosysadmins.com > [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] > On Behalf Of Greg > Whynott > > Sent: Monday, > March 08, 2010 2:50 > PM > > To: ' > discuss@studiosysadmins.com ' > > Subject: Re: > Researching: Isilon > and alternatives > > > >? ? > > What size is small? Just > wondering as from a fs level a meg > is considered large usually. > > G > > > > > > > > From: > studiosysadmins-discuss-bounces@mailman.studiosysadmins.com > studiosysadmins-discuss-bounces@mailman.studiosysadmins.com > > > To: > discuss@studiosysadmins.com

< discuss@studiosysadmins.com

Sent: Mon Mar 08 17:46:32 2010

Subject: RE: Researching: Isilon and alternatives?

Hi Ian,

?

We have both Isilon and Bluearc.? I have heard that a lot of people are quite happy with

their > Isilon system but > our experience didn't go that well.? I think a > lot has to do with > your file size.? At the time we had an enormous amount > of small files and > the Isilon didn't fare too well.? We had 24 > nodes at the time with > about 250 to 300 artists.? At the time they did not > have a backup node so > all we had were 1Gb connections available.? There were > no 10Gb nodes at > the time so balancing user's across nodes was another > issue.? > Troubleshooting problems with a lot of nodes wasn't a > lot of fun.? > They do have some nice features like quota's on a > directory or snapshots > but it just didn't handle our load.? We had > IQ3000 nodes. > > > ? > > Dave > > > > ? > > > > > > > > > > From: > studiosysadmins-discuss-bounces@mailman.studiosysadmins.com > [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] > On Behalf Of Ian > Haskin > > Sent: Monday, > March 08, 2010 2:01 > PM > > To: > discuss@studiosysadmins.com > > Subject: > Researching: Isilon and > alternatives > > > >? ? > > Hey everyone, > >? ? > > I've been looking into Isilon > storage and I'm quite impressed with > their sales-pitch.? Can anyone share their experiences > with Isilon products > or any other products that compete/compare with > them? > >? ? > >? ? > > Thanks, > >? ? > > Ian Haskin > > TOPIX > > > ? > > > > > > > > This > e-mail and any attachments are > intended only for use by the addressee(s) named herein and > may contain > confidential information. If you are not the intended > recipient of this e-mail, > you are hereby notified any dissemination, distribution or > copying of this > email and any attachments is strictly prohibited. If you > receive this email in > error, please immediately notify the sender by return email > and permanently > delete the original, any copy and any printout thereof. The > integrity and > security of e-mail cannot be guaranteed. > > > ? > > > > > > > > > This e-mail and any attachments are intended only for use > by the addressee(s) named herein and may contain > confidential information.? If you are not the intended > recipient of this e-mail, you are hereby notified any > dissemination, distribution or copying of this email and any > attachments is strictly prohibited.? If you receive this > email in error, please immediately notify the sender by > return email and permanently delete the original, any copy > and any printout thereof.? The integrity and security of > e-mail cannot be guaranteed. > > >? > > > > -----Inline Attachment Follows----- > > _ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@mailman.studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >

? ? ? Ask a question on any topic and get answers from real people. Go to Yahoo! Answers and share what you know at http://ca.answers.yahoo.com

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

-- This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information.? If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited.? If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof.? The integrity and security of e-mail cannot be guaranteed.

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/

That is a pretty crazy workload, but not unheard of. The Bio/Genome
Sequencing guys love their tiny files. I typically see an average
filesize of <100K in that space. This workload basically looks like a
transactional database and probably could be better served if treated
as such.

Cutting edge storage systems use a hybrid approach combining SSD, lots
of cache RAM and spinning disk to attempt to give decent performance
for this type of workload, but it is still a challenge in handling the
requests and mapping the files to the appropriate tier.

-RW

On Mar 9, 2010, at 10:17 AM, Andrew Klaassen wrote:

I bet if you threw your old workflow at the Bluearc it'd struggle
mightily, too... :-)

Andrew

--- On Tue, 3/9/10, Dave Algar dalgar@rainmaker.com wrote:

While it's true any vendor would struggle a bit with this Isilon only offered sata disks "at that time". Also I had to use nfs for backups as my Netbackup 5.1 was not certified by Isilon for ndmp and indeed it does not work very well. I use it but there are definite problems. I'm going to upgrade but not that I use BA as primary I can do so at my leisure. BA had FC disk, FC connectivity for ndmp and handles small files much better. I like my titan heads and their performance. Our pipeline has changed (thank the gods above) and things are much smoother now but I'm a real fan of ndmp backups for daily backups because of their simplicity and speed.

Dave

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Andrew Klaassen Sent: Tuesday, March 09, 2010 9:49 AM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

I think any of the vendors we're talking about here would struggle with a workload like that. It's not because they sell bad servers; it's the nature of platter-based hard drives. Let's do some math with the Fastest Hard Drive In The World (according to the press release), a 15K SCSI drive with a 3.2ms seek time:

1 second / 3.2 milliseconds = 312.5 seeks per second

312.5 seeks per second * 2KB per seek = 625KB/second per hard drive

Damn, that's low. Let's try with 4KB files:

312.5 seeks per second * 4KB per seek = 1250KB/second per hard drive

If you're constantly seeking for 4KB files, that's as fast as a single hard drive (a damn fast hard drive) is going to be able to pump out the data. Now, let's say you want to fill up a 1 GigE network pipe coming out of that server:

1Gb/s = 128MB/s = 131072KB/s

131072 / 1250 = 104.9 hard drives needed

With files that small, you'd need over one hundred 15K hard drives just to fill a single GigE network connection. I've never used Isilon, so I don't know if it's great or if it's shit, but in this case Isilon definitely can't be blamed for poor performance. Any vendor would've struggled with a workload like that.

Andrew

--- On Tue, 3/9/10, Dave Algar dalgar@rainmaker.com wrote:

The entire project used a little over 11TB and had 53 million files for an average of just over

200KB > for file size. > I don't have specifics on how small the files > actually got but I do > recall seeing a lot of files in the 2 to 4k range. At > the time the > decision was made by the pipeline developer to use a lot of > small files and > bake out a lot of things for quicker load times. > Maybe it worked for him > but it didn't for me. > > > > > Dave > > > > > > > > > > > > > > From: > studiosysadmins-discuss-bounces@mailman.studiosysadmins.com > [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] > On Behalf Of Greg > Whynott > > Sent: Monday, > March 08, 2010 2:50 > PM > > To: ' > discuss@studiosysadmins.com ' > > Subject: Re: > Researching: Isilon > and alternatives > > > > > > What size is small? Just > wondering as from a fs level a meg > is considered large usually. > > G > > > > > > > > From: > studiosysadmins-discuss-bounces@mailman.studiosysadmins.com > studiosysadmins-discuss-bounces@mailman.studiosysadmins.com > > > To: > discuss@studiosysadmins.com

< discuss@studiosysadmins.com

Sent: Mon Mar 08 17:46:32 2010

Subject: RE: Researching: Isilon and alternatives

Hi Ian,

We have both Isilon and Bluearc. I have heard that a lot of people are quite happy with

their > Isilon system but > our experience didn't go that well. I think a > lot has to do with > your file size. At the time we had an enormous amount > of small files and > the Isilon didn't fare too well. We had 24 > nodes at the time with > about 250 to 300 artists. At the time they did not > have a backup node so > all we had were 1Gb connections available. There were > no 10Gb nodes at > the time so balancing user's across nodes was another > issue. > Troubleshooting problems with a lot of nodes wasn't a > lot of fun. > They do have some nice features like quota's on a > directory or snapshots > but it just didn't handle our load. We had > IQ3000 nodes. > > > > > Dave > > > > > > > > > > > > > > From: > studiosysadmins-discuss-bounces@mailman.studiosysadmins.com > [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] > On Behalf Of Ian > Haskin > > Sent: Monday, > March 08, 2010 2:01 > PM > > To: > discuss@studiosysadmins.com > > Subject: > Researching: Isilon and > alternatives > > > > > > Hey everyone, > > > > I've been looking into Isilon > storage and I'm quite impressed with > their sales-pitch. Can anyone share their experiences > with Isilon products > or any other products that compete/compare with > them? > > > > > > Thanks, > > > > Ian Haskin > > TOPIX > > > > > > > > > > > This > e-mail and any attachments are > intended only for use by the addressee(s) named herein and > may contain > confidential information. If you are not the intended > recipient of this e-mail, > you are hereby notified any dissemination, distribution or > copying of this > email and any attachments is strictly prohibited. If you > receive this email in > error, please immediately notify the sender by return email > and permanently > delete the original, any copy and any printout thereof. The > integrity and > security of e-mail cannot be guaranteed. > > > > > > > > > > > > This e-mail and any attachments are intended only for use > by the addressee(s) named herein and may contain > confidential information. If you are not the intended > recipient of this e-mail, you are hereby notified any > dissemination, distribution or copying of this email and any > attachments is strictly prohibited. If you receive this > email in error, please immediately notify the sender by > return email and permanently delete the original, any copy > and any printout thereof. The integrity and security of > e-mail cannot be guaranteed. > > > > > > > -----Inline Attachment Follows----- > > _ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@mailman.studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >

Ask a question on any topic and get answers from real people. Go to Yahoo! Answers and share what you know at http://ca.answers.yahoo.com

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

-- This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

  <br />

Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

If you truly need small file multi-threaded performance talk with someone about the HP X9000 series.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Andrew Klaassen Sent: Tuesday, March 09, 2010 12:49 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

I think any of the vendors we're talking about here would struggle with a workload like that. It's not because they sell bad servers; it's the nature of platter-based hard drives. Let's do some math with the Fastest Hard Drive In The World (according to the press release), a 15K SCSI drive with a 3.2ms seek time:

1 second / 3.2 milliseconds = 312.5 seeks per second

312.5 seeks per second * 2KB per seek = 625KB/second per hard drive

Damn, that's low. Let's try with 4KB files:

312.5 seeks per second * 4KB per seek = 1250KB/second per hard drive

If you're constantly seeking for 4KB files, that's as fast as a single hard drive (a damn fast hard drive) is going to be able to pump out the data. Now, let's say you want to fill up a 1 GigE network pipe coming out of that server:

1Gb/s = 128MB/s = 131072KB/s

131072 / 1250 = 104.9 hard drives needed

With files that small, you'd need over one hundred 15K hard drives just to fill a single GigE network connection. I've never used Isilon, so I don't know if it's great or if it's shit, but in this case Isilon definitely can't be blamed for poor performance. Any vendor would've struggled with a workload like that.

Andrew

--- On Tue, 3/9/10, Dave Algar dalgar@rainmaker.com wrote:

The entire project used a little over 11TB and had 53 million files for an average of just over 200KB for file size.  I don’t have specifics on how small the files actually got but I do recall seeing a lot of files in the 2 to 4k range.  At the time the decision was made by the pipeline developer to use a lot of small files and bake out a lot of things for quicker load times.  Maybe it worked for him but it didn’t for me.

 

Dave

 

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Greg Whynott

Sent: Monday, March 08, 2010 2:50 PM

To: ' discuss@studiosysadmins.com '

Subject: Re: Researching: Isilon and alternatives

 

What size is small? Just wondering as from a fs level a meg is considered large usually.

G

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com studiosysadmins-discuss-bounces@mailman.studiosysadmins.com

To: discuss@studiosysadmins.com < discuss@studiosysadmins.com >

Sent: Mon Mar 08 17:46:32 2010

Subject: RE: Researching: Isilon and alternatives

Hi Ian,

 

We have both Isilon and Bluearc.  I have heard that a lot of people are quite happy with their Isilon system but our experience didn’t go that well.  I think a lot has to do with your file size.  At the time we had an enormous amount of small files and the Isilon didn’t fare too well.  We had 24 nodes at the time with about 250 to 300 artists.  At the time they did not have a backup node so all we had were 1Gb connections available.  There were no 10Gb nodes at the time so balancing user’s across nodes was another issue.  Troubleshooting problems with a lot of nodes wasn’t a lot of fun.  They do have some nice features like quota’s on a directory or snapshots but it just didn’t handle our load.  We had IQ3000 nodes.

 

Dave

 

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Ian Haskin

Sent: Monday, March 08, 2010 2:01 PM

To: discuss@studiosysadmins.com

Subject: Researching: Isilon and alternatives

 

Hey everyone,

 

I've been looking into Isilon storage and I'm quite impressed with their sales-pitch.  Can anyone share their experiences with Isilon products or any other products that compete/compare with them?

 

 

Thanks,

 

Ian Haskin

TOPIX

 

This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.

 

This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.

-----Inline Attachment Follows-----

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Ask a question on any topic and get answers from real people. Go to Yahoo! Answers and share what you know at http://ca.answers.yahoo.com

Dr. D Studios evaluated IBRIX roughly a year ago (along with Isilon and BlueArc). IBRIX faired well in testing (the Dell/EMC/IBRIX provided system had an EMC CLARiiON CX4-960 with half a tray of solid state disk for IBRIX file-system meta-data, and five Dell PE2950s each with 32GB of RAM)

IBRIX was subsequently purchased by HP and turned into X9000, and I'd have to agree with Phil that X9000 would is definitely worth a place in any evaluation round-up, but ultimately Dr. D chose BlueArc because we decided that it had:

• a lower level of complexity,
• a higher density of performance
• a lower cost to run and manage

...than the Dell/EMC/IBRIX solution

On Wed, Mar 10, 2010 at 5:58 AM, Haskell, Phil T phil.haskell@hp.com wrote: > If you truly need small file multi-threaded performance talk with someone about the HP X9000 series. > > -----Original Message----- > From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Andrew Klaassen > Sent: Tuesday, March 09, 2010 12:49 PM > To: discuss@studiosysadmins.com > Subject: RE: Researching: Isilon and alternatives > > I think any of the vendors we're talking about here would struggle with a workload like that. ?It's not because they sell bad servers; it's the nature of platter-based hard drives. ?Let's do some math with the Fastest Hard Drive In The World (according to the press release), a 15K SCSI drive with a 3.2ms seek time: > > 1 second / 3.2 milliseconds = 312.5 seeks per second > > 312.5 seeks per second * 2KB per seek = 625KB/second per hard drive > > Damn, that's low. ?Let's try with 4KB files: > > 312.5 seeks per second * 4KB per seek = 1250KB/second per hard drive > > If you're constantly seeking for 4KB files, that's as fast as a single hard drive (a damn fast hard drive) is going to be able to pump out the data. ?Now, let's say you want to fill up a 1 GigE network pipe coming out of that server: > > 1Gb/s = 128MB/s = 131072KB/s > > 131072 / 1250 = 104.9 hard drives needed > > With files that small, you'd need over one hundred 15K hard drives just to fill a single GigE network connection. ?I've never used Isilon, so I don't know if it's great or if it's shit, but in this case Isilon definitely can't be blamed for poor performance. ?Any vendor would've struggled with a workload like that. > > Andrew > > > --- On Tue, 3/9/10, Dave Algar dalgar@rainmaker.com wrote: > >> The entire >> project used a little over 11TB >> and had 53 million files for an average of just over 200KB >> for file size. >> I don?t have specifics on how small the files >> actually got but I do >> recall seeing a lot of files in the 2 to 4k range.? At >> the time the >> decision was made by the pipeline developer to use a lot of >> small files and >> bake out a lot of things for quicker load times. >> Maybe it worked for him >> but it didn?t for me. >> >> >> >> >> Dave >> >> >> >> >> >> >> >> >> >> >> >> >> >> From: >> studiosysadmins-discuss-bounces@mailman.studiosysadmins.com >> [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] >> On Behalf Of Greg >> Whynott >> >> Sent: Monday, >> March 08, 2010 2:50 >> PM >> >> To: ' >> discuss@studiosysadmins.com ' >> >> Subject: Re: >> Researching: Isilon >> and alternatives >> >> >> >> >> >> What size is small? Just >> wondering as from a fs level a meg >> is considered large usually. >> >> G >> >> >> >> >> >> >> >> From: >> studiosysadmins-discuss-bounces@mailman.studiosysadmins.com >> studiosysadmins-discuss-bounces@mailman.studiosysadmins.com >> >> >> To: >> discuss@studiosysadmins.com >> < discuss@studiosysadmins.com > >> >> Sent: Mon Mar >> 08 17:46:32 2010 >> >> Subject: RE: >> Researching: Isilon >> and alternatives >> >> Hi >> Ian, >> >> >> >> >> We have >> both Isilon and Bluearc.? I >> have heard that a lot of people are quite happy with their >> Isilon system but >> our experience didn?t go that well.? I think a >> lot has to do with >> your file size.? At the time we had an enormous amount >> of small files and >> the Isilon didn?t fare too well.? We had 24 >> nodes at the time with >> about 250 to 300 artists.? At the time they did not >> have a backup node so >> all we had were 1Gb connections available.? There were >> no 10Gb nodes at >> the time so balancing user?s across nodes was another >> issue. >> Troubleshooting problems with a lot of nodes wasn?t a >> lot of fun. >> They do have some nice features like quota?s on a >> directory or snapshots >> but it just didn?t handle our load.? We had >> IQ3000 nodes. >> >> >> >> >> Dave >> >> >> >> >> >> >> >> >> >> >> >> >> >> From: >> studiosysadmins-discuss-bounces@mailman.studiosysadmins.com >> [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] >> On Behalf Of Ian >> Haskin >> >> Sent: Monday, >> March 08, 2010 2:01 >> PM >> >> To: >> discuss@studiosysadmins.com >> >> Subject: >> Researching: Isilon and >> alternatives >> >> >> >> >> >> Hey everyone, >> >> >> >> I've been looking into Isilon >> storage and I'm quite impressed with >> their sales-pitch.? Can anyone share their experiences >> with Isilon products >> or any other products that compete/compare with >> them? >> >> >> >> >> >> Thanks, >> >> >> >> Ian Haskin >> >> TOPIX >> >> >> >> >> >> >> >> >> >> >> This >> e-mail and any attachments are >> intended only for use by the addressee(s) named herein and >> may contain >> confidential information. If you are not the intended >> recipient of this e-mail, >> you are hereby notified any dissemination, distribution or >> copying of this >> email and any attachments is strictly prohibited. If you >> receive this email in >> error, please immediately notify the sender by return email >> and permanently >> delete the original, any copy and any printout thereof. The >> integrity and >> security of e-mail cannot be guaranteed. >> >> >> >> >> >> >> >> >> >> >> >> This e-mail and any attachments are intended only for use >> by the addressee(s) named herein and may contain >> confidential information. ?If you are not the intended >> recipient of this e-mail, you are hereby notified any >> dissemination, distribution or copying of this email and any >> attachments is strictly prohibited. ?If you receive this >> email in error, please immediately notify the sender by >> return email and permanently delete the original, any copy >> and any printout thereof. ?The integrity and security of >> e-mail cannot be guaranteed. >> >> >> >> >> >> >> -----Inline Attachment Follows----- >> >> _ >> StudioSysAdmins-Discuss mailing list >> StudioSysAdmins-Discuss@mailman.studiosysadmins.com >> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >> > > > ? ? ? > Ask a question on any topic and get answers from real people. Go to Yahoo! Answers and share what you know at http://ca.answers.yahoo.com > > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@mailman.studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@mailman.studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >

I would consider this experimental, and you might have to get hacky
with how your backup software starts up an NDMP session on the BlueArc
to do it, but it looks like the BlueArc NDMP server supports a 'level
10' or 'level I' (letter "eye") backup, which sounds like what you're
asking. (The ADC replication system for replicating a filesystem makes
use of this level, and we use it for that purpose.)

For us, level 10 will: * take a snapshot send all blocks changed between the previous snapshot and the just- taken snapshot delete the previous snapshot

so it effectively functions as a rolling incremental backup/ replication level.

On the SMU, look at the text file /usr/local/bin/adc.readme, starting
with the section: "NDMP Environment Variables for BlueArc Servers"; also in Appendix B
of the PDF admin guide.

Not sure if this will work just as well to tape; if you try it out,
test the heck out of it and make sure you can restore files properly
after the maximum number of incrementals-between-fulls that you plan
to use. You should run the setup by the BlueArc support folks, too,
just to make extra sure, but this sounds like it may be what you're
talking about.

Also, YMMV, protection of your data is your own responsibility, and be
very careful and test restores before relying on this for data
protection (unless BlueArc endorses this method, but test backups/ restores even if they do endorse it.).

I think this is a BlueArc specific feature, though; not sure if other
manufacturers would support it.

-Sven

On Mar 8, 2010, at 3:31 PM, J. J. Franzen wrote:

Right, that's what we are doing now. My big complaint about the
setup is there is no way to do a true incremental. Which means you
either have do to a full every 7 days to keep the dump sizes down
(which for a single LTO4 is impossible to do a full levle 0 dump of
a 34 TB volume in less then a week), or after a while the level 1
dump size gets to be so big it takes days to do it anyways.. I am
not a big fan of the whole NDMP protocol's dump level design. It's
very coarse and not at all tunable. I actually miss the days when I
could do an incremental to tape every 4 hours. With NDMP, that's
almost completely impossible. Unless I'm missing something in the
protocol. If anyone knows of a way to get a true incremental out of
NDMP, I would become your biggest fan...

J^2

On Mar 8, 2010, at 3:21 PM, Greg Whynott wrote:

just to tocuh on something you said, you can back up BlueArc
over FC via NDMP. I had my back end BlueArc FC switch connected
directly to our tape libs and was doing backups via NDMP, totally
avoiding the ethernet.

-g

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [studiosysadmins-discuss-bounces@mailman.studiosysadmins.com ] On Behalf Of Greg Whynott [Greg.Whynott@oicr.on.ca] Sent: Monday, March 08, 2010 6:19 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

only NetApp of the ones you mentioned. BlueArc is all NAS, as is
Isilon.

-g

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [studiosysadmins-discuss-bounces@mailman.studiosysadmins.com ] On Behalf Of J. J. Franzen [jjfranzen@mac.com] Sent: Monday, March 08, 2010 6:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

Actually, do any of the big guys (Netapp, Isilon, etc.) allow
direct FC access to their volumes? I know Blue Arc does not and am
curious if anyone else does. I would so love to do away with NDMP
for my backups and be able to do true incremental backups like in
the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with
50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux
NAS. I'm looking for something scalable with high-bandwidth links for our
compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss- bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed
> with their > sales-pitch. Can anyone share their experiences with Isilon
> products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating
10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster
( 4 nodes ) quite happily. I think when you get up to the number of
nodes ( and accelerator nodes ) you need for a large cluster, some
problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

I don't want to press the edge on this blog (I respect the independence) but as an Ibrix Alumni am delighted with the simplification, density and pricing that being part of HP has enabled - and the performance is still there and increasing. (Dr. D's loss was a personal - very personal - disappointment)). Thanks for your tolerance.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Kim Pearce Sent: Tuesday, March 09, 2010 5:20 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

Dr. D Studios evaluated IBRIX roughly a year ago (along with Isilon and BlueArc). IBRIX faired well in testing (the Dell/EMC/IBRIX provided system had an EMC CLARiiON CX4-960 with half a tray of solid state disk for IBRIX file-system meta-data, and five Dell PE2950s each with 32GB of RAM)

IBRIX was subsequently purchased by HP and turned into X9000, and I'd have to agree with Phil that X9000 would is definitely worth a place in any evaluation round-up, but ultimately Dr. D chose BlueArc because we decided that it had:

• a lower level of complexity,
• a higher density of performance
• a lower cost to run and manage

...than the Dell/EMC/IBRIX solution

On Wed, Mar 10, 2010 at 5:58 AM, Haskell, Phil T phil.haskell@hp.com wrote: > If you truly need small file multi-threaded performance talk with someone about the HP X9000 series. > > -----Original Message----- > From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Andrew Klaassen > Sent: Tuesday, March 09, 2010 12:49 PM > To: discuss@studiosysadmins.com > Subject: RE: Researching: Isilon and alternatives > > I think any of the vendors we're talking about here would struggle with a workload like that. ?It's not because they sell bad servers; it's the nature of platter-based hard drives. ?Let's do some math with the Fastest Hard Drive In The World (according to the press release), a 15K SCSI drive with a 3.2ms seek time: > > 1 second / 3.2 milliseconds = 312.5 seeks per second > > 312.5 seeks per second * 2KB per seek = 625KB/second per hard drive > > Damn, that's low. ?Let's try with 4KB files: > > 312.5 seeks per second * 4KB per seek = 1250KB/second per hard drive > > If you're constantly seeking for 4KB files, that's as fast as a single hard drive (a damn fast hard drive) is going to be able to pump out the data. ?Now, let's say you want to fill up a 1 GigE network pipe coming out of that server: > > 1Gb/s = 128MB/s = 131072KB/s > > 131072 / 1250 = 104.9 hard drives needed > > With files that small, you'd need over one hundred 15K hard drives just to fill a single GigE network connection. ?I've never used Isilon, so I don't know if it's great or if it's shit, but in this case Isilon definitely can't be blamed for poor performance. ?Any vendor would've struggled with a workload like that. > > Andrew > > > --- On Tue, 3/9/10, Dave Algar dalgar@rainmaker.com wrote: > >> The entire >> project used a little over 11TB >> and had 53 million files for an average of just over 200KB >> for file size. >> I don't have specifics on how small the files >> actually got but I do >> recall seeing a lot of files in the 2 to 4k range.? At >> the time the >> decision was made by the pipeline developer to use a lot of >> small files and >> bake out a lot of things for quicker load times. >> Maybe it worked for him >> but it didn't for me. >> >> >> >> >> Dave >> >> >> >> >> >> >> >> >> >> >> >> >> >> From: >> studiosysadmins-discuss-bounces@mailman.studiosysadmins.com >> [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] >> On Behalf Of Greg >> Whynott >> >> Sent: Monday, >> March 08, 2010 2:50 >> PM >> >> To: ' >> discuss@studiosysadmins.com ' >> >> Subject: Re: >> Researching: Isilon >> and alternatives >> >> >> >> >> >> What size is small? Just >> wondering as from a fs level a meg >> is considered large usually. >> >> G >> >> >> >> >> >> >> >> From: >> studiosysadmins-discuss-bounces@mailman.studiosysadmins.com >> studiosysadmins-discuss-bounces@mailman.studiosysadmins.com >> >> >> To: >> discuss@studiosysadmins.com >> < discuss@studiosysadmins.com > >> >> Sent: Mon Mar >> 08 17:46:32 2010 >> >> Subject: RE: >> Researching: Isilon >> and alternatives >> >> Hi >> Ian, >> >> >> >> >> We have >> both Isilon and Bluearc.? I >> have heard that a lot of people are quite happy with their >> Isilon system but >> our experience didn't go that well.? I think a >> lot has to do with >> your file size.? At the time we had an enormous amount >> of small files and >> the Isilon didn't fare too well.? We had 24 >> nodes at the time with >> about 250 to 300 artists.? At the time they did not >> have a backup node so >> all we had were 1Gb connections available.? There were >> no 10Gb nodes at >> the time so balancing user's across nodes was another >> issue. >> Troubleshooting problems with a lot of nodes wasn't a >> lot of fun. >> They do have some nice features like quota's on a >> directory or snapshots >> but it just didn't handle our load.? We had >> IQ3000 nodes. >> >> >> >> >> Dave >> >> >> >> >> >> >> >> >> >> >> >> >> >> From: >> studiosysadmins-discuss-bounces@mailman.studiosysadmins.com >> [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] >> On Behalf Of Ian >> Haskin >> >> Sent: Monday, >> March 08, 2010 2:01 >> PM >> >> To: >> discuss@studiosysadmins.com >> >> Subject: >> Researching: Isilon and >> alternatives >> >> >> >> >> >> Hey everyone, >> >> >> >> I've been looking into Isilon >> storage and I'm quite impressed with >> their sales-pitch.? Can anyone share their experiences >> with Isilon products >> or any other products that compete/compare with >> them? >> >> >> >> >> >> Thanks, >> >> >> >> Ian Haskin >> >> TOPIX >> >> >> >> >> >> >> >> >> >> >> This >> e-mail and any attachments are >> intended only for use by the addressee(s) named herein and >> may contain >> confidential information. If you are not the intended >> recipient of this e-mail, >> you are hereby notified any dissemination, distribution or >> copying of this >> email and any attachments is strictly prohibited. If you >> receive this email in >> error, please immediately notify the sender by return email >> and permanently >> delete the original, any copy and any printout thereof. The >> integrity and >> security of e-mail cannot be guaranteed. >> >> >> >> >> >> >> >> >> >> >> >> This e-mail and any attachments are intended only for use >> by the addressee(s) named herein and may contain >> confidential information. ?If you are not the intended >> recipient of this e-mail, you are hereby notified any >> dissemination, distribution or copying of this email and any >> attachments is strictly prohibited. ?If you receive this >> email in error, please immediately notify the sender by >> return email and permanently delete the original, any copy >> and any printout thereof. ?The integrity and security of >> e-mail cannot be guaranteed. >> >> >> >> >> >> >> -----Inline Attachment Follows----- >> >> _ >> StudioSysAdmins-Discuss mailing list >> StudioSysAdmins-Discuss@mailman.studiosysadmins.com >> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >> > > > ? ? ? > Ask a question on any topic and get answers from real people. Go to Yahoo! Answers and share what you know at http://ca.answers.yahoo.com > > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@mailman.studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@mailman.studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >

Heterogeneous file system with SSD disks would solve the issue.
http://www.ramsan.com/?gclid=CPzbvZjlrKACFYEwpAod-naN3g Set up multiple tiers of disk: Tier1 - SSD Tier2 - HDD Tier3 - LTO Use data migration to truncate mature data to slower disks.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Andrew Klaassen Sent: Wednesday, 10 March 2010 4:49 AM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

I think any of the vendors we're talking about here would struggle with a workload like that. It's not because they sell bad servers; it's the nature of platter-based hard drives. Let's do some math with the Fastest Hard Drive In The World (according to the press release), a 15K SCSI drive with a 3.2ms seek time:

1 second / 3.2 milliseconds = 312.5 seeks per second

312.5 seeks per second * 2KB per seek = 625KB/second per hard drive

Damn, that's low. Let's try with 4KB files:

312.5 seeks per second * 4KB per seek = 1250KB/second per hard drive

If you're constantly seeking for 4KB files, that's as fast as a single hard drive (a damn fast hard drive) is going to be able to pump out the data. Now, let's say you want to fill up a 1 GigE network pipe coming out of that server:

1Gb/s = 128MB/s = 131072KB/s

131072 / 1250 = 104.9 hard drives needed

With files that small, you'd need over one hundred 15K hard drives just to fill a single GigE network connection. I've never used Isilon, so I don't know if it's great or if it's shit, but in this case Isilon definitely can't be blamed for poor performance. Any vendor would've struggled with a workload like that.

Andrew

--- On Tue, 3/9/10, Dave Algar dalgar@rainmaker.com wrote:

The entire project used a little over 11TB and had 53 million files for an average of just over 200KB for file size. I don’t have specifics on how small the files actually got but I do recall seeing a lot of files in the 2 to 4k range. At the time the decision was made by the pipeline developer to use a lot of small files and bake out a lot of things for quicker load times. Maybe it worked for him but it didn’t for me.

Dave

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Greg Whynott

Sent: Monday, March 08, 2010 2:50 PM

To: ' discuss@studiosysadmins.com '

Subject: Re: Researching: Isilon and alternatives

What size is small? Just wondering as from a fs level a meg is considered large usually.

G

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com studiosysadmins-discuss-bounces@mailman.studiosysadmins.com

To: discuss@studiosysadmins.com < discuss@studiosysadmins.com >

Sent: Mon Mar 08 17:46:32 2010

Subject: RE: Researching: Isilon and alternatives

Hi Ian,

We have both Isilon and Bluearc. I have heard that a lot of people are quite happy with their Isilon system but our experience didn’t go that well. I think a lot has to do with your file size. At the time we had an enormous amount of small files and the Isilon didn’t fare too well. We had 24 nodes at the time with about 250 to 300 artists. At the time they did not have a backup node so all we had were 1Gb connections available. There were no 10Gb nodes at the time so balancing user’s across nodes was another issue. Troubleshooting problems with a lot of nodes wasn’t a lot of fun. They do have some nice features like quota’s on a directory or snapshots but it just didn’t handle our load. We had IQ3000 nodes.

Dave

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Ian Haskin

Sent: Monday, March 08, 2010 2:01 PM

To: discuss@studiosysadmins.com

Subject: Researching: Isilon and alternatives

Hey everyone,

I've been looking into Isilon storage and I'm quite impressed with their sales-pitch. Can anyone share their experiences with Isilon products or any other products that compete/compare with them?

Thanks,

Ian Haskin

TOPIX

This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.

This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.

-----Inline Attachment Follows-----

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-

discuss >

Ask a question on any topic and get answers from real people. Go to Yahoo! Answers and share what you know at http://ca.answers.yahoo.com

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins- discuss

This is one thing I've been asked for a few times; the problem is that
NAS systems have their own filesystem on the NAS head, then present
the files of that filesystem out via NFS or CIFS (or whatever file- based protocol).

A fiber channel interface only knows how to speak blocks (via SCSI
commands); so there has to be a driver on top of that to implement a
filesystem (ext3, NTFS, CVFS, etc). Additionally, while the NAS head
manages competing requests for the same blocks, there's nothing like
that when you talk to a disk via Fiber Channel -- the disk will
happily take a write from one system that overwrites a change just
made by a different system. If one system decides to put a file where
the previous system just extended a directory entry, then your
filesystem is corrupt.

Even NDMP over fiber channel is similar; the NDMP server (which
understands the filesystem's layout) is telling the NDMP tape device
to read blocks in a particular order from the disk.

The only way for BlueArc, NetApp, etc. to present the same view of a
volume via fiber channel or iSCSI that they do via NFS or CIFS would
be if they: a) released a windows/linux/mac/whatever driver that implements WFS or
WAFL, their on-disk filesystem type, and b) set up these drivers and the NAS head to know how to communicate to
make sure that only one system is modifying a disk block at any one
time (which CVFS/StorNext handles for you).

It gets to be a can of worms, and (at least for now) if you need fiber
channel access to disk for multiple clients, StorNext seems to be the
best combination of FCP-and-network access, since you can connect an
FCP client to the filesystem, and have it re-serve the filesystem via
NFS, CIFS, or StorNext LAN client. There's also filesystem software
like (I think) GPFS, Lustre, and others, that I think do similar
things to StorNext/CVFS, but I haven't explored those too much.

If there are other and better options, I'd love to hear about them --
anything to make the bits move faster. }:>

-Sven

On Mar 8, 2010, at 3:18 PM, J. J. Franzen wrote:

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct
FC access to their volumes? I know Blue Arc does not and am curious
if anyone else does. I would so love to do away with NDMP for my
backups and be able to do true incremental backups like in the good
ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with
50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux
NAS. I'm looking for something scalable with high-bandwidth links for our
compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss- bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with
> their > sales-pitch. Can anyone share their experiences with Isilon
> products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating
10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Alas, yes, I've already tunneled down that rabbit hole. We are using BakBone's Netvault for tape backup, and their NDMP plugin has no concept of a level 10 dump. You can't put the value in via the GUI, and if you try to hack the job def file, it will throw an error. I've already sent a couple requests to BA to open up access to the level 10 NDMP extension they created and I've also made a request to BakBone to add support to their NDMP plugin for it. I would love to see a "movement" that would push the NDMP protocol into the modern age and have a "level 10" type dump actually become part of the standard NDMP protocol. Anyone wanna start a rally? :) Seriously though, with volume sizes growing into the 100 of TBs and even in some cases PBs (I'm looking at you WETA), there has to be a better why of implementing tape archive then the current NDMP standard that doesn't involve massive tape libraries with dozens of drives and insane levels of sub volume NDMP dumps. Maybe I should break this off into it's own thread...

J^2

On Mar 9, 2010, at 3:07 PM, Sven Nielsen wrote:

I would consider this experimental, and you might have to get hacky with how your backup software starts up an NDMP session on the BlueArc to do it, but it looks like the BlueArc NDMP server supports a 'level 10' or 'level I' (letter "eye") backup, which sounds like what you're asking. (The ADC replication system for replicating a filesystem makes use of this level, and we use it for that purpose.)

For us, level 10 will: * take a snapshot send all blocks changed between the previous snapshot and the just-taken snapshot delete the previous snapshot

so it effectively functions as a rolling incremental backup/replication level.

On the SMU, look at the text file /usr/local/bin/adc.readme, starting with the section: "NDMP Environment Variables for BlueArc Servers"; also in Appendix B of the PDF admin guide.

Not sure if this will work just as well to tape; if you try it out, test the heck out of it and make sure you can restore files properly after the maximum number of incrementals-between-fulls that you plan to use. You should run the setup by the BlueArc support folks, too, just to make extra sure, but this sounds like it may be what you're talking about.

Also, YMMV, protection of your data is your own responsibility, and be very careful and test restores before relying on this for data protection (unless BlueArc endorses this method, but test backups/restores even if they do endorse it.).

I think this is a BlueArc specific feature, though; not sure if other manufacturers would support it.

-Sven

On Mar 8, 2010, at 3:31 PM, J. J. Franzen wrote:

Right, that's what we are doing now. My big complaint about the setup is there is no way to do a true incremental. Which means you either have do to a full every 7 days to keep the dump sizes down (which for a single LTO4 is impossible to do a full levle 0 dump of a 34 TB volume in less then a week), or after a while the level 1 dump size gets to be so big it takes days to do it anyways.. I am not a big fan of the whole NDMP protocol's dump level design. It's very coarse and not at all tunable. I actually miss the days when I could do an incremental to tape every 4 hours. With NDMP, that's almost completely impossible. Unless I'm missing something in the protocol. If anyone knows of a way to get a true incremental out of NDMP, I would become your biggest fan...

J^2

On Mar 8, 2010, at 3:21 PM, Greg Whynott wrote:

just to tocuh on something you said, you can back up BlueArc over FC via NDMP. I had my back end BlueArc FC switch connected directly to our tape libs and was doing backups via NDMP, totally avoiding the ethernet.

-g

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Greg Whynott [Greg.Whynott@oicr.on.ca] Sent: Monday, March 08, 2010 6:19 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

only NetApp of the ones you mentioned. BlueArc is all NAS, as is Isilon.

-g

From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of J. J. Franzen [jjfranzen@mac.com] Sent: Monday, March 08, 2010 6:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with their > sales-pitch. Can anyone share their experiences with Isilon products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

I'm not trying to put an end to the thread, but I wanted to take this opportunity to thank everyone for their input.

Ian Haskin TOPIX

Hey Ian

Have you looked in to Gluster?

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Ian Haskin Sent: Wednesday, March 10, 2010 2:11 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

I'm not trying to put an end to the thread, but I wanted to take this opportunity to thank everyone for their input.

Ian Haskin TOPIX

I've got a quick one, I'd like to test out our Isilon cluster.

It seems as if I'm getting low performance at the moment and I was wondering if anyone else out there would be able to run a simple DD to compare.

Here's my output from 2 different machines.

[root@flame-09-ny-courtney Tom]# dd if=/dev/zero of=test3.2gb bs=1M count=2096 2197815296 bytes (2.2 GB) copied, 47.8699 seconds, 45.9 MB/s

[root@ny3d-24 Tom]# dd if=/dev/zero of=ny3d.4gb bs=1M count=20096 21072183296 bytes (21 GB) copied, 432.756 s, 48.7 MB/s

I'm currently running a 6 node 6000x cluster on version 5.5.4.21.

If anyone else could send me their output I'd appreciate it.

In related news, they've just announced a product called InsightIQ. It's a VM that shows incredible statistics about the cluster. I suggest chatting to your Isilon rep and getting a demo.

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Sven Nielsen Sent: Tuesday, March 09, 2010 6:25 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

This is one thing I've been asked for a few times; the problem is that NAS systems have their own filesystem on the NAS head, then present the files of that filesystem out via NFS or CIFS (or whatever file- based protocol).

A fiber channel interface only knows how to speak blocks (via SCSI commands); so there has to be a driver on top of that to implement a filesystem (ext3, NTFS, CVFS, etc). Additionally, while the NAS head manages competing requests for the same blocks, there's nothing like that when you talk to a disk via Fiber Channel -- the disk will happily take a write from one system that overwrites a change just made by a different system. If one system decides to put a file where the previous system just extended a directory entry, then your filesystem is corrupt.

Even NDMP over fiber channel is similar; the NDMP server (which understands the filesystem's layout) is telling the NDMP tape device to read blocks in a particular order from the disk.

The only way for BlueArc, NetApp, etc. to present the same view of a volume via fiber channel or iSCSI that they do via NFS or CIFS would be if they: a) released a windows/linux/mac/whatever driver that implements WFS or WAFL, their on-disk filesystem type, and b) set up these drivers and the NAS head to know how to communicate to make sure that only one system is modifying a disk block at any one time (which CVFS/StorNext handles for you).

It gets to be a can of worms, and (at least for now) if you need fiber channel access to disk for multiple clients, StorNext seems to be the best combination of FCP-and-network access, since you can connect an FCP client to the filesystem, and have it re-serve the filesystem via NFS, CIFS, or StorNext LAN client. There's also filesystem software like (I think) GPFS, Lustre, and others, that I think do similar things to StorNext/CVFS, but I haven't explored those too much.

If there are other and better options, I'd love to hear about them -- anything to make the bits move faster. }:>

-Sven

On Mar 8, 2010, at 3:18 PM, J. J. Franzen wrote:

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss- bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with > their > sales-pitch. Can anyone share their experiences with Isilon > products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

v1 http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&mt=8 If you experience problems downloading, please use the alternative link below: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&mt=8

36 node IQ3000 cluster, 4.7.8 version. No load other than this test. 1Gb connection to linux client.

time dd if=/dev/zero of=test3.2gb bs=1M count=2096 2096+0 records in 2096+0 records out

real 0m37.033s user 0m0.008s sys 0m5.801s

time dd if=/dev/zero of=ny3d.4gb bs=1M count=20096 20096+0 records in 20096+0 records out

real 5m56.733s user 0m0.098s sys 0m49.635s

Dave

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Tom Taylor Sent: Thursday, March 11, 2010 10:11 AM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

I've got a quick one, I'd like to test out our Isilon cluster.

It seems as if I'm getting low performance at the moment and I was wondering if anyone else out there would be able to run a simple DD to compare.

Here's my output from 2 different machines.

[root@flame-09-ny-courtney Tom]# dd if=/dev/zero of=test3.2gb bs=1M count=2096 2197815296 bytes (2.2 GB) copied, 47.8699 seconds, 45.9 MB/s

[root@ny3d-24 Tom]# dd if=/dev/zero of=ny3d.4gb bs=1M count=20096 21072183296 bytes (21 GB) copied, 432.756 s, 48.7 MB/s

I'm currently running a 6 node 6000x cluster on version 5.5.4.21.

If anyone else could send me their output I'd appreciate it.

In related news, they've just announced a product called InsightIQ. It's a VM that shows incredible statistics about the cluster. I suggest chatting to your Isilon rep and getting a demo.

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Sven Nielsen Sent: Tuesday, March 09, 2010 6:25 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

This is one thing I've been asked for a few times; the problem is that NAS systems have their own filesystem on the NAS head, then present the files of that filesystem out via NFS or CIFS (or whatever file- based protocol).

A fiber channel interface only knows how to speak blocks (via SCSI commands); so there has to be a driver on top of that to implement a filesystem (ext3, NTFS, CVFS, etc). Additionally, while the NAS head manages competing requests for the same blocks, there's nothing like that when you talk to a disk via Fiber Channel -- the disk will happily take a write from one system that overwrites a change just made by a different system. If one system decides to put a file where the previous system just extended a directory entry, then your filesystem is corrupt.

Even NDMP over fiber channel is similar; the NDMP server (which understands the filesystem's layout) is telling the NDMP tape device to read blocks in a particular order from the disk.

The only way for BlueArc, NetApp, etc. to present the same view of a volume via fiber channel or iSCSI that they do via NFS or CIFS would be if they: a) released a windows/linux/mac/whatever driver that implements WFS or WAFL, their on-disk filesystem type, and b) set up these drivers and the NAS head to know how to communicate to make sure that only one system is modifying a disk block at any one time (which CVFS/StorNext handles for you).

It gets to be a can of worms, and (at least for now) if you need fiber channel access to disk for multiple clients, StorNext seems to be the best combination of FCP-and-network access, since you can connect an FCP client to the filesystem, and have it re-serve the filesystem via NFS, CIFS, or StorNext LAN client. There's also filesystem software like (I think) GPFS, Lustre, and others, that I think do similar things to StorNext/CVFS, but I haven't explored those too much.

If there are other and better options, I'd love to hear about them -- anything to make the bits move faster. }:>

-Sven

On Mar 8, 2010, at 3:18 PM, J. J. Franzen wrote:

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss- bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with > their > sales-pitch. Can anyone share their experiences with Isilon > products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com

http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-disc uss >> >> __ >> StudioSysAdmins-Discuss mailing list >> StudioSysAdmins-Discuss@mailman.studiosysadmins.com >> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-disc uss > > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@mailman.studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-disc uss >

v1 http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704 758&mt=8 If you experience problems downloading, please use the alternative link below: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704 758&mt=8

-- This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.

From a Linux VM running on ESXi, 1Gb link to an Exanet cluster (2 nodes), under normal load:

time dd if=/dev/zero of=test3.2gb bs=1M count=2096 2096+0 records in 2096+0 records out 2197815296 bytes (2.2 GB) copied, 19.2054 seconds, 114 MB/s

real 0m19.374s user 0m0.002s sys 0m1.683s

time dd if=/dev/zero of=ny3d.4gb bs=1M count=20096 20096+0 records in 20096+0 records out 21072183296 bytes (21 GB) copied, 184.04 seconds, 114 MB/s

real 3m4.085s user 0m0.025s sys 0m22.366s

Will need to test with 10Gb...

Francis

-- Francis Provencher, IT Director Lumi?re VFX Email: francis@lumierevfx.com Phone: +1-514-316-1080x2116

This is from our single Blue Arc Mercury run on an 8-core MacPro under OS X 10.6.2. The Mercury is connected via ethernet over a 4 port aggregation (although for a test like that aggregation doesn't come into play).

gort:data root# dd if=/dev/zero of=test1.2gb bs=1m count=2096 2096+0 records in 2096+0 records out 2197815296 bytes transferred in 24.199800 secs (90819564 bytes/sec)

~ 90 MB/s

gort:data root# dd if=/dev/zero of=test1.4gb bs=1m count=20096 20096+0 records in 20096+0 records out 21072183296 bytes transferred in 243.662409 secs (86481059 bytes/sec)

~ 86 MB/s

Just for your reference.

J^2

On Mar 11, 2010, at 10:10 AM, Tom Taylor wrote:

I've got a quick one, I'd like to test out our Isilon cluster.

It seems as if I'm getting low performance at the moment and I was wondering if anyone else out there would be able to run a simple DD to compare.

Here's my output from 2 different machines.

[root@flame-09-ny-courtney Tom]# dd if=/dev/zero of=test3.2gb bs=1M count=2096 2197815296 bytes (2.2 GB) copied, 47.8699 seconds, 45.9 MB/s

[root@ny3d-24 Tom]# dd if=/dev/zero of=ny3d.4gb bs=1M count=20096 21072183296 bytes (21 GB) copied, 432.756 s, 48.7 MB/s

I'm currently running a 6 node 6000x cluster on version 5.5.4.21.

If anyone else could send me their output I'd appreciate it.

In related news, they've just announced a product called InsightIQ. It's a VM that shows incredible statistics about the cluster. I suggest chatting to your Isilon rep and getting a demo.

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Sven Nielsen Sent: Tuesday, March 09, 2010 6:25 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

This is one thing I've been asked for a few times; the problem is that NAS systems have their own filesystem on the NAS head, then present the files of that filesystem out via NFS or CIFS (or whatever file- based protocol).

A fiber channel interface only knows how to speak blocks (via SCSI commands); so there has to be a driver on top of that to implement a filesystem (ext3, NTFS, CVFS, etc). Additionally, while the NAS head manages competing requests for the same blocks, there's nothing like that when you talk to a disk via Fiber Channel -- the disk will happily take a write from one system that overwrites a change just made by a different system. If one system decides to put a file where the previous system just extended a directory entry, then your filesystem is corrupt.

Even NDMP over fiber channel is similar; the NDMP server (which understands the filesystem's layout) is telling the NDMP tape device to read blocks in a particular order from the disk.

The only way for BlueArc, NetApp, etc. to present the same view of a volume via fiber channel or iSCSI that they do via NFS or CIFS would be if they: a) released a windows/linux/mac/whatever driver that implements WFS or WAFL, their on-disk filesystem type, and b) set up these drivers and the NAS head to know how to communicate to make sure that only one system is modifying a disk block at any one time (which CVFS/StorNext handles for you).

It gets to be a can of worms, and (at least for now) if you need fiber channel access to disk for multiple clients, StorNext seems to be the best combination of FCP-and-network access, since you can connect an FCP client to the filesystem, and have it re-serve the filesystem via NFS, CIFS, or StorNext LAN client. There's also filesystem software like (I think) GPFS, Lustre, and others, that I think do similar things to StorNext/CVFS, but I haven't explored those too much.

If there are other and better options, I'd love to hear about them -- anything to make the bits move faster. }:>

-Sven

On Mar 8, 2010, at 3:18 PM, J. J. Franzen wrote:

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss- bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with > their > sales-pitch. Can anyone share their experiences with Isilon > products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

v1 http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&mt=8 If you experience problems downloading, please use the alternative link below: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&mt=8

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Out of curiosity, what type of switches are you using?

-----Original Message----- From: Tom Taylor [mailto:taylor@the-mill.com] Sent: Thursday, March 11, 2010 01:10 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

I've got a quick one, I'd like to test out our Isilon cluster.

It seems as if I'm getting low performance at the moment and I was wondering if anyone else out there would be able to run a simple DD to compare.

Here's my output from 2 different machines.

[root@flame-09-ny-courtney Tom]# dd if=/dev/zero of=test3.2gb bs=1M count=2096 2197815296 bytes (2.2 GB) copied, 47.8699 seconds, 45.9 MB/s

[root@ny3d-24 Tom]# dd if=/dev/zero of=ny3d.4gb bs=1M count=20096 21072183296 bytes (21 GB) copied, 432.756 s, 48.7 MB/s

I'm currently running a 6 node 6000x cluster on version 5.5.4.21.

If anyone else could send me their output I'd appreciate it.

In related news, they've just announced a product called InsightIQ. It's a VM that shows incredible statistics about the cluster. I suggest chatting to your Isilon rep and getting a demo.

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Sven Nielsen Sent: Tuesday, March 09, 2010 6:25 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

This is one thing I've been asked for a few times; the problem is that NAS systems have their own filesystem on the NAS head, then present the files of that filesystem out via NFS or CIFS (or whatever file- based protocol).

A fiber channel interface only knows how to speak blocks (via SCSI commands); so there has to be a driver on top of that to implement a filesystem (ext3, NTFS, CVFS, etc). Additionally, while the NAS head manages competing requests for the same blocks, there's nothing like that when you talk to a disk via Fiber Channel -- the disk will happily take a write from one system that overwrites a change just made by a different system. If one system decides to put a file where the previous system just extended a directory entry, then your filesystem is corrupt.

Even NDMP over fiber channel is similar; the NDMP server (which understands the filesystem's layout) is telling the NDMP tape device to read blocks in a particular order from the disk.

The only way for BlueArc, NetApp, etc. to present the same view of a volume via fiber channel or iSCSI that they do via NFS or CIFS would be if they: a) released a windows/linux/mac/whatever driver that implements WFS or WAFL, their on-disk filesystem type, and b) set up these drivers and the NAS head to know how to communicate to make sure that only one system is modifying a disk block at any one time (which CVFS/StorNext handles for you).

It gets to be a can of worms, and (at least for now) if you need fiber channel access to disk for multiple clients, StorNext seems to be the best combination of FCP-and-network access, since you can connect an FCP client to the filesystem, and have it re-serve the filesystem via NFS, CIFS, or StorNext LAN client. There's also filesystem software like (I think) GPFS, Lustre, and others, that I think do similar things to StorNext/CVFS, but I haven't explored those too much.

If there are other and better options, I'd love to hear about them -- anything to make the bits move faster. }:>

-Sven

On Mar 8, 2010, at 3:18 PM, J. J. Franzen wrote:

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss- bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with > their > sales-pitch. Can anyone share their experiences with Isilon > products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

v1 http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&mt=8 If you experience problems downloading, please use the alternative link below: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&mt=8

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

Everything is in an HP 2900 - 2 x 1Gb trunks into each cluster. (Soon to be Brocade or Extreme 10Gb)

Seems like the interfaces are not happy.

Port Total Bytes Total Frames Errors Rx Drops Tx Ctrl Limit

Thanks everyone for the test results!

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of bill@yuco.com Sent: Thursday, March 11, 2010 6:19 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

Out of curiosity, what type of switches are you using?

-----Original Message----- From: Tom Taylor [mailto:taylor@the-mill.com] Sent: Thursday, March 11, 2010 01:10 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

I've got a quick one, I'd like to test out our Isilon cluster.

It seems as if I'm getting low performance at the moment and I was wondering if anyone else out there would be able to run a simple DD to compare.

Here's my output from 2 different machines.

[root@flame-09-ny-courtney Tom]# dd if=/dev/zero of=test3.2gb bs=1M count=2096 2197815296 bytes (2.2 GB) copied, 47.8699 seconds, 45.9 MB/s

[root@ny3d-24 Tom]# dd if=/dev/zero of=ny3d.4gb bs=1M count=20096 21072183296 bytes (21 GB) copied, 432.756 s, 48.7 MB/s

I'm currently running a 6 node 6000x cluster on version 5.5.4.21.

If anyone else could send me their output I'd appreciate it.

In related news, they've just announced a product called InsightIQ. It's a VM that shows incredible statistics about the cluster. I suggest chatting to your Isilon rep and getting a demo.

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Sven Nielsen Sent: Tuesday, March 09, 2010 6:25 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

This is one thing I've been asked for a few times; the problem is that NAS systems have their own filesystem on the NAS head, then present the files of that filesystem out via NFS or CIFS (or whatever file- based protocol).

A fiber channel interface only knows how to speak blocks (via SCSI commands); so there has to be a driver on top of that to implement a filesystem (ext3, NTFS, CVFS, etc). Additionally, while the NAS head manages competing requests for the same blocks, there's nothing like that when you talk to a disk via Fiber Channel -- the disk will happily take a write from one system that overwrites a change just made by a different system. If one system decides to put a file where the previous system just extended a directory entry, then your filesystem is corrupt.

Even NDMP over fiber channel is similar; the NDMP server (which understands the filesystem's layout) is telling the NDMP tape device to read blocks in a particular order from the disk.

The only way for BlueArc, NetApp, etc. to present the same view of a volume via fiber channel or iSCSI that they do via NFS or CIFS would be if they: a) released a windows/linux/mac/whatever driver that implements WFS or WAFL, their on-disk filesystem type, and b) set up these drivers and the NAS head to know how to communicate to make sure that only one system is modifying a disk block at any one time (which CVFS/StorNext handles for you).

It gets to be a can of worms, and (at least for now) if you need fiber channel access to disk for multiple clients, StorNext seems to be the best combination of FCP-and-network access, since you can connect an FCP client to the filesystem, and have it re-serve the filesystem via NFS, CIFS, or StorNext LAN client. There's also filesystem software like (I think) GPFS, Lustre, and others, that I think do similar things to StorNext/CVFS, but I haven't explored those too much.

If there are other and better options, I'd love to hear about them -- anything to make the bits move faster. }:>

-Sven

On Mar 8, 2010, at 3:18 PM, J. J. Franzen wrote:

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss- bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with > their > sales-pitch. Can anyone share their experiences with Isilon > products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

v1 http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&mt=8 If you experience problems downloading, please use the alternative link below: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&mt=8

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

When I was doing HPC storage research (Lustre, specifically), I found that Extreme switches (Summit series) consistently outperformed the competition (Cisco, Juniper, and others) by sometimes as much as 5-10%, and the Extreme was able to reliably get me about 118 MBps every time, and was giving me as much as 95% of theoretical max without any tweaking.

As always, YMMV, but I've found Extreme to be extremely reliable in terms of switch throughput, and it comes without the "Cisco" sticker premium. There are other great performers out there as well (Foundry is one of them), but I didn't get a chance to punish one the same way so unfortunately I don't have numbers handy.

Klaus

On 3/11/10 3:23 PM, "Tom Taylor" taylor@the-mill.com etched on stone tablets:

Everything is in an HP 2900 - 2 x 1Gb trunks into each cluster. (Soon to be Brocade or Extreme 10Gb)

Seems like the interfaces are not happy.

Port Total Bytes Total Frames Errors Rx Drops Tx Ctrl Limit ------- -------------- -------------- ------------ ------------ ----- ------ 1-Trk1 2,512,407,710 2,727,621,561 86,495 11,466,568 off 0 2-Trk1 3,123,561,267 1,198,521,278 18,014 8,707,104 off 0 3-Trk2 798,322,066 982,377,621 49,155 11,695,269 off 0 4-Trk2 53,405,790 2,926,838,905 163,569 14,461,588 off 0 5-Trk3 553,600,861 3,618,989,149 82,960 7,622,094 off 0 6-Trk3 1,887,102,210 4,292,551,184 60,317 6,679,635 off 0 7-Trk4 1,137,517,211 3,081,662,500 43,361 8,464,187 off 0 8-Trk4 3,503,697,323 3,465,088,585 85,276 9,135,323 off 0 9-Trk5 2,296,075,992 1,128,920,646 72,032 7,489,051 off 0 10-Trk5 2,879,550,610 2,600,747,860 54,721 8,322,167 off 0 11-Trk6 2,748,755,020 3,165,756,973 24,034 183,152,905 off 0 12-Trk6 1,611,690,357 2,354,340,713 9430 64,319,051 off 0

Thanks everyone for the test results!

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of bill@yuco.com Sent: Thursday, March 11, 2010 6:19 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

Out of curiosity, what type of switches are you using?

-----Original Message----- From: Tom Taylor [mailto:taylor@the-mill.com] Sent: Thursday, March 11, 2010 01:10 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

I've got a quick one, I'd like to test out our Isilon cluster.

It seems as if I'm getting low performance at the moment and I was wondering if anyone else out there would be able to run a simple DD to compare.

Here's my output from 2 different machines.

[root@flame-09-ny-courtney Tom]# dd if=/dev/zero of=test3.2gb bs=1M count=2096 2197815296 bytes (2.2 GB) copied, 47.8699 seconds, 45.9 MB/s

[root@ny3d-24 Tom]# dd if=/dev/zero of=ny3d.4gb bs=1M count=20096 21072183296 bytes (21 GB) copied, 432.756 s, 48.7 MB/s

I'm currently running a 6 node 6000x cluster on version 5.5.4.21.

If anyone else could send me their output I'd appreciate it.

In related news, they've just announced a product called InsightIQ. It's a VM that shows incredible statistics about the cluster. I suggest chatting to your Isilon rep and getting a demo.

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Sven Nielsen Sent: Tuesday, March 09, 2010 6:25 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

This is one thing I've been asked for a few times; the problem is that NAS systems have their own filesystem on the NAS head, then present the files of that filesystem out via NFS or CIFS (or whatever file- based protocol).

A fiber channel interface only knows how to speak blocks (via SCSI commands); so there has to be a driver on top of that to implement a filesystem (ext3, NTFS, CVFS, etc). Additionally, while the NAS head manages competing requests for the same blocks, there's nothing like that when you talk to a disk via Fiber Channel -- the disk will happily take a write from one system that overwrites a change just made by a different system. If one system decides to put a file where the previous system just extended a directory entry, then your filesystem is corrupt.

Even NDMP over fiber channel is similar; the NDMP server (which understands the filesystem's layout) is telling the NDMP tape device to read blocks in a particular order from the disk.

The only way for BlueArc, NetApp, etc. to present the same view of a volume via fiber channel or iSCSI that they do via NFS or CIFS would be if they: a) released a windows/linux/mac/whatever driver that implements WFS or WAFL, their on-disk filesystem type, and b) set up these drivers and the NAS head to know how to communicate to make sure that only one system is modifying a disk block at any one time (which CVFS/StorNext handles for you).

It gets to be a can of worms, and (at least for now) if you need fiber channel access to disk for multiple clients, StorNext seems to be the best combination of FCP-and-network access, since you can connect an FCP client to the filesystem, and have it re-serve the filesystem via NFS, CIFS, or StorNext LAN client. There's also filesystem software like (I think) GPFS, Lustre, and others, that I think do similar things to StorNext/CVFS, but I haven't explored those too much.

If there are other and better options, I'd love to hear about them -- anything to make the bits move faster. }:>

-Sven

On Mar 8, 2010, at 3:18 PM, J. J. Franzen wrote:

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss- bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with > their > sales-pitch. Can anyone share their experiences with Isilon > products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

v1 http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&a mp;mt=8 If you experience problems downloading, please use the alternative link below: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&a mp;mt=8

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

__ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

holly smokes thats a lot of errors...

Everything is in an HP 2900 - 2 x 1Gb trunks into each cluster. (Soon to be Brocade or Extreme 10Gb)

Seems like the interfaces are not happy.

Port Total Bytes Total Frames Errors Rx Drops Tx Ctrl Limit

Thanks everyone for the test results!

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of bill@yuco.com Sent: Thursday, March 11, 2010 6:19 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

Out of curiosity, what type of switches are you using?

-----Original Message----- From: Tom Taylor [mailto:taylor@the-mill.com] Sent: Thursday, March 11, 2010 01:10 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

I've got a quick one, I'd like to test out our Isilon cluster.

It seems as if I'm getting low performance at the moment and I was wondering if anyone else out there would be able to run a simple DD to compare.

Here's my output from 2 different machines.

[root@flame-09-ny-courtney Tom]# dd if=/dev/zero of=test3.2gb bs=1M count=2096 2197815296 bytes (2.2 GB) copied, 47.8699 seconds, 45.9 MB/s

[root@ny3d-24 Tom]# dd if=/dev/zero of=ny3d.4gb bs=1M count=20096 21072183296 bytes (21 GB) copied, 432.756 s, 48.7 MB/s

I'm currently running a 6 node 6000x cluster on version 5.5.4.21.

If anyone else could send me their output I'd appreciate it.

In related news, they've just announced a product called InsightIQ. It's a VM that shows incredible statistics about the cluster. I suggest chatting to your Isilon rep and getting a demo.

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Sven Nielsen Sent: Tuesday, March 09, 2010 6:25 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

This is one thing I've been asked for a few times; the problem is that NAS systems have their own filesystem on the NAS head, then present the files of that filesystem out via NFS or CIFS (or whatever file- based protocol).

A fiber channel interface only knows how to speak blocks (via SCSI commands); so there has to be a driver on top of that to implement a filesystem (ext3, NTFS, CVFS, etc). Additionally, while the NAS head manages competing requests for the same blocks, there's nothing like that when you talk to a disk via Fiber Channel -- the disk will happily take a write from one system that overwrites a change just made by a different system. If one system decides to put a file where the previous system just extended a directory entry, then your filesystem is corrupt.

Even NDMP over fiber channel is similar; the NDMP server (which understands the filesystem's layout) is telling the NDMP tape device to read blocks in a particular order from the disk.

The only way for BlueArc, NetApp, etc. to present the same view of a volume via fiber channel or iSCSI that they do via NFS or CIFS would be if they: a) released a windows/linux/mac/whatever driver that implements WFS or WAFL, their on-disk filesystem type, and b) set up these drivers and the NAS head to know how to communicate to make sure that only one system is modifying a disk block at any one time (which CVFS/StorNext handles for you).

It gets to be a can of worms, and (at least for now) if you need fiber channel access to disk for multiple clients, StorNext seems to be the best combination of FCP-and-network access, since you can connect an FCP client to the filesystem, and have it re-serve the filesystem via NFS, CIFS, or StorNext LAN client. There's also filesystem software like (I think) GPFS, Lustre, and others, that I think do similar things to StorNext/CVFS, but I haven't explored those too much.

If there are other and better options, I'd love to hear about them -- anything to make the bits move faster. }:>

-Sven

On Mar 8, 2010, at 3:18 PM, J. J. Franzen wrote:

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss- bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with > their > sales-pitch. Can anyone share their experiences with Isilon > products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

v1 http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&mt=8 If you experience problems downloading, please use the alternative link below: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&mt=8

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

I think he is hitting a resource limit within the switch. you shouldn't see many if any TX errors. when you do it sometimes is an indication of a buffer depletion issue.

HP 8212's are great devices for HPC when you consider the cost per port and chassis capacity. Extreme and Force10 are both having a sale on 24 port 10Gbit switches this month. Personally I prefer Force10 over Extreme.

I' am in contact wiht many of North America's largest HPC admins, non of them that I am aware of have Fourndry deployed. nor could I recommend them. Especially now since they were bought by the kings of gouging. Soon you'll have to license your ethernet ports individualy, and pay a prenium of you want to use any feature, such as QoS, or trunking.

-g

When I was doing HPC storage research (Lustre, specifically), I found that Extreme switches (Summit series) consistently outperformed the competition (Cisco, Juniper, and others) by sometimes as much as 5-10%, and the Extreme was able to reliably get me about 118 MBps every time, and was giving me as much as 95% of theoretical max without any tweaking.

As always, YMMV, but I've found Extreme to be extremely reliable in terms of switch throughput, and it comes without the "Cisco" sticker premium. There are other great performers out there as well (Foundry is one of them), but I didn't get a chance to punish one the same way so unfortunately I don't have numbers handy.

Klaus

On 3/11/10 3:23 PM, "Tom Taylor" taylor@the-mill.com etched on stone tablets:

Everything is in an HP 2900 - 2 x 1Gb trunks into each cluster. (Soon to be Brocade or Extreme 10Gb)

Seems like the interfaces are not happy.

Port Total Bytes Total Frames Errors Rx Drops Tx Ctrl Limit ------- -------------- -------------- ------------ ------------ ----- ------ 1-Trk1 2,512,407,710 2,727,621,561 86,495 11,466,568 off 0 2-Trk1 3,123,561,267 1,198,521,278 18,014 8,707,104 off 0 3-Trk2 798,322,066 982,377,621 49,155 11,695,269 off 0 4-Trk2 53,405,790 2,926,838,905 163,569 14,461,588 off 0 5-Trk3 553,600,861 3,618,989,149 82,960 7,622,094 off 0 6-Trk3 1,887,102,210 4,292,551,184 60,317 6,679,635 off 0 7-Trk4 1,137,517,211 3,081,662,500 43,361 8,464,187 off 0 8-Trk4 3,503,697,323 3,465,088,585 85,276 9,135,323 off 0 9-Trk5 2,296,075,992 1,128,920,646 72,032 7,489,051 off 0 10-Trk5 2,879,550,610 2,600,747,860 54,721 8,322,167 off 0 11-Trk6 2,748,755,020 3,165,756,973 24,034 183,152,905 off 0 12-Trk6 1,611,690,357 2,354,340,713 9430 64,319,051 off 0

Thanks everyone for the test results!

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of bill@yuco.com Sent: Thursday, March 11, 2010 6:19 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

Out of curiosity, what type of switches are you using?

-----Original Message----- From: Tom Taylor [mailto:taylor@the-mill.com] Sent: Thursday, March 11, 2010 01:10 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

I've got a quick one, I'd like to test out our Isilon cluster.

It seems as if I'm getting low performance at the moment and I was wondering if anyone else out there would be able to run a simple DD to compare.

Here's my output from 2 different machines.

[root@flame-09-ny-courtney Tom]# dd if=/dev/zero of=test3.2gb bs=1M count=2096 2197815296 bytes (2.2 GB) copied, 47.8699 seconds, 45.9 MB/s

[root@ny3d-24 Tom]# dd if=/dev/zero of=ny3d.4gb bs=1M count=20096 21072183296 bytes (21 GB) copied, 432.756 s, 48.7 MB/s

I'm currently running a 6 node 6000x cluster on version 5.5.4.21.

If anyone else could send me their output I'd appreciate it.

In related news, they've just announced a product called InsightIQ. It's a VM that shows incredible statistics about the cluster. I suggest chatting to your Isilon rep and getting a demo.

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Sven Nielsen Sent: Tuesday, March 09, 2010 6:25 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

This is one thing I've been asked for a few times; the problem is that NAS systems have their own filesystem on the NAS head, then present the files of that filesystem out via NFS or CIFS (or whatever file- based protocol).

A fiber channel interface only knows how to speak blocks (via SCSI commands); so there has to be a driver on top of that to implement a filesystem (ext3, NTFS, CVFS, etc). Additionally, while the NAS head manages competing requests for the same blocks, there's nothing like that when you talk to a disk via Fiber Channel -- the disk will happily take a write from one system that overwrites a change just made by a different system. If one system decides to put a file where the previous system just extended a directory entry, then your filesystem is corrupt.

Even NDMP over fiber channel is similar; the NDMP server (which understands the filesystem's layout) is telling the NDMP tape device to read blocks in a particular order from the disk.

The only way for BlueArc, NetApp, etc. to present the same view of a volume via fiber channel or iSCSI that they do via NFS or CIFS would be if they: a) released a windows/linux/mac/whatever driver that implements WFS or WAFL, their on-disk filesystem type, and b) set up these drivers and the NAS head to know how to communicate to make sure that only one system is modifying a disk block at any one time (which CVFS/StorNext handles for you).

It gets to be a can of worms, and (at least for now) if you need fiber channel access to disk for multiple clients, StorNext seems to be the best combination of FCP-and-network access, since you can connect an FCP client to the filesystem, and have it re-serve the filesystem via NFS, CIFS, or StorNext LAN client. There's also filesystem software like (I think) GPFS, Lustre, and others, that I think do similar things to StorNext/CVFS, but I haven't explored those too much.

If there are other and better options, I'd love to hear about them -- anything to make the bits move faster. }:>

-Sven

On Mar 8, 2010, at 3:18 PM, J. J. Franzen wrote:

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss- bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with > their > sales-pitch. Can anyone share their experiences with Isilon > products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

v1 http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&a mp;mt=8 If you experience problems downloading, please use the alternative link below: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&a mp;mt=8

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

__ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

http://searchservervirtualization.techtarget.com/tip/0,289483,sid94_gci1376155_mem1,00.html

When I was doing HPC storage research (Lustre, specifically), I found that
Extreme switches (Summit series) consistently outperformed the competition
(Cisco, Juniper, and others) by sometimes as much as 5-10%, and the Extreme
was able to reliably get me about 118 MBps every time, and was giving me as
much as 95% of theoretical max without any tweaking.

As always, YMMV, but I've found Extreme to be extremely reliable in terms of
switch throughput, and it comes without the "Cisco" sticker premium. There
are other great performers out there as well (Foundry is one of them), but I
didn't get a chance to punish one the same way so unfortunately I don't have
numbers handy.

Klaus

On 3/11/10 3:23 PM, "Tom Taylor" TAYLOR@THE-MILL.COMetched on stone
tablets:

> Everything is in an HP 2900 - 2 x 1Gb trunks into each cluster. (Soon to be
> Brocade or Extreme 10Gb)
>
> Seems like the interfaces are not happy.
>
>
> Port Total Bytes Total Frames Errors Rx Drops Tx Ctrl Limit
> ------- -------------- -------------- ------------ ------------ ----- ------
> 1-Trk1 2,512,407,710 2,727,621,561 86,495 11,466,568 off 0
> 2-Trk1 3,123,561,267 1,198,521,278 18,014 8,707,104 off 0
> 3-Trk2 798,322,066 982,377,621 49,155 11,695,269 off 0
> 4-Trk2 53,405,790 2,926,838,905 163,569 14,461,588 off 0
> 5-Trk3 553,600,861 3,618,989,149 82,960 7,622,094 off 0
> 6-Trk3 1,887,102,210 4,292,551,184 60,317 6,679,635 off 0
> 7-Trk4 1,137,517,211 3,081,662,500 43,361 8,464,187 off 0
> 8-Trk4 3,503,697,323 3,465,088,585 85,276 9,135,323 off 0
> 9-Trk5 2,296,075,992 1,128,920,646 72,032 7,489,051 off 0
> 10-Trk5 2,879,550,610 2,600,747,860 54,721 8,322,167 off 0
> 11-Trk6 2,748,755,020 3,165,756,973 24,034 183,152,905 off 0
> 12-Trk6 1,611,690,357 2,354,340,713 9430 64,319,051 off 0
>
> Thanks everyone for the test results!
>
> Cheers,
> Tom
>
>
> -----Original Message-----
> From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com
> [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf
> Of bill@yuco.com
> Sent: Thursday, March 11, 2010 6:19 PM
> To: discuss@studiosysadmins.com
> Subject: Re: Researching: Isilon and alternatives
>
>
> Out of curiosity, what type of switches are you using?
>
>> -----Original Message-----
>> From: Tom Taylor [mailto:taylor@the-mill.com]
>> Sent: Thursday, March 11, 2010 01:10 PM
>> To: discuss@studiosysadmins.com
>> Subject: RE: Researching: Isilon and alternatives
>>
>> I've got a quick one, I'd like to test out our Isilon cluster.
>>
>> It seems as if I'm getting low performance at the moment and I was wondering
>> if anyone else out there would be able to run a simple DD to compare.
>>
>> Here's my output from 2 different machines.
>>
>>
>> [root@flame-09-ny-courtney Tom]# dd if=/dev/zero of=test3.2gb bs=1M
>> count=2096
>> 2197815296 bytes (2.2 GB) copied, 47.8699 seconds, 45.9 MB/s
>>
>> [root@ny3d-24 Tom]# dd if=/dev/zero of=ny3d.4gb bs=1M count=20096
>> 21072183296 bytes (21 GB) copied, 432.756 s, 48.7 MB/s
>>
>> I'm currently running a 6 node 6000x cluster on version 5.5.4.21.
>>
>> If anyone else could send me their output I'd appreciate it.
>>
>>
>> In related news, they've just announced a product called InsightIQ. It's a VM
>> that shows incredible statistics about the cluster. I suggest chatting to
>> your Isilon rep and getting a demo.
>>
>> Cheers,
>> Tom
>>
>>
>>
>>
>> -----Original Message-----
>> From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com
>> [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On
>> Behalf Of Sven Nielsen
>> Sent: Tuesday, March 09, 2010 6:25 PM
>> To: discuss@studiosysadmins.com
>> Subject: Re: Researching: Isilon and alternatives
>>
>> This is one thing I've been asked for a few times; the problem is that
>> NAS systems have their own filesystem on the NAS head, then present
>> the files of that filesystem out via NFS or CIFS (or whatever file-
>> based protocol).
>>
>> A fiber channel interface only knows how to speak blocks (via SCSI
>> commands); so there has to be a driver on top of that to implement a
>> filesystem (ext3, NTFS, CVFS, etc). Additionally, while the NAS head
>> manages competing requests for the same blocks, there's nothing like
>> that when you talk to a disk via Fiber Channel -- the disk will
>> happily take a write from one system that overwrites a change just
>> made by a different system. If one system decides to put a file where
>> the previous system just extended a directory entry, then your
>> filesystem is corrupt.
>>
>> Even NDMP over fiber channel is similar; the NDMP server (which
>> understands the filesystem's layout) is telling the NDMP tape device
>> to read blocks in a particular order from the disk.
>>
>> The only way for BlueArc, NetApp, etc. to present the same view of a
>> volume via fiber channel or iSCSI that they do via NFS or CIFS would
>> be if they:
>> a) released a windows/linux/mac/whatever driver that implements WFS or
>> WAFL, their on-disk filesystem type, and
>> b) set up these drivers and the NAS head to know how to communicate to
>> make sure that only one system is modifying a disk block at any one
>> time (which CVFS/StorNext handles for you).
>>
>> It gets to be a can of worms, and (at least for now) if you need fiber
>> channel access to disk for multiple clients, StorNext seems to be the
>> best combination of FCP-and-network access, since you can connect an
>> FCP client to the filesystem, and have it re-serve the filesystem via
>> NFS, CIFS, or StorNext LAN client. There's also filesystem software
>> like (I think) GPFS, Lustre, and others, that I think do similar
>> things to StorNext/CVFS, but I haven't explored those too much.
>>
>> If there are other and better options, I'd love to hear about them --
>> anything to make the bits move faster. }:>
>>
>> -Sven
>>
>> On Mar 8, 2010, at 3:18 PM, J. J. Franzen wrote:
>>
>>> Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct
>>> FC access to their volumes? I know Blue Arc does not and am curious
>>> if anyone else does. I would so love to do away with NDMP for my
>>> backups and be able to do true incremental backups like in the good
>>> ol days...
>>>
>>> J^2
>>>
>>> On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:
>>>
>>>> We're a small TV-commercial shop (3D/compositing, 20 people, with
>>>> 50 or so
>>>> machines, incl. 4 flames) that is outgrowing our home-built Linux
>>>> NAS. I'm
>>>> looking for something scalable with high-bandwidth links for our
>>>> compositing
>>>> nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com
>>>> [mailto:studiosysadmins-discuss-
>>>> bounces@mailman.studiosysadmins.com] On
>>>> Behalf Of Barry Robison
>>>> Sent: March-08-10 5:18 PM
>>>> To: discuss@studiosysadmins.com
>>>> Subject: Re: Researching: Isilon and alternatives
>>>>
>>>> On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin IAN@TOPIXFX.COMwrote:
>>>>> I've been looking into Isilon storage and I'm quite impressed with
>>>>> their
>>>>> sales-pitch. Can anyone share their experiences with Isilon
>>>>> products or
>>>> any
>>>>> other products that compete/compare with them?
>>>>>
>>>>
>>>> Hi Ian,
>>>>
>>>> What is your budget and expected workload?
>>>>
>>>> We evaluated both Isilon and BlueArc, with BlueArc winning our
>>>> business. For our expected artist count (200) and farm size ( ~6k
>>>> cores ), we liked BA better. We were sold on the data tiering,
>>>> disaster recovery, and throughput the hardware architecture allows.
>>>> From other shops we heard that BA was the only thing saturating
>>>> 10GbE.
>>>> We had issues getting Isilon working on 10GbE at all...
>>>>
>>>> At a previous facility I worked at we used a small Isilon cluster ( 4
>>>> nodes ) quite happily. I think when you get up to the number of nodes
>>>> ( and accelerator nodes ) you need for a large cluster, some problems
>>>> can be expected.
>>>>
>>>>
>>>> Good luck,
>>>> -Barry
>>>> _
>>>> StudioSysAdmins-Discuss mailing list
>>>> StudioSysAdmins-Discuss@mailman.studiosysadmins.com
>>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss
>>>>
>>>> _
>>>> StudioSysAdmins-Discuss mailing list
>>>> StudioSysAdmins-Discuss@mailman.studiosysadmins.com
>>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss
>>>
>>> _
>>> StudioSysAdmins-Discuss mailing list
>>> StudioSysAdmins-Discuss@mailman.studiosysadmins.com
>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss
>>>
>>
>> _
>> StudioSysAdmins-Discuss mailing list
>> StudioSysAdmins-Discuss@mailman.studiosysadmins.com
>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss
>>
>> v1
>> http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&a
>> mp;mt=8
>> If you experience problems downloading, please use the alternative link
>> below:
>> http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&a
>> mp;mt=8
>>
>> _
>> StudioSysAdmins-Discuss mailing list
>> StudioSysAdmins-Discuss@mailman.studiosysadmins.com
>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss
>>
>
>
> _
> StudioSysAdmins-Discuss mailing list
> StudioSysAdmins-Discuss@mailman.studiosysadmins.com
> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss
> _
> StudioSysAdmins-Discuss mailing list
> StudioSysAdmins-Discuss@mailman.studiosysadmins.com
> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_
StudioSysAdmins-Discuss mailing list
StudioSysAdmins-Discuss@mailman.studiosysadmins.com
http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss