I think he is hitting a resource limit within the switch. you shouldn't see many if any TX errors. when you do it sometimes is an indication of a buffer depletion issue. HP 8212's are great devices for HPC when you consider the cost per port and chassis capacity. Extreme and Force10 are both having a sale on 24 port 10Gbit switches this month. Personally I prefer Force10 over Extreme. I' am in contact wiht many of North America's largest HPC admins, non of them that I am aware of have Fourndry deployed. nor could I recommend them. Especially now since they were bought by the kings of gouging. Soon you'll have to license your ethernet ports individualy, and pay a prenium of you want to use any feature, such as QoS, or trunking. -g ________________________________________ From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Klaus Steden [klaus-s@moving-picture.com] Sent: Thursday, March 11, 2010 6:36 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives When I was doing HPC storage research (Lustre, specifically), I found that Extreme switches (Summit series) consistently outperformed the competition (Cisco, Juniper, and others) by sometimes as much as 5-10%, and the Extreme was able to reliably get me about 118 MBps every time, and was giving me as much as 95% of theoretical max without any tweaking. As always, YMMV, but I've found Extreme to be extremely reliable in terms of switch throughput, and it comes without the "Cisco" sticker premium. There are other great performers out there as well (Foundry is one of them), but I didn't get a chance to punish one the same way so unfortunately I don't have numbers handy. Klaus On 3/11/10 3:23 PM, "Tom Taylor" etched on stone tablets: > Everything is in an HP 2900 - 2 x 1Gb trunks into each cluster. (Soon to be > Brocade or Extreme 10Gb) > > Seems like the interfaces are not happy. > > > Port Total Bytes Total Frames Errors Rx Drops Tx Ctrl Limit > ------- -------------- -------------- ------------ ------------ ----- ------ > 1-Trk1 2,512,407,710 2,727,621,561 86,495 11,466,568 off 0 > 2-Trk1 3,123,561,267 1,198,521,278 18,014 8,707,104 off 0 > 3-Trk2 798,322,066 982,377,621 49,155 11,695,269 off 0 > 4-Trk2 53,405,790 2,926,838,905 163,569 14,461,588 off 0 > 5-Trk3 553,600,861 3,618,989,149 82,960 7,622,094 off 0 > 6-Trk3 1,887,102,210 4,292,551,184 60,317 6,679,635 off 0 > 7-Trk4 1,137,517,211 3,081,662,500 43,361 8,464,187 off 0 > 8-Trk4 3,503,697,323 3,465,088,585 85,276 9,135,323 off 0 > 9-Trk5 2,296,075,992 1,128,920,646 72,032 7,489,051 off 0 > 10-Trk5 2,879,550,610 2,600,747,860 54,721 8,322,167 off 0 > 11-Trk6 2,748,755,020 3,165,756,973 24,034 183,152,905 off 0 > 12-Trk6 1,611,690,357 2,354,340,713 9430 64,319,051 off 0 > > Thanks everyone for the test results! > > Cheers, > Tom > > > -----Original Message----- > From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com > [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf > Of bill@yuco.com > Sent: Thursday, March 11, 2010 6:19 PM > To: discuss@studiosysadmins.com > Subject: Re: Researching: Isilon and alternatives > > > Out of curiosity, what type of switches are you using? > >> -----Original Message----- >> From: Tom Taylor [mailto:taylor@the-mill.com] >> Sent: Thursday, March 11, 2010 01:10 PM >> To: discuss@studiosysadmins.com >> Subject: RE: Researching: Isilon and alternatives >> >> I've got a quick one, I'd like to test out our Isilon cluster. >> >> It seems as if I'm getting low performance at the moment and I was wondering >> if anyone else out there would be able to run a simple DD to compare. >> >> Here's my output from 2 different machines. >> >> >> [root@flame-09-ny-courtney Tom]# dd if=/dev/zero of=test3.2gb bs=1M >> count=2096 >> 2197815296 bytes (2.2 GB) copied, 47.8699 seconds, 45.9 MB/s >> >> [root@ny3d-24 Tom]# dd if=/dev/zero of=ny3d.4gb bs=1M count=20096 >> 21072183296 bytes (21 GB) copied, 432.756 s, 48.7 MB/s >> >> I'm currently running a 6 node 6000x cluster on version 5.5.4.21. >> >> If anyone else could send me their output I'd appreciate it. >> >> >> In related news, they've just announced a product called InsightIQ. It's a VM >> that shows incredible statistics about the cluster. I suggest chatting to >> your Isilon rep and getting a demo. >> >> Cheers, >> Tom >> >> >> >> >> -----Original Message----- >> From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com >> [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On >> Behalf Of Sven Nielsen >> Sent: Tuesday, March 09, 2010 6:25 PM >> To: discuss@studiosysadmins.com >> Subject: Re: Researching: Isilon and alternatives >> >> This is one thing I've been asked for a few times; the problem is that >> NAS systems have their own filesystem on the NAS head, then present >> the files of that filesystem out via NFS or CIFS (or whatever file- >> based protocol). >> >> A fiber channel interface only knows how to speak blocks (via SCSI >> commands); so there has to be a driver on top of that to implement a >> filesystem (ext3, NTFS, CVFS, etc). Additionally, while the NAS head >> manages competing requests for the same blocks, there's nothing like >> that when you talk to a disk via Fiber Channel -- the disk will >> happily take a write from one system that overwrites a change just >> made by a different system. If one system decides to put a file where >> the previous system just extended a directory entry, then your >> filesystem is corrupt. >> >> Even NDMP over fiber channel is similar; the NDMP server (which >> understands the filesystem's layout) is telling the NDMP tape device >> to read blocks in a particular order from the disk. >> >> The only way for BlueArc, NetApp, etc. to present the same view of a >> volume via fiber channel or iSCSI that they do via NFS or CIFS would >> be if they: >> a) released a windows/linux/mac/whatever driver that implements WFS or >> WAFL, their on-disk filesystem type, and >> b) set up these drivers and the NAS head to know how to communicate to >> make sure that only one system is modifying a disk block at any one >> time (which CVFS/StorNext handles for you). >> >> It gets to be a can of worms, and (at least for now) if you need fiber >> channel access to disk for multiple clients, StorNext seems to be the >> best combination of FCP-and-network access, since you can connect an >> FCP client to the filesystem, and have it re-serve the filesystem via >> NFS, CIFS, or StorNext LAN client. There's also filesystem software >> like (I think) GPFS, Lustre, and others, that I think do similar >> things to StorNext/CVFS, but I haven't explored those too much. >> >> If there are other and better options, I'd love to hear about them -- >> anything to make the bits move faster. }:> >> >> -Sven >> >> On Mar 8, 2010, at 3:18 PM, J. J. Franzen wrote: >> >>> Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct >>> FC access to their volumes? I know Blue Arc does not and am curious >>> if anyone else does. I would so love to do away with NDMP for my >>> backups and be able to do true incremental backups like in the good >>> ol days... >>> >>> J^2 >>> >>> On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote: >>> >>>> We're a small TV-commercial shop (3D/compositing, 20 people, with >>>> 50 or so >>>> machines, incl. 4 flames) that is outgrowing our home-built Linux >>>> NAS. I'm >>>> looking for something scalable with high-bandwidth links for our >>>> compositing >>>> nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network. >>>> >>>> >>>> -----Original Message----- >>>> From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com >>>> [mailto:studiosysadmins-discuss- >>>> bounces@mailman.studiosysadmins.com] On >>>> Behalf Of Barry Robison >>>> Sent: March-08-10 5:18 PM >>>> To: discuss@studiosysadmins.com >>>> Subject: Re: Researching: Isilon and alternatives >>>> >>>> On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin wrote: >>>>> I've been looking into Isilon storage and I'm quite impressed with >>>>> their >>>>> sales-pitch. Can anyone share their experiences with Isilon >>>>> products or >>>> any >>>>> other products that compete/compare with them? >>>>> >>>> >>>> Hi Ian, >>>> >>>> What is your budget and expected workload? >>>> >>>> We evaluated both Isilon and BlueArc, with BlueArc winning our >>>> business. For our expected artist count (200) and farm size ( ~6k >>>> cores ), we liked BA better. We were sold on the data tiering, >>>> disaster recovery, and throughput the hardware architecture allows. >>>> From other shops we heard that BA was the only thing saturating >>>> 10GbE. >>>> We had issues getting Isilon working on 10GbE at all... >>>> >>>> At a previous facility I worked at we used a small Isilon cluster ( 4 >>>> nodes ) quite happily. I think when you get up to the number of nodes >>>> ( and accelerator nodes ) you need for a large cluster, some problems >>>> can be expected. >>>> >>>> >>>> Good luck, >>>> -Barry >>>> _______________________________________________ >>>> StudioSysAdmins-Discuss mailing list >>>> StudioSysAdmins-Discuss@mailman.studiosysadmins.com >>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >>>> >>>> _______________________________________________ >>>> StudioSysAdmins-Discuss mailing list >>>> StudioSysAdmins-Discuss@mailman.studiosysadmins.com >>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >>> >>> _______________________________________________ >>> StudioSysAdmins-Discuss mailing list >>> StudioSysAdmins-Discuss@mailman.studiosysadmins.com >>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >>> >> >> _______________________________________________ >> StudioSysAdmins-Discuss mailing list >> StudioSysAdmins-Discuss@mailman.studiosysadmins.com >> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >> >> v1 >> http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&a >> mp;mt=8 >> If you experience problems downloading, please use the alternative link >> below: >> http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&a >> mp;mt=8 >> >> _______________________________________________ >> StudioSysAdmins-Discuss mailing list >> StudioSysAdmins-Discuss@mailman.studiosysadmins.com >> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >> > > > _______________________________________________ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@mailman.studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss > _______________________________________________ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@mailman.studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss _______________________________________________ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss _______________________________________________ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss