Sponsors
Sponsor Products
Broadcom / LSI MegaRAID with NVMe support
posted by Jean-Francois Panisset  on April 14, 2017, 4:55 p.m. (10 days ago)
6 Responses     0 Plus One's     0 Comments  
Apparently Broadcom/LSI is about to release (has released?) the 94xx series of MegaRAID controllers which add support for NVMe attached drives:

https://www.broadcom.com/products/storage/raid-controllers/#tab-span1

That could make for a pretty tasty Flame framestore for instance.

JF



Thread Tags:
  discuss-at-studiosysadmins 

Response from Ergi @ April 16, 2017, 12:58 a.m.

The PCI razor cards are getting very popular, SP is putting them in all their Ultra Storage line of server. We manage to do some benchmarks on the 24 NVME Samsung PM1725. Oracle ZFS striped/raid pool with all 24 drives. We got 7.7GB/sec using CLI DD tool. The architecture of razor cards and NVME drives to PCI bus is explained very well in this doc https://www.supermicro.com/white_paper/white_paper_NVMe2.pdf. To get that type of performance on a Flame box in theory 2 or 3  NVME drives will deliver 6+6=12GB/sec sequential reads http://www.samsung.com/semiconductor/global/file/insight/2015/11/pm1725-ProdOverview-2015-0.pdf  Where the NVME are really delivering the performance is in the IOPS not necessarily in higher bandwidth because of other variables like filesystem, pci bus, cpu, OS, block size, single/multithreaded,....  You get almost x7  (tested with fio-2.1.7 in the SP engineering lab) more IOPS than the traditional  industry standards SSDs. Flame requires a single stream of playback  so the NVME in practice 2-3 of those drives will deliver  up to 12GB/sec. I also hear that Flame is multithreaded if you use stone+wire. However in practice using cli tools  I have not see anything faster than 7.7GB/sec in all my testing.

To add to what Saker  about not needing the LSI controller. If you manage to get 3 pci slots x8 available  in you Flame box, i think in the Z840 the  3 slots x8 are used for the aja and nvidia  cards but if there is another motherboard that can give you three more x8 slots u can add those NVME drives and i would strongly recommend ZFS you will get that 7.7GB/sec that i am talking about.

 


0 Plus One's     0 Comments  
   

Response from Chris Hyman @ April 15, 2017, 7:35 p.m.
Hi,
Autodesk have certified HP, and  Dell have self certified the T7910 - https://knowledge.autodesk.com/support/flame-products/troubleshooting/caas/sfdcarticles/sfdcarticles/flame-sysreqs.html
I believe the latest IBM workstations are also certified or self certified but dont appear on the list yet however I remember reading something on Facebook logic group about IBM systems and being certified...
Regards

Chris
On 15 Apr 2017, at 16:02, Saker Klippsten <sakerk@gmail.com> wrote:
This card only allows for 2-4 NVMe Drives. Which can easily saturate the single X8 slot Gen3. Its meant I believe to be used as a caching tier...and sit in front of SAS/SATA or ssd-sata drives.. not sure how that would jive with the flame.If I were  going to use NVMe storage for a flame.. I would not bother with raid card at all.. and do a software raid. and have NVMe Devices connected directly to the pci-e bus.. 
Currently for our flames we use 16 Samsung EVO SSD 2TB ( in production for 2 years so far ) or  4TB drives ( 1 year )  using hardware raid via ADAPTEC. That is Until Capacity goes up and prices come down on NVMe drives.
Also HP needs to come out with a workstation box .. ( if autodesk is going to only certify their systems ) that has at least 16 NVMe ports on it.. otherwise I would put flame on a supermicro 2U but that can only support a single p6000 and Aja Card 
https://www.supermicro.com/products/system/2u/2028/SYS-2028U-TN24R4T_.cfm


The other option of course is to just go back to the idea of a SAN.. but use 100Gig between the storage and the flame(s) via iSCSI or NVMe-OF 
-S
Sent from my iPhone
On Apr 14, 2017, at 1:52 PM, Jean-Francois Panisset <panisset@gmail.com> wrote:

Apparently Broadcom/LSI is about to release (has released?) the 94xx series of MegaRAID controllers which add support for NVMe attached drives:

https://www.broadcom.com/products/storage/raid-controllers/#tab-span1

That could make for a pretty tasty Flame framestore for instance.

JF


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe
To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Mahlon E. Smith @ April 15, 2017, 1:30 p.m.
On Sat, Apr 15, 2017, Jorg-Ulrich Mohnen wrote: > Sounds amazing and great> BUT careful with Zpool/ZFS. We tried a > high end Samsung Evo Pro 2TB set of hard drives behind the LSI 3008 > with a direct connect MiniSAS connector on the card to SATA6 to each > hard drive, and the LSI 3008 could not get above speeds of 280MB/sec. > The ZFS/Zpool at this time was directly off of a set of apt-get in > debian/ubuntu. WOULD seriously buy one and test first before making > this a standard inside your facility. We were seeing similar numbers under FreeBSD NVMe with defaults. Not sure what the linux equivalent would be, but setting ashift *before* creating the zpool: sysctl vfs.zfs.min_auto_ashift=12 coupled with enabling lz4 compression on the dataset; We went from under 300MB/sec to 3.2G/sec on sequential write streams. # zdb -C [pool] | grep ashift ashift: 12 -- Mahlon E. Smith Senior Technologist LAIKA mahlon@laika.com / 503.906.5646

0 Plus One's     0 Comments  
   

Response from Saker Klippsten @ April 15, 2017, 11:05 a.m.
This card only allows for 2-4 NVMe Drives. Which can easily saturate the single X8 slot Gen3. Its meant I believe to be used as a caching tier...and sit in front of SAS/SATA or ssd-sata drives.. not sure how that would jive with the flame.If I were  going to use NVMe storage for a flame.. I would not bother with raid card at all.. and do a software raid. and have NVMe Devices connected directly to the pci-e bus.. 
Currently for our flames we use 16 Samsung EVO SSD 2TB ( in production for 2 years so far ) or  4TB drives ( 1 year )  using hardware raid via ADAPTEC. That is Until Capacity goes up and prices come down on NVMe drives.
Also HP needs to come out with a workstation box .. ( if autodesk is going to only certify their systems ) that has at least 16 NVMe ports on it.. otherwise I would put flame on a supermicro 2U but that can only support a single p6000 and Aja Card 
https://www.supermicro.com/products/system/2u/2028/SYS-2028U-TN24R4T_.cfm


The other option of course is to just go back to the idea of a SAN.. but use 100Gig between the storage and the flame(s) via iSCSI or NVMe-OF 
-S
Sent from my iPhone
On Apr 14, 2017, at 1:52 PM, Jean-Francois Panisset <panisset@gmail.com> wrote:

Apparently Broadcom/LSI is about to release (has released?) the 94xx series of MegaRAID controllers which add support for NVMe attached drives:

https://www.broadcom.com/products/storage/raid-controllers/#tab-span1

That could make for a pretty tasty Flame framestore for instance.

JF


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

0 Plus One's     0 Comments  
   

Response from Jorg-Ulrich Mohnen @ April 15, 2017, 10:56 a.m.

Sounds amazing and great> BUT careful with Zpool/ZFS. We tried a high end Samsung Evo Pro 2TB set of hard drives behind the LSI 3008 with a direct connect MiniSAS connector on the card to SATA6 to each hard drive, and the LSI 3008 could not get above speeds of 280MB/sec. The ZFS/Zpool at this time was directly off of a set of apt-get in debian/ubuntu. WOULD seriously buy one and test first before making this a standard inside your facility.

Jorg


0 Plus One's     0 Comments  
   

Response from Mussolini @ April 15, 2017, 9:43 a.m.

They already releaded it. Lets say it will be pretty tasty for everything you want, specially storages. 4GB of memory and NVMe support are awesome improvements.

Can't wait to touch one of those.

 

 


0 Plus One's     0 Comments