Articles with tag 'Storage'

Introduction

Please note that the following tests and results are purely for informational purposes only. The systems being compared are not "apples to apples" - each configuration being tested is in fact quite different from each other. The tests have been run over a 2 year period and you should expect better performance from new hardware and software code releases. All systems that have been tests have various hard drive configurations (drive controllers, drive speeds, storage size, and total number of spindles) as well; all the systems have different size network connections. In fact, the only thing that is consistent is the actual test being performed and the nodes that performed the actual tests.

In our case, we consider the following criteria to be the most positive - 1) Does the test complete without errors? 2) Does the performance obtained by the test appropriate for the solution required? 3) Is the total cost of ownership of the solution fit within our budget? I have not included pricing in this document as some of the tests were performed on equipment that is no longer available. Also, prices may vary in different parts of the world.

Finally, I must again stress the point that it was never the intention to see which system was the fastest or outperformed another system.  The set of tests were done on equipment we currently have (or had) in house (as a comparison for future expansion requirements) or on equipment we planned on purchasing for a specific function within our production pipeline.  For example, when testing the Isilon system, at the time, it was very important to be able to increase the capacity and/or performance in mid-production quickly without interrupting our artists.  I was worried that when I added a new node to the cluster, the AutoBalancing would decrease overall performance and had no idea how long it would take before performance would improve - by the tests below - I was clearly mistaken and I got a in increase in performance immediately.

When selecting the correct solution for your environment, you will have to take a lot of things under consideration. Especially support, maintenance and the specific features you require.

Network Storage Performance Testing

In order to test the network storage performance in our network environment, we have set up the following tests to capture the attached data. The fist test (multiple small files test) creates three (3) separate unique files from each render node with a size of 5MB, 10MB, and 20MB (TEST A and TEST B) before continuing through the remaining tests (TEST C, TEST D and TEST E). This test runs one time and reports the timing of each step. The second test (single large file test) will do the same as the first, with the exception of only creating a single file with a size of 1GB. Both the multiple small files and single large file test are run three (3) separate times on the following number of nodes: 1, 3, 4, 10, 16, 32, 64, 128, and 256. I know this seems like an odd number of nodes, but it was the best combination for the way our render nodes are connected to the CORE as well as testing the Isilon System with 4 nodes + an Accelerator for even distribution of network bandwidth.

TEST A

Create file(s) on the local disk to be copied across the network in TEST D.

-          The time required to complete Test A is not included in the calculations below as it does not use the network storage and only uses the local disk.

5MB, 10MB, and 20MB Files

dd if=/dev/sda2 of=/LOCAL_PATH/FILENAME bs=1024k count=5
dd if=/dev/sda2 of=/LOCAL_PATH/FILENAME bs=1024k count=10
dd if=/dev/sda2 of=/LOCAL_PATH/FILENAME bs=1024k count=20

-           if=/dev/sda2 is used to create a file with random contents - using /dev/zero allows cache to affect the final results

1GB File

dd if=/dev/sda2 of=/LOCAL_PATH/FILENAME bs=1024k count=1024

-          if=/dev/sda2 is used to create a file with random contents - using /dev/zero allows cache to affect the final results

TEST B

Create file(s) on the network disk to be copied across the network in TEST C and TEST E

-          A second file is created so to further eliminate the affects  of cache on the system

5MB, 10MB, and 20MB Files

dd if=/dev/sda2 of=/NETWORK_PATH/FILENAME bs=1024k count=5
dd if=/dev/sda2 of=/NETWORK_PATH/FILENAME bs=1024k count=10
dd if=/dev/sda2 of=/NETWORK_PATH/FILENAME bs=1024k count=20

-           if=/dev/sda2 is used to create a file with random contents - using /dev/zero allows cache to affect the final results

1GB File

dd if=/dev/sda2 of=/NETWORK_PATH/FILENAME bs=1024k count=1024

-           if=/dev/sda2 is used to create a file with random contents - using /dev/zero allows cache to affect the final results

TEST C

Copy file(s) from the network disk directly to the network disk.

cp /NETWORK_PATH/FILENAME /NETWORK_PATH/NEWFILENAME

TEST D

Copy file(s) from the local disk  to the network disk.

cp /LOCAL_PATH/FILENAME /NETWORK_PATH/NEWFILENAME

TEST E

Copy file(s) from the network disk to the local disk.

cp /NETWORK_PATH/FILENAME /LOCAL_PATH/NEWFILENAME

Amount of Network Data Accessed

During the tests the following amount of data is transferred across the network. Using these and the time calculations we can calculate the actual transfer speeds obtained.

Multiple Small Files Test (5MB, 10MB, and 20MB Files)

Large File Test (1GB File)

Network Access Totals:

The combined data transferred across the network from the combined running of all tests.

Node Result Data Points:

The data collected from the combined running of all tests.

Tested Configurations

Currently we have tested several filesystem solutions with the above mentioned tests (see results below).  I would love to test more, but it sometimes becomes difficult to arrange a testing window that does not compete with our production schedules.  Each of the above tests were run three (3) separate times and the sets of statistics have been recorded for each of the below documented configurations to help determine the average overall performance:

Configuration A
Tested on July 19^th, 2007

Standard Isilon 3-Node cluster

• 3 x Isilon IQ 1920i
• 3 x 12 SATA Shelves (12x3 = 36 Spindles)
• 3 x 1GB Network Pipes (Round Robin Access)

Configuration B
Tested on July 23^rd, 2007

Standard Isilon 4-Node cluster (with Auto Balancing running in background)

• 4 x Isilon IQ 1920i
• 4 x 12 SATA Shelves (12x4 = 48 Spindles)
• 4 x 1GB Network Pipes (Round Robin Access)

Configuration C
Tested on August 4^th, 2007

Standard Isilon 4-Node cluster

• 4 x Isilon IQ 1920i
• 4 x 12 SATA Shelves (12x4 = 48 Spindles)
• 4 x 1GB Network Pipes (Round Robin Access)

Configuration D
Tested on August 4^th, 2007

Standard Isilon 4-Node cluster w/Accelerator Node

• 4 x Isilon IQ 1920i
• 1 x Isilon Accelerator Node
• 4 x 12 SATA Shelves (12x4 = 48 Spindles)
• 5 x 1GB Network Pipes (Round Robin Access)

Configuration E
Tested on August 6^th, 2007

Standard BlueArc Titan 1 System

• BlueArc Titan 1 Head
• 4 x 14 SATA Shelves (14x4 = 56 Spindles)
• 4 x 1GB Trunked Network Pipe

Configuration F
Tested on September 20^th, 2007

Standard BlueArc Titan 2 System

• BlueArc Titan 2 Head
• 8 x 14 FC Shelves (14x8 = 112 Spindles)
• 4 x 1GB Trunked Network Pipe

Configuration G
Tested on September 1^st, 2008

Standard BlueArc Titan 3 w/Stone FS

• BlueArc Titan 3 Head
• 6 x 14 FC Shelves (14x6 = 84 Spindles)
• 1 x 10GB Network Pipe

Configuration H
Tested on September 1^st, 2008

Standard BlueArc Titan 3 w/Stone FS Using File Expansion

• BlueArc Titan 3 Head w/Stone FS
• 9 x 14 FC Shelves (14x9 = 126 Spindles)
  
  • Includes Filesystem in Configuration H
  • Plus expansion 3 x 14 FC Shelves (14x3 = 42 Spindles)
• 1 x 10GB Network Pipe

Configuration I
Tested on August 23^rd, 2009

Standard Sun Storage 7410

• Sun Storage 7410 Head w/500GB Flash-Readzila
• Build 10.3.0.1-1.6 with compression turned OFF on the share
• 2 x 24 Shelves (22x2 = 44 Spindles + 2x18GB Flash-Logzilla)
• 1x 10GB Network Pipe

Configuration J
Tested on August 30^th, 2009

Standard Sun Storage 7410

• Sun Storage 7410 Head w/500GB Flash-Readzila
• Build 10.5.0.1-1.20 with compression turned ON on the share
• 2 x 24 Shelves (22x2 = 44 Spindles + 2x18GB Flash-Logzilla)
• 1x 10GB Network Pipe

Render Node Configurations

For these tests we have been using three (3) different types of render nodes performing the tests which have been evenly distributed to ensure consistent network bandwidth to the storage being tested. We currently have newer nodes available and future tests will have to use newer nodes as the older one become no longer used and taken off our network.

IRNxxxxx Nodes (168 of 168 Render Nodes used)

• IBM Blade Center HS20
• 2 x 3.2GHz Intel Processors
• 4GB RAM

• Each set of 14 render nodes connected to network by 1GB.

ORNxxxxx Nodes (14 or 14 Render Nodes used)

• IBM Blade Center LS20
• 2 x Dual Core Opteron 2.2GHz Processors
• 4GB RAM

• Each set of 14 render nodes connected to network by 1GB.

SRNxxxxx Nodes (74 of 118 Render Nodes used)

• Sun Microsystems SunFire VX60
• 2 x 2.8Ghz Intel Processors
• 2GB RAM

• Each set of 39 render nodes connected to network by 2GB trunk.

Network Configuration

            We currently are using an HP ProCurve 8212 for our CORE switch which is connected to the render nodes with 2 x 10GB fiber connections (Test Configuration: I to J). All the previous tests (Test Configuration A to H) were tested on our previous network configuration (a Foundry Fast Iron 1500 with 1 x 10GB fiber to render farm).

Summary

All systems tested completed the jobs 100% with no errors on all nodes which is great. Several years ago this was definitely not the case. I remember testing old Network Appliances arrays, IBM JFS, and Maximum Throughput solutions with a high failure rate of nodes or locking up the system completely so nothing completed at all. Luckily we've come a long way since then and it is always improving - in production - our goal is always to increase capacity, improve performance, reduce costs, and reduce management.

As I mentioned earlier, when testing the Isilon IQ 1920i series nodes, I wanted to see what the affect of AutoBalancing had on the performance of the system. I was happily surprised to find that the performance increased when a new node was added to the cluster and even happier to see that once the AutoBalancing completed, the performance increased even more.  Although an Accelerator node improved overall performance, I noticed that the nodes connecting to the Accelerator completed much faster than the other nodes, which in turn, made those complete much slower overall. For my purposes, I decided that an Accelerator node was not required as I am looking for an even throughput across all my nodes.

When testing the BlueArc Titan 1, BlueArc Titan 2 and BlueArc Titan 3 we can see the progression of the updated heads giving us better performance as advertised. The new version of their filesystem - StoneFS - increased the performance even further and gave us the ability to add drives to a storage pool when more performance (not storage) was required. Past versions of their OS did not allow this option. I was very pleased when this new feature was finally added.

The SunStorage 7410 is the only system I have tested with a large amount of FLASH memory to help speed up performance. The results were surprisingly good for the amount of hardware being used.  Although only one head may have been a little too much for 256 nodes to be hammering it all at once - you can see the performance drop on large files when going from 64 to 128 nodes.  I suspect that the model I tested would be great for performance with around 80 nodes. One thing you will notice from the second test with LZJB compression turned on for the export, performance significantly improved which is very impressive. This also allows you to have more nodes accessing the filesystem simultaneously without as much performance degradation as the first test suggests.

Overall, I've been very happy with everything that I've tested. All the systems were very easy to setup, easy to manage and have great support. In production we currently use both the BlueArc and Isilion systems heavily and we have never run into issues that have affected production or even more importantly caused a loss of data.

I'd just like to add that one of the points of these tests were to try to overload the system for worse case scenario of a real life production scenario. In reality we have 300 users and over 512 render nodes (over 2400 processors) hammering the filesystems each and every day - and - the daily load average on each of the filesystems is much less then it was during the tests.  I am confident that any one of these systems could hold up under normal production with our current size and run even more effectively in smaller studios.

Not that long ago - storage was a big concern - what if the filesystem fails - am I safe - for the past 4+ years - the only thing I worry about is the users filling it up too fast - but I can sleep at night not worrying about failures.

Network Storage Overview

The following is a brief summary of how multiple vendor network storage is currently used at the Starz Animation facility in Toronto.  Our infrastructure has grown over the past couple of years from a small 30-60 person facility with 100 render nodes to a much larger facility that currently has approximately 300 people with just over 500 render nodes.  In the past, a lot of the storage for projects was all located on local attached disk (which is probably still true in many smaller shops).  As our facility grew and we had access to bigger budgets, we were able to completely get rid of all our old server attached storage and move to a better enterprise solution as shown in the diagram below. This by no means the best or only solution, and may not work at all in your facility; however; in this document, I would like to outline some of the benefits and reasons why we chose to set up our storage in this manner.

Protection Benefits

By moving away from server attached storage, we were able to protect our data in the following ways:

• Failover protection by creating filesystems with RAID-5 or RAID-6 so we can survive multiple drives failures without interruption of service.
• Enterprise level SNAPSHOTS so we can restore lost or corrupt data quickly without having to go to tape.
• Failover protection by user of CLUSTERING

SNAPSHOTS allows us to take a "picture" of the current filesystem and go back to retrieve files at a specific point in time. We currently have SNAPSHOT rules that take these "pictures" every 3 hours for a full day, every full day for a week, and every week for 3 weeks. This allows us to retrieve files very quickly when users remove or overwrite files by mistake (which seems to happen much too often).

With multiple storage systems, we were able to logically separate the data as if flows through our pipeline to spread the load of the filesystems across different areas. We also have the extra benefit of using CLUSTER capabilities of both BlueArc and Isilon so in the case that an entire head fails; another head will take over and pick up the load automatically.

For disaster recover purposed (backups and archives), we have added a large FC array of TIER-2 storage directly to our backup server. This gives us the ability to RSYNC all of the data we usually backup to tape and keep it on-line so it can quickly be retrieved without having to go to back tape (which is obviously much slower). This server also has an ADIC i2000 tape library attached with 4 LTO-3 tape drives that run incrementals each night with fulls once per month.

When projects are complete, we simply make two archives (one for on-site and one for off-site storage) of the full project. Then we run some custom scripts that replace the internal directory pointers on all the assets (mainly Maya files) and move them into our Library (located on the Isilon cluster) so they can be re-used for future projects.  Once all are archive is complete and has been tested, we delete the project completely from the BlueArc storage arrays.

Cost of Use Benefits

By moving to an enterprise storage solution we were able to reduce our cost of ownership in the following ways:

• Daily management of an enterprise solution is much easier - we spend less than 1 hour per week maximum (usually file retrievals).
• Both BlueArc and Isilon have proven to have solid filesystems with excellent support - we have never had a disaster to date (fingers crossed).
• Allowing us to incorporate a Cross-Platform Link-Based Directory Structure so we can further tighten of our pipeline scripts and standardize file locations.
• Less links to manage and no more moving data from filer to filer to find space at the last minute like the old days

Load Distribution

Evenly distributing the load of file access across multiple filesystems is very tricky especially with the highs and lows of your renderfarm.  Many places will assign a filer a project so that everything for that specific project is located in one place. We have chosen to split projects across all filers so that no matter what project is being worked on, the filers are always used about the same amount. The number of people and machines accessing the data remain fairly constant no matter which projects we are working on so we have built our filesystems to handle the worst case scenario (example: all artists and all render nodes hitting the filesystem simultaneously).  Some filers still are worked more than others, but even at the highest load, we are working the filers well under their tested performance capabilities.  In our facility (as seen in the diagram) we have split our data as follows:

ISILON CLUSTER

This storage library contains both NFS (Unix) and CIFS (Windows) exports for our Library (Software, Media files, Non-production related documents and Assets from non-current productions for reference and possible re-use), User Windows Roaming Profiles, User Home Directories (both Windows & Unix), and  Apache Web Services (See Load Balancing Network Services). This filesystem is not accessed by the renderfarm; it is strictly for all non-production data.

BLUEARC TITAN 3-A (Production)

The storage library contains all production related data - management, textures, Maya files, etc (anything a production person or artists touches for a project). All machines read from this filesystem, however, only artists write to it.

BLUEARC TITAN 3-B (Render Cache/Render Layers)

This storage library contains all Mentalray pre-render cache as well as all rendered layers. Only the renderfarm touches this filesystem as users rarely need to access these files. By separating these files out from the other filesystems, we greatly decreased the loads on other production files. When the renderfarm kicks in at full power in the middle of the day users are no longer affected by when loading and saving their files.

BLUEARC TITAN 3-B (Compositing)

This storage library contains all final composite frames; production related media files as well as the logs created by the renderfarm. The renderfarm is the only set of machines that write to this filesystem. Final compositing frames and render logs are only accessed by a small number of users. The media files created are accessed via our web-based production system.

What is a Cross-Platform Link-Based Directory Structure?

A link-based directory structure is created completely using links so that both UNIX and WINDOWS systems can easily access files in a similar manner. Using this method, you can easily maintain tools to automatically create directories and files in a consistent nature. You also gain the ablility to move directories and/or entire projects to new physical file systems while being totally invisible to the users.

What is needed to create a Cross-Platform Link-Based Directory Structure?

Nothing is required to be purchased. All you need is a UNIX server running SAMBA with MSDFS configured.

How to set-up a Cross-Platform Link-Based Directory Structure

First we choose a common drive letter to mount under WINDOWS (in our examples we will choose N:). Similarly, for mounting drives under UNIX, we select the same name as automount drive (/n). By choosing the same path, users will be able to navigate on either WINDOWS or UNIX by using the same standard paths. This also allows conversion mapping when files needs to be moved between WINDOWS and UNIX easily.

example smb.conf

[global]
netbios name = LINKSERVER
workgroup = YOUR_DOMAIN
host msdfs = yes
security = ads (or whatever you use)
encrypt passwords = Yes
password server = YOUR_DOMAIN_CONTROLLER (or password server)
max log size = 5000
log file = /opt/samba/var/log/%m
use sendfile = no
local master = yes
os level = 3
socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
log level = 3
log file = /opt/samba/var/log/%m
utmp = yes
utmp directory = /var/log/utmp
wtmp directory = /var/log/utmp
server string = DFS Server
guest ok = yes
[windows_n]
comment = Windows N: drive
path = /export/windows_n
msdfs root = yes
writeable = Yes
create mask = 0774
directory mask = 0775
guest ok = yes

example exports

/export/unix_n *(rw,no_root_squash,sync)

So on WINDOWS we would mount the SAMBA server LINKSERVERwindows_n as N: and on UNIX we would mount linkserver:/ /export/unix_n as /n

Creating a project under the Cross-Platform Link-Based Directory Structure

For this example lets create two projects (PROJECT1 and PROJECT2) with a simple directory structure: Remember that WINDOWS and UNIX both see different links, so we need to create the real directories on the filers the files will actually be located as well as two sets of links (one for WINDOWS and one for UNIX).

On the LINKSERVER you have two directorues /export/windows_n for WINDOWS links and /export/unix_n for UNIX links.

Create default project directories

mkdir /export/windows_n/projects
chmod 555 /export/windows_n/projects (So WINDOWS users can't create files at this level - otherwise won't be seen in UNIX)

mkdir /exportt/unix_n/projects
chmod 555 /export/unix_n/projects (So UNIX users can't create files at this level - otherwise won't be seen in WINDOWS)

Note: A physical projects directory doesn't need to be created this is just a fake level.

Create project directories

On your storage create real directories (At this point lets pretend we have only 1 storage filer called STORAGE1), this should be mounted under UNIX under /n if you use automounts. For this example we only have 3 top level directories to make the examples easier, in your studio, you can break down the levels as much as you need to. The more breakdown the better if you are forced to move data a later period of the project due to space constraints. One thing to consider is that users will not be able to create files are directories at the top level, so make sure you create ALL the top level directories that you are going to use.  The reason for this is simple, since the directories are actually links, WINDOWS users and UNIX users will not be looking at the same top level so one or the other will not be able to see what is created at that level.

Directories where files will actually be located:

mkdir /n/STORAGE1/PROJECT1

mkdir /n/STORAGE1/PROJECT1/Work
mkdir /n/STORAGE1/PROJECT1/Render
mkdir /n/STORAGE1/PROJECT1/Comp

mkdir /n/STORAGE1/PROJECT2

mkdir /n/STORAGE1/PROJECT2/Work
mkdir /n/STORAGE1/PROJECT2/Render
mkdir /n/STORAGE1/PROJECT2/Comp

WINDOWS Links: (no files ever located here) (pay attention to the double backslashes)

mkdir /export/windows_n/PROJECT1
chmod 555 /export/windows_n/PROJECT1 (So WINDOWS users can't create files at this level - otherwise won't be seen in UNIX)

cd /export/windows_n/PROJECT1
ln -s msdfs:STORAGE1PROJECT1Work Work
ln -s msdfs:STORAGE1PROJECT1Render Render
ln -s msdfs:STORAGE1PROJECT1Comp Comp

mkdir /export/windows_n/PROJECT2
chmod 555 /export/windows_n/PROJECT2 (So WINDOWS users can't create files at this level - otherwise won't be seen in UNIX)

cd /export/windows_n/PROJECT2
ln -s msdfs:STORAGE1PROJECT2Work Work
ln -s msdfs:STORAGE1PROJECT2Render Render
ln -s msdfs:STORAGE1PROJECT2Comp Comp

UNIX Links: (no files ever located here)

mkdir /export/unix_n/PROJECT1
chmod 555 /export/unix_n/PROJECT1 (So UNIX users can't create files at this level - otherwise won't be seen in WINDOWS)

cd /export/unix_n/PROJECT1
ln -s /n/STORAGE1/PROJECT1/Work Work
ln -s /n/STORAGE1/PROJECT1/Render Render
ln -s /n/STORAGE1/PROJECT1/Comp Comp

mkdir /export/unix_n/PROJECT2
chmod 555 /export/unix_n/PROJECT2 (So UNIX users can't create files at this level - otherwise won't be seen in WINDOWS)

cd /export/unix_n/PROJECT2
ln -s /n/STORAGE1/PROJECT2/Work Work
ln -s /n/STORAGE1/PROJECT2/Render Render
ln -s /n/STORAGE1/PROJECT2/Comp Comp

That's all you need to get started. In WINDOWS if you can navigate to N:projects and see two projects, where as in UNIX you can navigate /n/projects and see the same directories.

Moving Your Data

Now that your projects are created in the Cross-Platform Link-Based Directory Structure you are able to move your data whenever you like and be total invisible to the users.

Above we created two projects (PROJECT1 and PROJECT2) on our single storage filer STORAGE1. Well, the projects are getting two big and STORAGE1 is running out of room, we need to move the data.

In this example, let's say we have a second storage filer available (STORAGE2).

You do have some decisions to make, do you want to move an entire project to the new filer or do you want to move data from both projects? Either way you are covered.

Let's say we decided to move just "Render" and "Comp" directories for all projects to the new storage and keep the "Work" directories where they are.

No matther what your choice is, you will have to sync the data to the new storage filer and make sure the directories are set up the same as the original.

Note: Don't touch the links until the sync in complete! Also, you will need to take the directories down for a bit to do a final sync and make sure you don't lose any data!

Updating Windows LInks:

cd /export/windows_n/PROJECT1
rm -Rf Render
ln -s msdfs:STORAGE2PROJECT1Render Render
rm -Rf Comp
ln -s msdfs:STORAGE2PROJECT1Comp Comp

cd /export/windows_n/PROJECT2
rm -Rf Render
ln -s msdfs:STORAGE2PROJECT2Render Render
rm -Rf Comp
ln -s msdfs:STORAGE2PROJECT2Comp Comp

Updating UNIX Links:

cd /export/unix_n/PROJECT1
rm -Rf Render
ln -s /n/STORAGE2/PROJECTS1/Render Render
rm -Rf Comp
ln -s /n/STORAGE2/PROJECTS1/Comp Comp

cd /export/unix_n/PROJECT2
rm -Rf Render
ln -s /n/STORAGE2/PROJECTS2/Render Render
rm -Rf Comp
ln -s /n/STORAGE2/PROJECTS2/Comp Comp

That's all you need to do. WINDOWS and UNIX users will both see the projects the same way even though the data has been physically moved.

All the above can be scripted so that a single command will create a new project, all the physical directories and both sets of links to your default storage locations.  Also, you should customize your pipeline tools so that the link-based system is used and stop any tools looking at specific mounts and/or filers.