NetApp SnapMirror


NetApp SnapMirror is a pretty easy utility to use, but offers great power. NetApp snapview snapshots a LUN to a given destination either on a demand basis or a schedule. You can set this up via the UI or you can run commands via SSH. In the following example, we will be setting up snapview on a LUN.

However, there is one key concept here. You can dedicate different interfaces to handle different roles. So if you have your production data accessible on say e0a, then consider setting up your snapshot traffic on e0b. This way you do not flood the interface or the upstream network components. Also you are spreading the load.

Here’s a quick overview or some key concepts you need to consider:

We have a volume

ontap> vol status cloud01

Volume  State
cloud01 online

We have a lun

ontap> lun show

/vol/cloud01/cloud01-na-lun01    100g (107374182400)  (r/w,online,mapped)

And this is our satatus of SnapMirror

ontap> snapmirror status Snapmirror is on. 
ontap> snapmirror destinations snapmirror destination: no known destinations

We need to edit the /etc/snapmirror.conf file and place some information in ro configure a snapmirror job. Here is a good article on what options you can use.

The following snapmirror.conf entry indicates that filer ontap1′s volume cloud04_mirror will mirror volume cloud04 via the ontap0-gig interface. The ontap0-gig interface is whatever IP address ontap1 can resolve that name to. In this case, it might be a gigabit ethernet link on filer ontap0. The mirror is updated at 9:30 a.m., 1:30 p.m., and 7:30 p.m., Monday through Friday. The asterisk means that the data replication schedule is not affected by the day of month; it is the same as entering numbers 1 through 31 (comma-separated) in that space. It is actually similar to a cron format.

ontap0-gig:cloud01 ontap1:cloud01_mirror – 30 9, 13, 19 * 1, 2, 3, 4, 5

The important part here is you can tell snapmirror.conf the interface name to send the snapmirror replication traffic down. You may also need to set the /etc/snapmirror.allow to allow snapview connetions, otherwise you may see a connection denied issue.

Here is the commands you need to look at

ontap0> wrfile /etc/snapmirror.allow ontap1

This will configure ontap0 to allow snapmirror connections to ontap1. As long as the name resolution is set correctly, it will traverse the interface as generic IP routing will take care of the traffic flow.

The other iimportant thing you can consider is the bandwidth shaping. The dash (–) in the command above, which is actually at the argu_ments field location, indicates that both the kbs and restart arguments are set to default.

So what is the ksb?

Taken from the link above (I said it was good) is this extract:

The value for this argument specifies the maximum speed (in kilobytes per second) at which SnapMirror data is transferred over the network. The kbs setting is used to throttle network bandwidth consumed, disk I/O, and CPU usage. By default, the filer transfers the data as fast as it can. The throttle value is not used while synchronously mirroring.

You can set it something like this:


This means the transfer speed is set at a maximum rate of 2, 000 kilobytes per second.

Asyn, Sync or semi sync

In async mode, snapshot copies of the volume are created periodically on the source. Only blocks that have changed or have been newly created since the last replication cycle are transferred to the target, making this method very efficient in terms of storage system overhead and network bandwidth.

Sync mode sends updates from the source to the destination as they occur, rather than according to a predetermined schedule. This helps data written on the source system to be protected on the destination even if the entire source system fails. NVLOG forwarding and consistency point (CP) forwarding are used to keep the target completely up to date. NVLOG forwarding enables data from the write log that is normally cached in NVRAM on NetApp storage to be synchronized with the target. Consistency point forwarding keeps the on-disk file system images synchronized.

Semi-sync mode differs from sync mode in two ways. Writes to the source aren’t required to wait for acknowledgement from the target before they are committed and acknowledged, and NVLOG forwarding is not used. These two changes speed up application response with only a very small hit in terms of achievable recovery point objective (RPO).

SnapMirror status?

We have can see the status through both the CLI and the UI. Here is the CLI

ontap0> snapmirror status
Snapmirror is on.
Source         Destination            State          Lag       Status
ontap0:cloud01 ontap1:cloud01_mirror  Snapmirrored   00:00:29  Idle

Here you can see the command snapmirror status shows the output of the status of the SnapMirror replication. Pretty simple right?

The UI shows a little more information as shown here:

Nice images 😦 but you get the drift.

There lots more to this and maybe I will add to this blog at a later date, but you can see how SnapMirror can provide real time data by using synchronous mirroring for your volumes. Also, using methods of traffic shaping and IP routing, you can traverse your snapmirror vols over a specific interface. Lots of good options. Enjoy!

Puppet integration with NetApp

I was look at the puppet forge the other day and noticed that NetApp integrates with Puppet which made me smile. Take a look here:

I have copied some of the readme notes. Here is an extract

NetApp operations

As part of this module, there is a defined type called ‘netapp::vqe’, which can be used to create a volume, add a qtree and create an NFS export. An example of this is:

netapp::vqe { 'volume_name':
ensure        => present,
size          => '1t',
aggr          => 'aggr2',
spaceres      => 'volume',
snapresv      => 20,
autoincrement => true,
persistent    => true

This will create a NetApp volume called ‘v_volume_name’ with a qtree called ‘q_volume_name’. The volume will have an initial size of 1 Terabyte in Aggregate aggr2. The space reservation mode will be set to volume, and snapshot space reserve will be set to 20%. The volume will be able to auto increment, and the NFS export will be persistent. To be honest, that is awesome if you need to build up and automate say infrastructure deployments.

I have used many auto deployments tools in my time, but I have never seen such a great adoption as I have done with Puppet. This just proves it. Why is this? Well, I think the main reasons are that puppet is open source and it doesn’t require you to lock your self in with a particular product, i.e. it runs on CentOS, WIndows, Ubuntu etc. It is also very flexbile and you code it in a way that doesn’t require integration with the target node. So in the example, NetApp is completely unaware of puppet, so NetApp needs no integration. This is because puppet can interact with the NetApp Manageability SDK Ruby libraries. How cool is that?

A great example of how good opensource can really be.

Hoorah to Puppet and NetApp!




NetApp storage solutions

I have worked with many different storage solutions really since the beginning of my journey with VMware, which was at the end of 2005 with VMware ESX 2.5, no virtual center – imagine that!. The first SAN I worked with still stays close to my heart as one of the best products out on the market, Data Core SAN Symphony. It was a great virtualisation SAN where you plugged in vendor neutral storage arrays in to a physical server, which was running Windows Server and the DataCore SAN Symphony management software. This in turn then managed the vendor neutral storage arrays. It offered both iSCSI and Fabric and at the time, it was a really great product to have got involved with. I have used many other products, including the EMC’s CX-120, the DS Series from IBM and mainly over the last few years, the Dell Equallogics PS series. I’ve also built my own SAN using Solaris ZFS and using just plain old CentOS with iscsi targets and tgtadm, running off big disks on a buinch of super micros servers. I also use the Nextenta community edition in a lab which, out of the box, is pretty darn good. However, my other favourite SAN technology that I have used is NetApp. It just oozes a quality and what I like is that it seems to have taken all the good features from all of the other SANs I have used and put them altogether.

I was given a chance to use NetApp when I was working for a large service provider down under, however it didn’t make the cut due to the price points and also due to the fact that it was too difficult to migrate from our existing SAN that was running our cloud offering at the time, so we ended up persevering with our existing solution and carried on ploughing our investment into what was a flakey product at the time (no names mentioned).

One of the ways you can get your hands dirty with NetApp is to run the evaluation product. A great colleague of mine, @wadeis, taught me all the tricks there is to know with the NetApps. Here are a couple of key offerings the NetApp solution has:

The NetApp Data ONTAP 8 Operating System

This is the operating system for your storage devices.. You can run this in 2 modes, cluster mode and 7 mode. It runs on FreeBSD, like most provders, Dell, IBM etc (Juniper run also FreeBSD on the network devices) so its a popular choice.. It also has a great command line utility which enables which seems to be very popular. To manage ONTAP, you need to run System Manager which runs on both Windows or using Linux and its a nice UI. You connect to your ONTAP device and the have all the management tools available to you depending on the license you have. Here is a screen shot of System Manager running on Windows managing an ONTAP device.

It pretty nicely laid out. There are a few key concepts with ONTAP which I want to run through:


NetApp Aggregates is a way of aggregating your disks from disk arrays together to provide you raw usable disk space. Then you assign logical volumes on top of this raw space. It can hold multiple disk from different disk array but it is highly recommended to use the same type of disk. I.e., don’t mismatch SAS with SATA, 300GB with 144GB for example. Keep them the same, but the key is you can have many spindles that make up an aggregate. There are some limitations you need to consider, for example, the maximum aggregate size.

Here are some concepts you’ll need to undertstand when mangaging your aggregates

  • Benefits of keeping your RAID groups homogenous for disk size and speed
  • What types of disks can be used together (FCAL and SAS, SATA and ATA)
  • How to add disks to aggregates from a heterogenous disk pool
  • The requirement to add disks owned by the same system and pool
  • Best practices for providing hot spares


Qtrees stand for quote based trees. It is a concept that is confusing to start with, but you need not overthink how qtrees work. It stands for quota based trees where you can assign qtrees to volumes and slice up quota’s for resources created under the volume. It’s a way of partitioning volumes up in to quota’s for various purposes.

For example, you may create a volume, vol01, and share that out using cifs. Then you create 2 qtree’s, one called qtree_one and the other qtree_two. You can set quota’s on both the qtrees but still share out the volumes. If you lock down a user to only be able to use a certain amount of space in qtree_one, well they can’t add more than their set quota. Qtree_two can have a different quota to qtree_one. You would need to have the ontap device connected to your domain so it can pick up domain groups and user to apply the quotas to.

Another way of using quotas on a volume to quota how large your LUNs can be. For example, you might want to assign volume space to various database administrators and allow them to create and manage their own LUNs. You can organize the volume into qtrees with quotas and enable the individual database administrators to manage the space they have been allocated.

If you organize your LUNs in qtrees with quotas, make sure the quota limit can accommodate the sizes of the LUNs you want to create. Data ONTAP does not allow you to create a LUN in a qtree with a quota if the LUN size exceeds the quota.

I created a volume with 100GB space and then created a qtree under the volume with a quota limit of 10GB. I then tried to create a 10GB LUN and gto this error:

It rounded down the LUN size to 9.96GB but it shows how you can manage space allocation for LUNS using qtrees.


Wafl is the file layout used by the netapp filers. It stands for write any file location. Apparently it is not classifgied as a file system but does act like one. WAFL supports all different kinds of storage filsystems. For example, it can handle CIFS for window and UNIX shares, NFS for network file system shares and block base storage for iscsi and FC. So it need to handle a few different types of files. WAFL is best thought of a tree of blocks and at the root of the tree if the root inode. The root inode describe the inode file and the inode file describes the rest of the files in the file system, including bthe block-map and indoe-map files. When WAFL loads, it need to locate the root inode, so this needs to be in a fixed located, which is the only exception to the write anywhere rule.

Here is a diagram that shows how a root inode tree can be made up.

Now, one of the greatest benefits of using this type of technology is how snapshots work. NetApp can snapshot LUNs instantaneously and it does this by copying the the root inode. If a snapshot or clone changes a block, it remaps the new block in the inode tree, as if it was l.ike a new branch. See this:

So this give NetApp some really great flexibilty when it comes to snapshot s and cloning techniques.

So it’s now time to discuss some other tools that NetApp uses to help DR and backups

NetApp SnapShot

NetApp SnapShot software enables you to protect your data with no performance impact and minimal consumption of storage space.

NetApp Snapshot technology enables you to create point-in-time copies of file systems, which you can use to protect data—from a single file to a complete disaster recovery solution.

SnapShot key points:

  • Perform instant backups by copying the inode root tree.
  •  You can have up to 255 snapshot copies per volume
  •  You can combine other technologies such as SnapMirror to build a data protection solution


NetApp SyncMirror ensures your data is available and up-to-date at all times. By maintaining two copies of data online, SyncMirror protects your data against all types of hardware outages. This is similar to SAN Symphonies mirrored volumes.

The above is a good diagram on how SyncMirror can work. Its best the have 2 arrays which copies its blocks of data destined for the volume to multiple arrays, therefore you can los a who array and the ONTAP cluster will seemlessly fail over to the other array, without any intervention. This is the same methodolgy as SAN Symphony. The only catch here, is you are duplicating you data set as you are effectively mirroring the volumes. You can by all means mirror on the the same ONTAP device.

SyncMirror allows you to split the data copies so that the mirrored data can be used by another application. This allows for doing backup, application testing, or data mining using up-to-date production data but on the passive mirrored volumes and you can perform these background tasks using the mirrored data without affecting your production environment


You can schedule snapshots in NetApp at regular intervals. This is handy again for DR stratgey. The snapshots can


Again, using WAFLs inode tree structure FlexClone is able to create instant clones. You can use these clones many different purposes, for example, setting up a dev environment, micking production data, or a test environment or a DR strategy. You cannot commit the changes of the clone back in to the producion LUN though, so once it is cloned, you can only branch the dataset.

Here are some more functionality of NetApp which also explains the benefits of using their technologies.

NetApp Data Compression

– Transparent inline data compression for data reduction

– Reduces the amount of storage you need to purchase and maintain

NetApp Deduplication

– General-purpose deduplication for removal of redundant data objects

– Reduces the amount of storage you need to purchase and maintain


– Caches NFS volumes for accelerated file access in remote offices and for server compute farms

– Improves your system’s performance, response times, and data availability for particular workloads


– Instantaneously creates file, LUN, and volume clones without requiring additional storage

– Enables you to save time in testing and development and increases your storage capacity


– Prioritizes storage resource allocation to highest value workloads on a heavily loaded system

– Provides you with better performance for designated high-priority applications


– Creates flexibly sized LUNs and volumes across a large pool of disks and one or more RAID groups

– Enables your storage systems to be used at maximum efficiency and reduces your hardware investment


– An integrated high-availability/disaster recovery solution for campus and metro-area deployments

– Enables you to have immediate data availability if a site fails


– Securely partitions a storage system into multiple virtual storage appliances

– Allows you to consolidate multiple domains and file servers

Operations Manager

– Manages multiple NetApp systems from a single administrative console

– Simplifies your NetApp deployment and allows you to consolidate management of multiple NetApp systems

 Protection Manager

– Backup and replication management software for NetApp disk-to-disk environments

– Lets you automate data protection, enabling you to have mistake-free backup

 System Manager

– Provides setup, provisioning, and configuration management of a Data ONTAP storage system

– Simplifies out-of-box setup and device management using an intuitive Windows-based interface


– Provides host-based data management of NetApp storage from Windows, UNIX, and Linux servers

– Allows you to initiate error-free system restores should servers fail


– Provides host-based data management of NetApp storage for databases and business applications

– Lets you automate error-free data restores and provides you with application-aware disaster recovery


– Enables automatic, incremental data replication between systems: synchronous or asynchronous

– Provides you with flexibility and efficiency when mirroring for data distribution and disaster recovery


– Enables rapid reassignment of disks between controllers within a system, without disruption

– Lets you load balance an active-active controller system with no disruption to data flow


– Rapidly restores single files, directories, or entire LUNs and volumes from any Snapshot copy backup

– Instantaneously recovers your files, databases, and complete volumes from your backup


– Makes incremental, data-in-place, point-in-time copies of a LUN or volume with minimal performance impact

– Enables you to create frequent, space-efficient backups with no disruption to data traffic


– Maximizes data integrity for Oracle


– Allows you to enhance the resiliency of Oracle Data-bases so they comply with Oracle HARD initiative


– Exports Snapshot copies to another NetApp system, providing an incremental block-level backup solution

– Provides you with cost-effective, long-term backups of disk-based data


– Maintains two online copies of data with RAID-DP protection on each side of the mirror

– Protects your system from all types of hardware outages, including triple disk failure

The summary for me is the NetApp are way up their for offering performance with functionality but it does come as a cost. The product reminds me of SAN Symphony which is also a failry hefty cost but done of data size.  I had real fun working with the NetApps and would stick my hand up anytime if offered another opportunity.

Have fun!