Network Data Management Protocol (NDMP)

NDMP is a standard that was created to allow Network Attached Storage devices that do not allow applications to run on them to be backed up remotely.

Clients

Note: You can install the NDMP feature support on either windows or a Linux client. To do this you will need to use the UPSTREAM windows configurator.

We strongly recommend that when possible that you plan on using an NDMP client on the storage server (z/Linux for a z⁄OS storage server or directly on your Reservoir storage server) to avoid double transmission of the same data. If this is not possible, you can still use UPSTREAM / SOS or the SAN Express to avoid transmission over the wire.

UPSTREAM for NDMP supports first time full backups, incremental, and differential backups.

Note: Full Merge backups are NOT supported for NDMP.

When an incremental backup is requested, only those files which have changed since the last backup are included.

Security

An NDMP server is secured and a NDMP valid user name and password must be passed. To do this a NDMP profile is created in the UPSTREAM Client Configurator that has the NDMP validation information.

Client Installation and Configuration

#

Install UPSTREAM on a Windows client and you will need to install a separate service or daemon to configure NDMP backups and restores.

This new service is needed to configure a NDMP profile to allow security validation to the NDMP server. This creation is done in the Windows client configurator.

Note: To install the NDMP feature support on either a x86 or zLINUX client, copy the *.NDMP file from the Windows client work-path to the Linux client work-path.

Note: All fields are stored as encrypted within the profile files.

NDMP Profile Name

A name to be used when performing the NDMP backup or restore.

Server IP Address

The IP address or host name of the NDMP server.

User Name

The secured user name of the user to be logged in.

Password

The password that goes with the user name to be logged in. The password must meet the requirements for an NDMP password as to length and complexity.

When you have completed entering the above information, click the SAVE and CLOSE buttons.

If you later need to modify a profile (change a password or correct a misspelling), you must stop the service first, then highlight a NDMP profile in the NDMP Profiles list box; that causes the fields below to be filled in with the values from the profile. In Windows, press the CHANGE button to modify the fields.

Backups

Using the Director

If using the Director, v7.0.00 or higher must be used.

• 129_List1_Bullet_1127530

  • Select the Target you wish to backup.

• 129_List1_Bullet_1129617• You can expand a server definition to the volume level.
  
• 129_List1_Bullet_1128216 • If you highlight a whole server/appliance and press Add Directory/Volume, that adds a single spec that specifies the backup of the entire server. Given the capacity of these devices, backup of the entire device in one job is not recommended.
  
• 129_List1_Bullet_1129990 • If you then wish to exclude a single volume once you have added a server, highlight that volume and press Exclude Directory to exclude that specific volume. You can also exclude directories or files from the volume backup.
  
• 129_List1_Bullet_1129908 • If you highlight a single volume you can press Add Directory/Volume to add just that one volume.
  
• 129_List1_Bullet_1130116 • If you do not wish to target the entire Volume, you can select a directory, or files, to backup.
  

The Differential check-box is described below under.

Incremental Backups

UPSTREAM supports full, incremental, and differential backups. The first backup needs to be a first time full backup. Following the First time full, all subsequent backups must either be incremental backups or differentials.

The default NDMP behavior forces every 9th incremental to actually be a differential. Optionally you can control the frequency of differential backups. To enable this feature ensure the box labeled Level Based Incremental Backup is UNCHECKED during the configuration.

Note that since the level ends at 9, the system automatically does a consolidated differential at that point. If you wish to force the differential at some other time, check the Differential check-box in the NDMP tab of the Options button. If specifying the parameter, the name is DIFFERENTIAL and if set to Y, a differential will be done rather than a regular incremental. Note that a differential resets the count back to one which allows you to continue doing incremental backups.

Retention

You will need to manage your retention for these NDMP backups differently than all other UPSTREAM backups. If using GDGs to control retention schemes you will need to retain a ALL your incremental backups back to and including your last full backup, or differential.

Note: For NDMP backup the differential is treated like a incremental backup for retention policy's.

Thus most users will do fulls infrequently (monthly or perhaps even less often, but at least once per year); Differentials, weekly, and incremental backups (Incremental backups without the Differential option set) daily.

You can see if a backup is a differential in the Profiles tab Type column or in the Restore tab, the icon is similar to an incremental, but with a ‘D’ in the cup.

Because the full is taken less frequently than is usual in UPSTREAM and incremental backups and differentials depend on it so heavily, it is strongly recommended that the full be protected by duplicating it with a vault.

Note: In Reservoir 3.9.0 and above, for the purposes of retention, differentials are treated like fulls. But since they are not fulls, the full that they depend on is never deleted unless all of the incremental backups and differentials that depend on it are also deleted. See the Reservoir 3.9 manual for additional details.

Using a Batch Job

Performing an NDMP backup is similar to performing a standard UPSTREAM backup. You must specify at least the following:

• 129_List1_Bullet_1108145 • For the overall parameters:
  
  • 129_List2_Bullet_1108179 – Backup profile to identify the data in the backup (parameter: BACKUPPROFILE).
    
  • 129_List2_Bullet_1108258 – The NDMP profile name defined above (parameter NDMPPROFILE). Backup type. Since NDMP controls the contents of the backup.
    
  • 129_List2_Bullet_1140301 – Backup type there are only fulls (parameter: MERGE 1 or 3, which are the same) or incremental backups (parameter: MERGE 2).
    
  • 129_List2_Bullet_1108364 – Storage type (parameter: STORAGETYPE 2 for disk and 3 for tape).
    
  • 129_List2_Bullet_1108411 – Action type (parameter: ACTION 1 for backup and 0 for restore).
    
  • 129_List2_Bullet_1116311 – The NDMP parameters listed below are optional.
    
• 129_List1_Bullet_1101882 • For repeating parameters (separated by SPECNUMBER):
  
  • 129_List2_Bullet_1108625 – The file specification (parameter: FILES). Most users start with \\. which indicates the entire NAS device. However, you can specify each NDMP (NAS) volume or files in the volume independently. On Windows, for example, to backup the /TestDir on the /vol/vol0 volume, specify:
    

FILES \\.\vol\vol0\TestDir\*.*

For UNIX the same specification would be:

FILES /vol/vol0/TestDir/*

Note: In Windows and in UNIX, case is sensitive and must be correct in NDMP backup and restore specifications past the volume level.

• 129_List2_Bullet_1125038 – The plugin (parameter: PLUGIN NDMP)
  

Thus a simple example JCL for a complete full backup to tape might be:

{{id name="NetworkDataManagementProtocol(NDMP)-129_JCL_1109609"/}}

{{id name="NetworkDataManagementProtocol(NDMP)-1109609"/}}

BACKUPPROFILE SERVER
NDMPPROFILE IBM
MERGE 1
STORAGETYPE 3
ACTION 1
SPECNUMBER 1
FILES ~\~\.
PLUGIN NDMP\\As the backup proceeds, NDMP messages are written to the upstream log. You may notice that multiple backups occur internally to a single backup request. This is normal and should be expected.

Also, backups are performed without performing an initial backup scan. Scans of this type are not supported in NDMP. Similarly, the definition of the files to be included in an incremental are defined in NDMP and are outside the control of the UPSTREAM client.

Size estimates logged by NDMP are just that, estimates. The actual amount of data written may be substantially different, particularly on incremental backups. Again this is normal and should be expected.

22.4 Restores

Using the Director

Unlike the backup, the NDMP Profile for the restore is automatically added when you select the profile to restore. It can be changed in the Options button after adding a spec.

Otherwise, a restore with NDMP is very similar to a restore in your existing environment.

Using a Batch Job

Restores are specified similarly to backups. The primary difference is that you typically restore only a single file, directory, or volume.

To see what is backed up, you can use the Director which displays the volumes, directories, and files that were included in the backup. Note that the default restore method is a Direct Access Restore (NDMPDAR=Y). You should only turn it off when restoring a full volume (see below).

Thus, to restore just the /vol/vol0/TestDir directory in Windows specify JCL similar to the following:

{{id name="NetworkDataManagementProtocol(NDMP)-129_JCL_1134027"/}}

{{id name="NetworkDataManagementProtocol(NDMP)-1134027"/}}

BACKUPPROFILE SERVER
NDMPPROFILE IBM
ACTION 0
SPECNUMBER 1
FILES ~\~\.\vol\vol0\TestDir\
PLUGIN NDMP\\Some things to note above:

• 129_List1_Bullet_1134035 • You do not need to specify MERGE or STORAGETYPE. Those are backup specific parameters.
  
• 129_List1_Bullet_1111546 • The FILES specification is case sensitive with \TestDir\. Note the trailing backslash which indicates that you wish the contents of that directory restored.
  

Restores are performed in multiple phases. You should expect a restore to first identify the files to recover and then recover the blocks where those files are stored. Since this might be in multiple backups, there may be many phases before all of the data is completely restored.

Non Direct Access Restores

By default, restores are Direct Access Restores (DAR). UPSTREAM does a restore of the list of the files to be restored and then uses that to recover, just the blocks needed. However, if you need to recover a full volume, you should use a non-direct access restore (NDMPDAR=N).

When restoring an Entire Volume, (for example for Disaster Recovery) use Non-Direct Access. This has a performance benefit as it is an efficient block Level restore.

NDMP Overall Optional Parameters

Valid NDMP overall (non-repeating) optional parameters include:

DIFFERENTIAL

Only used when performing an incremental, if set to Y forces a consolidated differential (changes since the full) and resets the incremental counter.
Default: N (normal incremental)

NDMPBLOCKSIZEMB

Data is transferred from the NDMP server as a single stream. To aid in individual file recovery, it is broken down by UPSTREAM into individual files of the size specified here. Most users leave the default. However, if you have a really fast server and fast network you may wish to increase the size of this value. It is a backup only value.

Default: 10 (MB)

NDMPDAR

This stands for Direct Access Restore and should be used whenever you are recovering anything smaller than a full volume. Specify Y or N

Default: Y

NDMPDARMAXFILES

When performing a Direct Access Restore, each file that is to be recovered is saved and then requested later in a block restore. The storage server has a maximum number of files which can be requested in a single restore and the default of 16384 fits within that number. Most users do not need to change this value; only change it based on the request of technical support.

Default: 16384

NDMPLOCALPORT

NDMP actually uses two separate connections from the UPSTREAM client to the NDMP server. The first is initiated by UPSTREAM. The second is requested by UPSTREAM but is opened by the NDMP server. UPSTREAM uses the port specified here for this connection (which is used for block data). The default of 10000 can be left by most users. If you are running simultaneous operations, and the port is in use, it will be dynamically incremented until an open one is found. So most users will not need to adjust this.

Default: 10000

NDMPPROFILE

The NDMP profile configured in the UPSTREAM Client Configurator (see above) This is a required parameter, but is listed here as well.

NDMPSERVERADDR

By default the UPSTREAM client uses the server address configured in the NDMP profile definition. However, you can override this value here.

NDMPSERVERPORT

NDMP has a well known port for the UPSTREAM client to connect to (10000). However, if you override the port in the NDMP server address, you need to specify the port number here.

NDMPTIMER

Actually an environment variable, how long the client will wait for the NDMP data session to start, in seconds. The default of 7200 may need to be increased ONLY if this is an incremental of a very large, very busy system.

Disaster Recovery

When recovering an NDMP system in a disaster environment, we strongly recommend that you have an NDMP server from the same vendor and with the same version of the operating system. Disaster recovery to systems of different vendors or different operating system levels are not supported.

The vendor will often make recommendations on disaster recovery. For example, with Network Appliance, they recommend fast, non-history restores (See Chapter22 “Non Direct Access Restores” above). This implies that in system restore you create volume definitions with the same names and at least the same sizes.

In all cases the NAS vendor’s documentation needs to be followed closely.

When you create the machines, we strongly recommend that you use the same information for the machine and the machine’s login as you did for the original system. If you do this, the original NDMP definition can be used. In this case you can simply specify the NDMP Profile name from the original backup (which is filled in automatically in the Director) and if the NDMP profile definition does not exist on the target machine it will be recovered from the original backup and used (reducing by one the things you need to worry about).

Otherwise, on the UPSTREAM Client target machine, you can use the configurator to redefine the NDMP Profiles.

Using Unicode

Unicode is a standard designed to allow the display of characters outside of the 8-bit ASCII character set implemented in several ways. NDMP does not support Unicode as part of the standard. However, an extension to ASCII was been used by a number of vendors named UTF-8 which uses 8-bit characters to display file names which actually need more resolution than 8-bits can provide.

If you have unicode characters you should activate UTF-8 before doing the backup. New versions of most systems already have it activated.

To activate unicode, for IBM or Network Appliance, set the language to include a UTF-8 suffix. You can see the defined language for a volume by specifying from the NAS administrative console (usually accessed by using SSH or Telnet to the server itself:

{{id name="NetworkDataManagementProtocol(NDMP)-129_JCL_1132723"/}}

{{id name="NetworkDataManagementProtocol(NDMP)-1132723"/}}

vol lang <volume name>\\For example you would enter for vol1:

{{id name="NetworkDataManagementProtocol(NDMP)-129_JCL_1132761"/}}

{{id name="NetworkDataManagementProtocol(NDMP)-1132761"/}}

vol lang vol1\\...and it might respond with:

{{id name="NetworkDataManagementProtocol(NDMP)-129_JCL_1132837"/}}

{{id name="NetworkDataManagementProtocol(NDMP)-1132837"/}}

Volume language is en_US (English (US))\\That would generally mean that UTF-8 is not on. To enable it, enter:

{{id name="NetworkDataManagementProtocol(NDMP)-129_JCL_1132967"/}}

{{id name="NetworkDataManagementProtocol(NDMP)-1132967"/}}

vol land vol1 en_US.UTF-8\\If you change a volume mapping you generally need to schedule a reboot.