Netapp V-series tape drive path lost on rebooting Tape library

Issue : When you reboot the Tape library(Quantum i6k in our case) the Netapp v-series nodes tape drive paths are lost. All backup jobs will fail.

netappnode> storage show tape
No tape devices found.

Solution :

[1] Reboot the netapp node(not recommended).

[2] Disable/Reenable the fc adapter ports on the netapp nodes that are used for backups.

Steps to fix as follows :

(a) First list the fc adapter ports that are configured for backups.(1a & 1c in our case)

netappnode >fcadmin config                   Local
Adapter Type      State                  Status
—————————————————
0a   initiator  CONFIGURED             online
0b   initiator  CONFIGURED             online
0c   initiator  CONFIGURED             online
0d   initiator  CONFIGURED             online
  1a   initiator  CONFIGURED.            online
  1b   initiator  CONFIGURED.            offline
1c   initiator  CONFIGURED.            online
  1d   initiator  CONFIGURED.            offline
11a  initiator  CONFIGURED.            offline
11b  initiator  CONFIGURED.            offline

(2) Use sysconfig to verify that you have the right adapter – there should be NO devices showing under them.

netappnode> sysconfig

        slot 1: FC Host Adapter 1a
slot 1: FC Host Adapter 1b
slot 1: FC Host Adapter 1c
slot 1: FC Host Adapter 1d
(3) Disable the tape drive ports and then Enable them :
netappnode> storage disable adapter 1a
netappnode> storage disable adapter 1c
netappnode> storage enable adapter 1a

netappnode> storage enable adapter 1c

(4) Check that the configured drives are back:

netappnode> sysconfig -t
    Tape drive (Switch ip)  IBM LTO 5 ULTRIUM
rst4l  –  rewind device,        format is: LTO-3(ro)/4 4/800GB
nrst4l –  no rewind device,     format is: LTO-3(ro)/4 4/800GB
urst4l –  unload/reload device, format is: LTO-3(ro)/4 4/800GB
rst4m  –  rewind device,        format is: LTO-3(ro)/4 8/1600GB cmp
nrst4m –  no rewind device,     format is: LTO-3(ro)/4 8/1600GB cmp
urst4m –  unload/reload device, format is: LTO-3(ro)/4 8/1600GB cmp
rst4h  –  rewind device,        format is: LTO-5 1600GB
nrst4h –  no rewind device,     format is: LTO-5 1600GB
urst4h –  unload/reload device, format is: LTO-5 1600GB
rst4a  –  rewind device,        format is: LTO-5 3200GB cmp
nrst4a –  no rewind device,     format is: LTO-5 3200GB cmp
urst4a –  unload/reload device, format is: LTO-5 3200GB cmp
    Tape drive (switch ip))  IBM LTO 5 ULTRIUM
rst5l  –  rewind device,        format is: LTO-3(ro)/4 4/800GB
nrst5l –  no rewind device,     format is: LTO-3(ro)/4 4/800GB
urst5l –  unload/reload device, format is: LTO-3(ro)/4 4/800GB
rst5m  –  rewind device,        format is: LTO-3(ro)/4 8/1600GB
netappnode> sysconfig

slot 1: FC Host Adapter 1a
    2 Tapes:            ULTRIUM-TD5
                                    ULTRIUM-TD5
slot 1: FC Host Adapter 1b
slot 1: FC Host Adapter 1c
    2 Tapes:            ULTRIUM-TD5
                                    ULTRIUM-TD5
slot 1: FC Host Adapter 1d

(5) Kill all leftover ndmp jobs.

netappnode> ndmpd killall

netappnode> ndmpd status
ndmpd ON.
No ndmpd sessions active.

Now the drive paths are up on the netapp and backups will go through fine.

 

EMC VNX NDMP Backup with NetBackup 6.5.x fails with Status 229 – Image inconsistency error

We have EMC VNX 7500 in our environment that we use as a NAS appliance.

Some ndmp based backup jobs failed with error code 229 – image inconsistency error.

Below is the workaround  to fix the issue (This is specific to EMC VNX ) :

Add the following entry under backup selection of the policy  :

SET type=tar

 

Symantec Netbackup

In this post i  will be discussing some of the issues that i have come across while administering a netbackup environment.

Restore job stuck as the tape goes into frozen state. If you unfreeze the tape from GUI it complaints with error “Already Allocated” and restore job will not move further.

Here is how we can fix this type of issue.

[1] From your master server go to following location and run command :

/usr/openv/netbackup/bin/admincmd/Nbrbutil –dump|grep < tape barcode >

example : masterserver$ nbrbutil -dump|grep 109026

MdsAllocation: allocationKey=41791 jobType=2 mediaKey=4001630 mediaId=109026 driveKey=0 driveName= drivePath= stuName= masterServerName=ebars04 mediaServerName=ebars04 ndmpTapeServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0

Note : masterserver is the master server of your backup environment and you need to grep for the media id that is getting frozen.(109026 is the media id in this case)

[2] Note the allocation key highlighted above and run below command to release it

nbrbutil -releaseMDS 41791

Now you can rerun the restore and it will run without issues.