Tuesday, May 1, 2012

Update DNS Server Entries on Multiple Servers

Recently we had upgraded our Windows 2000 Domain Controllers with Windows 2008 Domain Controllers and had to replace the DNS Servers  listed in Network Properties across all the member servers running windows 2000, Windows 2003 & Windows 2008 Servers. I created the following script to replace DNS Entries.. Replace "a.b.c.d"  with the actual dns server IP's in the script, create a file "d:\DNS_Order\servers.txt" with the server names listed in it. I ran it from command line as below: test.txt will record the output from the script which lists the DNS Entries of each and every server before and after completing the replacement..

C:\ cscript dns_replace_Serversfile.vbs >test.txt

dns_replace_Serversfile.vbs
Const ForReading = 1
Set objNetwork = CreateObject("Wscript.Network")
'Open the File System Object to Feed the Script
Set objFSO = CreateObject("Scripting.FileSystemObject")

'This line needs to be customized for the servers.txt localtion text file
Set objTextFile = objFSO.OpenTextFile ("d:\DNS_Order\servers.txt", ForReading)

'Starts the process reading thru the input file line by line until EOF
Do Until objTextFile.AtEndOfStream
strComputer = objTextFile.ReadLine
Wscript.Echo ""
Wscript.Echo "********************** Gathering Data On **********************"
Wscript.Echo " Computer Name From TXT File : "& strComputer
Wscript.Echo "********************** Gathering Data On **********************"


On Error Resume Next


arrNewDNSServerSearchOrder = Array("a.b.c.d", "a.b.c.d1", "a.b.c.d2", "a.b.c.d3")

Set objWMIService = GetObject("winmgmts:" _
 & "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
Set colNicConfigs = objWMIService.ExecQuery _
 ("SELECT * FROM Win32_NetworkAdapterConfiguration WHERE IPEnabled = True")

WScript.Echo VbCrLf & "Computer: " & strComputer

For Each objNicConfig In colNicConfigs
  WScript.Echo VbCrLf & "  Network Adapter " & objNicConfig.Index
  WScript.Echo "    DNS Server Search Order - Before:"
  If Not IsNull(objNicConfig.DNSServerSearchOrder) Then
    For Each strDNSServer In objNicConfig.DNSServerSearchOrder
      WScript.Echo "        " & strDNSServer
    Next
  End If
  intSetDNSServers = _
   objNicConfig.SetDNSServerSearchOrder(arrNewDNSServerSearchOrder)
  If intSetDNSServers = 0 Then
    WScript.Echo "    Replaced DNS server search order list."
  Else
    WScript.Echo "    Unable to replace DNS server search order list."
  End If
Next

WScript.Echo VbCrLf & String(80, "-")

Set colNicConfigs = objWMIService.ExecQuery _
 ("SELECT * FROM Win32_NetworkAdapterConfiguration WHERE IPEnabled = True")

For Each objNicConfig In colNicConfigs
  WScript.Echo VbCrLf & "  Network Adapter " & objNicConfig.Index
  WScript.Echo "    DNS Server Search Order - After:"
  If Not IsNull(objNicConfig.DNSServerSearchOrder) Then
    For Each strDNSServer In objNicConfig.DNSServerSearchOrder
      WScript.Echo "        " & strDNSServer
    Next
  End If
Next
Loop

Thursday, April 26, 2012

Update AIX 7.1 Service Packs

Make sure that all current filesets are commited by running installp -s or lslpp -l
# installp -s
   installp:  No filesets were found in the Software
        Vital Product Database in the APPLIED state.
To cleanup  the repository images of your installation directory. If you have committed all the fixes and don't need these files anymore..
#/usr/lib/instl/lppmgr -d /usr/sys/inst.images -l -b -x  -r -e
  lppmgr: Source table of contents location is /usr/sys/inst.images/.toc
  lppmgr: Building table of contents in /usr/sys/inst.images ..
  dsm.core.7.1.1.1.U
  mcr.rte.7.1.1.1.U
  security.pkcs11.7.1.1.1.U
  sysmgt.cfgassist.7.1.0.15.U
======================== end list ==============================

Run the inutoc command, this creates the .toc file in Directory. If a .toc file already exists, it is recreated with new information. The default installation image Directory is/usr/sys/inst.images. The inutoc command adds table of contents entries in the .toc file for every installation image in Directory. To create .toc file for /usr/sys/inst.images
# inutoc /usr/sys/inst.images
Optional--Not needed to run --To rename the downloaded files to their fileset name, run the bffcreate command. After renaming the files, run the inutoc command again.
# bffcreate -c -d /usr/sys/inst.images
# inutoc /usr/sys/inst.images
Run smit update_all to install all updates from this package that apply to the installed filesets on your system, select the Installation Directory, make sure expand filesystem as required..
# smit update_all
   



 Reboot the system if required

Monday, March 26, 2012

Update Firmware on 9111-285 (power5 System)

Download Required Firmware - In this case Recommended Firmware is SF240-417
FTP the rpm file (01SF240_417_382.rpm) to the Server that Needs to be Upgraded

Run the command below to extract the flash image file in the rpm file:

rpm -Uvh --ignoreos /tmp/01SF240_202_201.rpm

Before installing check the existing firmware level. From AIX, use the command lsmcode.  This command resides in the diagnostic directory.  An example of the output of the lsmcode command is as follows:
----------------------------------------------------------------------------------------------------------
DISPLAY MICROCODE LEVEL                                                   802811
IBM,9111-285

The current permanent system firmware image is SF240_202
The current temporary system firmware image is SF240_202
The system is currently booted from the temporary firmware image.

Use Enter to continue.
--------------------------------------------------------------------------------------------------------------
Next, run the update_flash command to upgrade firmware:
# ls /tmp/fwupdate
01SF240_417_382
# /usr/lpp/diagnostics/bin/update_flash -f /tmp/fwupdate/01SF240_417_382
The image is valid and would update the temporary image to SF240_417.
The new firmware level for the permanent image would be SF240_202.

The current permanent system firmware image is SF240_202.
The current temporary system firmware image is SF240_202.


***** WARNING: Continuing will reboot the system! *****

Do you wish to continue?
Enter 1=Yes or 2=No
SHUTDOWN PROGRAM
Mon Mar 26 11:01:35 EDT 2012
Stopping The LWI Nonstop Profile...
Waiting for The LWI Nonstop Profile to exit...
Waiting for The LWI Nonstop Profile to exit...
Stopped The LWI Nonstop Profile.
0513-044 The sshd Subsystem was requested to stop.
Wait for 'Rebooting...' before stopping.
Error logging stopped...
Advanced Accounting has stopped...
Process accounting stopped...
Stopping NFS/NIS Daemons
0513-044 The nfsd Subsystem was requested to stop.
0513-044 The biod Subsystem was requested to stop.
0513-044 The rpc.lockd Subsystem was requested to stop.
0513-044 The rpc.statd Subsystem was requested to stop.
0513-004 The Subsystem or Group, gssd, is currently inoperative.
0513-004 The Subsystem or Group, nfsrgyd, is currently inoperative.
0513-044 The rpc.mountd Subsystem was requested to stop.
0513-004 The Subsystem or Group, ypbind, is currently inoperative.


Connection to host lost.
After the System is back up, ran lsmcode and found that the system booted from the new Temporary Firmware as shown below
DISPLAY MICROCODE LEVEL                                                   802811
IBM,9111-285

The current permanent system firmware image is SF240_202
The current temporary system firmware image is SF240_417
The system is currently booted from the temporary firmware image.

Use Enter to continue.
To move the Temporary Image
# /usr/lpp/diagnostics/bin/update_flash -c 

Friday, March 23, 2012

Rename hdisk in AIX

Recently we had an issue with 2 hdisks going bad in AIX LPAR, both in different volume groups (hdisk15 & hdisk78)., The Volume groups had disks mirrored (hdisk0 to hdisk39 mirrored to hdisk40 to hdisk79). After replacing the drives, found that  the hdisks names got swapped as shown below before and after replacement. Since I do not want to create any confusion in future, I had to rename them back to original state before adding them to voulme groups as below..

 Before Replacing 



#lscfg -vl hdisk15
  hdisk15          U5791.001.992055M-P2-T5-L11-L0  16 Bit LVD SCSI Disk Drive (7
3400 MB)



#lscfg -vl hdisk78
  hdisk78          U5791.001.9920546-P2-T6-L10-L0  16 Bit LVD SCSI Disk Drive (7
3400 MB)

After Replacing



#lscfg -vl hdisk78
  hdisk78          U5791.001.992055M-P2-T5-L11-L0  16 Bit LVD SCSI Disk Drive (7
3400 MB)



#lscfg -vl hdisk15
  hdisk15          U5791.001.9920546-P2-T6-L10-L0  16 Bit LVD SCSI Disk Drive (7
3400 MB)

Swapping Back to Original State

Removed both the drives
#rmdev -dl hdisk78
#rmdev -dl hdisk15

Ran cfgmgr in the order I wanted

#cfgmgr -l scsi3 
#cfgmgr -l scsi23

Verified that they are swapped back

#lsdev -Cc disk |grep hdisk15
hdisk15   Available 0P-08-00-11,0 16 Bit LVD SCSI Disk Drive 
$ lsdev -Cc adapter |grep 0P-08
scsi3     Available 0P-08 Wide/Ultra-3 SCSI I/O Controller



#lscfg -vl hdisk15
  hdisk15          U5791.001.992055M-P2-T5-L11-L0  16 Bit LVD SCSI Disk Drive (7
3400 MB)




#lsdev -Cc disk |grep hdisk78
hdisk78   Available 0t-08-00-10,0 16 Bit LVD SCSI Disk Drive 
# lsdev -Cc adapter |grep  0t-08  
scsi23    Available 0t-08 Wide/Ultra-3 SCSI I/O Controller
#lscfg -vl hdisk78
  hdisk78          U5791.001.9920546-P2-T6-L10-L0  16 Bit LVD SCSI Disk Drive (7
3400 MB)

Replace Faulty Disk in a SCSI Mirrored VG in AIX

Found a Faulty SCSI Disk in a volume group on one of the LPAR in p595, Volume Group has 40 SCSI Disks mirrored and one Logical Volume and it is marked stale.,
Following are the steps taken to replace the Faulty Disk

Error Report Details

# errpt |more

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
EAA3D429   0321085312 U S LVDD           PHYSICAL PARTITION MARKED STALE
EAA3D429   0321084712 U S LVDD           PHYSICAL PARTITION MARKED STALE

16F35C72   0321044812 P H hdisk78        DISK OPERATION ERROR
16F35C72   0321023812 P H hdisk78        DISK OPERATION ERROR

$ errpt -a -j F7DDA124 |more
---------------------------------------------------------------------------
LABEL:          LVM_SA_PVMISS
IDENTIFIER:     F7DDA124

Date/Time:       Wed Mar 21 04:48:29 EDT 2012
Sequence Number: 99327
Machine Id:      00CFEFAF4C00
Node Id:         test01t
Class:           H
Type:            UNKN
WPAR:            Global
Resource Name:   LVDD
Resource Class:  NONE
Resource Type:   NONE
Location:

Description
PHYSICAL VOLUME DECLARED MISSING

Probable Causes
POWER, DRIVE, ADAPTER, OR CABLE FAILURE

Detail Data
MAJOR/MINOR DEVICE NUMBER
8000 0013 0000 0051
SENSE DATA
00CF EFAF 0000 4C00 0000 0113 D553 FECC 00CF EFAF 9018 3EC4 0000 0000 0000 0000

$ errpt -a -j 16F35C72 |more
---------------------------------------------------------------------------
LABEL:          DISK_ERR2
IDENTIFIER:     16F35C72

Date/Time:       Wed Mar 21 04:48:29 EDT 2012
Sequence Number: 99325
Machine Id:      00CFEFAF4C00
Node Id:         test01t
Class:           H
Type:            PERM
WPAR:            Global
Resource Name:   hdisk78
Resource Class:
Resource Type:
Location:
VPD:
        Manufacturer................IBM
        Machine Type and Model......ST373454LC
        FRU Number..................00P2685
        ROS Level and ID............43373137
        Serial Number...............0005D90D
        EC Level....................H13092
        Part Number.................26K5280
        Device Specific.(Z0)........000004129F00013E
        Device Specific.(Z1)........0721C717
        Device Specific.(Z2)........0002
        Device Specific.(Z3)........05179
        Device Specific.(Z4)........0001
        Device Specific.(Z5)........22
        Device Specific.(Z6)........H13092

Description
DISK OPERATION ERROR

Probable Causes
DASD DEVICE

Failure Causes
DISK DRIVE
DISK DRIVE ELECTRONICS

$ lscfg -vl hdisk78
  hdisk78          U5791.001.9920546-P2-T6-L10-L0  16 Bit LVD SCSI Disk Drive (7
3400 MB)

        Manufacturer................IBM
        Machine Type and Model......ST373454LC
        FRU Number..................00P2685
        ROS Level and ID............43373137
        Serial Number...............0005D90D
        EC Level....................H13092
        Part Number.................26K5280
        Device Specific.(Z0)........000004129F00013E
        Device Specific.(Z1)........0721C717
        Device Specific.(Z2)........0002
        Device Specific.(Z3)........05179
        Device Specific.(Z4)........0001
        Device Specific.(Z5)........22
        Device Specific.(Z6)........H13092


Monday, November 7, 2011

Replacing Fiber Adapter (HBA) in IBM AIX 5.2.0 on p595

One of the lpar running on p595 servers had Started reporting  Errors on one of the two Fiber Adapters (fscsi1) in Error report, the errors appeared as Type:TEMP as shown below.
825849BF   1106105211 T H fcs1           ADAPTER ERROR
B8FBD189   1106105211 T S fscsi1         SOFTWARE PROGRAM ERROR

LABEL:          FSCSI_ERR6
IDENTIFIER:     B8FBD189

Date/Time:       Sun Nov  6 10:52:31 EST
Sequence Number: 872005
Machine Id:      XXXXXX
Node Id:        XXXXXXX
Class:           S
Type:            TEMP
Resource Name:   fscsi1

LABEL:          FCS_ERR2
IDENTIFIER:     825849BF

Date/Time:       Sun Nov  6 10:52:31 EST
Sequence Number: 873473
Machine Id:      XXXXXXXX
Node Id:         XXXXXXX
Class:           H
Type:            TEMP
Resource Name:   fcs1
Resource Class:  adapter
Resource Type:   df1000fa
Location:        U5791.001.992083W-P1-C05-T1
VPD:

Description
SOFTWARE PROGRAM ERROR

Probable Causes
ADAPTER MICROCODE
SOFTWARE PROGRAM
SOFTWARE DEVICE DRIVER

Failure Causes
ADAPTER MICROCODE
SOFTWARE PROGRAM
SOFTWARE DEVICE DRIVER

        Recommended Actions
        IF PROBLEM PERSISTS THEN DO THE FOLLOWING
        CONTACT APPROPRIATE SERVICE REPRESENTATIVE

This server was running DB2 and DB2 crashed around the same time when we started getting these alerts. Found that DB2 crashed due to super block corruption in the filesystem and DB2 was unable to create files..,Unmounted the Filesystem,  corrected superblock corruption by copying secondary superblock.
dd count=1 bs=4k skip=31 seek=1 if=/dev/LVXX of=/dev/LVXX
dd: 1+0 records in.
dd: 1+0 records out.
Had contacted SAN Team to look into any SAN issues and they did not find any issues on SAN.
Found that it works for a while and goes into degraded state and eventually into Failed state.
Placed a service call with IBM to replace the Adapter and replacing the adapter fixed the issue.. Steps followed to replace the Fiber Adapter..

  • datapath query adapter --- To identify the faulty or degraded adapter
Adpt# Name State Mode Select Errors Paths Active
0 fscsi1 FAILED ACTIVE 2059041211 711 20 0
1 fscsi3 NORMAL ACTIVE 231765446 37 20 18

  • datapath remove adapter 0 --- To remove the adapter from SDD
  • rmdev -Rdl fcs1 ---To remove adapter from ODM. This will also remove all child devices ( hdiskXX). This step is not needed as the following step will do this for you, hotplug mgr will do that for you, having the device still there allows you to identify it easier
  • diag, choose Task Selection, Hot Plug Task, PCI Hot Plug Manager.... tasks to replace the adapter
  • lscfg -vl fcs1 ---To identify new wwn. Have SAN Team assign LUN's to this adapter
  • cfgmgr -vl fscsi1 --- Run cfgmgr to rediscover the deleted paths to LUNs
    cfgmgr -v
  • addpaths ---add paths to SDD
  • datapath query device ----verify all LUNS have same number of active paths
     datapath query adapter

Saturday, August 6, 2011

AiX Commads

  1. ssa_ela -h 7 ; To get SRN Status for ssa (output
    ssa0 SRN 45002
    enclosure0 SRN 80221)
  2.  lscfg -vlpdisk3 -- To get details of pdisk3
  3.   lsdev –C –c pdisk –s ssar –H –To list all pdisks or use smit Devices->SSA Disks—>SSA Physical Disks->List All Defined SSA Physical Disks 
  4. lsdev –C –t hdisk –c disk –s ssar –H – To list SSA Logical Disks or use smit Devices->SSA Disks—> SSA Logical Disks->List All Defined SSA Logical Disks
  5.   smit -> SSA Dsks—>SSA Physical Disks ->Show Physical to Logical SSA Disk Relationship or--for I in $(lsdev –CS1 –c pdisk –ssar –F name); do; echo “$i: “$ssaxlate –l $i);done
  6.  lsdev -P -H  --To list all devices in the Predefined Devices object class with column headers, type--The system displays a message similar to the following:(class type  subclass   description--logical_volume vgtype vgsubclass Volume group ) 
  7.  To Remove pdisk3 follow the steps below 

    pdisk3, which is  hdisk5
    go into diags
    Task Selection (Diagnostics, Advanced Diagnostics, Service Aids, etc.
    SSA Service Aids
    Set Service Mode
    <ServerName>:pdisk3
    Do Identify on pdisk3 to find out where it's located

    Once you find it backout to the service mode menu  again and do
    Set or Reset Service Mode on pdisk3

    This is when the drive light should come on solid and can be replaced
    Pull old drive out and put new drive in

    Then
    rmdev -l pdisk3 -d
    rmdev -l hdisk5 -d

    cfgmgr

    lscfg -vl pdisk3

    You should see the new pdisk3