Secure connection HTTP Server Admin and WAS Adminconsole

Export Web Admin Certificate from Web Admin Keystore and add it to cellDefaultTrustStore as signer certificate.

Change WASADMIN Console Session Timeout Time

1. Edit /opt/IBM/WebSphere/AppServer/profiles/Dmgr/config/cells/AUTOCell/applications/isclite.ear/deployments/isclite/deployment.xml
2.Locate the xml statement <tuningParams xmi:id="TuningParams_1088453565469" maxInMemorySessionCount="1000" allowOverflow="true" writeFrequency="TIME_BASED_WRITE" writeInterval="10" writeContents="ONLY_UPDATED_ATTRIBUTES" invalidationTimeout="30">
3.Change the invalidationTimeout value to the desired session timeout. The default is 30.
4.Save the deployment.xml file.
5.Restart the Deployment Manager.

IHS Diagnose

http://www-01.ibm.com/support/docview.wss?uid=swg24008409

WAS Jython Functions

AdminControl
completeObjectName
                Returns a String version of an object name given a template
                name
getAttribute_jmx
                Given ObjectName and name of attribute, returns value of
                attribute
getAttribute    Given String version of ObjectName and name of attribute,
                returns value of attribute
getAttributes_jmx
Given ObjectName and array of attribute names, returns
AttributeList.
getAttributes   Given String version of ObjectName and attribute names,
returns String of name value pairs. Given String version
of ObjectName, returns String of all name value pairs.
getCell         Returns the cell name of the connected server
getConfigId     Given String version of ObjectName, return a configuration
ID for the corresponding configuration object, if any
getDefaultDomain
Returns "WebSphere"
getDomainName   Returns "WebSphere"
getHost         Returns String representation of connected host
getMBeanCount   Returns number of registered beans
getMBeanInfo_jmx
                Given ObjectName, returns MBeanInfo structure for MBean
getNode         Returns the node name of the connected server
getObjectInstance
                Given String version of ObjectName, returns ObjectInstance
                object that match.
getPort         Returns String representation of port in use
getType         Returns String representation of connection type in use
help            Shows help information
invoke_jmx      Given ObjectName, name of method, array of parameters and
                signature, invokes method on MBean specified
invoke          Invokes a method on the specified MBean
isRegistered_jmx
                Returns true if supplied ObjectName is registered
isRegistered    Returns true if supplied String version of ObjectName is
                registered
makeObjectName  Returns an ObjectName built with the given string
queryNames_jmx  Given ObjectName and QueryExp, retrieves set of ObjectNames
                that match.
queryNames      Given String version of ObjectName, retrieves String of
                ObjectNames that match
queryMBeans     Given String version of ObjectName, returns a set of
                ObjectInstances object that match
reconnect       Reconnects with server
setAttribute_jmx
                Given ObjectName and Attribute object, sets attribute for MBean
                specified
setAttribute    Given String version of ObjectName, attribute name and
                attribute value, sets attribute for MBean specified
setAttributes_jmx
                Given ObjectName and AttributeList object, sets attributes for
                the MBean specified
setAttributes   Given String version of ObjectName, attribute name
                and value pairs, sets attributes for the MBean specified
startServer     Given the name of a server, starts that server
stopServer      Given the name of a server, stops that server
testConnection  Tests the connection to a DataSource object
trace           Sets the wsadmin trace specification

AdminApp
deleteUserAndGroupEntries
                Deletes all the user/group information for all the roles and
                all the user name/password information for RunAs roles for a
                given application
edit            Edits the properties of an application
editInteractive Edits the properties of an application interactively
export          Exports application to a file
exportDDL       Exports DDL from application to a directory
exportFile      Exports content of a single file from an application to a file
getDeployStatus Returns the combined Deployment status of the application
help            Shows help information
install         Installs an application, given a file name and an option string
installInteractive
                Installs an application in interactive mode, given a file name
                and an option string
isAppReady      Checks whether the application is ready to be run
list            Lists all installed applications
listModules     Lists the modules in a specified application
options         Shows the options available, for a given file, application,
                or in general
renameApplication       Rename a given application
publishWSDL     Publishs WSDL files for a given application
searchJNDIReferences
                Lists application that refers to the given JNDIName on a given
                node
taskInfo        Shows detailed information pertaining to a given install task
                for a given file
uninstall       Uninstalls an application, given an application name and
                an option string
update          Updates an installed application
updateAccessIDs Updates the user/group binding information with accessID
                from user registry for a given application
updateInteractive       Updates an installed application interactively
view            Views an application or module, given an application or
                module name

AdminConfig

attributes      Shows the attributes for a given type
checkin         Checks a file into the the configuration repository
convertToCluster
                Converts a server to be the first member of a new ServerCluster
create          Creates a configuration object, given a type, a parent, and
                a list of attributes, and optionally an attribute name for the
                new object
createClusterMember
                Creates a new server that is a member of an existing cluster
createDocument  Creates a new document in the configuration repository
createUsingTemplate
                Creates an object using a particular template type
defaults        Displays the default values for attributes of a given type
deleteDocument  Deletes a document from the configuration repository
existsDocument  Tests for the existence of a document in the configuration
                repository
extract         Extracts a file from the configuration repository
getCrossDocumentValidationEnabled
Returns true if cross-document validation is enabled
getid           Shows the configuration ID of an object, given a string version
of its containment
getObjectName   Given a configuration ID, returns a string version of the
                ObjectName for the corresponding running MBean, if any
getObjectType   Given a configuration ID, returns a string version of the
                object type
getSaveMode     Returns the mode used when "save" is invoked
getValidationLevel
                Returns the validation used when files are extracted from the
                repository.
getValidationSeverityResult
                Returns the number of messages of a given severity from the
                most recent validation
hasChanges      Returns true if unsaved configuration changes exist
help            Shows help information
installResourceAdapter
                Installs a J2C resource adapter with the given rar file name
                and an option string in the node
list            Lists all configuration objects of a given type
listTemplates   Lists all available configuration templates of a given type
modify          Changes specified attributes of a given configuration object
parents         Shows the objects which contain a given type
queryChanges    Returns a list of unsaved files
remove          Removes the specified configuration object
required        Displays the required attributes of a given type.
reset           Discards unsaved configuration changes
resetAttributes Resets value of the specified attributes.
save            Commits unsaved changes to the configuration repository
setCrossDocumentValidationEnabled
                Sets the cross-document validation enabled mode.
setSaveMode     Changes the mode used when "save" is invoked
setValidationLevel
                Sets the validation used when files are extracted from the
                repository.
show            Shows the attributes of a given configuration object
showall         Recursively shows the attributes of a given configuration
                object, and all the objects contained within each attribute.
showAttribute   Displays only the value for the single attribute specified.
types           Shows the possible types for configuration
uninstallResourceAdapter
                Uninstalls a J2C resource adapter with the given resource
                adapter configuration ID.
unsetAttributes Unsets value of the specified attributes.
validate        Invokes validation

Solaris: Create new swap partition

Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c0t0d0 <DEFAULT cyl 8921 alt 2 hd 255 sec 63>
          /pci@0,0/pci1022,7450@2/pci1000,3060@3/sd@0,0
       1. c0t1d0 <DEFAULT cyl 8921 alt 2 hd 255 sec 63>
          /pci@0,0/pci1022,7450@2/pci1000,3060@3/sd@1,0
Specify disk (enter its number): 1
selecting c0t1d0
[disk formatted]
Warning: Current Disk has mounted partitions.
/dev/dsk/c0t1d0s0 is currently mounted on /var/sadm. Please see umount(1M).
/dev/dsk/c0t1d0s4 is currently mounted on /var/opt/SUNWuce. Please see umount(1M).
/dev/dsk/c0t1d0s5 is currently mounted on /opt/backup. Please see umount(1M).

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        fdisk      - run the fdisk program
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !<cmd>     - execute <cmd>, then return
        quit

format> p

PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        !<cmd> - execute <cmd>, then return
        quit
partition> p
Current partition table (original):
Total disk cylinders available: 8921 + 2 (reserved cylinders)
Part      Tag    Flag     Cylinders        Size            Blocks
  0        var    wm    1177 - 2482       10.00GB    (1306/0/0)  20980890
  1 unassigned    wu    2483 - 2613        1.00GB    (131/0/0)    2104515
  2     backup    wm       0 - 8920       68.34GB    (8921/0/0) 143315865
  3       swap    wu    2614 - 3136        4.01GB    (523/0/0)    8401995
  4   reserved    wm       1 -  653        5.00GB    (653/0/0)   10490445
  5   reserved    wm     654 - 1176        4.01GB    (523/0/0)    8401995
  6 unassigned    wm       0               0         (0/0/0)            0
  7 unassigned    wm       0               0         (0/0/0)            0
  8       boot    wu       0 -    0        7.84MB    (1/0/0)        16065
  9 unassigned    wm       0               0         (0/0/0)            0

partition> 3  
Part      Tag    Flag     Cylinders        Size            Blocks
  3 unassigned    wm       0               0         (0/0/0)            0
Enter partition id tag[unassigned]: swap
Enter partition permission flags[wm]: wu
Enter new starting cyl[1]: 2614
Enter partition size[0b, 0c, 2614e, 0.00mb, 0.00gb]: 4gb
partition> p
Current partition table (unnamed):
Total disk cylinders available: 8921 + 2 (reserved cylinders)
Part      Tag    Flag     Cylinders        Size            Blocks
  0        var    wm    1177 - 2482       10.00GB    (1306/0/0)  20980890
  1 unassigned    wu    2483 - 2613        1.00GB    (131/0/0)    2104515
  2     backup    wm       0 - 8920       68.34GB    (8921/0/0) 143315865
  3       swap    wu    2614 - 3136        4.01GB    (523/0/0)    8401995
  4   reserved    wm       1 -  653        5.00GB    (653/0/0)   10490445
  5   reserved    wm     654 - 1176        4.01GB    (523/0/0)    8401995
  6 unassigned    wm       0               0         (0/0/0)            0
  7 unassigned    wm       0               0         (0/0/0)            0
  8       boot    wu       0 -    0        7.84MB    (1/0/0)        16065
  9 unassigned    wm       0               0         (0/0/0)            0
partition> h
Expecting one of the following: (abbreviations ok):
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        !<cmd> - execute <cmd>, then return
        quit
partition> label
Ready to label disk, continue? y
partition> p
Current partition table (unnamed):
Total disk cylinders available: 8921 + 2 (reserved cylinders)
Part      Tag    Flag     Cylinders        Size            Blocks
  0        var    wm    1177 - 2482       10.00GB    (1306/0/0)  20980890
  1 unassigned    wu    2483 - 2613        1.00GB    (131/0/0)    2104515
  2     backup    wm       0 - 8920       68.34GB    (8921/0/0) 143315865
  3       swap    wu    2614 - 3136        4.01GB    (523/0/0)    8401995
  4   reserved    wm       1 -  653        5.00GB    (653/0/0)   10490445
  5   reserved    wm     654 - 1176        4.01GB    (523/0/0)    8401995
  6 unassigned    wm       0               0         (0/0/0)            0
  7 unassigned    wm       0               0         (0/0/0)            0
  8       boot    wu       0 -    0        7.84MB    (1/0/0)        16065
  9 unassigned    wm       0               0         (0/0/0)            0

List Swap Space:
# swap -l
swapfile             dev    swaplo   blocks     free
/dev/dsk/c6t0d0s1   79,1         8  1048568  1048568

Turn off swap:
# swap -d /dev/dsk/c6t0d0s1
/dev/dsk/c6t0d0s1 was dump device --
invoking dumpadm(1M) -d swap to select new dump device
dumpadm: no swap devices are available

Notice you have no dump device
Disable swap from coming back on reboot:
#vi /etc/vfstab
there's a line to mount the swap space, remove it.. Look for your swap device, i.e. c6t0d0s1.

Add swap space (partition):
#swap -a /dev/desk/c6t0d0s1

Adding file based swap space:
#mkfile 250m /opt/myswapfile
#swap -a /opt/myswapfile

MQ Series: Configuration - Linux

Configuration


Prepare Share Storage for MQ Multi-Instance

Mount nfs volume to the servers for shared file system.
nfsserver:/vol/mqshare/GW  to  mqsvr01 and mqsvr02
nfsserver:/vol/mqshare/QM1 to  mqsvr03 and mqsvr04
nfsserver:/vol/mqshare/QM2 to  mqsvr05 and mqsvr06

7.3 Configure MQ Multi-Instance
7.3.1 Verify MQ Shared File System
NOTE: Execute following steps as user “mqm”
• Run amqmfsck on both App server per site (mqsvr01, mqsvr02), (mqsvr03, mqsvr04) (mqsvr05, mqsvr06)
amqmfsck /var/mqm/share
The tests on the directory completed successfully.
• Run amqmfsck -c
NOTE: Run on BOTH App server per site (mqsvr01, mqsvr02), (mqsvr03, mqsvr04) (mqsvr05, mqsvr06) at the SAME TIME
amqmfsck -c /var/mqm/share
Writing to test file. This will normally complete within about 60 seconds.
...........................................................
The tests on the directory completed successfully.
• Run amqmfsck -w
NOTE: Run on BOTH App server per site (mqsvr01, mqsvr02), (mqsvr03, mqsvr04) (mqsvr05, mqsvr06) at the SAME TIME
On one node you should see “File lock acquired” while on other node, it’s “Waiting for the file lock”
Hit Enter on the node which gets the file lock to release the lock. On the other node, you should see “File lock acquired”
amqmfsck -w /var/mqm/share
Start a second copy of this program with the same parameters on another server.
File lock acquired.
Press Enter or terminate this process to release the lock.
File lock released.
The tests on the directory completed successfully.
amqmfsck -w /var/mqm/share
Waiting for the file lock.
Waiting for the file lock.
Waiting for the file lock.
Waiting for the file lock.
Waiting for the file lock.
Waiting for the file lock.
File lock acquired.
Press Enter or terminate this process to release the lock.
File lock released.
The tests on the directory completed successfully.
7.3.2 Configure MQ Multi-Instance for ART MQ GateWay:
NOTE: Execute following steps as user “mqm”
• Create folder for MQ instance
mkdir -p /var/mqm/share/log
mkdir -p /var/mqm/share/qmgrs
• Create MQ instance on GW (mqsvr01)
crtmqm -ll -lf 32000 -lp 10 -ls 3 -u DL.ART.POCGW.QMGR01.DEAD.LETTER.QUEUE  \
-ld /var/mqm/share/log -md /var/mqm/share/qmgrs ART.POCGW.QMGR01

WebSphere MQ queue manager created.
Directory '/var/mqm/share/qmgrs/ART!POCGW!QMGR01' created.
Creating or replacing default objects for ART.POCGW.QMGR01.
Default objects statistics : 65 created. 0 replaced. 0 failed.
Completing setup.
Setup completed.
• Verify MQ instance on (mqsvr01)
strmqm ART.POCGW.QMGR01
WebSphere MQ queue manager 'ART.POCGW.QMGR01' starting.
11 log records accessed on queue manager 'ART.POCGW.QMGR01' during the log replay phase.
Log replay for queue manager 'ART.POCGW.QMGR01' complete.
Transaction manager state recovered for queue manager 'ART.POCGW.QMGR01'.
WebSphere MQ queue manager 'ART.POCGW.QMGR01' started.
dspmq
QMNAME(ART.POCGW.QMGR01)                                  STATUS(Running)
• Stop MQ instance on (mqsvr01)
endmqm ART.POCGW.QMGR01

Quiesce request accepted. The queue manager will stop when all outstanding work
is complete.
dspmq
QMNAME(ART.POCGW.QMGR01)                    STATUS(Ended normally)
• Verify MQ instance on (mqsvr01) with Multi-Instance Mode
strmqm -x ART.POCGW.QMGR01  

WebSphere MQ queue manager 'ART.POCGW.QMGR01' starting.
11 log records accessed on queue manager 'ART.POCGW.QMGR01' during the log replay phase.
Log replay for queue manager 'ART.POCGW.QMGR01' complete.
Transaction manager state recovered for queue manager 'ART.POCGW.QMGR01'.
WebSphere MQ queue manager 'ART.POCGW.QMGR01' started.

dspmq -x -m ART.POCGW.QMGR01
QMNAME(CTPQMGR1)                                          STATUS(Running)
    INSTANCE(ctp-bcclprapp03) MODE(Active)
• Start Listener for MQ instance on (mqsvr01)
NOTE: Change the port 1415 to production value
nohup runmqlsr -p 1414 -t tcp -m ART.POCGW.QMGR01 > /dev/null 2>&1 &

• Import MQ instance on (mqsvr02)
addmqinf -v Name= ART.POCGW.QMGR01 -v Directory= ART.POCGW.QMGR01 -v Prefix=/var/mqm \
-v DataPath=/var/mqm/share/qmgrs/ ART.POCGW.QMGR01
WebSphere MQ configuration information added.
• Verify MQ instance configuration on BOTH servers (mqsvr01 and mqsvr02)
NOTE: the output should be identical
dspmqinf ART.POCGW.QMGR01
QueueManager:
   Name=ART.POCGW.QMGR01
   Directory=ART!POCGW!QMGR01
   Prefix=/var/mqm
   DataPath=/var/mqm/share/qmgrs/ART!POCGW!QMGR01
• Start MQ instance on mqsvr02 with Multi-Instance Mode
strmqm -x ART.POCGW.QMGR01
WebSphere MQ queue manager 'CTPQMGR1' starting.
A standby instance of queue manager 'CTPQMGR1' has been started. The active
instance is running elsewhere.
• Check MQ instance on Both server (mqsvr01 and mqsvr02)
dspmq -x -m ART.POCGW.QMGR01
QMNAME(CTPQMGR1)                                          STATUS(Running as standby)
    INSTANCE(ctp-bcclprapp03) MODE(Active)
    INSTANCE(ctp-bcclprapp04) MODE(Standby)
• Start Listener for MQ instance on mqsvr02
NOTE: Change the port 1415 to production value
nohup runmqlsr -p 1414 -t tcp -m ART.POCGW.QMGR01 > /dev/null 2>&1 &
7.3.3 Verify Multi-Instance Failover
• Stop MQ instance on the Active node (mqsvr01)
endmqm -s ART.POCGW.QMGR01
Quiesce request accepted. The queue manager will stop when all outstanding work
is complete, permitting switchover to a standby instance.

dspmq -x -m ART.POCGW.QMGR01
QMNAME(ART.POCGW.QMGR01)                                  STATUS(Quiescing)
    INSTANCE(mqsvr01) MODE(Active)
    INSTANCE(mqsvr02) MODE(Standby)

dspmq -x -m ART.POCGW.QMGR01
QMNAME(ART.POCGW.QMGR01)                                  STATUS(Running elsewhere)
    INSTANCE(mqsvr02) MODE(Active)
7.3.4 Configure MQ for JMS
• Run following on the Active Node of MQ as mqm
runmqsc ART.POCGW.QMGR01 < /opt/mqm/java/bin/MQJMS_PSQ.mqsc

8 MQSC commands read.
No commands have a syntax error.
All valid MQSC commands were processed.
7.3.5 Configure MQ Multi-Instance for ART MQ B1
NOTE: Execute following steps as user “mqm”
• Create folder for MQ instance
mkdir -p /var/mqm/share/log
mkdir -p /var/mqm/share/qmgrs
• Create MQ instance on GW (mqsvr03)
crtmqm -ll -lf 32000 -lp 10 -ls 3 -u DL.ART.POCB1.QMGR01.DEAD.LETTER.QUEUE  \
-ld /var/mqm/share/log -md /var/mqm/share/qmgrs ART.POCB1.QMGR01

WebSphere MQ queue manager created.
Directory '/var/mqm/share/qmgrs/ART!POCB1!QMGR01' created.
Creating or replacing default objects for ART.POCB1.QMGR01.
Default objects statistics : 65 created. 0 replaced. 0 failed.
Completing setup.
Setup completed.
• Verify MQ instance on (mqsvr03)
strmqm ART.POCB1.QMGR01
WebSphere MQ queue manager 'ART.POCB1.QMGR01' starting.
11 log records accessed on queue manager 'ART.POCB1.QMGR01' during the log replay phase.
Log replay for queue manager 'ART.POCB1.QMGR01' complete.
Transaction manager state recovered for queue manager 'ART.POCB1.QMGR01'.
WebSphere MQ queue manager 'ART.POCB1.QMGR01' started.
dspmq
QMNAME(ART.POCB1.QMGR01)                      STATUS(Running)
• Stop MQ instance on (mqsvr03)
endmqm ART.POCB1.QMGR01

Quiesce request accepted. The queue manager will stop when all outstanding work
is complete.
dspmq
QMNAME(ART.POCB1.QMGR01)               STATUS(Ended normally)
• Verify MQ instance on (mqsvr03) with Multi-Instance Mode
strmqm -x ART.POCB1.QMGR01  

WebSphere MQ queue manager 'ART.POCB1.QMGR01' starting.
11 log records accessed on queue manager 'ART.POCB1.QMGR01' during the log replay phase.
Log replay for queue manager 'ART.POCB1.QMGR01' complete.
Transaction manager state recovered for queue manager 'ART.POCB1.QMGR01'.
WebSphere MQ queue manager 'ART.POCB1.QMGR01' started.

dspmq -x -m ART.POCB1.QMGR01
QMNAME(ART.POCB1.QMGR01)                                  STATUS(Running)
    INSTANCE(mqsvr03) MODE(Active)
• Start Listener for MQ instance on (mqsvr03)
NOTE: Change the port 1415 to production value
nohup runmqlsr -p 1414 -t tcp -m ART.POCB1.QMGR01 > /dev/null 2>&1 &
• Import MQ instance on (mqsvr04)
addmqinf -v Name=ART.POCB1.QMGR01 -v Directory='ART!POCB1!QMGR01' -v Prefix=/var/mqm -v DataPath='/var/mqm/share/qmgrs/ART!POCB1!QMGR01'
WebSphere MQ configuration information added.
• Verify MQ instance configuration on BOTH servers (mqsvr03 and mqsvr04)
NOTE: the output should be identical
dspmqinf ART.POCB1.QMGR01

QueueManager:
   Name=ART.POCB1.QMGR01
   Directory=ART!POCB1!QMGR01
   Prefix=/var/mqm
   DataPath=/var/mqm/share/qmgrs/ART!POCB1!QMGR01
• Start MQ instance on mqsvr04 with Multi-Instance Mode
strmqm -x ART.POCB1.QMGR01

WebSphere MQ queue manager 'ART.POCB1.QMGR01' starting.
A standby instance of queue manager 'ART.POCB1.QMGR01' has been started. The
active instance is running elsewhere.
• Check MQ instance on Both server (mqsvr03 and mqsvr04)
dspmq -x -m ART.POCB1.QMGR01
QMNAME(ART.POCB1.QMGR01)                                  STATUS(Running as standby)
    INSTANCE(mqsvr03) MODE(Active)
    INSTANCE(mqsvr04) MODE(Standby)
• Start Listener for MQ instance on mqsvr04
NOTE: Change the port 1415 to production value
nohup runmqlsr -p 1414 -t tcp -m ART.POCB1.QMGR01 > /dev/null 2>&1 &
7.3.6 Verify Multi-Instance Failover
• Stop MQ instance on the Active node (mqsvr03)
endmqm -s ART.POCB1.QMGR01
Quiesce request accepted. The queue manager will stop when all outstanding work
is complete, permitting switchover to a standby instance.

dspmq -x -m ART.POCB1.QMGR01
QMNAME(ART.POCB1.QMGR01)                                  STATUS(Quiescing)
    INSTANCE(mqsvr03) MODE(Active)
    INSTANCE(mqsvr04) MODE(Standby)

dspmq -x -m ART.POCB1.QMGR01

QMNAME(ART.POCB1.QMGR01)                                  STATUS(Running elsewhere)
    INSTANCE(mqsvr04) MODE(Active)

7.3.7 Configure MQ for JMS
• Run following on the Active Node of MQ as mqm
runmqsc ART.POCB1.QMGR01 < /opt/mqm/java/bin/MQJMS_PSQ.mqsc
8 MQSC commands read.
No commands have a syntax error.
All valid MQSC commands were processed.

7.3.8 Configure MQ Multi-Instance for ART MQ B2
NOTE: Execute following steps as user “mqm”
• Create folder for MQ instance
mkdir -p /var/mqm/share/log
mkdir -p /var/mqm/share/qmgrs
• Create MQ instance on GW (mqsvr05)
crtmqm -ll -lf 32000 -lp 10 -ls 3 -u DL.ART.POCB2.QMGR01.DEAD.LETTER.QUEUE  \
-ld /var/mqm/share/log -md /var/mqm/share/qmgrs ART.POCB2.QMGR01

WebSphere MQ queue manager created.
Directory '/var/mqm/share/qmgrs/ART!POCB2!QMGR01' created.
Creating or replacing default objects for ART.POCB2.QMGR01.
Default objects statistics : 65 created. 0 replaced. 0 failed.
Completing setup.
Setup completed.
• Verify MQ instance on (mqsvr05)
strmqm ART.POCB2.QMGR01
WebSphere MQ queue manager 'ART.POCB2.QMGR01' starting.
11 log records accessed on queue manager 'ART.POCB2.QMGR01' during the log replay phase.
Log replay for queue manager 'ART.POCB2.QMGR01' complete.
Transaction manager state recovered for queue manager 'ART.POCB2.QMGR01'.
WebSphere MQ queue manager 'ART.POCB2.QMGR01' started.
dspmq
QMNAME(ART.POCB2.QMGR01)                                  STATUS(Running)
• Stop MQ instance on (mqsvr05)
endmqm ART.POCB2.QMGR01

Quiesce request accepted. The queue manager will stop when all outstanding work
is complete.
dspmq
QMNAME(ART.POCB2.QMGR01)                  STATUS(Ended normally)
• Verify MQ instance on (mqsvr05) with Multi-Instance Mode
strmqm -x ART.POCB2.QMGR01  

WebSphere MQ queue manager 'ART.POCB2.QMGR01' starting.
11 log records accessed on queue manager 'ART.POCB2.QMGR01' during the log replay phase.
Log replay for queue manager 'ART.POCB2.QMGR01' complete.
Transaction manager state recovered for queue manager 'ART.POCB2.QMGR01'.
WebSphere MQ queue manager 'ART.POCB2.QMGR01' started.

dspmq -x -m ART.POCB2.QMGR01
QMNAME(ART.POCB2.QMGR01)                                  STATUS(Running)
    INSTANCE(mqsvr05) MODE(Active)
• Start Listener for MQ instance on (mqsvr05)
NOTE: Change the port 1415 to production value
nohup runmqlsr -p 1414 -t tcp -m ART.POCB2.QMGR01 > /dev/null 2>&1 &
• Import MQ instance on (mqsvr06)
addmqinf -v Name=ART.POCB2.QMGR01 -v Directory='ART!POCB2!QMGR01' -v Prefix=/var/mqm -v DataPath='/var/mqm/share/qmgrs/ART!POCB2!QMGR01'

WebSphere MQ configuration information added.
• Verify MQ instance configuration on BOTH servers (mqsvr05 and mqsvr06)
NOTE: the output should be identical
dspmqinf ART.POCB2.QMGR01
QueueManager:
   Name=ART.POCB2.QMGR01
   Directory=ART!POCB2!QMGR01
   Prefix=/var/mqm
   DataPath=/var/mqm/share/qmgrs/ART!POCB2!QMGR01

• Start MQ instance on mqsvr06 with Multi-Instance Mode
strmqm -x ART.POCB2.QMGR01
WebSphere MQ queue manager 'ART.POCB2.QMGR01' starting.
A standby instance of queue manager 'ART.POCB2.QMGR01' has been started. The
active instance is running elsewhere.
• Check MQ instance on Both server (mqsvr05 and mqsvr06)
dspmq -x -m ART.POCB2.QMGR01
QMNAME(ART.POCB2.QMGR01)                                  STATUS(Running as standby)
    INSTANCE(mqsvr05) MODE(Active)
    INSTANCE(mqsvr06) MODE(Standby)
• Start Listener for MQ instance on mqsvr06
NOTE: Change the port 1415 to production value
nohup runmqlsr -p 1414 -t tcp -m ART.POCB2.QMGR01 > /dev/null 2>&1 &
7.3.9 Verify Multi-Instance Failover:
• Stop MQ instance on the Active node (mqsvr05)
endmqm -s ART.POCB2.QMGR01
Quiesce request accepted. The queue manager will stop when all outstanding work
is complete, permitting switchover to a standby instance.

dspmq -x -m ART.POCB2.QMGR01
QMNAME(ART.POCB2.QMGR01)                                  STATUS(Quiescing)
    INSTANCE(mqsvr05) MODE(Active)
    INSTANCE(mqsvr06) MODE(Standby)


dspmq -x -m ART.POCB2.QMGR01
QMNAME(ART.POCB2.QMGR01)                                  STATUS(Ended normally)
    INSTANCE(mqsvr06) MODE(Active)

7.3.10 Configure MQ for JMS:
• Run following on the Active Node of MQ as mqm
runmqsc ART.POCB2.QMGR01 < /opt/mqm/java/bin/MQJMS_PSQ.mqsc
8 MQSC commands read.
No commands have a syntax error.
All valid MQSC commands were processed.


7.3.11 Configure Channels and Local Ques for Gateway MQ:
Run these on running instance QMgr.
• Define Server Channel:
DEFINE CHANNEL('SYSTEM.ADMIN.SVRCONN') CHLTYPE(SVRCONN) TRPTYPE(TCP)
• Define Dead Letter Queue:
DEFINE QLOCAL(DL.ART.POCGW.QMGR01.DEAD.LETTER.QUEUE) DESCR('Dead Letter Queue')
• Define Transmission Queue:
DEFINE QLOCAL('TQ.ART.POCB1.QMGR01')  TRIGGER INITQ(SYSTEM.CHANNEL.INITQ) TRIGDATA(POCGW.TO.POCB1) PROCESS(start.sndr.channel) DESCR('Transmission Queue to B1 Qmgr') USAGE(XMITQ) DEFPSIST(YES)

DEFINE QLOCAL('TQ.ART.POCB2.QMGR01') TRIGGER INITQ(SYSTEM.CHANNEL.INITQ) TRIGDATA(POCGW.TO.POCB1) PROCESS(start.sndr.channel.b2) DESCR('Transmission Queue to B2 Qmgr') USAGE(XMITQ) DEFPSIST(YES)
• Define local Queues:
DEFINE QLOCAL('LQ.ART.POCGW1') DESCR('Local Que ART.POCGW.QMGR01') DEFPSIST(YES) SHARE NOTRIGGER USAGE(NORMAL)
DEFINE QLOCAL('LQ.ART.POCGW2') DESCR('Local Que ART.POCGW.QMGR01') DEFPSIST(YES) SHARE NOTRIGGER USAGE(NORMAL)

• Define Remote Ques:
DEFINE QREMOTE('RQ.LQ.ART.POCB1') DESCR('Remote Q GWQmgr to B1Qmgr') RNAME('LQ.ART.POCB') RQMNAME('ART.POCB1.QMGR01') XMITQ('TQ.ART.POCB1.QMGR01') DEFPSIST(YES)

DEFINE QREMOTE('RQ.LQ.ART.POCB2') DESCR('Remote Q GWQmgr to B2Qmgr') RNAME('LQ.ART.POCB') RQMNAME('ART.POCB2.QMGR01') XMITQ('TQ.ART.POCB2.QMGR02') DEFPSIST(YES)
• Define Receiver Channels
DEFINE CHANNEL(POCB1.TO.POCGW) CHLTYPE(RCVR) TRPTYPE(TCP) DESCR('Receiver Channel from B1 queue manager')
DEFINE CHANNEL(POCB2.TO.POCGW) CHLTYPE(RCVR) TRPTYPE(TCP) DESCR('Receiver Channel from B1 queue manager')
• Define Sender Channels:
DEFINE CHANNEL('POCGW.TO.POCB1') CHLTYPE(SDR) TRPTYPE(TCP) CONNAME('10.193.28.12(1414),10.193.28.13(1414)') XMITQ('TQ.ART.POCB1.QMGR01') DESCR('Sender Channel to B1 queue manager')
DEFINE CHANNEL('POCGW.TO.POCB2') CHLTYPE(SDR) TRPTYPE(TCP) CONNAME('10.193.28.14(1414),10.193.28.15(1414)') XMITQ('TQ.ART.POCB2.QMGR01') DESCR('Sender Channel to B2 queue manager')

• Define auto sender channel start process:
DEFINE PROCESS(start.sndr.channel) DESCR('Autostart Sender Channel when message arrives') APPLTYPE(UNIX) APPLICID('CSQX START') USERDATA('POCGW.TO.POCB1')
DEFINE PROCESS(start.sndr.channel.b2) DESCR('Autostart Sender Channel when message arrives') APPLTYPE(UNIX) APPLICID('CSQX START') USERDATA('POCGW.TO.POCB2')


7.3.12 Configure Channels and Local Ques for B1 MQ:
Run these on running instance QMgr.
• Define Server Channel:
DEFINE CHANNEL('SYSTEM.ADMIN.SVRCONN') CHLTYPE(SVRCONN) TRPTYPE(TCP)
• Define Dead Letter Queue:
DEFINE QLOCAL(DL.ART.POCB1.QMGR01.DEAD.LETTER.QUEUE) DESCR('Dead Letter Queue')
• Define Transmission Queue:
DEFINE QLOCAL('TQ.ART.POCGW.QMGR01') TRIGGER INITQ(SYSTEM.CHANNEL.INITQ) TRIGDATA(POCB1.TO.POCGW) PROCESS(start.sndr.channel) DESCR('Transmission Queue') USAGE(XMITQ) DEFPSIST(YES)
• Define local Queues:
DEFINE QLOCAL('LQ.ART.POCB') DESCR('Local Que ART.POCB1.QMGR01') MAXDEPTH(10000) MAXMSGL(64512) MSGDLVSQ(FIFO) SHARE NOTRIGGER DEFPSIST(YES) USAGE(NORMAL)
• Define local Queue Alias:
DEFINE QALIAS('ART.POCB') DESCR('Local ART Que for B1Qmgr') TARGQ('LQ.ART.POCB')
• Define Remote Queues:
DEFINE QREMOTE('RQ.LQ.ART.POCGW1') DESCR('Remote Q B1Qmgr to GWQmgr') RNAME('LQ.ART.POCGW1') RQMNAME('ART.POCGW.QMGR01')
XMITQ('TQ.ART.POCGW.QMGR01') DEFPSIST(YES)
• Define Receiver Channel:
DEFINE CHANNEL(POCGW.TO.POCB1) CHLTYPE(RCVR) TRPTYPE(TCP) DESCR('Receiver Channel GW to B1 queue manager')
• Define Sender Channel:
DEFINE CHANNEL('POCB1.TO.POCGW') CHLTYPE(SDR) TRPTYPE(TCP) CONNAME('10.193.28.10(1414),10.193.28.11(1414)') XMITQ('TQ.ART.POCB1.01') DESCR('Sender Channel to Gateway queue manager')
• Define Process to auto start sender channel:
DEFINE PROCESS(start.sndr.channel)  DESCR(‘Autostart Sender Channel when message arrives’) APPLTYPE(UNIX) APPLICID(‘CSQX START’) USERDATA(‘POCB1.TO.POCGW’)

7.3.13 Configure Channels and Local Ques for B2 MQ:
Run these on running instance QMgr.
• Define Server Channel:
DEFINE CHANNEL('SYSTEM.ADMIN.SVRCONN') CHLTYPE(SVRCONN) TRPTYPE(TCP)
• Define Dead Letter Queue:
DEFINE QLOCAL(DL.ART.POCB2.QMGR01.DEAD.LETTER.QUEUE) DESCR('Dead Letter Queue')
• Define Transmission Queue:
DEFINE QLOCAL('TQ.ART.POCGW.QMGR01') TRIGGER INITQ(SYSTEM.CHANNEL.INITQ) TRIGDATA(POCB2.TO.POCGW) PROCESS(start.sndr.channel) DESCR('Transmission Queue to GW Qmgr') USAGE(XMITQ) DEFPSIST(YES)
• Define local Queues:
DEFINE QLOCAL('LQ.ART.POCB') DESCR('Local Que ART.POCB2.QMGR01') MAXDEPTH(10000) MAXMSGL(64512) MSGDLVSQ(FIFO) SHARE NOTRIGGER DEFPSIST(YES) USAGE(NORMAL)
• Define local Queue Alias:
DEFINE QALIAS('ART.POCB') DESCR('Local ART Que for B2Qmgr') TARGQ('LQ.ART.POCB')
• Define Remote Queues:
DEFINE QREMOTE('RQ.LQ.ART.POCGW2') DESCR('Remote Q B2Qmgr to GWQmgr') RNAME('LQ.ART.POCGW2') RQMNAME('ART.POCGW.QMGR01') XMITQ('TQ.ART.POCGW.QMGR01') DEFPSIST(YES)
• Define Receiver Channel:
DEFINE CHANNEL(POCGW.TO.POCB2) CHLTYPE(RCVR) TRPTYPE(TCP) DESCR('Receiver Channel GW to B2 queue manager')
• Define Sender Channel:
DEFINE CHANNEL('POCB2.TO.POCGW') CHLTYPE(SDR) TRPTYPE(TCP) CONNAME('10.193.28.10(1414),10.193.28.11(1414)') XMITQ('TQ.ART.POCGW.QMGR01') DESCR('Sender Channel to Gateway queue manager')
• Define Process to auto start Sender Channel:
DEFINE PROCESS(start.sndr.channel)  DESCR(‘Autostart Sender Channel when message arrives’) APPLTYPE(UNIX) APPLICID(‘CSQX START’) USERDATA(‘POCB2.TO.POCGW’)

7.3.14 Generate .binding files for GW MQ:
1 – Edit JMSAdmin.config file at /opt/mqm/java/bin/JMSAdmin.config
INITIAL_CONTEXT_FACTORY=com.sun.jndi.fscontext.RefFSContextFactory
PROVIDER_URL=file:/var/mqm/jndi
SECURITY_AUTHENTICATION=none
2 – Create jndi directory under /var/mqm
$mkdir /var/mqm/jndi
3 – Generate binding file
Cd /opt/mqm/java/bin
./setjmsenv64
MQ_JAVA_DATA_PATH is /var/mqm
MQ_JAVA_LIB_PATH is /opt/mqm/java/lib64
CLASSPATH is :/opt/mqm/java/lib/com.ibm.mq.jar:/opt/mqm/java/lib/com.ibm.mqjms.jar:/opt/mqm/samp/jms/samples:/opt/mqm/samp/wmqjava/samples
4 – Run JMSAdmin from /opt/mqm/java/bin
$./JMSAdmin
(if  you get NoClassDefFoundError Exception export CLASSPATH manually)
$export CLASSPATH=/opt/mqm/java/lib/com.ibm.mq.jar:/opt/mqm/java/lib/com.ibm.mqjms.jar:/opt/mqm/samp/jms/samples:/opt/mqm/samp/wmqjava/samples
$echo $CLASSPATH
/opt/mqm/java/lib/com.ibm.mq.jar:/opt/mqm/java/lib/com.ibm.mqjms.jar:/opt/mqm/samp/jms/samples:/opt/mqm/samp/wmqjava/samples
Run JMSAdmin again
$./JMSAdmin
(Create QCF and Q at InitCtx prompt)
InitCtx> DEF QCF(ARTQCF) QMGR(ART.POCGW.QMGR01) TRAN(CLIENT) CHAN(SYSTEM.DEF.SVRCONN) HOST(10.193.28.10) PORT(1414)
InitCtx> DEF Q(ARTQ) QMGR(ART.POCGW.QMGR01) QUEUE(LQ.ART.POCGW1)
InitCtx> end
5 – Copy JMS binding file onto Client (BW):
Copy binding file from /var/mqm/jndi/.binding
If required copy following libraries from /opt/mqm/java/lib folder
com.ibm.mq.jar
com.ibm.mq.jmqi.jar
com.ibm.mqjms.jar
connector.jar
dhbcore.jar
fscontext.jar
jms.jar
jndi.jar
jta.jar
providerutil.jar

7.3.15 Generate .binding files for B1 MQ:
1 – Edit JMSAdmin.config file at /opt/mqm/java/bin/JMSAdmin.config
INITIAL_CONTEXT_FACTORY=com.sun.jndi.fscontext.RefFSContextFactory
PROVIDER_URL=file:/var/mqm/jndi
SECURITY_AUTHENTICATION=none
2 – Create jndi directory under /var/mqm
$mkdir /var/mqm/jndi
3 – Generate binding file
Cd /opt/mqm/java/bin
./setjmsenv64
MQ_JAVA_DATA_PATH is /var/mqm
MQ_JAVA_LIB_PATH is /opt/mqm/java/lib64
CLASSPATH is :/opt/mqm/java/lib/com.ibm.mq.jar:/opt/mqm/java/lib/com.ibm.mqjms.jar:/opt/mqm/samp/jms/samples:/opt/mqm/samp/wmqjava/samples
4 – Run JMSAdmin from /opt/mqm/java/bin
$./JMSAdmin
(if  you get NoClassDefFoundError Exception export CLASSPATH manually)
$export CLASSPATH=/opt/mqm/java/lib/com.ibm.mq.jar:/opt/mqm/java/lib/com.ibm.mqjms.jar:/opt/mqm/samp/jms/samples:/opt/mqm/samp/wmqjava/samples
$echo $CLASSPATH
/opt/mqm/java/lib/com.ibm.mq.jar:/opt/mqm/java/lib/com.ibm.mqjms.jar:/opt/mqm/samp/jms/samples:/opt/mqm/samp/wmqjava/samples
Run JMSAdmin again
$./JMSAdmin
(Create QCF and Q at InitCtx prompt)
InitCtx> DEF QCF(ARTQCF) QMGR(ART.POCB1.QMGR01) TRAN(CLIENT) CHAN(SYSTEM.DEF.SVRCONN) HOST(10.193.28.12) port(1414)
InitCtx> DEF Q(ARTQ) QMGR(ART.POCB1.QMGR01) QUEUE(LQ.ART.POCB)
InitCtx> end
5 – Copy JMS binding file onto Client (BW):
Copy binding file from /var/mqm/jndi/.binding
If required copy following libraries from /opt/mqm/java/lib folder
com.ibm.mq.jar
com.ibm.mq.jmqi.jar
com.ibm.mqjms.jar
connector.jar
dhbcore.jar
fscontext.jar
jms.jar
jndi.jar
jta.jar
providerutil.jar

7.3.16 Generate .binding files for B2 MQ:
1 – Edit JMSAdmin.config file at /opt/mqm/java/bin/JMSAdmin.config
INITIAL_CONTEXT_FACTORY=com.sun.jndi.fscontext.RefFSContextFactory
PROVIDER_URL=file:/var/mqm/jndi
SECURITY_AUTHENTICATION=none
2 – Create jndi directory under /var/mqm
$mkdir /var/mqm/jndi
3 – Generate binding file
Cd /opt/mqm/java/bin
./setjmsenv64
MQ_JAVA_DATA_PATH is /var/mqm
MQ_JAVA_LIB_PATH is /opt/mqm/java/lib64
CLASSPATH is :/opt/mqm/java/lib/com.ibm.mq.jar:/opt/mqm/java/lib/com.ibm.mqjms.jar:/opt/mqm/samp/jms/samples:/opt/mqm/samp/wmqjava/samples
4 – Run JMSAdmin from /opt/mqm/java/bin
$./JMSAdmin
(if  you get NoClassDefFoundError Exception export CLASSPATH manually)
$export CLASSPATH=/opt/mqm/java/lib/com.ibm.mq.jar:/opt/mqm/java/lib/com.ibm.mqjms.jar:/opt/mqm/samp/jms/samples:/opt/mqm/samp/wmqjava/samples
$echo $CLASSPATH
/opt/mqm/java/lib/com.ibm.mq.jar:/opt/mqm/java/lib/com.ibm.mqjms.jar:/opt/mqm/samp/jms/samples:/opt/mqm/samp/wmqjava/samples
Run JMSAdmin again
$./JMSAdmin
(Create QCF and Q at InitCtx prompt)
InitCtx> DEF QCF(ARTQCF) QMGR(ART.POCB2.QMGR01) TRAN(CLIENT) CHAN(SYSTEM.DEF.SVRCONN) HOST(10.193.28.14) port(1414)
InitCtx> DEF Q(ARTQ) QMGR(ART.POCB2.QMGR01) QUEUE(LQ.ART.POCB)
InitCtx> end
5 – Copy JMS binding file onto Client (BW):
Copy binding file from /var/mqm/jndi/.binding
If required copy following libraries from /opt/mqm/java/lib folder
com.ibm.mq.jar
com.ibm.mq.jmqi.jar
com.ibm.mqjms.jar
connector.jar
dhbcore.jar
fscontext.jar
jms.jar
jndi.jar
jta.jar
providerutil.jar

MQ Series: Installation - Linux

Install MQ Series

mkdir -p /export/home/mqm

groupadd -g 358 mqm

useradd -g 358 -u 246 -d /export/home/mqm -s /bin/bash mqm

chown -R mqm:mqm /export/home/mqm

 Prepare Installable

tar xzvf mq.tar.gz

Below system packages are required

compat-gcc 
compat-gcc-c++ 
compat-libstdc++
libstdc++ 
glibc-devel 
glibc-headers
Tune  System Parameters:
# /etc/sysctl.conf
# MQ
kernel.msgmni = 1024
kernel.shmmni = 4096
kernel.sem = 500 256000 250 1024
fs.file-max = 32768
net.ipv4.tcp_keepalive_time = 300
then run “sysctl –p” to take effect the new settings.

Update GSK7
cd /opt/installbinaries/MQ
rpm -Uvh gsk7bas64-7.0-4.23.x86_64.rpm
rpm -Uvh gsk7bas-7.0-4.23.i386.rpm

Process MQ License Agreement
cd /opt/installbinaries/MQ
./mqlicense.sh

Install MQ Binary
cd /opt/installbinaries/MQ
rpm -ivh MQSeriesRuntime-7.0.1-0.x86_64.rpm
rpm -ivh MQSeriesServer-7.0.1-0.x86_64.rpm
rpm -ivh MQSeriesJava-7.0.1-0.x86_64.rpm
rpm -ivh MQSeriesKeyMan-7.0.1-0.x86_64.rpm
rpm -ivh MQSeriesJRE-7.0.1-0.x86_64.rpm
rpm -ivh MQSeriesMan-7.0.1-0.x86_64.rpm
rpm -ivh MQSeriesSamples-7.0.1-0.x86_64.rpm
rpm -ivh MQSeriesClient-7.0.1-0.x86_64.rpm
Apply MQ Patch
cd /opt/installbinaries/MQ/fixpack
rpm -Uvh gsk7bas*
rpm -Uvh MQSeriesRuntime*
rpm -Uvh MQSeriesServer*
rpm -Uvh MQSeriesJava*
rpm -Uvh MQSeriesKeyMan*
rpm -Uvh MQSeriesJRE*
rpm -Uvh MQSeriesMan*
rpm -Uvh MQSeriesSamples*
rpm -Uvh MQSeriesClient*
Verify the MQ installation
rpm -qa | grep MQ | sort./
MQSeriesClient-7.0.1-0
MQSeriesClient-U827231-7.0.1-1
MQSeriesJava-7.0.1-0
MQSeriesJava-U827231-7.0.1-1
MQSeriesJRE-7.0.1-0
MQSeriesJRE-U827231-7.0.1-1
MQSeriesKeyMan-7.0.1-0
MQSeriesKeyMan-U827231-7.0.1-1
MQSeriesMan-7.0.1-0
MQSeriesMan-U827231-7.0.1-1
MQSeriesRuntime-7.0.1-0
MQSeriesRuntime-U827231-7.0.1-1
MQSeriesSamples-7.0.1-0
MQSeriesSamples-U827231-7.0.1-1
MQSeriesServer-7.0.1-0
MQSeriesServer-U827231-7.0.1-1
su – mqm
dspmqver
Name:        WebSphere MQ
Version:     7.0.1.5
CMVC level:  p701-105-110419
BuildType:   IKAP - (Production)

WAS Application Server Hang during stop

WAS Application Server Hang during stop

action taken: reviewed submitted doc.                                  
  Two of the four running threads in each Javacore are associated to   
  the pop-up window asking for user credentials:                       
    "Signal Dispatcher" TID:0x0000000111FEB600, j9thread_t:0x0000000   
    "AWT-XAWT" TID:0x0000000114BC0700, j9thread_t:0x00000001147C2060   
  If I don't provide the credentials within one minute, then I see     
  an access denied message on the console and a security message in    
  the SystemOut.log.                                                   
  The other two threads are:                                           
  1. "Attach API wait loop" J9VMThread:0x0000000010639900, j9thread_t: 
        at com/ibm/tools/attach/javaSE/IPC.waitSemaphore(Native Method 
        at com/ibm/tools/attach/javaSE/CommonDirectory.waitSemaphore   
          (CommonDirectory.java:220)                                   
        at com/ibm/tools/attach/javaSE/AttachHandler$WaitLoop.         
          waitForNotification(AttachHandler.java:329)                  
        at com/ibm/tools/attach/javaSE/AttachHandler$WaitLoop.run      
          (AttachHandler.java:396)                                     
    IZ73533 is a known problem with the Attach API, fixed in 6.0.0     
    SR8.  Fei is already at SR9 and this thread is not consuming       
    excessive cpu (according to the topdashH output), however it       
    would be useful to understand if disabling the Attach API would    
    have an impact on the problem.                                     
     IBM IZ73533: JAVA ATTACH API THREAD CAN CAUSE HIGH CPU UTILIZAT   
       https://www-304.ibm.com/support/docview.wss?uid=isg1IZ73533     
    I suggest disabling the java attach api function by setting  the   
    following as a webcontainer custom property:                       
            -Dcom.ibm.tools.attach.enable=no                                See this link for further instructions:                            
     IBM Setting webcontainer custom properties                        
     http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21284395  
  2.   "main" J9VMThread:0x0000000010411E00, j9thread_t:0x00000000     
       Java callstack:                                                 
           at java/net/PlainSocketImpl.socketAccept(Native Method)     
           at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:451 
           at java/net/ServerSocket.implAccept(ServerSocket.java:464)  
           at java/net/ServerSocket.accept(ServerSocket.java:432)      
           at com/ibm/ws/management/tools/WsServerController.          
             waitForServerInit(WsServerController.java:238)            
           at com/ibm/ws/management/tools/WsServerStop.runTool         
             (WsServerStop.java:417)...                                
  The stopServer.sh script spawns another Java process that does the   
  work. Typically when a stopserver attempt is made, a message         
  similar to this is appended to the SystemOut.log:                    
    AdminTool     A   ADMU3201I: Server stop request issued. Waiting   
        for stop status                                                
  However, the  SystemOut.log doesn't show this message, so if the     
  suggestion in step 1 above does not yield results, I suggest         
  capturing a trace of the shutdown as  described in "collecting data  
  manually - step 3" of the Mustgather for Stop Server issues:         
   http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21201014    
  The change is to add this to the server's server.xml file:           
    startupTraceSpecification="*=info:com.ibm.ws.*=all"                
  The -trace option can also be added to command line if               
  stopServer.sh is being used on the command line. Please let me know  
  if you have any questions about this.  Corinne                       
action plan: requeue for Matt's review.      

Java Memory Leak Analyze

Tools
1. Eclipse Memory Analyzer + Diagnostic Tool Framework for Java
ftp://ftp.software.ibm.com/software/java/support/tools/dtfj/dtfj-updatesite.zip

http://www.ibm.com/developerworks/java/jdk/tools/dtfj.html


2. IBM Support Assistant (ISA)

Thin Client (Servlet + Browser).

http://www-01.ibm.com/support/docview.wss?uid=swg27024922

Network Monitoring

http://www.brendangregg.com/Perf/network.html

 Network Monitoring
The following is a guide to monitoring network interfaces and the network from the Solaris operating system.
Note: This document is not written by Sun.
Brendan Gregg, 09-Oct-2005, version 0.70.

Is your network busy?
The network often gets blamed when things are performing poorly, and perhaps this is correct - your network interfaces may be running at 100% utilisation.
What command will tell you how busy the network interface is? Many sysadmins suggest using netstat -i to find out,

$ netstat -i 1 input hme0 output input (Total) output packets errs packets errs colls packets errs packets errs colls 70820498 6 73415337 0 0 113173825 6 115768664 0 0 242 0 149 0 0 246 0 153 0 0 1068 0 552 0 0 1075 0 559 0 0 [...]

This shows packet counts per second, the first line is the summary since boot. How many packets would mean the interface is busy? 100/sec, 1000/sec?
What we do know is the speed of the network interface, for this one it is 100 Mb/sec. However we have no way of telling the size of the packets - they may be 56 byte packets or 1500 bytes. This means that the packet count is not useful, perhaps it is useful as a yardstick of activity only. What we really need is Kb/sec...

Contents

• By System - monitoring network interface usage.
• netstat - netstat, the Solaris kitchen sink network tool.
• kstat - the Kernel Statistics framework.
• nx.se - SE Toolkit's nx.se.
• nicstat - nicstat for network interface utilisation.
• SNMP - SNMP based tools include MRTG.
• Across Network - analysing external network performance.
• ping - the classic network probe tool.
• traceroute - timing hops to destination.
• TTCP - creates test load between two hosts.
• pathchar - traceroute with throughputs, an amazing tool.
• ntop - comprhensive statistics for snooped traffic.
• By Process - determining the process responsible for traffic.
• tcptop - TCP PID summary.
• tcpsnoop - watch TCP traffic live with PID.

By System
How to monitor network usage for the entire system, usually by network interface.

---

netstat
The Solaris netstat command is where a number of different network status programs have been dropped, it's the kitchen sink of network tools.
netstat -i as mentioned earlier, only prints packet counts. We don't know if they are big packets or small packets, and we can't use them to accurately determine how utilised the network interface is. There are other performance monitoring tools that plot this as a "be all and end all" value - this is wrong.
netstat -s dumps various network related counters from Kstat, the Kernel Statistics framework. This shows that Kstat does track at least some details in terms of bytes,


$ netstat -s | grep Bytes tcpOutDataSegs =37367847 tcpOutDataBytes =166744792 tcpRetransSegs =153437 tcpRetransBytes =72298114 tcpInAckSegs =25548715 tcpInAckBytes =148658291 tcpInInorderSegs =35290928 tcpInInorderBytes =3637819567 tcpInUnorderSegs =324309 tcpInUnorderBytes =406912945 tcpInDupSegs =152795 tcpInDupBytes =73998299 tcpInPartDupSegs = 7896 tcpInPartDupBytes =5821485 tcpInPastWinSegs = 38 tcpInPastWinBytes =971347352

However the byte values above are for TCP in total, including loopback traffic that never travelled via the network interfaces.
netstat -k on Solaris 9 and earlier dumped all Kstat counters,


$ netstat -k | awk '/^hme0/,/^$/'hme0: ipackets 70847004 ierrors 6 opackets 73438793 oerrors 0 collisions 0 defer 0 framing 0 crc 0 sqe 0 code_violations 0 len_errors 0 ifspeed 100000000 buff 0 oflo 0 uflo 0 missed 6 tx_late_collisions 0 retry_error 0 first_collisions 0 nocarrier 0 nocanput 0 allocbfail 0 runt 0 jabber 0 babble 0 tmd_error 0 tx_late_error 0 rx_late_error 0 slv_parity_error 0 tx_parity_error 0 rx_parity_error 0 slv_error_ack 0 tx_error_ack 0 rx_error_ack 0 tx_tag_error 0 rx_tag_error 0 eop_error 0 no_tmds 0 no_tbufs 0 no_rbufs 0 rx_late_collisions 0 rbytes 289601566 obytes 358304357 multircv 558 multixmt 73411 brdcstrcv 3813836 brdcstxmt 1173700 norcvbuf 0 noxmtbuf 0 newfree 0 ipackets64 70847004 opackets64 73438793 rbytes64 47534241822 obytes64 51897911909 align_errors 0 fcs_errors 0 sqe_errors 0 defer_xmts 0 ex_collisions 0 macxmt_errors 0 carrier_errors 0 toolong_errors 0 macrcv_errors 0 link_duplex 0 inits 31 rxinits 0 txinits 0 dmarh_inits 0 dmaxh_inits 0 link_down_cnt 0 phy_failures 0 xcvr_vendor 524311 asic_rev 193 link_up 1

Great - so bytes by network interface are indeed tracked. However netstat -k was an undocumented switch that has now been dropped in Solaris 10. That's ok, as there are better ways to get to Kstat, including the C library that tools such as vmstat use - libkstat.



---

kstat
The Solaris Kernel Statistics framework does track network usage, and as of Solaris 8 there has been a /usr/bin/kstat command to fetch Kstat details,


$ kstat -p 'hme:0:hme0:*bytes64' 1 hme:0:hme0:obytes64 51899673435 hme:0:hme0:rbytes64 47536009231 hme:0:hme0:obytes64 51899673847 hme:0:hme0:rbytes64 47536009709 [...]

Now we just need a tool to present this in a more meaningful way.



---

nx.se
The SE Toolkit provides a language, SymbEL, that lets us write our own performance monitoring tools. It also contained a collection of example tools, including nx.se which lets us identify network utilisation,


$ se nx.se 1 Current tcp RtoMin is 400, interval 1, start Sun Oct 9 10:36:42 2005 10:36:43 Iseg/s Oseg/s InKB/s OuKB/s Rst/s Atf/s Ret% Icn/s Ocn/s tcp 841.6 4.0 74.98 0.27 0.00 0.00 0.0 0.00 0.00 Name Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s hme0 845.5 420.8 119.91 22.56 0.000 0.000 0.0 0.00 0.00 10:36:44 Iseg/s Oseg/s InKB/s OuKB/s Rst/s Atf/s Ret% Icn/s Ocn/s tcp 584.2 5.0 77.97 0.60 0.00 0.00 0.0 0.00 0.00 Name Ipkt/s Opkt/s InKB/s OuKB/s IErr/s OErr/s Coll% NoCP/s Defr/s hme0 579.2 297.1 107.95 16.16 0.000 0.000 0.0 0.00 0.00 [...]

Having KB/s values lets us determine exactly how busy our network interfaces are. There is other useful information printed above, including Coll% - collisions, NoCP/s - no can puts, and Defr/s defers, which may be evidence of network saturation.



---

nicstat
nicstat is a freeware tool written in C to print out network utilisation and saturation by interface,


$ nicstat 1 Time Int rKb/s wKb/s rPk/s wPk/s rAvs wAvs %Util Sat 10:48:30 hme0 4.02 4.39 6.14 6.36 670.73 706.50 0.07 0.00 10:48:31 hme0 0.29 0.50 3.00 4.00 98.00 127.00 0.01 0.00 10:48:32 hme0 1.35 4.23 14.00 15.00 98.79 289.00 0.05 0.00 10:48:33 hme0 67.73 19.08 426.00 207.00 162.81 94.39 0.71 0.00 10:48:34 hme0 315.22 128.91 1249.00 723.00 258.44 182.58 3.64 0.00 10:48:35 hme0 529.96 67.53 2045.00 1046.00 265.37 66.11 4.89 0.00 10:48:36 hme0 454.14 62.16 2294.00 1163.00 202.72 54.73 4.23 0.00 10:48:37 hme0 93.55 15.78 583.00 295.00 164.31 54.77 0.90 0.00 10:48:38 hme0 74.84 32.41 516.00 298.00 148.52 111.38 0.88 0.00 10:48:39 hme0 0.76 4.17 7.00 9.00 111.43 474.00 0.04 0.00 [...]

Fantastic. There is also an older Perl version of nicstat available.
The following are the switches available from version 0.90 of the C version,


$ nicstat -h USAGE: nicstat [-hsz] [-i int[,int...]] | [interval [count]] -h # help -i interface # track interface only -s # summary output -z # skip zero value lines eg, nicstat # print summary since boot only nicstat 1 # print every 1 second nicstat 1 5 # print 5 times only nicstat -z 1 # print every 1 second, skip zero lines nicstat -i hme0 1 # print hme0 only every 1 second

The utilisation measurement is based on the maximum speed of the interface (if available via Kstat), divided by the current throughput. The saturation measurement is a value that reflects errors due to saturation (no can puts, etc).



---

SNMP
It's worth mentionining that there is also useful data available in SNMP, which is used by software such as MRTG. Here we use Net-SNMP's snmpget to fetch some interface values,


$ snmpget -v1 -c public localhost ifOutOctets.2 ifInOctets.2 IF-MIB::ifOutOctets.2 = Counter32: 10016768 IF-MIB::ifInOctets.2 = Counter32: 11932165

These values are the outbound and inbound bytes for our main interface. In Solaris 10 a full description of the IF-MIB values can be found at /etc/sma/snmp/mibs/IF-MIB.txt.



---

Across Network
Analysing the performance of the external network.

---

ping
ping is the classic network probe tool,


$ ping -s mars PING mars: 56 data bytes 64 bytes from mars (192.168.1.1): icmp_seq=0. time=0.623 ms 64 bytes from mars (192.168.1.1): icmp_seq=1. time=0.415 ms 64 bytes from mars (192.168.1.1): icmp_seq=2. time=0.464 ms ^C ----mars PING Statistics---- 3 packets transmitted, 3 packets received, 0% packet loss round-trip (ms) min/avg/max/stddev = 0.415/0.501/0.623/0.11

So I discover that mars is up, and it responds within 1 ms. Solaris 10 enhanced ping to print 3 decimal places for the times.
ping is handy to see if a host is up, but that's about all. Some people use it to test whether their application server is ok. Hmm. ICMP is handled in the kernel without needing to call a user based process, so it's possible that a server will ping ok while the application either responds slowly or not at all.



---

traceroute
traceroute sends a series of UDP packets with an increasing TTL, and by watching the ICMP time expired replies can discover the hops to a host (assuming the hops actually decrement the TTL),


$ traceroute www.sun.com traceroute: Warning: Multiple interfaces found; using 260.241.10.2 @ hme0:1 traceroute to www.sun.com (209.249.116.195), 30 hops max, 40 byte packets 1 tpggate (260.241.10.1) 21.224 ms 25.933 ms 25.281 ms 2 172.31.217.14 (172.31.217.14) 49.565 ms 27.736 ms 25.297 ms 3 syd-nxg-ero-zeu-2-gi-3-0.tpgi.com.au (220.244.229.9) 25.454 ms 22.066 ms 26.237 ms 4 syd-nxg-ibo-l3-ge-0-2.tpgi.com.au (220.244.229.132) 42.216 ms * 37.675 ms 5 220-245-178-199.tpgi.com.au (220.245.178.199) 40.727 ms 38.291 ms 41.468 ms 6 syd-nxg-ibo-ero-ge-1-0.tpgi.com.au (220.245.178.193) 37.437 ms 38.223 ms 38.373 ms 7 Gi11-2.gw2.syd1.asianetcom.net (202.147.41.193) 24.953 ms 25.191 ms 26.242 ms 8 po2-1.gw1.nrt4.asianetcom.net (202.147.55.110) 155.811 ms 169.330 ms 153.217 ms 9 Abovenet.POS2-2.gw1.nrt4.asianetcom.net (203.192.129.42) 150.477 ms 157.173 ms * 10 so-6-0-0.mpr3.sjc2.us.above.net (64.125.27.54) 240.077 ms 239.733 ms 244.015 ms 11 so-0-0-0.mpr4.sjc2.us.above.net (64.125.30.2) 224.560 ms 228.681 ms 221.149 ms 12 64.125.27.102 (64.125.27.102) 241.229 ms 235.481 ms 238.868 ms 13 * *^C

The times may give me some idea where a network bottleneck is. We must also remember that networks are dynamic, and this may not be the permanent path to that host.



---

TTCP
Test TCP is a freeware tool to test the throughput between two hops. It needs to be run on both the source and destination, and there is a Java version of TTCP which will run on many different operating systems. Beware, it will flood the network with traffic to perform it's test.
The following is run on one host as a reciever. The options used make the test run for a reasonable duration - around 60 seconds,


$ java ttcp -r -n 65536 Receive: buflen= 8192 nbuf= 65536 port= 5001

Then the following was run on the second host as the transmitter,


$ java ttcp -t jupiter -n 65536 Transmit: buflen= 8192 nbuf= 65536 port= 5001 Transmit connection: Socket[addr=jupiter/192.168.1.5,port=5001,localport=46684]. Transmit: 536870912 bytes in 46010 milli-seconds = 11668.57 KB/sec (93348.56 Kbps).

This shows the speed between these hosts for this test is around 11.6 Megabytes per second.



---

pathchar
After writing traceroute, Van Jacobson then went on to write pathchar - an amazing tool that identifys network bottlenecks. It operates like traceroute, but rather than printing response time to each hop it prints bandwidth between each pair of hops.


# pathchar 192.168.1.10 pathchar to 192.168.1.1 (192.168.1.1) doing 32 probes at each of 64 to 1500 by 32 0 localhost | 30 Mb/s, 79 us (562 us) 1 neptune.drinks.com (192.168.2.1) | 44 Mb/s, 195 us (1.23 ms) 2 mars.drinks.com (192.168.1.1) 2 hops, rtt 547 us (1.23 ms), bottleneck 30 Mb/s, pipe 7555 bytes

This tool works by sending "shaped" traffic over a long interval and carefully measuring the response times. It doesn't flood the network like TTCP does.



---

ntop
ntop is a tool that sniffs network traffic and provides comprehensive reports via a web interface. It is also available on sunfreeware.com.


# ntop ntop v.1.3.1 MT [sparc-sun-solaris2.8] listening on [hme0,hme0:0,hme0:1]. Copyright 1998-2000 by Luca Deri <deri@ntop.org> Get the freshest ntop from http://www.ntop.org/ Initialising... Loading plugins (if any)... WARNING: Unable to find the plugins/ directory. Waiting for HTTP connections on port 3000... Sniffying...

Now you connect via a web browser to localhost:3000.



---

By Process
How to monitor network usage by process. Recently the addition of DTrace to Solaris 10 has allowed the creation of the first network by process tools.

---

tcptop
This is a DTrace based tool from the freeware DTraceToolkit which gives a summary of TCP traffic by system and by process,


# tcptop 10 Sampling... Please wait. 2005 Jul 5 04:55:25, load: 1.11, TCPin: 2 Kb, TCPout: 110 Kb UID PID LADDR LPORT FADDR FPORT SIZE NAME 100 20876 192.168.1.5 36396 192.168.1.1 79 1160 finger 100 20875 192.168.1.5 36395 192.168.1.1 79 1160 finger 100 20878 192.168.1.5 36397 192.168.1.1 23 1303 telnet 100 20877 192.168.1.5 859 192.168.1.1 514 115712 rcp

This version of tcptop will examine newly connected sessions (while tcptop has been running). In the above we can see PID and SIZE columns, this is tracking TCP traffic that has travelled on external interfaces. The TCPin and TCPout summaries also tracks localhost TCP traffic.



---

tcpsnoop
This is a DTrace based tool from the DTraceToolkit which prints TCP packets live by process,


# tcpsnoop UID PID LADDR LPORT DR RADDR RPORT SIZE CMD 100 20892 192.168.1.5 36398 -> 192.168.1.1 79 54 finger 100 20892 192.168.1.5 36398 <- 192.168.1.1 79 66 finger 100 20892 192.168.1.5 36398 -> 192.168.1.1 79 54 finger 100 20892 192.168.1.5 36398 -> 192.168.1.1 79 56 finger 100 20892 192.168.1.5 36398 <- 192.168.1.1 79 54 finger 100 20892 192.168.1.5 36398 <- 192.168.1.1 79 606 finger 100 20892 192.168.1.5 36398 -> 192.168.1.1 79 54 finger 100 20892 192.168.1.5 36398 <- 192.168.1.1 79 54 finger 100 20892 192.168.1.5 36398 -> 192.168.1.1 79 54 finger 100 20892 192.168.1.5 36398 -> 192.168.1.1 79 54 finger 100 20892 192.168.1.5 36398 <- 192.168.1.1 79 54 finger 0 242 192.168.1.5 23 <- 192.168.1.1 54224 54 inetd 0 242 192.168.1.5 23 -> 192.168.1.1 54224 54 inetd [...]

This version of tcpsnoop will examine newly connected sessions (while tcpsnoop has been running). In the above we can see a PID column and packet details, this is tracking TCP traffic that has travelled on external interfaces.



---

Back to Brendan Gregg's Homepage

---

Created: 09-Oct-2005
Last updated: 09-Oct-2005
Copyright (c) 2005 Brendan Gregg

VERITAS

bash-3.00# ls -l /etc/VRTSvcs/conf/config/*.cf
-rw-------   2 root     root         466 Nov  1  2010 /etc/VRTSvcs/conf/config/CFSTypes.cf
-rw-------   1 root     sys          789 Jan  6  2006 /etc/VRTSvcs/conf/config/ClusterConnectorConfigType.cf
-rw-------   2 root     root         749 Nov  1  2010 /etc/VRTSvcs/conf/config/CVMTypes.cf
-rw-------   2 root     root        9186 Mar 24 12:28 /etc/VRTSvcs/conf/config/main.cf
-rw-------   1 root     root        2107 Nov 13  2007 /etc/VRTSvcs/conf/config/master.main.cf
-rw-------   2 root     root        7696 Nov  2  2010 /etc/VRTSvcs/conf/config/types.cf
-rw-r--r--   1 root     sys          244 Jul 11  2003 /etc/VRTSvcs/conf/config/VRTSWebAppType.cf


Commands:
bash-3.00# ls /opt/VRTS/bin
bmcmap              fsckpt_restore      hagui               mount               vxdco               vxinfo              vxrepquota
cfscluster          fsckptadm           haipswitch          mv                  vxdctl              vxiod               vxrestore
cfsdgadm            fsclustadm          halog               ncheck              vxddladm            vxlicinst           vxrlink
cfsmntadm           fsdb                halogin             pal33               vxdg                vxlicrep            vxrvg
cfsmount            fsenvadm            hamsg               qioadmin            vxdisk              vxlictest           vxscriptlog
cfsumount           fsmap               hanotify            qiomkfile           vxdiskadd           vxlsino             vxsd
cp                  fsppadm             hareg               qiostat             vxdiskadm           vxmake              vxsnap
cpio                fstyp               hares               setext              vxdiskconfig        vxmemstat           vxspcshow
df                  fsvmap              hasim               sparcv9             vxdmpadm            vxmend              vxstat
fcladm              fsvoladm            hastart             sqlite3             vxdump              vxmirror            vxsvc
ff                  gabconfig           hastatus            umount              vxedit              vxmsadm             vxsvcctrl
fiostat             gabport             hastop              vea                 vxedquota           vxnotify            vxtask
fsadm               getext              hasys               vradmin             vxenablef           vxplex              vxtrace
fsapadm             glmconfig           hatype              vrnotify            vxfsckd             vxprint             vxtranslog
fscat               haagent             hauser              vrport              vxfsconvert         vxquot              vxtune
fscdsadm            haattr              hawizard            vrstat              vxfsstat            vxquota             vxtunefs
fscdsconv           hacf                lltconfig           vxassist            vxgetinterfaceinfo  vxquotaoff          vxupgrade
fscdstask           haclus              lltstat             vxcache             vxgetmsg            vxquotaon           vxvol
fsck                haconf              ls                  vxclustadm          vxgettext           vxrecover
fsckall             hagrp               mkfs                vxcmdlog            vxibc               vxrelayout

# hastatus
bash-3.00# hastatus
attempting to connect....connected
group           resource             system               message            
--------------- -------------------- -------------------- --------------------
                                     cipgwuat2ap1         RUNNING            
                                     cipgwuat2ap2         RUNNING            
                                     cipgwuat2br1         RUNNING            
                                     cipgwuat2br2         RUNNING            
-------------------------------------------------------------------------
CIPG_AI_ZONE                         cipgwuat2ap2         ONLINE             
CIPG_AI_ZONE                         cipgwuat2ap1         ONLINE             
CIPG_BR_ZONE                         cipgwuat2ap1         ONLINE             
CIPG_BR_ZONE                         cipgwuat2ap2         ONLINE             
CIPG_DATA                            cipgwuat2ap1         ONLINE             
-------------------------------------------------------------------------
CIPG_DATA                            cipgwuat2ap2         ONLINE             
CIPG_DATA                            cipgwuat2br1         ONLINE             
CIPG_DATA                            cipgwuat2br2         ONLINE             
CIPG_IS_ZONE1                        cipgwuat2ap1         PARTIAL            
CIPG_IS_ZONE1                        cipgwuat2ap2         PARTIAL            
-------------------------------------------------------------------------
# hagrp -list
bash-3.00# hagrp -list
CIPGSHARE                       cipgwuat2ap1
CIPGSHARE                       cipgwuat2ap2
CIPGSHARE                       cipgwuat2br1
CIPGSHARE                       cipgwuat2br2
CIPG_AI_ZONE                    cipgwuat2ap1
CIPG_AI_ZONE                    cipgwuat2ap2
CIPG_BR_ZONE                    cipgwuat2ap1
CIPG_BR_ZONE                    cipgwuat2ap2
CIPG_DATA                       cipgwuat2ap1
CIPG_DATA                       cipgwuat2ap2
CIPG_DATA                       cipgwuat2br1
CIPG_DATA                       cipgwuat2br2
CIPG_IS_ZONE1                   cipgwuat2ap1
CIPG_IS_ZONE1                   cipgwuat2ap2
CIPG_IS_ZONE3                   cipgwuat2ap1
CIPG_IS_ZONE3                   cipgwuat2ap2
CIPG_SHARED_DISK                cipgwuat2ap1
CIPG_SHARED_DISK                cipgwuat2ap2
ClusterService                  cipgwuat2ap1
ClusterService                  cipgwuat2ap2
MQnfsSG                         cipgwuat2ap1
MQnfsSG                         cipgwuat2ap2
VxSS                            cipgwuat2ap1
VxSS                            cipgwuat2ap2
WmBroker                        cipgwuat2ap1
WmBroker                        cipgwuat2ap2
WmBroker2                       cipgwuat2ap1
WmBroker2                       cipgwuat2ap2
cvm                             cipgwuat2ap1
cvm                             cipgwuat2ap2
cvm                             cipgwuat2br1
cvm                             cipgwuat2br2