Oracle Trouble Shooting Startup Hang

$ truss -Daefo /tmp/startup.txt sqlplus / as sysdba
SQL> startup mount
SQL> alter session set tracefile_identifier='10046';
SQL> Alter session set timed_statistics = true;
SQL> Alter session set statistics_level=all;
SQL> Alter session set max_dump_file_size = unlimited;
SQL> Alter session set events '10046 trace name context forever, level 12';

SQL> alter database open;
-- Wait 5-10 minutes then kill process and shutdown immediate


+++

Please collect the Hanganalyse report from another session to identify the blocking chains

SQL> sqlplus " / as sysdba"
SQL> oradebug setmypid
SQL> oradebug unlimit
SQL> oradebug hanganalyze 3
-- wait 90 seconds
SQL> oradebug hanganalyze 3
SQL> oradebug tracefile_name
SQL> exit



++ Upload the startup.txt , 10046 trace file and Hang analyze

Solaris 11 Roll Back Patch

 # pkg info entire
          Name: entire                                         
       Summary: entire incorporation including Support Repository Update (Oracle Solaris 11.2.13.6.0).
   Description: This package constrains system package versions to the same                         
                build.  WARNING: Proper system update and correct package                           
                selection depend on the presence of this incorporation.                             
                Removing this package will result in an unsupported system.  For                    
                more information see:                                                               
                https://support.oracle.com/rs?type=doc&id=1672221.1                                 
      Category: Meta Packages/Incorporations                                                        
         State: Installed                                                                           
     Publisher: solaris                                                                             
       Version: 0.5.11 (Oracle Solaris 11.2.13.6.0)                                                 
 Build Release: 5.11                                                                                
        Branch: 0.175.2.13.0.6.0                                                                    
Packaging Date: August 10, 2015 04:16:29 PM                                                         
          Size: 5.46 kB                                                                             
          FMRI: pkg://solaris/entire@0.5.11,5.11-0.175.2.13.0.6.0:20150810T161629Z    

# beadm list          
BE        Flags Mountpoint Space  Policy Created                      
--        ----- ---------- -----  ------ -------                      
solaris   -     -          21.34M static 2016-05-27 06:46             
solaris-1 NR    /          33.96G static 2016-05-27 07:03  

# beadm activate solaris
# reboot
# pkg info entire
          Name: entire
       Summary: Incorporation to lock all system packages to the same build
   Description: This package constrains system package versions to the same
                build.  WARNING: Proper system update and correct package
                selection depend on the presence of this incorporation.
                Removing this package will result in an unsupported system.
      Category: Meta Packages/Incorporations
         State: Installed
     Publisher: solaris
       Version: 0.5.11
 Build Release: 5.11
        Branch: 0.175.2.0.0.42.0
Packaging Date: June 24, 2014 07:38:32 PM
          Size: 5.46 kB
          FMRI: pkg://solaris/entire@0.5.11,5.11-0.175.2.0.0.42.0:20140624T193832Z

ZFS Tuning Up

# http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Identify_ZFS_Tuning_Changes
set zfs:zfs_nocacheflush=1