RAC One Failed because archivelog is full

$ srvctl start database -d RACTST
PRCR-1079 : Failed to start resource ora.ractst.db
CRS-5017: The resource action "ora.ractst.db start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 3208
Session ID: 577 Serial number: 1
. For details refer to "(:CLSN00107:)" in "/u01/11.2.0/grid/log/gw6/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.ractst.db' on 'gw6' failed
CRS-5017: The resource action "ora.ractst.db start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 3200
Session ID: 253 Serial number: 1
. For details refer to "(:CLSN00107:)" in "/u01/11.2.0/grid/log/gw8/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.ractst.db' on 'sun08' failed
CRS-2632: There are no more servers to try to place resource 'ora.ractst.db' on that would satisfy its placement policy


Trouble Shooting
$ srvctl start database -d RACTST -o mount

$ rman target sys/********@scan-name:1521/RACTST
RMAN> list archivelog all;
RMAN> delete archivelog all;

$ srvctl stop database -d RACTST
$ srvctl start database -d RACTST

Oracle Grant Privileges to ETL Account

Create Read Only Role

Login as Objects Owner
CREATE ROLE OWNER_READONLY;
GRANT SELECT ANY TABLE TO OWNER_READONLY;
GRANT SELECT ANY TRANSACTION TO OWNER_READONLY;

Grant Role to user

Login as sys
GRANT OWNER_READONLY TO DW_ETL;

NFS Version 4 Connection

# mount -t nfs4 vsun01:/ishare /mnt
on Solaris, nfsstat will show the connection statistics


# nfsstat                   

Server rpc:
Connection oriented:
calls      badcalls   nullrecv   badlen     xdrcall    dupchecks  dupreqs   
74683      0          0          0          0          2308       0         
Connectionless:                                                             
calls      badcalls   nullrecv   badlen     xdrcall    dupchecks  dupreqs   
9          0          0          0          0          0          0         

Server NFSv2:
calls      badcalls   referrals  referlinks
3          0          0          0         

Server NFSv3:
calls      badcalls   referrals  referlinks
7230       0          0          0         

Server NFSv4:
calls      badcalls   referrals  referlinks
54         0          0          0         
Version 2: (3 calls)                       
null     getattr  setattr  root     lookup   readlink read     wrcache 
3 100%   0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     0 0%    
write    create   remove   rename   link     symlink  mkdir    rmdir   
0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     0 0%    
readdir  statfs                                                        
0 0%     0 0%                                                          
Version 3: (7132 calls)                                                
null        getattr     setattr     lookup      access      readlink   
6 0%        1220 17%    0 0%        280 3%      530 7%      0 0%       
read        write       create      mkdir       symlink     mknod      
1758 24%    2166 30%    66 0%       0 0%        0 0%        0 0%       
remove      rmdir       rename      link        readdir     readdirplus
66 0%       0 0%        0 0%        0 0%        0 0%        10 0%      
fsstat      fsinfo      pathconf    commit                             
960 13%     2 0%        1 0%        67 0%                              
Version 4: (54 calls)                                                  
null                compound                                           
2 3%                52 96%                                             
Version 4: (115 operations)                                            
reserved            access              close               commit             
0 0%                4 3%                1 0%                0 0%               
create              delegpurge          delegreturn         getattr            
0 0%                0 0%                0 0%                41 35%             
getfh               link                lock                lockt              
7 6%                0 0%                0 0%                0 0%               
locku               lookup              lookupp             nverify            
0 0%                4 3%                0 0%                0 0%               
open                openattr            open_confirm        open_downgrade     
1 0%                0 0%                1 0%                0 0%               
putfh               putpubfh            putrootfh           read               
43 37%              0 0%                2 1%                0 0%               
readdir             readlink            remove              rename             
2 1%                0 0%                1 0%                0 0%               
renew               restorefh           savefh              secinfo            
1 0%                0 0%                0 0%                0 0%               
setattr             setclientid         setclientid_confirm verify             
1 0%                3 2%                3 2%                0 0%               
write               release_lockowner   illegal                                
0 0%                0 0%                0 0%                                   

Server nfs_acl:
Version 2: (0 calls)
null        getacl      setacl      getattr     access      getxattrdir
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%       
Version 3: (0 calls)                                                   
null        getacl      setacl      getxattrdir                        
0 0%        0 0%        0 0%        0 0%                               

Client rpc:
Connection oriented:
calls      badcalls   badxids    timeouts   newcreds   badverfs   timers    
688837     0          0          0          0          0          0         
cantconn   nomem      interrupts                                            
0          0          0                                                     
Connectionless:                                                             
calls      badcalls   retrans    badxids    timeouts   newcreds   badverfs  
0          0          0          0          0          0          0         
timers     nomem      cantsend                                              
0          0          0                                                     

Client nfs:
calls      badcalls   clgets     cltoomany 
688819     0          0          0         
Version 2: (0 calls)
null     getattr  setattr  root     lookup   readlink read     wrcache
0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     0 0%
write    create   remove   rename   link     symlink  mkdir    rmdir
0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     0 0%
readdir  statfs
0 0%     0 0%
Version 3: (0 calls)
null        getattr     setattr     lookup      access      readlink
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%
read        write       create      mkdir       symlink     mknod
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%
remove      rmdir       rename      link        readdir     readdirplus
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%
fsstat      fsinfo      pathconf    commit
0 0%        0 0%        0 0%        0 0%
Version 4: (688815 calls)
null                compound
0 0%                688815 100%
Version 4: (1785319 operations)
reserved            access              close               commit
0 0%                71039 3%            67444 3%            71049 3%
create              delegpurge          delegreturn         getattr
0 0%                0 0%                67420 3%            367238 20%
getfh               link                lock                lockt
74340 4%            0 0%                0 0%                0 0%
locku               lookup              lookupp             nverify
0 0%                7729 0%             7 0%                8742 0%
open                openattr            open_confirm        open_downgrade
67455 3%            9 0%                20 0%               0 0%
putfh               putpubfh            putrootfh           read
651851 36%          0 0%                21 0%               121 0%
readdir             readlink            remove              rename
30 0%               0 0%                28 0%               588 0%
renew               restorefh           savefh              secinfo
37539 2%            6871 0%             8310 0%             9 0%
setattr             setclientid         setclientid_confirm verify
70887 3%            3 0%                3 0%                19 0%
write
206547 11%

Client nfs_acl:
Version 2: (0 calls)
null        getacl      setacl      getattr     access      getxattrdir
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%
Version 3: (0 calls)
null        getacl      setacl      getxattrdir
0 0%        0 0%        0 0%        0 0%

Open LDAP troubleshooting

# certutil -d /etc/openldap/cacerts.pem -L

# ldapsearch -H ldaps://ldap.goweekend.ca -d 255 -x uid=aniu