Hi Subbu
1. Check the transaction code or any report why its goes to PRIV mode? you can enable to ST12 "Single transaction trace analysis" find the over the performance of the transaction or report, Kindly refer the below link how to use the transaction code ST12 http://scn.sap.com/docs/DOC-44143
2. Why the Transaction code or reports are going to PRIV mode? for SAP link reference
Private Memory (SAP Library - SAP Memory Management (BC-CST-MM))
3. AIX base OS & SAP on AIX Memory parameter tuning Note & links
http://www.ibm.com/developerworks/aix/library/au-aix6tuning/
https://www-950.ibm.com/events/wwe/grp/grp017.nsf/vLookupPDFs/AIX_Oracle_Tuning_2013/$file/AIX_Oracle_Tuning_2013.pdf
973227 - AIX Virtual Memory Management: Tuning Recommendations
1780029 - High physical memory utilization on AIX
941735 - SAP memory management for 64-bit Linux systems
Regards
Sriram