Hi Reddy,
Open the Config tool , and increase the Heap Memory.
Also you need to check the four other parameter in java parameter of the instance
Can you please paste the content or screenshot of window.
You have not define the java parameter properly so GC is not able to get the appropriate memory for it execution.
Also check the license as suggested.
With Regards
Ashutosh Chaturvedi