Configuring Memory Parameters on AIX
for Sybase EAServer
Author: Samir Nigam, EAServer Engineering, Sybase
This document describes tuning options available to customers running EAServer 4.2 on AIX. These recommendations are based on research conducted in a lab environment using EAServer 4.2 running IBM JRE 1.3.1 build ca131-20021107 on AIX 4.3.3 and 5.1. This discussion applies to IBM JRE 1.4.x as well.
This document is not a comprehensive listing of all available options. For further details, see the IBM Web site, the AIX system documentation, and the other references listed at the end of this document.
Who should read this document?
If you are planning on deploying your EAServer application to AIX machine, then you should read this. Intended audience for this document is system administrators, application testers, and to some extent application developers.
Sybase EAServer is built using a mix of code written in Java and C/C++ to utilize the best of both native code (C/C++) and platform independent code (Java). Therefore, some parts of EAServer make JNI calls (calling Java code from C/C++ code, or vice versa).
On all UNIX systems, the operating system reserves a pool of memory when the EAServer process starts, out of which JVM reserves a chunk and does its own memory management to process Java memory allocations and garbage collection. The size of the JVM memory chunk is based on setting of minimum and maximum heap sizes specified in the EAServer server properties.
EAServer relies on the OS kernel attributes and properties to pre-specify its memory needs. The size of available memory to a process is guided by the standard OS kernel settings and further tuned by using limit or ulimit commands. However, on AIX for large memory requirements these are most often not enough and you must configure the additional settings described here.
AIX programs using more than 256MB of memory must use the “Large Address Space” model to request the necessary amount of memory. In this model, AIX breaks memory into data segments of 256MB. You can reserve a number of data segments for a process based on the value of o_maxdata attribute in XCOFF header of the process executable. This value can be set either when creating the executable by passing –bmaxdata option to ld or by setting MAXDATA value in LDR_CNTRL environment variable. These segments are used by OS for memory requests that come from malloc/new and not mmap type allocations within a process. EAServer uses malloc/new, whereas the IBM JVM uses either depending on the size of the JVM maximum heap size setting (-Xmx).
The EAServer executable jagsrv is built with –bmaxdata=0x40000000 (4 data segments) which enables memory allocation up to 1GB (4 x 256MB) total for both the native (C/C++) side and java side (JVM memory). For most applications this is sufficient but in some cases it may be required to further tune this value to avoid problems. You can do so by adding following line in $JAGUAR/bin/user_setenv.sh script.
LDR_CNTRL=MAXDATA=0xN0000000 export LDR_CNTRL
Where N is number of data segments. If $JAGUAR/bin/user_setenv.sh does not exist, create the file in a text editor and assign it executable permissions.
The maximum value of MAXDATA is 0x80000000, allowing you to reserve 8 segments. There are total of 10 data segments available to a process, meaning that 2 of the available segments cannot be reserved. Some of these 2 segments’ memory is used for process stack and text, and the remainder is available for application memory.
IBM proposes the following formula to calculate the value of MAXDATA:
Total Memory = Native Memory Requirements + Java Memory Requirements
Number Of Data Segments Required = (Total Memory + 255MB)/256MB
The “Number Of Data Segments Required” is a rough estimate of data segments which can be passed as “N” value for MAXDATA. However, as explained below, this rule does not hold for some JVM heap size values. If MAXDATA is configured incorrectly, memory problems can arise as this is total memory available to EAServer to satisfy both native memory requirements and java memory requirements.
Setting MAXDATA results in reserving memory which will be used to satisfy all memory allocation needs of the process. Since this memory is used to serve both the native and java side requests, the tuning of JVM minimum and maximum heap size should be done carefully. You must also consider the dual-personality heap allocation of IBM’s JVM 1.3.x. By default the JVM heap is allocated using malloc, which means MAXDATA reserved data segments are used for java memory allocation. However, when the heap size is greater than 1GB (actually 999MB in our testing), JVM uses mmap which comes out of data segments not reserved by MAXDATA.
Let’s consider an example in which your java application running on EAServer requires about 1M memory per client with expected peak load of 1200 concurrent clients. This load pushes the peak memory requirement to about 1.5GB (including allocations internal to EAServer).
In this example, the default setting of MAXDATA for EAServer process (0x40000000) will not appear to be enough to satisfy the peak load. So, we would change MAXDATA to 0x60000000 or above. Also, the JVM maximum heap size to allow java memory growth to 1.5GB . By default EAServer JVM maximum heap size is 1.5GB, so we don’t need to change it. Now, let’s examine what happens when the EAServer process starts.
The OS kernel, based on the EAServer MAXDATA setting, reserves 6 segments for the EAServer process., leaving 4 unreserved. EAServer then initializes the JVM with maximum heap size setting of 1.5GB. Since the heap size is greater than 1GB, JVM uses mmap to reserve 1.5GB of heap memory. Recall that mmap obtains memory from segments not reserved by MAXDATA. The 4 unreserved data segments are not enough to mmap memory of 1.5GB. The EAServer will fail to initialize the JVM and following message is printed on EAServer console.
Unable to mmap Java heap of requested size, perhaps the maxdata value is too large - see Java README.HTML for more information.
To address this, you actually must decrease the MAXDATA back to 0x40000000 (EAServer default) which leaves 6 unreserved data segments, thereby allowing the JVM to call mmap to get 1.5 GB. This setting also leaves 4 data segments (1GB) for sole use by the native code in EAServer.
This example may seem counterintuitive, but you must consider the technical background to arrive at the correct tuning of MAXDATA for your application.
Let’s consider another example involving a Java application making native calls to C code. Assume the java side allocates ½ MB and C code allocates another 1 MB per client request. Assuming there could be 800 concurrent clients, the total memory requirement would roughly be 1.5 GB: 512 MB for Java plus 1GB for native.
In this example, JVM heap size can be set to 512MB, which will be allocated from MAXDATA reserved segments. With default EAServer MAXDATA value of 0x40000000, the process can allocate up to 1GB out of which 512MB of JVM memory will take 2 data segments. Therefore, with the default MAXDATA value, there is a good chance of memory problems in the native code under peak loading conditions because native has only 512MB available (from the remaining two reserved data segments) while it needs 1GB for peak load.
For this example, MAXDATA to 0x60000000 reserves 1.5GB out of which 1GB from 4 data segments (other 2 used by JVM heap to allocate 512MB) are used for the native code memory requests under peak load.
Consider another scenario where Java needs 1.75GB memory. In such case, the JVM calls mmap (since JVM heap size is 1.75GB) and succeeds only if at least 7 data segments are unreserved. Therefore, the MAXDATA value must be 0x30000000 or less. This setting leaves about 768MB or less available to the native code.
Another case occurs if you set JVM heap size for which MAXDATA value is not sufficient. For example, consider setting JVM heap size to 868MB and MAXDATA to 0x30000000. You will see the following error as EAServer initializes the JVM:
Unable to alloc heap of requested size, perhaps the maxdata value is too small - see README.HTML for more information.
**Out of memory, aborting**
*** panic: JVMST017: Cannot allocate memory in initializeMarkAndAllocBits(markbits1)
You can resolve this error by either reducing the max heap size to 768MB or less, or by increasing the MAXDATA to 0x40000000 or more. These changes leave at least 256MB available for use by native code. You might try a heap size of 768M and MAXDATA of 0x30000000 to allow the server and JVM to start, but these settings leave no data segments available to native code (other than the 2 that cannot be reserved). Therefore this configuration is prone to unexpected problems.
The table below lists feasible values of MAXDATA and JVM heap size based on java and native memory requirements.
Native Memory Considerations
As described above, the MAXDATA and JVM heap size settings together determine how much memory is available for the native code memory requests in EAServer. The EAServer kernel is written in C/C++ and therefore it is highly important that sufficient memory is always available for allocation in the EAServer’s own internal native code as well the application’s native code if there is any.
Following problems can occur under stress load conditions due to potential memory issues in case of inadequate JVM heap size and/or MAXDATA setting:
1. Unexpected exceptions such as java.lang.OutOfMemory, java.lang.ClassCastException, java.lang.UnsatisfiedLinkError, java.lang.ArrayIndexOutOfBoundsException, java.lang.ArrayStoreException.
2. Silent JVM crash with native thread stack in JVM garbage collector code.
3. Any of the following: frequent GC cycles, long GC cycles, long GC pauses.
While testing on AIX platform, we found that for most common situations the default EAServer MAXDATA setting of 0x40000000 works fine with the default JVM heap sizes (minimum: 64MB and maximum: 1.5GB). However, when we put EAServer under heavy load and JVM is initialized with “JIT enabled”, we found setting the minimum heap size higher (256MB or above) helps avoid unexplained server crashs. We did not need to tune any other garbage collection parameters such as min free, and so forth to resolve the unexplained crash. However, tuning garbage collection parameters may help overall performance by reducing garbage collection overhead.
Sybase recommendations for customers running EAServer on AIX are listed below.
These recommendations have been arrived at in consultation with IBM and the
IBM documentation references listed at the end of this document:
1. Carefully evaluate application as well as EAServer internal memory requirements. EAServer internal memory requirements are based on server properties such as maximum connections (pre-allocates memory based on this value) and on performance features such as caching, instance pooling, etc. Even though an application may only consist of java components or just native components, both JVM and native memory requests are made in EAServer internally. So it is important to look at both aspects of memory usage and tune appropriately.
2. Increase your EAServer JVM minimum heap size (Property name com.sybase.jaguar.server.jvm.minHeapSize). You can turn on JVM verbose GC option to see more details on garbage collection cycles and behavior for application runtime memory usage characteristics.
3. Set LDR_CNTRL=MAXDATA based on the application needs using the formula and examples described above, including the table of feasible values for the JVM heap size and MAXDATA.
4. If unexplained server problems such as hang, crash continue to occur, try running EAServer with JIT disabled (Property name: com.sybase.jaguar.server.nojit=true) and see if that helps while still meeting your performance requirements.
5. Please check with IBM to see availability of any new AIX patch (APAR) for the JVM.
If the problem still persists, customers with valid
support contract can call Sybase technical support at 1 800-8-SYBASE. You can
also post your questions to the support newsgroups below, available at no charge
on news server
To purchase technical support from Sybase's award-winning support organization call 1-800-8-SYBASE.
1. IBM document on AIX Large Program Support http://publib.boulder.ibm.com/doc_link/en_US/a_doc_lib/aixprggd/genprogc/lrg_prg_support.htm.
2. AIX Performance Management Guide http://www16.boulder.ibm.com/pseries/en_US/aixbman/prftungd/prftungd02.htm
IBM Document on Porting Java Applications to AIX
Fine Tuning Java Garbage Collection Performance
5. IBM AIX JDK 1.3.1 Release notes documentation.