Java process memory: JVM heap size constraints in 32 bit environment

In theory, on a given computer system, the addressable memory is 2^b bytes where b is bit of the processor architecture. (Although there is much more to it like PAE and stuff. )

So maximum physical memory in a 32 bit is:
2^32 = 4X(1024)X(1024)X(1024) Bytes = 4096 MB = 4 GB

Out of this 4 GB some of the memory is reserved for OS, amount varies from OS to OS. So the maximum memory any process can occupy is less than 4GB. Further, in case of java process there are other components to the memory usage - JVM heap, PermGen space and native threads.

Roughly,
JVM Heap Size + PermGen Space + (Thread stack size*number of thread) < Maximum space OS allows for processes.

So there is limit on the maximum heap size you can have practically which depends on the underlying OS.

I did a few tests to check what is the result of requesting different Max heap size values on different systems. Following table lists result of different Xmx values for each of those systems:

Xmx (CentOS 32 bit)	Xmx (Ubuntu 32 bit)	Xmx (RHEL 64 bit)	Xmx (Win7 64 bit)	Xmx (Win XP 32 bit)	Xmx (SunOS 32 bit)	Result
> 4095m	> 4095m	> 4095m	> 4095m	> 4095m	> 4095m	The specified size exceeds the maximum representable size.
4095m	4095m	4095m	4095m	4095m	4095m	Incompatible minimum and maximum heap sizes specified
4030m - 4094m	4030m - 4094m	4030m - 4094m	4000m - 4094m	4000m - 4094m	4000m - 4094m	The size of the object heap + VM data exceeds the maximum representable size
2700m - 4030m	2685m - 4030m	3730m - 4030m	1500m - 4000m	1460m - 4000m	3750m - 4000m	Could not reserve enough space for object heap
< 2700m (approx)	< 2685m (approx)	< 3730m (approx)	< 1500m (approx)	< 1460m (approx)	< 3750m	JVM initialized

Little more on the results:

1. So when you try to go above 4096M (ie 4GB) the  error is "The specified size exceeds the maximum representable size." Which is expected since 4GB is maximum addressable memory.
2. At 4095M, strangely it says "Incompatible minimum and maximum heap sizes specified", looks like value 4095 is regarded same as 0. I am not sure on this, need to do more research.
3. From roughly 4000m to 4094m it says "The size of the object heap + VM data exceeds the maximum representable size" which is again understandable since as mentioned before java process has other components too so overall java process size is going beyond 4096M (the limit on addressable memory)
4. Going lower on heap, we have "Could not reserve enough space for object heap" so now we see those OS specific limits. Each OS allows a different amount of memory for a process and subtracting the non-heap part of java process we get the limit on heap size. Even the non heap part (thread stack size and limit on threads) is OS specific.

All of these tests were performed using 32 bit jdk1.7.0_04 with default PermGen space and default thread stack size and thread limits. If you tweak these heap size can be increased a little.
So in the limited test environments (with default settings) the trend is roughly as following:

• OS being 32 bit or 64 bit has very little bearing on the results.
• Server class Unix based OS (SunOS and RHEL) > Desktop Linux (Ubuntu, CentOS) > Windows Desktop (Win7, WinXP)

Need to check this on Windows Servers too but i think results will be between Windows desktop and Linux desktop values but closer to windows desktop OS.

Evaluate XPATH in Java

This is straight forward and I think better than iterating through the document tree. There is a evaluate method available in XPATH interface.

Here is a sample code for it:

import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;

import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;


public class ApplyXPATH {

    public static void main(String[] args) {
        XPath xpath = XPathFactory.newInstance().newXPath();
        InputSource inputSource = new InputSource("XMLs/MultiNode.xml");
        String expression = "//node/data";
        NodeList outPut=null;
        try {
            outPut = (NodeList) xpath.evaluate(expression, inputSource, XPathConstants.NODESET);
        } catch (XPathExpressionException e) {
            e.printStackTrace();
        }
        for(int i=0; i<outPut.getLength();i++) {
            System.out.println(outPut.item(i).getFirstChild().getNodeValue());
        }
    }
}

Sample Input:

<root>
    <node>
        <data>123</data>
        <data>456</data>
        <junk>abc</junk>
    </node>
    <junknode>
        <junk>def</junk>
    </junknode>
    <node>
        <data>789</data>
        <junk>ghi</junk>
        <data>012</data>
    </node>
</root> 

Sample Output:

123
456
789
012 

Here I am just printing the node values but you can easily create a new XML document and append this nodelist to it.

PS: if you change the expression to //node/data/text() then to access the value you just need to do outPut.item(i).getNodeValue()

Recursive XSLT: Concat Multiple Nodes

Sharing this XSLT which I used couple of years back to concat a node set. The usecase was to extract a specific node set and concatenate their values.

Sample Input:

<root>
    <node>
        <data>123</data>
        <data>456</data>
        <junk>abc</junk>
    </node>
    <junknode>
        <junk>def</junk>
    </junknode>
    <node>
        <data>789</data>
        <junk>ghi</junk>
        <data>012</data>
    </node>
</root>

Desired Output:

<out>
  <mssg>123456789012</mssg>
  <junk>abcdefghi</junk>
</out>

XSLT:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:output indent="yes" />
  <xsl:template match="/">
    <out>
      <xsl:variable name="mssg_data">
        <xsl:call-template name="ConCat">
          <xsl:with-param name="List" select="/root/node/data" />
        </xsl:call-template>
      </xsl:variable>
      <xsl:variable name="junk_data">
        <xsl:call-template name="ConCat">
          <xsl:with-param name="List" select="//junk" />
        </xsl:call-template>
      </xsl:variable>
      <mssg>
        <xsl:value-of select="$mssg_data" disable-output-escaping="no" />
      </mssg>
      <junk>
        <xsl:value-of select="$junk_data"/> 
      </junk>
    </out>
  </xsl:template>
  <xsl:template name="ConCat">
    <xsl:param name="List" />
    <xsl:param name="result" select="''" />
    <xsl:choose>
      <xsl:when test="$List">
        <xsl:call-template name="ConCat">
          <xsl:with-param name="List" select="$List[position() &gt; 1]" />
          <xsl:with-param name="result" select="concat($result, $List[1])" />
        </xsl:call-template>
      </xsl:when>
      <xsl:otherwise>
        <xsl:value-of select="$result" disable-output-escaping="no" />
      </xsl:otherwise>
    </xsl:choose>
  </xsl:template>
</xsl:stylesheet>

Note: The similar result can be achieved by using XPATH evaluator to get text of data node set (ie /root/node/data/text()), depending on the requirement one or the other approach may be more suitable.

Looping in XSLT: Recursive Template Calls

To loop in XSLT you need to call a template recursively.

In past I have done this using two approaches:

Option-1: While Loop
Logic is similar to a while loop. So the counter is decremented till it is greater than 0.
 

 
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:output indent="no" />
  <xsl:template match="/">
    <LoopInput>
      <xsl:call-template name="CreateLoop">
        <xsl:with-param name="count" select="${numLoop}" />
      </xsl:call-template>
    </LoopInput>
  </xsl:template>
  <xsl:template name="CreateLoop">
    <xsl:param name="count" />
    <xsl:if test="$count &gt; 0">
      <xsl:call-template name="CreateLoop">
        <xsl:with-param name="count" select="$count - 1" />
      </xsl:call-template>
      <Line>
        <xsl:text disable-output-escaping="no">Line Data: </xsl:text>
        <xsl:value-of select="$count" disable-output-escaping="no" />
      </Line>
    </xsl:if>
  </xsl:template>
</xsl:stylesheet>

Option-2: For Loop
Here counter is incremented from a initial count till it reaches the desired loop count.

 
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:output indent="no" />
  <xsl:template match="/">
    <LoopInput>
      <xsl:call-template name="CreateLoop">
        <xsl:with-param name="MaxLoop" select="${numLoop}" />
        <xsl:with-param name="iLoop" select="1" />
      </xsl:call-template>
    </LoopInput>
  </xsl:template>
  <xsl:template name="CreateLoop">
    <xsl:param name="MaxLoop" />
    <xsl:param name="iLoop" />
    <Line>
      <xsl:text disable-output-escaping="no">Line Data: </xsl:text>
      <xsl:value-of select="$iLoop" disable-output-escaping="no" />
    </Line>
    <xsl:if test="$iLoop &lt; $MaxLoop">
      <xsl:call-template name="CreateLoop">
        <xsl:with-param name="MaxLoop" select="$MaxLoop" />
        <xsl:with-param name="iLoop" select="$iLoop + 1" />
      </xsl:call-template>
    </xsl:if>
  </xsl:template>
</xsl:stylesheet>

Event Driven Architecture: Using log entries as events

Let's say you want to monitor multiple applications and based on certain log events automatically trigger some mechanism - say send an email to application admin or may be start a server which went down etc etc. The easiest way to make this event driven monitoring setup, I think, is to redirect application log entries to JMS destinations instead of a logfile. Then have monitoring application subscribe to the JMS destination.

This is straightforward in log4j. Simply specify appender as JMSAppender. Below is sample configuration which is for a JMS destination of type Topic and ActiveMQ as JMS broker but this can be used for any JMS broker, all you need is client jar for it and find value of InitialContextFactoryName for it.

Once the application is restarted its logs will start going to the specified JMS destination. Here is a sample log entry as it shows up on my HermesJMS.

Recursive XSLT to filter nodes

We all know that one can surely do a lot of cool stuff with XSL but still sometimes you'll come across some small piece of code which will impress you with its simplicity and yet effectiveness. Here is my favorite piece of  XSLT code:

 
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output indent="yes"/>
  <xsl:template match="node()|@*">
    <xsl:copy>
      <xsl:apply-templates select="node()|@*"/>
    </xsl:copy>
  </xsl:template>
</xsl:stylesheet>

All it does it create a copy of input XML. But it can be modified to do a lot of neat stuff. Here are a few examples:

----------------------------- Example-1: Convert attributes to fields -----------------------------

Source XML:

<root att1="att1">
  <child1  att2="att2">
    <child2  name="field_name1" value="field_value1"/>
    <child2  name="field_name2" value="field_value2"/>
  </child1>
</root>

Lets say my target application does not like attributes and it wants data as XML nodes. So the desired output is:

<?xml version="1.0" encoding="UTF-8"?>

<root att1="att1">
    <child1 att2="att2">
        <field_name1>field_value1</field_name1>
        <field_name2>field_value2</field_name2>
    </child1>
</root>

Here is the XSL code:

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output indent="yes"/>

  <xsl:template match="node()|@*">
    <xsl:copy>
      <xsl:apply-templates select="node()|@*"/>
    </xsl:copy>
  </xsl:template>

  <xsl:template match="child2">
    <xsl:element name="{@name}">
      <xsl:value-of select="@value"/>
    </xsl:element>
  </xsl:template>

</xsl:stylesheet>

----------------------------- Example-2 Filter Specific Data -----------------------------

Source XML:

<?xml version="1.0" encoding="UTF-8"?>
<root>
  <node>
    <data1>null</data1>
    <data2>data</data2>
    <data3/>
    <data4>null</data4>
    <data5></data5>
  </node>
</root>

Desired Output is:

<?xml version="1.0" encoding="UTF-8"?>
<root>
  <node>
    <data2>data</data2>
    <data3/>
    <data5/>
  </node>
</root>

This is very common in integration scenarios. Application-A gives out data will no value as 'null' values but Application-B takes this 'null' as a string. So best way forward is to filter out these 'null' nodes.

Here is the XSL code:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output indent="yes"/>

  <xsl:template match="node()|@*">
          <xsl:if test=". !='null'">
            <xsl:copy>
              <xsl:apply-templates select="node()|@*"/>
            </xsl:copy>
          </xsl:if>
  </xsl:template>


</xsl:stylesheet>

Awesome isn't it? Other way would be to put a check on each node for value 'null' before it is sent to Application-B, which would be just painfully annoying and plainly a bad approach.


----------------------------- Example-3: Update Selective Nodes -----------------------------

How about the other way around. So now we have some empty nodes and we want their value set as 'null'

Source XML:

<root att1="a1">
  <child att2="a2">
    <data att3="a3"></data>
    <data att3="a3">data</data>
    <data att3="a3"/>
  </child>
</root>

Desired Output:

<?xml version="1.0" encoding="UTF-8"?>
<root att1="a1">
    <child att2="a2">
        <data>null</data>
        <data att3="a3">data</data>
        <data>null</data>
    </child>
</root>

Here is the XSL code:

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output indent="yes"/>

  <xsl:template match="node()|@*">
          <xsl:choose>
          <xsl:when test="string-length(.) = 0">
            <xsl:element name="{name()}">
                <xsl:text disable-output-escaping="no">null</xsl:text>
            </xsl:element>
          </xsl:when>
          <xsl:otherwise>
            <xsl:copy>
              <xsl:apply-templates select="node()|@*"/>
            </xsl:copy>
          </xsl:otherwise>
        </xsl:choose>
  </xsl:template>
</xsl:stylesheet>

These are just a few examples but we can use this XSL in many ways to reformat/restructure a XML. There is always a possibility for a better approach. So guys if you know anything better then do update me, either in comments or just shoot me a mail.

32 bit, 64 bit confusions: ODBC connection

Often I have seen people get confused in an mixed 32 bit, 64 bit environment. Mostly people will have 64 bit OS like Windows 7, 64 bit with 32 bit applications like MS office 2007. Like recently I found suddenly a simple JDBC code (using JDBC-ODBC bridge) which was reading Excel file, started failing in a new environment. Just listing down things to keep in mind:

- Check if you have 32 bit Java or 64 bit Java?
- Check if you have 32 bit ODBC drivers or 64 bit ODBC drivers.
- Finally check if you are using JDBC connection URL for the version of ODBC drivers you have.

So, in 32 bit environment:
run C:\windows\sysWOW64\odbcad32.exe and see if the drivers are present for excel or not.
If not then install AccessDatabaseEngine.exe from Microsoft Download Center.

Now the connection URL in the JDBC code will be:
jdbc:odbc:Driver={Microsoft Excel Driver (*.xls)};DBQ=<Path to .xls file>

And in a 64 bit environment:
Run C:\windows\system32\odbcad32.exe and see if the drivers are present.
If not then install AccessDatabaseEngine_x64.exe from the link mentioned above.

Now the connection URL will be:
jdbc:odbc:DRIVER={Microsoft Excel Driver (*.xls, *.xlsx, *.xlsm, *.xlsb)};DBQ=<Path to .xls file>

Sample code:

Class.forName( "sun.jdbc.odbc.JdbcOdbcDriver" );

//using DSN-less connection in 32 bit env
con = DriverManager.getConnection( "jdbc:odbc:Driver={Microsoft Excel Driver (*.xls)};DBQ=C:/Abhinay/test.xls")

//using DSN-less connection in 64 bit env

//con = DriverManager.getConnection( "jdbc:odbc:DRIVER={Microsoft Excel Driver (*.xls, *.xlsx, *.xlsm, *.xlsb)};DBQ=C:/Abhinay/test.xls");

stmnt = con.createStatement();
String query = "select * from [Sheet1$];";
ResultSet rs = stmnt.executeQuery( query );