linuxProblems

Okay so you want check out a tomcat (or any other java process for that matter) in jvisualvm, but the tomcat is stuck out in the DMZ and the firewalls are blocking the RMI. Fear not friends for there is a way, and I've scripted it to make it even easier :-)

Server Side

Copy the following script to the remote server (note you may need to tune some of the settings in the config section) and chmod 755:

[quick@centos ~]$ cat /tmp/jserver.sh 
#!/bin/sh

#---------------------CONFIG--------------------------------------------#

JAVA_HOME=/usr/java/latest
PATH=$JAVA_HOME/bin:$PATH

policyfile=/tmp/statd.all.policy
jstatport=1099

#------------------END OF CONFIG----------------------------------------#

# write out the policy file
cat> $policyfile <<EOF
grant codebase "file:\${java.home}/../lib/tools.jar" {
          permission java.security.AllPermission;
       };
EOF

# * start up rmiregistry if the default port 1099 is already used on the server
# * remember to set jstatport above to a different port and uncomment this line
#rmiregistry $jport &

# start jstatd and remember its pid
jstatd -J-Djava.security.policy=$policyfile -p $jstatport &
pid=$!

# give jstatd time to start, then get the rmi port
sleep 2;
rmiport=`ss -lpn | grep $pid | grep -v $jstatport | awk '{print \$3}' | sed 's/.*://'`


echo "Now run the following commands to set up tunnels on the client:"
echo "ssh -Nf -L $jstatport:localhost:$jstatport " `hostname`
echo "ssh -Nf -L $rmiport:localhost:$rmiport " `hostname`

Now execute the script on the remote server as root or the user running the process you're interested in. Unless you're a privileged user, you won't see any other processes apart from your own:

[quick@centos ~]$ sudo /tmp/jserver.sh
Now run the following commands to set up tunnels on the client:
ssh -Nf -L 1099:localhost:1099  centos.linuxproblems.org
ssh -Nf -L 50722:localhost:50722  centos.linuxproblems.org

Client Side

Now on your client desktop run the commands as you were told by the script above, for example:

[quick@localhost ~]$ ssh -Nf -L 1099:localhost:1099  centos.linuxproblems.org
[quick@localhost ~]$ ssh -Nf -L 50722:localhost:50722  centos.linuxproblems.org

Once that is done, you're ready to run jvisualvm.

[quick@localhost ~]$ export JAVA_HOME=/usr/java/latest
[quick@localhost ~]$ export PATH=$JAVA_HOME/bin:$PATH
[quick@localhost ~]$ jvisualvm

In jvisualvm, add a jstatd connection on the localhost branch for port 1099 (default) or if you customised jstatport in the jserver.sh script above, use that. Hopefully you see your DMZ tomcat process.

If you had any problems with that (as I did), it may help to move/delete ~/.visualvm and start jvisualvm again.

Jedi Level

Now you have set up the ssh tunnels to your desktop, experience the power of the command line!

[quick@localhost ~]$ jps -l -m -v rmi://localhost
11410 sun.tools.jstatd.Jstatd -p 1099 -Dapplication.home=/usr/java/jdk1.7.0_02 -Xms8m -Djava.security.policy=/tmp/statd.all.policy
24373 org.apache.catalina.startup.Bootstrap start -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -Dcatalina.base=/usr/share/tomcat6 -Dcatalina.home=/usr/share/tomcat6 -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat6/temp -Djava.util.logging.config.file=/usr/share/tomcat6/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager

[quick@localhost ~]$ jstat -gcutil 24373@localhost 1000
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT   
  0.00   0.00   1.93  64.51  99.97    437    1.227   407  119.854  121.081
  0.00   0.00   1.93  64.51  99.97    437    1.227   407  119.854  121.081
  0.00   0.00   1.93  64.51  99.97    437    1.227   407  119.854  121.081

I tried these commands too but the force was weak, possibly java1.7 talking to openjdk 1.6?

jmap 24373@localhost
jinfo 24373@localhost

And failed with

[quick@localhost ~]$ jinfo 24373@localhost
Attaching to remote server 24373@localhost, please wait...
Error attaching to remote server: java.rmi.NotBoundException: SARemoteDebugger_24373

Another way

Set up a socks proxy. Anything connecting to port 32001 locally, will be forwarded to the remote host (centos).

[quick@localhost ~]$ ssh -Nf -D 32001 centos

[quick@localhost ~]$ /usr/java/latest/bin/jstat -gcutil -J-DsocksProxyHost=localhost -J-DsocksProxyPort=32001 24373@centos:1099
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT   
  0.00   0.00  10.15  64.51  99.97    428    1.214   398  117.263  118.478