Popular @
Forum Controls
Spotlight Features
The Rich Engineering Heritage Behind Dependency Injection
Andrew McVeigh takes us on a tour of the rich heritage behind dependency injection, what it represents, and tells us why its here to stay.
NetBeans 6: Matisse Updates
NetBeans 6 delivers great updates to the Matisse GUI builder. Spend a few minutes with Roman Strobl and get an expert briefing on what's new and what has changed. (sponsored)
Introduction to Groovy Part 3
In this, the third and final installation of Andres' Introduction to Groovy series, you learn about how Groovy handles variable numbers of arguments, named parameters, currying, and more about Groovy operators. Including, some new operators.
Easier Custom Components with Swing Fuse
Swing Fuse (actually just Fuse), is a framework designed to make it easier to create your own custom desktop components. In this article, Daniel Spiewak shows you how to get started and provides sample source code you can download.
Benchmark Analysis: Guice vs Spring
Willam Louth shows how he uses JXInsight Probes to investigate probable performance issues with code bases that he is not familiar with. He also highlights possible pitfalls in creating a benchmark, as well as in the analysis of results.
JRoller Memory Headaches: Any Ideas From Readers?
At 8:22 AM on Jul 30, 2007, Michael Urban wrote:
Fresh Jobs for Developers Post a job opportunity
In a nutshell, it goes something like this:
.. 471859200 bytes .. (Sun Jul 29 11:26:50 EDT 2007)
.. 471859200 bytes .. (Sun Jul 29 11:26:55 EDT 2007)
.. 471859200 bytes .. (Sun Jul 29 11:27:00 EDT 2007)
.. 471859200 bytes .. (Sun Jul 29 11:27:05 EDT 2007)
.. 554211328 bytes .. (Sun Jul 29 11:27:11 EDT 2007)
.. 648875008 bytes .. (Sun Jul 29 11:27:16 EDT 2007)
.. 695051776 bytes .. (Sun Jul 29 11:27:21 EDT 2007)
.. 812158464 bytes .. (Sun Jul 29 11:27:26 EDT 2007)
.. 882390528 bytes .. (Sun Jul 29 11:27:31 EDT 2007)
.. 955197952 bytes .. (Sun Jul 29 11:27:36 EDT 2007)
.. 1112053760 bytes .. (Sun Jul 29 11:27:41 EDT 2007)
.. 1193363456 bytes .. (Sun Jul 29 11:27:46 EDT 2007)
.. 1286749696 bytes .. (Sun Jul 29 11:27:51 EDT 2007)
.. 1393844224 bytes .. (Sun Jul 29 11:27:57 EDT 2007)
.. 1492004352 bytes .. (Sun Jul 29 11:28:03 EDT 2007)
.. 1492004352 bytes .. (Sun Jul 29 11:28:08 EDT 2007)
As you can see, it chugs along fine... And sometimes will go for hours just fine, using right around 450 - 500 Mb of RAM.
Then, all of the sudden, something happens where it just starts to shoot up out rapidly for some reason we have yet to be able to track down. As you can see, memory went from about 472 Mb, up to 1492 Mb in just one minute.
We are pretty sure the problem is originating within the Velocity template system somewhere. One thing we are considering is the possibility of one very misbehaved template somewhere that is causing this when that particular template gets hit.
However, tracking that down is proving difficult. What would be ideal, would be if it were possible to write an extension to JMX or the JDI that would allow us to see exactly what template got accessed when the memory use suddenly spirals out of control. But it doesn't seem that either JMX or JDI has the ability to do this.
Anyone out there have any debugging-fu suggestions on how we might be able to track this elusive memory problem down?
17 replies so far (
Post your own)
Re: JRoller Memory Headaches: Any Ideas From Readers?
I've had similar memory nightmares in the past, and the tool that always guided me was jmap/jhat (both come with Java 5, 6 etc). jmap does a full heap dump (you can even include weakly reachable things) of a running java VM process, and jhat allows you to analyze the produced heap dump.jhat comes with a handy web server so you can navigate the heap analysis using a browser. It also provides useful statistics like memory taken per object class, and allows you to examine ownership chains. Just make sure you have lots of RAM memory on the machine you'll be performing the analysis.
http://java.sun.com/javase/6/docs/technotes/tools/share/jmap.html
In the past, I've used this pair to track similar issues (which in my case turned to be because some server-side MySQL PreparedStatements weren't being closed, and each of them retained quite a lot of info)
Re: JRoller Memory Headaches: Any Ideas From Readers?
Hi,I would try to intercept the calls to the Velocity engine when the template is going to be merged with the context.
Then you could log the current template name and the current memory consumption. Even if this is not very accurate it will help you to determine if you are looking for the right subsystem.
Probably some AOP will help here. Creating an around advice could easily track the needed information before the merge happens and right after the the template was filled with the context.
Good luck!
Re: JRoller Memory Headaches: Any Ideas From Readers?
It's not clear in your original post if this is a memory leak or just a rapid rise in memory usage. Does this ever get GCed? Do you get out of memory errors?Re: JRoller Memory Headaches: Any Ideas From Readers?
Its a rapid rise of memory. Under the IBM JVM, we rarely see any significant increase that isn't GC'd until this rapid climb. Once this rapid climb occurs, the system is unable to GC in both Sun's JVM and IBMs and it flops over.bestuff.com - the best stuff in the world
Re: JRoller Memory Headaches: Any Ideas From Readers?
How about doing what Alexander suggested, and with some tricks to allow you to home in on the offending references. Run until you get that run-away condition, then run jmap (no need to run the JVM with any special options, because the JVM allows for instrumentation from the local machine).Replace pid with the process id of the jvm. 'jps' is a handy ps list of the java processes. I don't know if you're using windows and I don't know if 'jps' works on windows.
Run jhat using the dump file. Then on the first webpage page, near the bottom, pick "Show instance counts for all classes (excluding platform)".
What I do often is I run
Run the app some more, wait for more memory to be allocated, then do another dump, restart jhat, and do another wget, this time saving to counts.2
I then do a side by side diff on counts.1 and counts.2 (tkdiff or vim -d ?) and see what's different.
If *that* doesn't help home in on the offending references, consider using (buying) JProbe. It's a got all kinds of ways of making snapshots and analyzing them.
Re: JRoller Memory Headaches: Any Ideas From Readers?
Oh, and I forgot. In Java 6 at least, you have the-XX:+HeapDumpOnOutOfMemoryError
JVM option, which produces a HeapDump automatically whenever a process runs out of memory.
Re: JRoller Memory Headaches: Any Ideas From Readers?
Might want to limit your heap to something manageable, maybe around 768MB..Then enable -XX:+HeapDumpOnOutOfMemoryError, then use it with jhat.
I personally don't like the JHAT tool, I'd rather use IBM's heap analyzer, or HP's JMeter. The only problem is that those two tools don't work with the binary output produced by the flag
Re: JRoller Memory Headaches: Any Ideas From Readers?
The HeapDumpOnOutOfMemory is also available on 1.4.2_12 and up.Re: JRoller Memory Headaches: Any Ideas From Readers?
But then you only get a dump when it runs out of memory, not?Manually running jmap lets you take a dump between operations, and compare to see which one is browner
Re: JRoller Memory Headaches: Any Ideas From Readers?
> But then you only get a dump when it runs out of> memory, not?
>
> Manually running jmap lets you take a dump between
> operations, and compare to see which one is browner
> ;-)
Correct, but sometimes you don't know when the server is going to flop. In my case at least, the server exhibited normal behaviour and memory use right until the minute or so before it crashed. Since I didn't know when a crash was going to happen, I used that option.
Re: JRoller Memory Headaches: Any Ideas From Readers?
If you can run JRockit, I hightly recommend it.I've had better luck with their Memory Leak Detection tool (yes, it is actually called that) than with anything else.
Re: JRoller Memory Headaches: Any Ideas From Readers?
Could this be a capacity issue and not just a particular rogue template?The following article shows how to go about resolving out of memory exceptions in such cases.
http://www.jinspired.com/products/jxinsight/outofmemoryexceptions.html
You could leave out the memory allocation tracking and instead focus on profiling what was currently executing by writing a JXInsight terminal script that recorded a profile and metric snapshot every few minutes followed by a reset. With our servlet trace extension you would then be able to determine the actual http request patterns during these intervals.
http://www.jinspired.com/products/jxinsight/servletconfig.html
Our new early access builds of JXInsight 5.5 also make this relatively easy and provide a comprehensive diagnostics request imaging similar to what you might have attempted with JDI.
http://blog.jinspired.com
With the level of traffic hitting many of the JL network sites do you not think it is time to get serious about service management? What tools do you currently use in development, testing and production?
regards,
William Louth
JXInsight Product Architect
CTO, JINSPIRED
"Performance monitoring and problem diagnostics for Java EE, SOA, and Grid Computing"
http://www.jinspired.com
Re: JRoller Memory Headaches: Any Ideas From Readers?
Hopefully you don't have a complex interdependency where a person enters value X on page 1 and then subsequently page 2 causes the memory blowout.Assuming that's not the case, I'd use two approaches:
1) Log when a page request comes in and also when it is complete. Each with a unique ID. Then check for requests that come in and don't finish. This will pick up any page that enters but doesn't leave.
2) Write a shell script that loops and every 1 - 5 seconds checks the memory usage of your VM. If it exceeds your normal, send a SIGQUIT to dump the stack and then run jmap for a baseline. Continue running these to get a differential and fine the problem.
Assuming that the problem is confined to a single page, those should find it.
Good luck!
David
Re: JRoller Memory Headaches: Any Ideas From Readers?
It would be nice to find out how you end up finding the cause and solution you used.