xmlgraphics-fop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Douglas" <edoug...@blockhouse.com>
Subject RE: Fop Memory Use
Date Fri, 20 May 2011 18:44:13 GMT
Appears to be solved.
I believe I've found where the memory went.
The PNGRenderer needs a lot of memory to transform large documents.
If I'm reading this right, from the results of JVisualVM, the memory use is the pixel size.
The PNGRender stores every page as an image, which can be retrieved with getPageImage(pageNumber).
That image translates to 72 pixels per inch.
For an 11" x 8.5" document that's 11 (792) x 8.5 (612) = 484704.
It appears the images contain those int[] arrays and use up that amount (484704) x 4 + 16
bytes ( = 1938832)
I'm saving those images for print preview and loading them into a GUI window with a zoom.
For the zoom I'm just resizing the image with Java's Graphics2D.drawImage.
If I redraw that to a larger size it gets a bit blurry.  To help reduce that I was sizing
the initial image.
Fop generates a larger image if I set the methods on the FOUserAgent (setSourceResolution(72),
Plug that into the same calculation and it quadruples the memory use ((792 * 2 = 1584) * (612
* 2 = 1224)) * 4 + 16 = 7755280 bytes per page.
For a 100 page report that's a ton.
I am generating my own input (it's a custom report writer) so I know exactly what fits on
a page.  I'm writing everything with hard page breaks and absolute positioning.
I already had a memory problem just trying to create a PDF when I got around 150 pages so
I solved that by breaking it up.
I now transform all output 10 pages at a time, creating multiple PDFs (in memory) with xsl's
initial-page-number then use pdfbox to put the pages together.
The FO actually wasn't using much memory, just something Fop was doing when I tried to use
that FO to generate a parge PDF all at once.
So, I just save my FO files in an array and generate the PNG page images no more than 10 at
a time and I've limited memory use!
I'm using the PNG to split the process so I can generate output on one machine (server) and
display it on another (client).
If I only need to transform part of the document, and it's not taking much more than the Graphics2D
redraw, I can just call the render again for new page requests so no more fuzzy images.


From: Georg Datterl [mailto:georg.datterl@geneon.de] 
Sent: Thursday, May 19, 2011 12:22 AM
To: fop-dev@xmlgraphics.apache.org
Subject: AW: Fop Memory Use

Hi Eric,


That sounds interesting. If you run the transformer for each page and set a breakpoint after
the first run, there (IMHO) should not be a reference to any fop object. Ignore the int[]s
first, they are used everywhere. Concentrate on the fop objects which should not be there.
You could as well run your transformation X times and then investigate all objects which exists
exactly X (or Y*X) times in memory. Those  are probably accumulated over many runs and crash
your application sooner or later.




Georg Datterl


------ Kontakt ------


Georg Datterl


Geneon media solutions gmbh

Gutenstetter Straße 8a

90449 Nürnberg


HRB Nürnberg: 17193

Geschäftsführer: Yong-Harry Steiert 


Tel.: 0911/36 78 88 - 26

Fax: 0911/36 78 88 - 20




Weitere Mitglieder der Willmy MediaGroup:


IRS Integrated Realization Services GmbH:    www.irs-nbg.de <http://www.irs-nbg.de>

Willmy PrintMedia GmbH:                      www.willmy.de <http://www.willmy.de> 

Willmy Consult & Content GmbH:               www.willmycc.de <http://www.willmycc.de>


Von: Eric Douglas [mailto:edouglas@blockhouse.com] 
Gesendet: Mittwoch, 18. Mai 2011 19:43
An: fop-dev@xmlgraphics.apache.org
Betreff: RE: Fop Memory Use


This test run isn't using SVG at all.  The PDFRenderer is working, the PNGRenderer runs out
of memory, so it is using images but as output.

I already broke up the input to multiple FOs with multiple calls to the transform to generate
a large document by combining small documents using the initial-page-number.

As the program runs it just keeps increasing memory use.

I tried running a profile with Java's VisualVM though I'm not sure what exactly I'm looking
at or what to do with it.

The number one item showing memory hog in the profiler, as of my last snapshot was: class
int[], live bytes 23,273,952 B, live objects 382, generations 10

After the program crashed with the profiler running I had an additional file opened, Java2DRenderer.class,
so I'm assuming it's doing something that breaks PNGRenderer.

My class doesn't have any int[] references.

After that first reference the sizes drop off sharply in the profiler.  The next class reference
is char[], then org.apache.fop.area.inline.SpaceArea.




From: Peter Hancock [mailto:peter.hancock@gmail.com] 
Sent: Wednesday, May 18, 2011 12:05 PM
To: fop-dev@xmlgraphics.apache.org
Subject: Re: Fop Memory Use

Hi Eric,

Does your document contain many large SVG's?  If so take a look at  Bugzilla #46360.  This
issue was resolved in rev 997602 of FOP trunk.


On Wed, May 18, 2011 at 5:10 PM, Adrian Cumiskey <adrian.cumiskey@gmail.com> wrote:

Hi Eric,


Fop calculates layout in page sequence chunks, so try breaking up your pages into chunks of
page sequences.  Pages should be available for garbage collection once the page sequence has
been rendered.

Cheers, Adrian.

On May 18, 2011, at 7:24 AM, Michael Rubin <mrubin@thunderhead.com> wrote:

	Just a wild thought. But is there a way you could possibly get the JVM to garbage collect
between each run? Maybe that might free the memory up?
	On 18/05/11 13:20, Eric Douglas wrote: 

		I am using Fop 1.0. 
		I tried using Fop to transform a single document.  When I got a little over 100 pages my
FO file was over 5 MB.  The transform crashed with a Java heap out of memory error.

		I managed to break the input down, as I'm using embedded code generating the input programmatically,
and the PDF output is a lot smaller.

		So I'm currently transforming 10 pages at a time, setting the initial-page-number to the
next sequence (1, 11, 21, etc).

		Then I save all the generated PDFs in memory and merge them using pdfbox.  So far this is
working great. 

		I tried to do the same thing with the PNGRenderer, just calling a method to transform 10
pages at a time and save the output images in an array.

		The PNGRenderer is created locally in the method.  It should be getting released when the
method ends but the java process never releases any memory.

		I tested a 90 page report and the memory use was over 1 GB.  I tested on another machine
where the memory limit is apparently lower and it crashed on page 24.

		Everything about the method to render to PNG is the same as the method to render to PDF
aside from the Renderer. 
		Is there a problem with this renderer or something I could need to do different? 



Michael Rubin


Thunderhead Logo

Fehler! Es wurde kein Dateiname angegeben.

Fehler! Es wurde kein Dateiname angegeben.






+44 20 8238 7400 

+44 20 8238 7401 



www.thunderhead.com <http://www.thunderhead.com> 


Thunderhead featured in The Sunday Times Profit Track 100 league table of companies with fastest-growing
profits. Click here <http://www.fasttrack.co.uk/fasttrack/press/pt11-lon.pdf>  to read

Fehler! Es wurde kein Dateiname angegeben. <http://www.linkedin.com/companies/25033/Thunderhead>
Fehler! Es wurde kein Dateiname angegeben. <http://twitter.com/Thunderheadon> Fehler!
Es wurde kein Dateiname angegeben. <http://www.thunderhead.com/rss/rss.php> Fehler!
Es wurde kein Dateiname angegeben. <http://www.youtube.com/user/ThunderheadOn> Fehler!
Es wurde kein Dateiname angegeben. <http://thunderheadinnovate.wordpress.com/> Fehler!
Es wurde kein Dateiname angegeben. <http://thunderhead.com/about/careers.php> 

The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it.


View raw message