ant-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject Re: [PATCH] Proposed StreamPumper Hacks
Date Mon, 09 Sep 2002 16:35:25 GMT

Check out bug 5003.  I am writing some CVS related tasks when I have
opportunity.  In using the CVS task by composition the StreamPumper comes
into play.  I had problems getting the entire results of a cvs log into my
parsing code until I applied a few patches and made one myself.  The
details are in the bug log and the newsgroup postings I have made.

An archive of ant-dev can be found at:

James Lee Carpenter
Software Engineer
Household Technical Services
6602 Convoy Court
San Diego, CA 92111

ph: 858-609-2461

                           Jim Wright                  To:     Ant Developers List <>
                           <>        cc:                         
                                                       Subject:       [PATCH] Proposed StreamPumper
                           09/08/2002 04:42 PM                                           
                           Please respond to                                             
                           "Ant Developers List"                                         


I've been integrating Ant into my own tools and
noticed from the code that StreamPumper
sleeps for 5 ms after every 128 bytes
limiting throughput to approx 25kb/s.
I made some tentative changes aimed at removing
what I saw as distractions in the code but have
since realised that this 5ms delay can make quite
a difference.

For example I list the contents of large backup tar
archives to file by effectively running:

<exec output="archive.lis" executable="tar">

etc., but I actually do it in Java. Without the patch
for a 3MB list of filenames it takes a couple of
minutes longer than it should -  apparently
10-20 seconds with the patch.

I don't feel it is necessary for a StreamPumper thread
to yield explicitly in any way because it is likely to
frequently block on reading from the input stream and
otherwise on writing to the output stream, assuming
the related processing is not both trivial and performed
selfishly. But I have actually changed the sleep
to a harmless Thread.yield() to be on the safe side.

While there I remove a couple of TODO comments.

One was replaced with explanatory comment and
I have included a proposed implementation of the

Instead of a status flag I have added a method
exceptionCheck() that throws any IOException that
occurred. The idea is to achieve an effect consistent
with what would happen if there was no thread in the
implementation. With hindsight this might have been
combined with waitFor().

If you accept this proposal then a call to
exceptionCheck() should probably be added to (easy) and is more difficult.

The fact that I've done this TODO does not
affect calling code. It just provides the option of
additional error handling. In the case of logging
there is not much point in attempting to *report*
failure ;-) But for cases like the example above it
would appear that PumpStreamHandler should be

I have not got as far as working out if this class should be
deprecated in favour of LogStreamHandler (or
vice-versa). Nor have I searched the mailing list for
history of StreamPumper development discussion etc.
(Actually I couldn't find an online search of ant-dev.)
So use your informed judgement.



RCS file:

retrieving revision 1.7
diff -u -r1.7
---          15 Apr 2002 14:56:29 -0000          1.7
+++          8 Sep 2002 23:26:26 -0000
@@ -66,14 +66,15 @@
 public class StreamPumper implements Runnable {

-    // TODO: make SIZE and SLEEP instance variables.
-    // TODO: add a status flag to note if an error occured in run.
-    private static final int SLEEP = 5;
-    private static final int SIZE = 128;
+    // This value is not critical. Assuming one or both streams use native
+    // code to copy bytes, the most we want to do is avoid much looping
+    // before input or ouput stream methods block.
+    // No need for runtime configuration.
+    private static final int SIZE = 128;    // 8k loops per MB
     private InputStream is;
     private OutputStream os;
     private boolean finished;
+    private IOException runException;

      * Create a new stream pumper.
@@ -86,7 +87,6 @@
         this.os = os;

      * Copies data from the input stream to the output stream.
@@ -104,11 +104,10 @@
         try {
             while ((length = > 0) {
                 os.write(buf, 0, length);
-                try {
-                    Thread.sleep(SLEEP);
-                } catch (InterruptedException e) {}
+                Thread.yield();
         } catch (IOException e) {
+            runException = e;
         } finally {
             synchronized (this) {
                 finished = true;
@@ -135,4 +134,18 @@
+    /**
+     * If pumping was stopped by an exception then throw it.
+     *
+     * <p>This method should ideally always be called before
+     * discarding an instance.</p>
+     */
+    public void exceptionCheck() throws IOException {
+        if (runException != null) {
+            throw runException;
+        }
+    }

To unsubscribe, e-mail:   <>
For additional commands, e-mail: <>

To unsubscribe, e-mail:   <>
For additional commands, e-mail: <>

View raw message