perl-docs-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From s...@apache.org
Subject cvs commit: modperl-docs/src/search .swishcgi.conf README SwishSpiderConfig.pl search.tt spider.pl swish.cgi swish.conf
Date Sun, 03 Mar 2002 11:27:22 GMT
stas        02/03/03 03:27:22

  Modified:    src/search .swishcgi.conf README SwishSpiderConfig.pl
                        search.tt spider.pl swish.cgi swish.conf
  Log:
  - updating search utility and configs
  Submitted by:	Bill Moseley <moseley@hank.org>
  Reviewed by:	stas
  
  Revision  Changes    Path
  1.2       +16 -0     modperl-docs/src/search/.swishcgi.conf
  
  Index: .swishcgi.conf
  ===================================================================
  RCS file: /home/cvs/modperl-docs/src/search/.swishcgi.conf,v
  retrieving revision 1.1
  retrieving revision 1.2
  diff -u -r1.1 -r1.2
  --- .swishcgi.conf	30 Jan 2002 06:35:00 -0000	1.1
  +++ .swishcgi.conf	3 Mar 2002 11:27:21 -0000	1.2
  @@ -6,5 +6,21 @@
               options         => {
                   INCLUDE_PATH    => '.',
               },
  +    },
  +    select_by_meta  => {
  +        #method      => 'radio_group',  # pick: radio_group, popup_menu, or checkbox_group
  +        method      => 'checkbox_group',
  +        #method      => 'popup_menu',
  +        columns     => 6,
  +        metaname    => 'section',     # Can't be a metaname used elsewhere!
  +        values      => [qw/about contribute docs download maillist products stats stories support/],
  +        labels  => {
  +            about   => 'About mod_perl',
  +            doc     => 'Documentation',
  +            stories => 'Sucess Stories',
  +            support => 'Support',
           },
  +        description => 'Limit search to these areas: ',
  +    },
  +    
   };
  
  
  
  1.2       +39 -6     modperl-docs/src/search/README
  
  Index: README
  ===================================================================
  RCS file: /home/cvs/modperl-docs/src/search/README,v
  retrieving revision 1.1
  retrieving revision 1.2
  diff -u -r1.1 -r1.2
  --- README	4 Feb 2002 09:22:27 -0000	1.1
  +++ README	3 Mar 2002 11:27:22 -0000	1.2
  @@ -12,20 +12,28 @@
   Indexing:
   ---------
   
  -1. normally build the site:
  +1. Set an environment variable to the path of the site:
   
  -  % bin/build -f (-d to build pdfs)
  +    export MODPERL_SITE='http://perl.org'
   
  -which among other things creates the dir: dst_html/search
  +or
  +
  +    export MODPERL_SITE='http://localhost:4000/dst_html'
  +
  +This is used as the base for spidering, plus is used to determine
  +the sections of the site (for limiting the site to those sections.
  +    
   
  -2. check that swish.conf points to the right base URL, e.g.:
  +2. normally build the site:
   
  -  SwishProgParameters default http://localhost/modperl-site/
  +  % bin/build -f (-d to build pdfs)
  +
  +which among other things creates the dir: dst_html/search
   
   3. Index the site
   
     % cd dst_html/search
  -  % swish-e -S prog -c swish.conf
  +  % ./swish-e -S prog -c swish.conf
   
   You should see something like:
   
  @@ -81,3 +89,28 @@
   
   
   
  +How does indexing work?
  +-----------------------
  +
  +Swish is run with a config file, and is run in a mode that says
  +to use an external program to fetch documents.  That external program
  +is called spider.pl (part of the swish-e distribution).
  +
  +spider.pl uses a config file (by default) of SwishSpiderConfig.pl.  This file
  +builds an array of hashes (in this case a sinlge hash in the array).  This hash
  +is the config.
  +
  +Part of the config are call-back functions that spider.pl will call while spidering.
  +One says to skip image files.  Another one is a bit more tricky.  It splits a document into
  +sections, creates new "sub-pages" that are complete HTML pages, and calls the function in spider.pl
  +that sends those off to swish for indexing.  (That function then returns false to tell swish not to
  +index that document since the sections have already been indexed.)
  +
  +That's about it.
  +
  +One trick.  For debugging you can run the spider without indexing.
  +
  +   ./spider.pl > bigfile.out
  +
  +Another trick, you can send SIGHUP to spider.pl while indexing and
  +it will stop spidering, but let swish index what's been read so far.
  
  
  
  1.3       +20 -8     modperl-docs/src/search/SwishSpiderConfig.pl
  
  Index: SwishSpiderConfig.pl
  ===================================================================
  RCS file: /home/cvs/modperl-docs/src/search/SwishSpiderConfig.pl,v
  retrieving revision 1.2
  retrieving revision 1.3
  diff -u -r1.2 -r1.3
  --- SwishSpiderConfig.pl	7 Feb 2002 07:26:15 -0000	1.2
  +++ SwishSpiderConfig.pl	3 Mar 2002 11:27:22 -0000	1.3
  @@ -2,13 +2,19 @@
   #
   # a few custom callbacks are located after the @servers definition section.
   
  +
  +
  +my $base_path = $ENV{MODPERL_SITE} || die "must set \$ENV{MODPERL_SITE}";
  +
  +die "Don't use trailing slash in MODPERL_SITE" if $base_path =~ m!/$!;
  +
  +
   @servers = (
       {
  -        base_url        => 'http://mardy:40994/dst_html/index.html',
  +        base_url        => "$base_path/index.html",
   
           # Debugging -- see perldoc spider.pl
   
  -        #base_url        => 'http://mardy.hank.org:40994/dst_html/docs/guide/index.html',
           #max_depth => 1,
           #debug => DEBUG_HEADERS,
           #debug => DEBUG_URL|DEBUG_SKIPPED|DEBUG_INFO,
  @@ -21,12 +27,9 @@
   
           delay_min       => .0001,
   
  +
           # Ignore images files
  -        test_url => sub {
  -            return if $_[0]->path =~ /\.(?:gif|jpeg|.png|.gz)$/i;
  -            return unless $_[0]->path =~ m!^/preview/modperl-site!;
  -            return 1;
  -        },
  +        test_url => sub { return $_[0]->path !~ /\.(?:gif|jpeg|.png|.gz)$/i },
   
           # Only index text/html
           test_response   => sub { return $_[2]->content_type =~ m[text/html] },
  @@ -35,7 +38,7 @@
           filter_content  => \&split_page,
   
           # optionally validate external links
  -        validate_links => 1,
  +        validate_links  => $ENV{VALIDATE_LINKS} || 0,
       },
   
   );
  @@ -92,11 +95,20 @@
           $head->push_content( $title );
       }
   
  +    # Extract out part of the path to use for limiting searches to parts of the document tree.
  +
  +    if ( $uri =~ m!$base_path/([^/]+)/.+$! ) {
  +        my $meta = HTML::Element->new('meta', name=> 'section', content => $1);
  +        $head->push_content( $meta );
  +    }
  +        
  +
       my $body = HTML::Element->new('body');
       my $doc  = HTML::Element->new('html');
   
       $body->push_content( $section );
       $doc->push_content( $head, $body );
  +
   
       my $new_content = $doc->as_HTML(undef,"\t");
       output_content( $params->{server}, \$new_content,
  
  
  
  1.4       +1 -0      modperl-docs/src/search/search.tt
  
  Index: search.tt
  ===================================================================
  RCS file: /home/cvs/modperl-docs/src/search/search.tt,v
  retrieving revision 1.3
  retrieving revision 1.4
  diff -u -r1.3 -r1.4
  --- search.tt	4 Feb 2002 07:16:43 -0000	1.3
  +++ search.tt	3 Mar 2002 11:27:22 -0000	1.4
  @@ -15,6 +15,7 @@
           [% PROCESS search_form %]
           [% PROCESS nav_bar %]
           [% PROCESS results_list %]
  +        [% IF search.navigation('hits') > search.config('page_size'); PROCESS nav_bar; END %]
       [% END %]
   
   [% END %]
  
  
  
  1.3       +222 -30   modperl-docs/src/search/spider.pl
  
  Index: spider.pl
  ===================================================================
  RCS file: /home/cvs/modperl-docs/src/search/spider.pl,v
  retrieving revision 1.2
  retrieving revision 1.3
  diff -u -r1.2 -r1.3
  --- spider.pl	31 Jan 2002 01:51:50 -0000	1.2
  +++ spider.pl	3 Mar 2002 11:27:22 -0000	1.3
  @@ -2,7 +2,7 @@
   use strict;
   
   
  -# $Id: spider.pl,v 1.2 2002/01/31 01:51:50 stas Exp $
  +# $Id: spider.pl,v 1.3 2002/03/03 11:27:22 stas Exp $
   #
   # "prog" document source for spidering web servers
   #
  @@ -23,7 +23,7 @@
   use HTML::Tagset;
   
   use vars '$VERSION';
  -$VERSION = sprintf '%d.%02d', q$Revision: 1.2 $ =~ /: (\d+)\.(\d+)/;
  +$VERSION = sprintf '%d.%02d', q$Revision: 1.3 $ =~ /: (\d+)\.(\d+)/;
   
   use vars '$bit';
   use constant DEBUG_ERRORS   => $bit = 1;    # program errors
  @@ -36,10 +36,13 @@
   
   
   use constant MAX_SIZE       => 5_000_000;   # Max size of document to fetch
  +use constant MAX_WAIT_TIME  => 30;          # request time.
   
   #Can't locate object method "host" via package "URI::mailto" at ../prog-bin/spider.pl line 473.
   #sub URI::mailto::host { return '' };
   
  +
  +# This is not the right way to do this.
   sub UNIVERSAL::host { '' };
   sub UNIVERSAL::port { '' };
   sub UNIVERSAL::host_port { '' };
  @@ -62,7 +65,7 @@
       print STDERR "$0: Reading parameters from '$config'\n";
   
       my $abort;
  -    local $SIG{HUP} = sub { $abort++ };
  +    local $SIG{HUP} = sub { warn "Caught SIGHUP\n"; $abort++ } unless $^O =~ /Win32/i;
   
       my %visited;  # global -- I suppose would be smarter to localize it per server.
   
  @@ -74,8 +77,9 @@
               die "You must specify 'base_url' in your spider config settings\n";
           }
   
  -        for (ref $s->{base_url} eq 'ARRAY' ? @{$s->{base_url}} : $s->{base_url} ) {
  -            $s->{base_url} = $_;
  +        my @urls = ref $s->{base_url} eq 'ARRAY' ? @{$s->{base_url}} :( $s->{base_url});
  +        for my $url ( @urls ) {
  +            $s->{base_url} = $url;
               process_server( $s );
           }
       }
  @@ -100,12 +104,18 @@
       # set defaults
   
       $server->{debug} ||= 0;
  -    $server->{debug} = 0 unless $server->{debug} =~ /^\d+$/;
  +    die "debug parameter '$server->{debug}' must be a number\n" unless $server->{debug} =~ /^\d+$/;
   
   
       $server->{max_size} ||= MAX_SIZE;
       die "max_size parameter '$server->{max_size}' must be a number\n" unless $server->{max_size} =~ /^\d+$/;
   
  +
  +    $server->{max_wait_time} ||= MAX_WAIT_TIME;
  +    die "max_wait_time parameter '$server->{max_wait_time}' must be a number\n" if $server->{max_wait_time} !~ /^\d+/;
  +
  +
  +
       $server->{link_tags} = ['a'] unless ref $server->{link_tags} eq 'ARRAY';
       $server->{link_tags_lookup} = { map { lc, 1 } @{$server->{link_tags}} };
   
  @@ -139,14 +149,32 @@
       my $uri = URI->new( $server->{base_url} );
       $uri->fragment(undef);
   
  +    if ( $uri->userinfo ) {
  +        die "Can't specify parameter 'credentials' because base_url defines them\n"
  +            if $server->{credentials};
  +        $server->{credentials} = $uri->userinfo;
  +        $uri->userinfo( undef );
  +    }
  +
  +
       print STDERR "\n -- Starting to spider: $uri --\n" if $server->{debug};
   
       
   
       # set the starting server name (including port) -- will only spider on server:port
       
  -    $server->{authority} = $uri->authority;
  -    $server->{same} = [ $uri->authority ];
  +
  +    # All URLs will end up with this host:port
  +    $server->{authority} = $uri->canonical->authority;
  +
  +    # All URLs must match this scheme ( Jan 22, 2002 - spot by Darryl Friesen )
  +    $server->{scheme} = $uri->scheme;
  +
  +
  +
  +    # Now, set the OK host:port names
  +    $server->{same} = [ $uri->canonical->authority ];
  +    
       push @{$server->{same}}, @{$server->{same_hosts}} if ref $server->{same_hosts};
   
       $server->{same_host_lookup} = { map { $_, 1 } @{$server->{same}} };
  @@ -169,8 +197,9 @@
   
   
       my $ua;
  +
       if ( $server->{ignore_robots_file} ) {
  -        $ua = LWP::UserAgent->new( );
  +        $ua = LWP::UserAgent->new;
           return unless $ua;
           $ua->agent( $server->{agent} );
           $ua->from( $server->{email} );
  @@ -181,6 +210,9 @@
           $ua->delay( $server->{delay_min} || 0.1 );
       }
   
  +    # Set the timeout on the server and using Windows.
  +    $ua->timeout( $server->{max_wait_time} ) if $^O =~ /Win32/i;
  +
           
       $server->{ua} = $ua;  # save it for fun.
       # $ua->parse_head(0);   # Don't parse the content
  @@ -224,6 +256,56 @@
       }
   }
   
  +
  +#-----------------------------------------------------------------------
  +# Deal with Basic Authen
  +
  +
  +
  +# Thanks Gisle!
  +sub get_basic_credentials {
  +    my($uri, $server, $realm ) = @_;
  +    my $netloc = $uri->canonical->host_port;
  +
  +    my ($user, $password);
  +
  +    eval {
  +        local $SIG{ALRM} = sub { die "timed out\n" };
  +        alarm( $server->{credential_timeout} || 30 ) unless $^O =~ /Win32/i;
  +
  +        if (  $uri->userinfo ) {
  +            print STDERR "\nSorry: invalid username/password\n";
  +            $uri->userinfo( undef );
  +        }
  +            
  +
  +        print STDERR "Need Authentication for $uri at realm '$realm'\n(<Enter> skips)\nUsername: ";
  +        $user = <STDIN>;
  +        chomp($user);
  +        die "No Username specified\n" unless length $user;
  +
  +        alarm( $server->{credential_timeout} || 30 ) unless $^O =~ /Win32/i;
  +
  +        print STDERR "Password: ";
  +        system("stty -echo");
  +        $password = <STDIN>;
  +        system("stty echo");
  +        print STDERR "\n";  # because we disabled echo
  +        chomp($password);
  +
  +        alarm( 0 ) unless $^O =~ /Win32/i;
  +    };
  +
  +    return if $@;
  +
  +    return join ':', $user, $password;
  +
  +
  +}
  +            
  +
  +        
  +
   #----------- Non recursive spidering ---------------------------
   
   sub spider {
  @@ -275,9 +357,30 @@
       $server->{no_index} = 0;
       $server->{no_spider} = 0;
   
  +
  +    # Set basic auth if defined - use URI specific first, then credentials
  +    if ( my ( $user, $pass ) = split /:/, ( $uri->userinfo || $server->{credentials} || '' ) ) {
  +        $request->authorization_basic( $user, $pass );
  +    }
  +
  +
  +    
  +
       my $been_here;
       my $callback = sub {
   
  +        # Reset alarm;
  +        alarm( $server->{max_wait_time} ) unless $^O =~ /Win32/i;
  +
  +
  +        # Cache user/pass
  +        if ( $server->{cur_realm} && $uri->userinfo ) {
  +             my $key = $uri->canonical->host_port . ':' . $server->{cur_realm};
  +             $server->{auth_cache}{$key} =  $uri->userinfo;
  +        }
  +
  +        $uri->userinfo( undef ) unless $been_here;
  +
           die "test_response" if !$been_here++ && !check_user_function( 'test_response', $uri, $server, $_[1], \$_[0]  );
               
   
  @@ -290,12 +393,55 @@
   
       };
   
  -    my $response = $ua->simple_request( $request, $callback, 4096 );
  +    my $response;
  +
  +    eval {
  +        local $SIG{ALRM} = sub { die "timed out\n" };
  +        alarm( $server->{max_wait_time} ) unless $^O =~ /Win32/i;
  +        $response = $ua->simple_request( $request, $callback, 4096 );
  +        alarm( 0 ) unless $^O =~ /Win32/i;
  +    };
   
   
       return if $server->{abort};
   
   
  +    if ( $response && $response->code == 401 && $response->header('WWW-Authenticate') && $response->header('WWW-Authenticate') =~ /realm="([^"]+)"/i ) {
  +        my $realm = $1;
  +
  +        my $user_pass;
  +
  +        # Do we have a cached user/pass for this realm?
  +        my $key = $uri->canonical->host_port . ':' . $realm;
  +
  +        if ( $user_pass = $server->{auth_cache}{$key} ) {
  +
  +            # If we didn't just try it, try again
  +            unless( $uri->userinfo && $user_pass eq $uri->userinfo ) {
  +                $uri->userinfo( $user_pass );
  +                return process_link( $server, $uri, $parent, $depth );
  +            }
  +        }
  +
  +        # otherwise, prompt:
  +
  +
  +        if ( $user_pass = get_basic_credentials( $uri, $server, $realm ) ) {
  +            $uri->userinfo( $user_pass );
  +
  +            $server->{cur_realm} = $realm;  # save so we can cache
  +            my $links = process_link( $server, $uri, $parent, $depth );
  +            delete $server->{cur_realm};
  +
  +            return $links;
  +        }
  +        print STDERR "Skipping $uri\n";
  +    }
  +
  +    $uri->userinfo( undef );
  +        
  +
  +
       # Log the response
       
       if ( ( $server->{debug} & DEBUG_URL ) || ( $server->{debug} & DEBUG_FAILED && !$response->is_success)  ) {
  @@ -322,6 +468,7 @@
           return;
       }
   
  +    $response->request->uri->userinfo( undef );
   
   
       # skip excluded by robots.txt
  @@ -501,13 +648,15 @@
   
           # which tags to use ( not reported in debug )
   
  -        print STDERR " ?? Looking at extracted tag '$tag'\n" if $server->{debug} & DEBUG_LINKS;
  +        my $attr = join ' ', map { qq[$_="$attr{$_}"] } keys %attr;
  +
  +        print STDERR "\nLooking at extracted tag '<$tag $attr>'\n" if $server->{debug} & DEBUG_LINKS;
   
           unless ( $server->{link_tags_lookup}{$tag} ) {
           
               # each tag is reported only once per page
               print STDERR
  -                " ?? <$tag> skipped because not one of (",
  +                "   <$tag> skipped because not one of (",
                   join( ',', @{$server->{link_tags}} ),
                   ")\n" if $server->{debug} & DEBUG_LINKS && !$skipped_tags{$tag}++;
   
  @@ -539,18 +688,14 @@
                   next unless check_link( $u, $server, $base, $tag, $attribute );
                   
                   push @links, $u;
  -                print STDERR qq[ ++ <$tag $attribute="$u"> Added to list of links to follow\n] if $server->{debug} & DEBUG_LINKS;
  +                print STDERR qq[   $attribute="$u" Added to list of links to follow\n] if $server->{debug} & DEBUG_LINKS;
                   $found++;
               }
           }
   
   
           if ( !$found && $server->{debug} & DEBUG_LINKS ) {
  -            my $s = "<$tag";
  -            $s .= ' ' . qq[$_="$attr{$_}"] for sort keys %attr;
  -            $s .= '>';
  -                
  -            print STDERR " ?? tag $s did not include any links to follow\n";
  +            print STDERR "  tag did not include any links to follow or is a duplicate\n";
           }
           
       }
  @@ -599,15 +744,15 @@
   
       # Here we make sure we are looking at a link pointing to the correct (or equivalent) host
   
  -    unless ( $server->{same_host_lookup}{$u->authority} ) {
  +    unless ( $server->{scheme} eq $u->scheme && $server->{same_host_lookup}{$u->canonical->authority} ) {
   
  -        print STDERR qq[ ?? <$tag $attribute="$u"> skipped because different authority (server:port)\n] if $server->{debug} & DEBUG_LINKS;
  +        print STDERR qq[ ?? <$tag $attribute="$u"> skipped because different host\n] if $server->{debug} & DEBUG_LINKS;
           $server->{counts}{'Off-site links'}++;
           validate_link( $server, $u, $base ) if $server->{validate_links};
           return;
       }
       
  -    $u->authority( $server->{authority} );  # Force all the same host name
  +    $u->host_port( $server->{authority} );  # Force all the same host name
   
       # Allow rejection of this URL by user function
   
  @@ -661,10 +806,10 @@
           my $request = HTTP::Request->new('HEAD', $uri->canonical );
   
           eval {
  -            $SIG{ALRM} = sub { die "timed out\n" };
  -            alarm 5;
  +            local $SIG{ALRM} = sub { die "timed out\n" };
  +            alarm( $server->{max_wait_time} ) unless $^O =~ /Win32/i;
               $response = $ua->simple_request( $request );
  -            alarm 0;
  +            alarm( 0 ) unless $^O =~ /Win32/i;
           };
   
           if ( $@ ) {
  @@ -729,7 +874,6 @@
   }
   
   sub default_urls {
  -    die "$0: Must list URLs when using 'default'\n" unless @ARGV;
   
       my $validate = 0;
       if ( $ARGV[0] eq 'validate' ) {
  @@ -737,6 +881,9 @@
           $validate = 1;
       }
   
  +    die "$0: Must list URLs when using 'default'\n" unless @ARGV;
  +
  +
       my @content_types  = qw{ text/html text/plain };
   
       return map {
  @@ -786,9 +933,18 @@
           },
       );
   
  -  begin indexing:
  +Begin indexing:
  +
       swish-e -S prog -c swish.config
   
  +Note: When running on some versions of Windows (e.g. Win ME and Win 98 SE)
  +you may need to index using the command:
  +
  +    perl spider.pl | swish-e -S prog -c swish.conf -i stdin
  +
  +This pipes the output of the spider directly into swish.
  +   
  +
   =head1 DESCRIPTION
   
   This is a swish-e "prog" document source program for spidering
  @@ -1013,6 +1169,19 @@
   
       base_url => [qw! http://swish-e.org/ http://othersite.org/other/index.html !],
   
  +You may specify a username and password:
  +
  +    base_url => 'http://user:pass@swish-e.org/index.html',
  +
  +but you may find that to be a security issue.  If a URL is protected by Basic Authentication
  +you will be prompted for a username and password.  This might be a slighly safer way to go.
  +
  +The parameter C<max_wait_time> controls how long to wait for user entry before skipping the
  +current URL.
  +
  +See also C<credentials> below.
  +
  +
   =item same_hosts
   
   This optional key sets equivalent B<authority> name(s) for the site you are spidering.
  @@ -1034,9 +1203,9 @@
   
       http://www.mysite.edu/path/to/file.html
   
  -Note: This should probably be called B<same_authority> because it compares the URI C<authority>
  +Note: This should probably be called B<same_host_port> because it compares the URI C<host:port>
   against the list of host names in C<same_hosts>.  So, if you specify a port name in you will
  -probably want to specify the port name in the the list of hosts in C<same_hosts>:
  +want to specify the port name in the the list of hosts in C<same_hosts>:
   
       my %serverA = (
           base_url    => 'http://sunsite.berkeley.edu:4444/',
  @@ -1076,6 +1245,16 @@
   but in general you will probably want it much smaller.  But, check with
   the webmaster before using too small a number.
   
  +=item max_wait_time
  +
  +This setting is the number of seconds to wait for data to be returned from
  +the request.  Data is returned in chunks to the spider, and the timer is reset each time
  +a new chunk is reported.  Therefore, documents (requests) that take longer than this setting
  +should not be aborted as long as some data is received every max_wait_time seconds.
  +The default it 30 seconds.
  +
  +NOTE: This option has no effect on Windows.
  +
   =item max_time
   
   This optional key will set the max minutes to spider.   Spidering
  @@ -1204,6 +1383,19 @@
   Just a hack.  If you set this true the spider will do HEAD requests all links (e.g. off-site links), just
   to make sure that all your links work.
   
  +=item credentials
  +
  +You may specify a username and password to be used automatically when spidering:
  +
  +    credentials => 'username:password',
  +
  +A username and password supplied in a URL will override this setting.
  +
  +=item credential_timeout
  +
  +Sets the number of seconds to wait for user input when prompted for a username or password.
  +The default is 30 seconds.
  +
   =back
   
   =head1 CALLBACK FUNCTIONS
  @@ -1445,7 +1637,7 @@
   files to index only the document titles.
   
   As shown above, you can turn this feature on for specific documents by setting a flag in
  -the server hash passed into the C<test_response> or C<filter_contents> subroutines.
  +the server hash passed into the C<test_response> or C<filter_content> subroutines.
   For example, in your configuration file you might have the C<test_response> callback set
   as:
   
  @@ -1466,7 +1658,7 @@
   HTML I<and> a title is found in the html document.
   
   Note: In most cases you probably would not want to send a large binary file to swish, just
  -to be ignored.  Therefore, it would be smart to use a C<filter_contents> callback routine to
  +to be ignored.  Therefore, it would be smart to use a C<filter_content> callback routine to
   replace the contents with single character (you cannot use the empty string at this time).
   
   A similar flag may be set to prevent indexing a document at all, but still allow spidering.
  
  
  
  1.3       +978 -314  modperl-docs/src/search/swish.cgi
  
  Index: swish.cgi
  ===================================================================
  RCS file: /home/cvs/modperl-docs/src/search/swish.cgi,v
  retrieving revision 1.2
  retrieving revision 1.3
  diff -u -r1.2 -r1.3
  --- swish.cgi	4 Feb 2002 09:19:39 -0000	1.2
  +++ swish.cgi	3 Mar 2002 11:27:22 -0000	1.3
  @@ -2,17 +2,20 @@
   package SwishSearch;
   use strict;
   
  -use lib qw( modules );  ### This must be adjusted!
  +use lib qw( modules );  ### This may need to be adjusted!
  +                        ### It should point to the location of the
  +                        ### associated script modules directory
   
   
  -####################################################################################
  +
  +###################################################################################
   #
   #    If this text is displayed on your browser then your web server
   #    is not configured to run .cgi programs.  Contact your web server administrator.
   #
   #    To display documentation for this program type "perldoc swish.cgi"
   #
  -#    swish.cgi $Revision: 1.2 $ Copyright (C) 2001 Bill Moseley swishscript@hank.org
  +#    swish.cgi $Revision: 1.3 $ Copyright (C) 2001 Bill Moseley swishscript@hank.org
   #    Example CGI program for searching with SWISH-E
   #
   #    This example program will only run under an OS that supports fork().
  @@ -31,14 +34,13 @@
   #
   #    The above lines must remain at the top of this program
   #
  -#    $Id: swish.cgi,v 1.2 2002/02/04 09:19:39 stas Exp $
  +#    $Id: swish.cgi,v 1.3 2002/03/03 11:27:22 stas Exp $
   #
   ####################################################################################
   
   # This is written this way so the script can be used as a CGI script or a mod_perl
   # module without any code changes.
   
  -
   # use CGI ();  # might not be needed if using Apache::Request
   
   #=================================================================================
  @@ -59,50 +61,11 @@
       }
   
   
  -#=================================================================================
  -#   mod_perl entry point
  -#
  -#   As an example, you might use a PerlSetVar to point to paths to different
  -#   config files, and then cache the different configurations by path.
  -#
  -#=================================================================================
  -
  -my %cached_configs;
  -
  -sub handler {
  -    my $r = shift;
  -
  -    if ( my $config_path = $r->dir_config( 'Swish_Conf_File' ) ) {
  -
  -        # Already cached?
  -        if ( $cached_configs{ $config_path } ) {
  -            process_request( $cached_configs{ $config_path } );
  -            return Apache::Constants::OK();
  -        }
  -
  -        # Else, load config
  -        my $config = default_config();
  -        $config->{config_file} = $config_path;
  -
  -        # Merge with disk config file.
  -        $cached_configs{ $config_path } = merge_read_config( $config );
  -
  -        process_request( $cached_configs{ $config_path } );
  -        return Apache::Constants::OK();
  -    }
  -
  -
  -    # Otherwise, use hard-coded config
  -    process_request( default_config() );
  -
  -    return Apache::Constants::OK();
  -
  -}
  -
   
   
   #==================================================================================
  -#   This sets the default configuration
  +#   This sets the default configuration parameters
  +#
   #   Any configuration read from disk is merged with these settings.
   #
   #   Only a few settings are actually required.  Some reasonable defaults are used
  @@ -140,18 +103,13 @@
   
   sub default_config {
   
  -    # make the search of the swish-e executable more flexible. First
  -    # search in the PATH, then in the current dir.
  -    my $exec = `which swish-e`;
  -#warn "found exec: $exec";
  -    chomp $exec;
  -    $exec ||= './swish-e';
  -    die "Cannot find swish-e" unless -x $exec;
   
  +    
       ##### Configuration Parameters #########
   
       #---- This lists all the options, with many commented out ---
       # By default, this config is used -- see the process_request() call below.
  +    
       # You should adjust for your site, and how your swish index was created.
   
       ##>>
  @@ -159,11 +117,24 @@
       ##>>
       ##>>  Send a small example, without all the comments.
   
  -    # Items beginning with an "x" or "#" are commented out
  -
  +    #======================================================================
  +    # NOTE: Items beginning with an "x" or "#" are commented out
  +    #       the "x" form simply renames (hides) that setting.  It's used
  +    #       to make it easy to disable a mult-line configuation setting.
  +    #
  +    #   If you do not understand a setting then best to leave the default.
  +    #
  +    #   Please follow the documentation (perldoc swish.cgi) and set up
  +    #   a test using the defaults before making changes.  It's much easier
  +    #   to modify a working example than to try to get a modified example to work ;)
  +    #
  +    #   Again, this is a Perl hash structure.  Commas are important.
  +    #======================================================================
  +    
       return {
  -        title           => 'Search our site',  # Title of your choice.
  -        swish_binary    => $exec,              # Location of swish-e binary
  +        title           => 'Search our site',  # Title of your choice.  Displays on the search page
  +        swish_binary    => './swish-e',        # Location of swish-e binary
  +
   
           # By default, this script tries to read a config file.  You should probably
           # comment this out if not used save a disk stat
  @@ -175,7 +146,7 @@
           # If you have more than one index to search then specify an array
           # reference.  e.g. swish_index =>[ qw/ index1 index2 index3 /],
           
  -        swish_index     => 'index.swish-e',  # Location of your index file
  +        swish_index     => 'index.swish-e',    # Location of your index file
                                                   
                                                  # See "select_indexes" below for how to
                                                  # select more than one index.
  @@ -188,6 +159,8 @@
           # But you can specify any PropertyName defined in your document.
           # By default, swish will return the pathname for documents that do not
           # have a title.
  +        # In other words, this is used for the text of the links of the search results.
  +        #  <a href="prepend_path/swishdocpath">title_property</a>
   
           title_property => 'swishtitle',
   
  @@ -283,6 +256,9 @@
   
           timeout         => 10,    # limit time used by swish when fetching results - DoS protection.
   
  +        max_query_length => 100,  # limit length of query string.  Swish also has a limit (default is 40)
  +                                  # You might want to set swish-e's limit higher, and use this to get a
  +                                  # somewhat more friendly message.
   
   
           # These settings will use some crude highlighting code to highlight search terms in the
  @@ -337,7 +313,7 @@
           #swish_index     => [ qw/ index.swish-e index.other index2.other index3.other index4.other / ], 
   
           Xselect_indexes  => {
  -            #method  => 'radio_group',  # pico radio_group, popup_menu, or checkbox_group
  +            #method  => 'radio_group',  # pick radio_group, popup_menu, or checkbox_group
               method  => 'checkbox_group',
               #method => 'popup_menu',
               columns => 3,
  @@ -357,7 +333,7 @@
           
   
           Xselect_by_meta  => {
  -            #method      => 'radio_group',  # pico radio_group, popup_menu, or checkbox_group
  +            #method      => 'radio_group',  # pick: radio_group, popup_menu, or checkbox_group
               method      => 'checkbox_group',
               #method      => 'popup_menu',
               columns     => 3,
  @@ -409,12 +385,42 @@
           },
   
   
  +
  +        # The "on_intranet" setting is just a flag that can be used to say you do
  +        # not have an external internet connection.  It's here because the default
  +        # page generation includes links to images on swish-e.or and on www.w3.org.
  +        # If this is set to one then those images will not be shown.
  +        # (This only effects the default ouput module TemplateDefault)
  +
  +        on_intranet => 0,
  +
  +
  +
  +        # Here you can hard-code debugging options.  The will help you find
  +        # where you made your mistake ;)
  +        # Using all at once will generate a lot of messages to STDERR
  +        # Please see the documentation before using these.
  +        # Typically, you will set these from the command line instead of in the configuration.
  +        
  +        # debug_options => 'basic, command, headers, output, summary, dump',
  +
  +
  +
           # This defines the package object for reading CGI parameters
           # Defaults to CGI.  Might be useful with mod_perl.
           # request_package     => 'CGI',
           # request_package     => 'Apache::Request',
   
   
  +        
  +        # Minor adjustment to page display.  The page navigation normally looks like:
  +        #                          Page: 1 5 6 7 8 9 24
  +        # where the first page and last page are always displayed.  These can be disabled by
  +        # by setting to true values ( 1 )
  +
  +        no_first_page_navigation   => 0,
  +        no_last_page_navigation    => 0,
  +
   
   
   
  @@ -458,6 +464,52 @@
   
   }
   
  +#^^^^^^^^^^^^^^^^^^^^^^^^^ end of user config ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  +#========================================================================================
  +
  +
  +
  +#=================================================================================
  +#   mod_perl entry point
  +#
  +#   As an example, you might use a PerlSetVar to point to paths to different
  +#   config files, and then cache the different configurations by path.
  +#
  +#=================================================================================
  +
  +my %cached_configs;
  +
  +sub handler {
  +    my $r = shift;
  +
  +    if ( my $config_path = $r->dir_config( 'Swish_Conf_File' ) ) {
  +
  +        # Already cached?
  +        if ( $cached_configs{ $config_path } ) {
  +            process_request( $cached_configs{ $config_path } );
  +            return Apache::Constants::OK();
  +        }
  +
  +        # Else, load config
  +        my $config = default_config();
  +        $config->{config_file} = $config_path;
  +
  +        # Merge with disk config file.
  +        $cached_configs{ $config_path } = merge_read_config( $config );
  +
  +        process_request( $cached_configs{ $config_path } );
  +        return Apache::Constants::OK();
  +    }
  +
  +
  +    # Otherwise, use hard-coded config
  +    process_request( default_config() );
  +
  +    return Apache::Constants::OK();
  +
  +}
  +
  +
   #============================================================================
   #   Read config settings from disk, and merge
   #   Note, all errors are ignored since by default this script looks for a
  @@ -467,16 +519,82 @@
   sub merge_read_config {
       my $config = shift;
   
  +    set_default_debug_flags();
  +
  +    set_debug($config);  # get from config or from %ENV
  +
       return $config unless $config->{config_file};
   
       my $return = do $config->{config_file};
   
       return $config unless ref $return eq 'HASH';
   
  +    if ( $config->{debug} || $return->{debug} ) {
  +        require Data::Dumper;
  +        print STDERR "\n---------- Read config parameters from '$config->{config_file}' ------\n",
  +              Data::Dumper::Dumper($return),
  +              "-------------------------\n";
  +    }
  +
  +    set_debug( $return );
  +        
  +
       # Merge settings
       return { %$config, %$return };
   }
   
  +#--------------------------------------------------------------------------------------------------
  +sub set_default_debug_flags {
  +    # Debug flags defined
  +
  +    $SwishSearch::DEBUG_BASIC       = 1;  # Show command used to run swish
  +    $SwishSearch::DEBUG_COMMAND     = 2;  # Show command used to run swish
  +    $SwishSearch::DEBUG_HEADERS     = 4;  # Swish output headers
  +    $SwishSearch::DEBUG_OUTPUT      = 8;  # Swish output besides headers
  +    $SwishSearch::DEBUG_SUMMARY     = 16;  # Summary of results parsed
  +    $SwishSearch::DEBUG_DUMP_DATA   = 32;  # dump data that is sent to templating modules
  +}
  +
  +
  +
  +
  +#---------------------------------------------------------------------------------------------------
  +sub set_debug {
  +    my $conf = shift;
  +
  +    unless ( $ENV{SWISH_DEBUG} ||$conf->{debug_options} ) {
  +        $conf->{debug} = 0;
  +        return;
  +    }
  +    
  +    my %debug = (
  +        basic       => [$SwishSearch::DEBUG_BASIC,   'Basic debugging'],
  +        command     => [$SwishSearch::DEBUG_COMMAND, 'Show command used to run swish'],
  +        headers     => [$SwishSearch::DEBUG_HEADERS, 'Show headers returned from swish'],
  +        output      => [$SwishSearch::DEBUG_OUTPUT,  'Show output from swish'],
  +        summary     => [$SwishSearch::DEBUG_SUMMARY, 'Show summary of results'],
  +        dump        => [$SwishSearch::DEBUG_DUMP_DATA, 'Show all data available to templates'],
  +    );
  +
  +
  +    $conf->{debug} = 1;
  +
  +    for ( split /\s*,\s*/, $ENV{SWISH_DEBUG} ) {
  +        if ( exists $debug{ lc $_ } ) {
  +            $conf->{debug} |= $debug{ lc $_ }->[0];
  +            next;
  +        }
  +
  +        print STDERR "Unknown debug option '$_'.  Must be one of:\n",
  +             join( "\n", map { sprintf('  %10s: %10s', $_, $debug{$_}->[1]) } sort { $debug{$a}->[0] <=> $debug{$b}->[0] }keys %debug),
  +             "\n\n";
  +        exit;
  +    }
  +
  +    print STDERR "Debug level set to: $conf->{debug}\n";
  +}
  +        
  +
   #============================================================================
   #
   #   This is the main entry point, where a config hash is passed in.
  @@ -491,17 +609,64 @@
       $request_package =~ s[::][/]g;
       require "$request_package.pm";
   
  +    my $request_object = $conf->{request_package} ? $conf->{request_package}->new : CGI->new;
  +
  +    if ( $conf->{debug} ) {
  +        print STDERR 'Enter a query [all]: ';
  +        my $query = <STDIN>;
  +        $query =~ tr/\r//d;
  +        chomp $query;
  +        unless ( $query ) {
  +            print STDERR "Using 'not asdfghjklzxcv' to match all records\n";
  +            $query = 'not asdfghjklzxcv';
  +        }
  +
  +        $request_object->param('query', $query );
  +
  +        print STDERR 'Enter max results to display [1]: ';
  +        my $max = <STDIN>;
  +        chomp $max;
  +        $max = 1 unless $max && $max =~/^\d+$/;
  +
  +        $conf->{page_size} = $max;
  +    }
  +        
  +
   
       # create search object
       my $search = SwishQuery->new(
           config    => $conf,
  -        request   => ($conf->{request_package} ? $conf->{request_package}->new : CGI->new),
  +        request   => $request_object,
       );
   
   
       # run the query
       my $results = $search->run_query;  # currently, results is the just the $search object
   
  +    if ( $conf->{debug} ) {
  +        if ( $conf->{debug} & $SwishSearch::DEBUG_DUMP_DATA ) {
  +            require Data::Dumper;
  +            print STDERR "\n------------- Results structure passed to template ------------\n",
  +                  Data::Dumper::Dumper( $results ),
  +                  "--------------------------\n";
  +        } elsif ( $conf->{debug} & $SwishSearch::DEBUG_SUMMARY ) {
  +            print STDERR "\n------------- Results Summary ------------\n";
  +            if ( $results->{hits} ) {
  +                require Data::Dumper;
  +                print STDERR "Showing $results->{navigation}{showing} of $results->{navigation}{hits}\n",
  +                    Data::Dumper::Dumper( $results->{_results} );
  +            } else {
  +                print STDERR "** NO RESULTS **\n";
  +            }
  +
  +            print STDERR "--------------------------\n";
  +        } else {
  +            print STDERR ( ($results->{hits} ? "Found $results->{hits} results\n" : "Failed to find any results\n" . $results->errstr . "\n" ),"\n" );
  +        }
  +    }
  +    
  +    
  +
       my $template = $conf->{template} || { package => 'TemplateDefault' };
   
       my $package = $template->{package};
  @@ -509,7 +674,21 @@
       my $file = "$package.pm";
       $file =~ s[::][/]g;
   
  -    require $file;
  +    eval { require $file };
  +    if ( $@ ) {
  +        warn "$0 $@\n";
  +        print <<EOF;
  +Content-Type: text/html
  +
  +<html>
  +<head><title>Software Error</title></head>
  +<body><h2>Software Error<h2><p>Please check error log</p></body>
  +</html>
  +EOF
  +
  +    exit;
  +}
  +
       $package->show_template( $template, $results );
   }
   
  @@ -522,6 +701,10 @@
   #==================================================================================================
   
   use Carp;
  +# Or use this instead -- PLEASE see perldoc CGI::Carp for details
  +# <opinion>CGI::Carp doesn't help that much</opinion>
  +#use CGI::Carp; # qw(fatalsToBrowser);
  +
   
   #--------------------------------------------------------------------------------
   # new() doesn't do much, just create the object
  @@ -626,7 +809,6 @@
       my $conf = $self->{config};
   
   
  -
       # Sets the query string, and any -L limits.
       return $self unless $self->build_query;
   
  @@ -656,24 +838,20 @@
   
   
   
  -    # Trap the call - not portable.
  -
  -    my $timeout = $self->config('timeout');
  -
  -    if ( $timeout ) {
  -        eval {
  -            local $SIG{ALRM} = sub { die "Timed out\n" };
  -            alarm ( $self->config('timeout') || 5 );
  -            $self->run_swish;
  -            alarm 0;
  -        };
  +    my $timeout = $self->config('timeout') || 0;
   
  -        if ( $@ ) {
  -            $self->errstr( $@ );
  -            return $self;
  -        }
  -    } else {
  +    eval {
  +        local $SIG{ALRM} = sub { die "Timed out\n" };
  +        alarm $timeout if $timeout && $^O !~ /Win32/i;
           $self->run_swish;
  +        alarm 0  unless $^O =~ /Win32/i;
  +        waitpid $self->{pid}, 0 if $self->{pid};  # for IPC::Open2
  +    };
  +
  +    if ( $@ ) {
  +        warn "$0 $@"; # if $conf->{debug};
  +        $self->errstr( "Service currently unavailable" );
  +        return $self;
       }
   
   
  @@ -764,7 +942,9 @@
           $self->errstr('Please enter a query string') if $q->param('submit');
           return;
       }
  -    if ( length( $query ) > 100 ) {
  +
  +
  +    if ( length( $query ) > $self->{config}{max_query_length} ) {
           $self->errstr('Please enter a shorter query');
           return;
       }
  @@ -871,9 +1051,13 @@
   
       eval { require DateRanges };
       if ( $@ ) {
  -        $self->errstr( $@ );
  +        print STDERR "\n------ Can't use DateRanges feature ------------\n",
  +                     "\nScript will run, but you can't use the date range feature\n", 
  +                     $@,
  +                     "\n--------------\n" if $conf->{debug};
  +            
           delete $conf->{date_ranges};
  -        return;
  +        return 1;
       }
       
       my $q = $self->{q};
  @@ -931,15 +1115,17 @@
       # Now set sort option - if a valid option submitted (or you could let swish-e return the error).
       my %sorts = map { $_, 1 } @$sorts_array;
   
  -    if ( $q->param('sort') && $sorts{ $q->param('sort') } ) {
  +    my $sortby =  $q->param('sort') || 'swishrank';
  +
  +    if ( $sortby && $sorts{ $sortby } ) {
   
  -        my $direction = $q->param('sort') eq 'swishrank'
  +        my $direction = $sortby eq 'swishrank'
               ? $q->param('reverse') ? 'asc' : 'desc'
               : $q->param('reverse') ? 'desc' : 'asc';
                   
  -        $self->swish_command( '-s', $q->param('sort'), $direction );
  +        $self->swish_command( '-s', $sortby, $direction );
   
  -        if ( $conf->{secondary_sort} && $q->param('sort') ne $conf->{secondary_sort}[0] ) {
  +        if ( $conf->{secondary_sort} && $sortby ne $conf->{secondary_sort}[0] ) {
                   $self->swish_command(ref $conf->{secondary_sort} ? @{ $conf->{secondary_sort} } : $conf->{secondary_sort} );
           }
   
  @@ -1017,8 +1203,8 @@
               }
               
               @pages = $current_page..$current_page + $max_pages - 1;
  -            unshift @pages, 0 if $current_page;
  -            push @pages, $pages unless $current_page + $max_pages - 1 == $pages;
  +            unshift @pages, 0 if $current_page && !$self->{config}{no_first_page_navigation};
  +            push @pages, $pages unless $current_page + $max_pages - 1 == $pages || $self->{config}{no_last_page_navigation}
           }
   
       
  @@ -1080,7 +1266,6 @@
   #       or possibly a scalar with an error message.
   #
   
  -use Symbol;
   
   sub run_swish {
   
  @@ -1091,8 +1276,6 @@
       my $conf    = $self->{config};
       my $q       = $self->{q};
   
  -
  -
       
       my @properties;
       my %seen;
  @@ -1116,15 +1299,10 @@
       $self->swish_command( -x => join( '\t', map { "<$_>" } @properties ) . '\n' );
       $self->swish_command( -H => 9 );
   
  -    # Run swish 
  -    my $fh = gensym;
  -    my $pid = open( $fh, '-|' );
  +    my $fh = $^O =~ /Win32/i
  +             ? windows_fork( $conf, $self )
  +             : real_fork( $conf, $self );
   
  -    die "Failed to fork: $!\n" unless defined $pid;
  -
  -    if ( !$pid ) {  # in child
  -        exec $self->{prog},  $self->swish_command or die "Failed to exec '$self->{prog}' Error:$!";
  -    }
   
       $self->{COMMAND} = join ' ', $self->{prog},  $self->swish_command;
   
  @@ -1142,13 +1320,20 @@
       # Loop through values returned from swish.
   
       my %stops_removed;
  -    
  +
  +    my $unknown_output = '';
  +
  +
       while (<$fh>) {
   
           chomp;
  +        tr/\r//d;
   
           # This will not work correctly with multiple indexes when different values are used.
           if ( /^# ([^:]+):\s+(.+)$/ ) {
  +
  +            print STDERR "$_\n" if $conf->{debug} & $SwishSearch::DEBUG_HEADERS;
  +
               my $h = lc $1;
               my $value = $2;
               $self->{_headers}{$h} = $value;
  @@ -1156,12 +1341,18 @@
               push @{$self->{_headers}{'removed stopwords'}}, $value if $h eq 'removed stopword' && !$stops_removed{$value}++;
   
               next;
  +        } elsif ( $conf->{debug} & $SwishSearch::DEBUG_OUTPUT ) {
  +            print STDERR "$_\n";
           }
  +        
   
   
  -        # return errors as text
  +        # return swish errors as a mesage to the script
           $self->errstr($1), return if /^err:\s*(.+)/;
   
  +        # Or, if you want to log the errors and just say "Service Unavailable" use this:
  +        #die "$1\n" if /^err:\s*(.+)/;
  +
   
           # Found a result
           if ( /^\d/ ) {
  @@ -1189,7 +1380,8 @@
   
                       eval { require "$package.pm" };
                       if ( $@ ) {
  -                        $self->errstr( $@ );
  +                        $self->errstr( "Failed to load Highlighting Module - check error log" );
  +                        warn "$0: $@";
                           $highlight = '';
                           next;
                       } else {
  @@ -1216,19 +1408,93 @@
                       $h{$trim_prop} = substr( $h{$trim_prop}, 0, $max) . ' <b>...</b>';
                   }
               }
  +
  +            next;
       
  +        } elsif ( /^\.$/ ) {
  +            last;
  +
  +        } else {
  +            next if /^#/;
           }
   
  -        # Might check for "\n." for end of results.
  +        $unknown_output .= "'$_'\n";
  +
  +
   
           
       }
   
  +    die "Swish returned unknown output: $unknown_output\n" if $unknown_output;
  +
       $self->{hits} = @results;
       $self->{_results} = \@results if @results;
           
   }
   
  +#==================================================================
  +# Run swish-e by forking
  +#
  +
  +use Symbol;
  +
  +sub real_fork {
  +    my ( $conf, $self ) = @_;
  +
  +
  +    # Run swish 
  +    my $fh = gensym;
  +    my $pid = open( $fh, '-|' );
  +
  +    die "Failed to fork: $!\n" unless defined $pid;
  +
  +     
  +
  +    if ( !$pid ) {  # in child
  +        if ( $conf->{debug} & $SwishSearch::DEBUG_COMMAND ) {
  +            print STDERR "---- Running swish with the following command and parameters ----\n";
  +            print STDERR join( "  \\\n", map { /[^\/.\-\w\d]/ ? qq['$_'] : $_ }  $self->{prog}, $self->swish_command );
  +            print STDERR "\n-----------------------------------------------\n";
  +        }
  +
  +
  +        unless ( exec $self->{prog},  $self->swish_command ) {
  +            warn "Child process Failed to exec '$self->{prog}' Error: $!";
  +            print "Failed to exec Swish";  # send this message to parent.
  +            exit;
  +        }
  +    }
  +
  +    return $fh;
  +}
  +
  +
  +#=====================================================================================
  +#   Windows work around
  +#   from perldoc perlfok -- na, that doesn't work.  Try IPC::Open2
  +#
  +sub windows_fork {
  +    my ( $conf, $self ) = @_;
  +
  +    if ( $conf->{debug} & $SwishSearch::DEBUG_COMMAND ) {
  +        print STDERR "---- Running swish with the following command and parameters ----\n";
  +        print STDERR join( ' ', map { /[^.\-\w\d]/ ? qq["$_"] : $_ } map { s/"/\\"/g; $_ }  $self->{prog}, $self->swish_command );
  +        print STDERR "\n-----------------------------------------------\n";
  +    }
  +    
  +
  +    require IPC::Open2;
  +    my ( $rdrfh, $wtrfh );
  +
  +    # Ok, I'll say it.  Windows sucks.
  +    my @command = map { s/"/\\"/g; $_ }  $self->{prog}, $self->swish_command;
  +    my $pid = IPC::Open2::open2($rdrfh, $wtrfh, @command );
  +
  +
  +    $self->{pid} = $pid;
  +
  +    return $rdrfh;
  +}
   
   #=====================================================================================
   # This method parses out the query from the "Parsed words" returned by swish
  @@ -1347,138 +1613,309 @@
   
   =head1 DESCRIPTION
   
  -C<swish.cgi> is an example CGI script for searching with the SWISH-E search engine version 2.1-dev and above.
  +C<swish.cgi> is a CGI script for searching with the SWISH-E search engine version 2.1-dev and above.
   It returns results a page at a time, with matching words from the source document highlighted, showing a
   few words of content on either side of the highlighted word.
   
  -The standard configuration should work with most swish index files.  Customization of the parameters will be
  +The script is highly configurable; you can search multiple (or selectable) indexes, limit searches to
  +part of the index, allow sorting by a number of different properties, limit results to a date range, and so on.
  +
  +The standard configuration (i.e. not using a config file) should work with most swish index files.
  +Customization of the parameters will be
   needed if you are indexing special meta data and want to search and/or display the meta data.  The
   configuration can be modified by editing this script directly, or by using a configuration file (.swishcgi.conf
   by default).
   
  +You are strongly encouraged to get the default configuration working before making changes.  Most problems
  +using this script are the result of configuration modifications.
  +
   The script is modular in design.  Both the highlighting code and output generation is handled by modules, which
   are included in the F<example/modules> directory.  This allows for easy customization of the output without
   changing the main CGI script.  A module exists to generate standard HTML output.  There's also modules and
  -template examples to use with the popular templating systems HTML::Template and Template-Toolkit.  This allows
  +template examples to use with the popular Perl templating systems HTML::Template and Template-Toolkit.  This allows
   you to tightly integrate this script with the look of an existing template-driven web site.
  +HTML::Template and Template-Toolkit are available from the CPAN (http://search.cpan.org).
   
   This scipt can also run basically unmodified as a mod_perl handler, providing much better performance than
   running as a CGI script.
   
  -Due to the forking nature of this program and its use of signals,
  -this script probably will not run under Windows without some modifications.
  -There's plan to change this soon.
  +Please read the rest of the documentation.  There's a C<DEBUGGING> section, and a C<FAQ> section.
  +
  +This script should work on Windows, but security may be an issue.
  +
  +=head1 REQUIREMENTS
  +
  +You should be running a reasonably current version of Perl.  5.00503 or above is recommended (anything older
  +will not be supported).
  +
  +If you wish to use the date range feature you will need to install the Date::Calc module.  This is available
  +from http://search.cpan.org.
   
   
   =head1 INSTALLATION
   
  -Installing a CGI application is dependent on your specific web server's configuration.
  -For this discussion we will assume you are using Apache in a typical configuration.  For example,
  -a common location for the DocumentRoot is C</usr/local/apache/htdocs>.  If you are installing this
  -on your shell account, your DocumentRoot might be C<~yourname/public_html>.
  +Here's an example installation session.  Please get a simple installation working before modifying the
  +configuration file.  Most problems reported for using this script have been due to improper configuration.
   
  -For the sake of this example we will assume the following:
  +The script's default settings are setup for initial testing.  By default the settings expect to find
  +most files and the swish-e binary in the same directory as the script.
   
  -    /usr/local/apache/htdocs        - Document root
  -    /usr/local/apache/cgi-bin       - CGI directory
  +For I<security> reasons, once you have tested the script you will want to change settings to limit access
  +to some of these files by the web server
  +(either by moving them out of web space, or using access control such as F<.htaccess>).
  +An example of using F<.htaccess> on Apache is given below.
   
  -=head2 Move the files to their locations
  +It's expected that you have already unpacked the swish-e distribution
  +and built the swish-e binary (if using a source distribution).
  +
  +Below is a (unix) session where we create a directory, move required files into this directory, adjust
  +permissions, index some documents, and symlink into the web server.
   
   =over 4
   
  -=item Copy the swish.cgi file to your CGI directory
  +=item 1 Move required files into their own directory.
   
  -Most web servers have a directory where CGI programs are kept.
  -Copy the C<swish.cgi> perl script into that directory if this is the case on your
  -server.  You will need to provide read
  -and execute permisssions to the file.  Exactly what permissions are needed again depends on
  -your specific configuration.  For example, under Unix:
  +This assumes that swish-e was unpacked and build in the ~/swish-e directory.    
   
  -    chmod 0755 swish.cgi
  +    ~ >mkdir swishdir
  +    ~ >cd swishdir
  +    ~/swishdir >cp ~/swish-e/example/swish.cgi .
  +    ~/swishdir >cp -rp ~/swish-e/example/modules .
  +    ~/swishdir >cp ~/swish-e/src/swish-e .
  +    ~/swishdir >chmod 755 swish.cgi
  +    ~/swishdir >chmod 644 modules/*
   
  -This gives the file owner (that's you) write access, and everyone read and execute access.    
   
  -Note that you are not required to use a cgi-bin directory with Apache.  You may place the
  -CGI script in any directory accessible via the web server and
  -enable it as a CGI script with something like the following
  -(place either in httpd.conf or in .htaccess):
  +=item 2 Create an index
   
  -    <Files swish.cgi>
  -        Allow from all
  -        SetHandler cgi-script
  -        Options +ExecCGI
  -    </Files>        
  +This step you will create a simple configuration file.  In this example the Apache documentation
  +is indexed.  Last we run a simple query to test swish.
   
  -Using this method you don't even need to use the C<.cgi> extension.  For example, rename
  -the script to "search" and then use that in the C<Files> directive.  Take to your web
  -administrator for further information.
  -
  -=item Copy the modules directory
  -
  -Copying the modules directory is optional, but the script needs to find additional modules so you will
  -need to edit the script to point to the modules directory.  Unlike CPAN modules that need to
  -be uncompressed, built, and installed, all you need to do is make sure the modules are some place where
  -the web server can read them.  You may decide to leave them where you uncompressed the swish-e distribution,
  -or you may wish to move them to your perl library.
  +    ~/swishdir >cat swish.conf            
  +    IndexDir /usr/local/apache/htdocs
  +    IndexOnly .html .htm
  +    DefaultContents HTML
  +    StoreDescription HTML <body> 200000
  +    MetaNames swishdocpath swishtitle
   
  -=head1 CONFIGURATION
  +    ~/swishdir >./swish-e -c swish.conf   
  +    Indexing Data Source: "File-System"
  +    Indexing "/usr/local/apache/htdocs"
  +    Removing very common words...
  +    no words removed.
  +    Writing main index...
  +    Sorting words ...
  +    Sorting 7005 words alphabetically
  +    Writing header ...
  +    Writing index entries ...
  +      Writing word text: Complete
  +      Writing word hash: Complete
  +      Writing word data: Complete
  +    7005 unique words indexed.
  +    5 properties sorted.                                              
  +    124 files indexed.  1485844 total bytes.  171704 total words.
  +    Elapsed time: 00:00:02 CPU time: 00:00:02
  +    Indexing done!
  +
  +Now, verify that the index can be searched:
  +
  +    ~/swishdir >./swish-e -w install -m 1
  +    # SWISH format: 2.1-dev-25
  +    # Search words: install
  +    # Number of hits: 14
  +    # Search time: 0.001 seconds
  +    # Run time: 0.040 seconds
  +    1000 /usr/local/apache/htdocs/manual/dso.html "Apache 1.3 Dynamic Shared Object (DSO) support" 17341
  +    .
  +
  +Let's see what files we have in our directory now:
  +
  +    ~/swishdir >ls -1 -F
  +    index.swish-e
  +    index.swish-e.prop
  +    modules/
  +    swish-e*
  +    swish.cgi*
  +    swish.conf
   
  -=head2 Configure the swish.cgi program
  +=item 3 Test the CGI script
   
  -Use a text editor and open the C<swish.cgi> program.
  +This is a simple step, but often overlooked.  You should test from the command line instead of jumping
  +ahead and testing with the web server.  See the C<DEBUGGING> section below for more information.
   
  -=over 4
  +    ~/swishdir >./swish.cgi | head
  +    Content-Type: text/html; charset=ISO-8859-1
  +
  +    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
  +    <html>
  +        <head>
  +           <title>
  +              Search our site
  +           </title>
  +        </head>
  +        <body>
  +
  +The above shows that the script can be run directly, and generates a correct HTTP header and HTML.
   
  -=item 1 Check the C<shebang> line
  +If you run the above and see something like this:
   
  -The first line of the program must point to the location of your perl program.  Typical
  -examples are:
  +    ~/swishdir >./swish.cgi
  +    bash: ./swish.cgi: No such file or directory
   
  +then you probably need to edit the script to point to the correct location of your perl program.
  +Here's one way to find out where perl is located (again, on unix):
  +
  +    ~/swishdir >which perl
  +    /usr/local/bin/perl
  +
  +    ~/swishdir >/usr/local/bin/perl -v   
  +    This is perl, v5.6.0 built for i586-linux
  +    ...
  +
  +Good! We are using a reasonably current version of perl.  You should be running
  +at least perl 5.005 (5.00503 really).  You will may have problems otherwise.
  +
  +Now that we know perl is at F</usr/local/bin/perl> we can adjust the "shebang" line
  +in the perl script (e.g. the first line of the script):
  +
  +    ~/swishdir >pico swish.cgi
  +    (edit the #! line)
  +    ~/swishdir >head -1 swish.cgi
       #!/usr/local/bin/perl -w
  -    #!/usr/bin/perl -w
  -    #!/opt/perl/bin/perl -w
   
  -=item 2 Set the perl library path
  +=item 4 Test with your web server
   
  -The script must find the modules that the script is distributed with.  These modules handle
  -the highlighting of the search terms, and the output generation.  Again, where you place the
  -modules is up to you, and the only requirement is that the web server can access those files.
  +How you do this is completely dependent on your web server, and you may need to talk to your web
  +server admin to get this working.  Often files with the .cgi extension are automatically set up to
  +run as CGI scripts, but not always.  In other words, this step is really up to you to figure out!
   
  -You tell perl the location of the modules with the "use lib" directive.  The default for this script is:
  +First, I create a symlink in Apache's document root to point to my test directory "swishdir".  This will work
  +because I know my Apache server is configured to follow symbolic links.
   
  -    use lib qw( modules );
  +    ~/swishdir >su -c 'ln -s /home/bill/swishdir /usr/local/apache/htdocs/swishdir'
  +    Password: *********
   
  -This says to look for the modules in the F<modules> directory of the current directory.
  +If your account is on an ISP and your web directory is F<~/public_html> the you might just move the entire
  +directory:
   
  -For example, say you want to leave the modules where you unpacked the swish-e distribution.  If
  -you unpacked in your home directory of F</home/yourname/swish-e> then you must add this to the
  -script:
  +    mv ~/swishdir ~/public_html
   
  -    use lib qw( /home/yourname/swish-e/example/modules );
  +Now, let's make a real HTTP request.  I happen to have Apache setup on a local port:
   
  -    
  +    ~/swishdir >GET http://localhost:8000/swishdir/swish.cgi | head -3
  +    #!/usr/local/bin/perl -w
  +    package SwishSearch;
  +    use strict;
   
  -=item 3 Set the configuration parameters
  +Oh, darn. It looks like Apache is not running the script and instead returning it as a
  +static page.  I need to tell Apache that swish.cgi is a CGI script.
   
  -To make things somewhat simple, the configuration parameters are included at the top of the program.
  -The parameters are all part of a perl C<hash> structure, and the comments at the top of the program should
  -get you going.
  +In my case F<.htaccess> comes to the rescue:
  +
  +    ~/swishdir >cat .htaccess 
  +
  +    # Deny everything by default
  +    Deny From All
  +
  +    # But allow just CGI script
  +    <files swish.cgi>
  +        Options ExecCGI
  +        Allow From All
  +        SetHandler cgi-script
  +    </files>
  +
  +Let's try the request one more time:    
  +
  +    ~/swishdir >GET http://localhost:8000/swishdir/swish.cgi | head
  +    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
  +    <html>
  +        <head>
  +           <title>
  +              Search our site
  +           </title>
  +        </head>
  +        <body>
  +            <h2>
  +            <a href="http://swish-e.org">
  +
  +That looks better!  Now use your web browser to test.
  +
  +Make sure you look at your web server's error log file while testing the script.
  +
  +BTW - "GET" is a program included with Perl's LWP library.  If you do no have this you might
  +try something like:
  +
  +    wget -O - http://localhost:8000/swishdir/swish.cgi | head
  +
  +and if nothing else, you can always telnet to the web server and make a basic request.
  +
  +    ~/swishtest > telnet localhost 8000
  +    Trying 127.0.0.1...
  +    Connected to localhost.
  +    Escape character is '^]'.
  +    GET /swishtest/swish.cgi http/1.0
  +
  +    HTTP/1.1 200 OK
  +    Date: Wed, 13 Feb 2002 20:14:31 GMT
  +    Server: Apache/1.3.20 (Unix) mod_perl/1.25_01
  +    Connection: close
  +    Content-Type: text/html; charset=ISO-8859-1
  +
  +    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
  +    <html>
  +        <head>
  +           <title>
  +              Search our site
  +           </title>
  +        </head>
  +        <body>
  +
  +This may seem like a lot of work compared to using a browser, but browsers
  +are a poor tool for basic CGI debugging.
  +        
  +
  +=back
  +
  +If you have problems check the C<DEBUGGING> section below.
   
  -You will probably need to specify at least the location of the swish-e binary, your index file or files,
  -and a title.
  +=head1 CONFIGURATION
  +
  +If you want to change the location of the swish-e binary or the index file, use multiple indexes, add additional metanames and properties,
  +change the default highlighting behavior, etc., you will need to adjust the script's configuration settings.
  +
  +Please get a test setup working with the default parameters before making changes to any configuration settings.
  +Better to debug one thing at a time...
  +
  +In general, you will need to adjust the script's settings to match the index file you are searching.  For example,
  +if you are indexing a hypermail list archive you may want to make the script
  +use metanames/properties of Subject, Author, and, Email address.  Or you may wish to provide a way to limit
  +searches to parts of your index file (e.g. parts of your directory tree).
  +
  +To make things somewhat "simple", the configuration parameters are included near the top of the swish.cgi program.
  +That is the only place that the individual parameters are defined and explained, so you will need to open up
  +the swish.cgi script in an editor to view the options.  Further questions about individual settings should
  +be referred to the swish-e discussion list.
   
  -You have two options for changing the configuration settings from their default:
  +The parameters are all part of a perl C<hash> structure, and the comments at the top of the program should
  +get you going.  The perl hash structure may seem a bit confusing, but it makes it easy to create nested and complex
  +parameters. 
  +
  +You have two options for changing the configuration settings from their default values:
   you may edit the script directly, or you may use a configuration file.  In either case, the configuration
   settings are a basic perl hash reference.
   
  -Using a configuration file is described below.
  +Using a configuration file is described below, but contains the same hash structure.
   
  -The configuration settings might look like:
  +There are many configuration settings, and some of them are commented out either by using
  +a "#" symbol, or by simply renaming the configuration directive (e.g. by adding an "x" to the parameter
  +name).
  +
  +A very basic configuration setup might look like:
   
       return {
           title           => 'Search the Swish-e list',   # Title of your choice.
           swish_binary    => './swish-e',                 # Location of swish-e binary
  -        swish_index     => '../index.swish-e',          # Location of your index file
  +        swish_index     => 'index.swish-e',             # Location of your index file
       };
   
   Or if searching more than one index:
  @@ -1486,31 +1923,25 @@
       return {
           title           => 'Search the Swish-e list',
           swish_binary    => './swish-e',
  -        swish_index     => ['../index.swish-e', '../index2'],
  +        swish_index     => ['index.swish-e', 'index2'],
       };
   
  -Both of these examples return a reference to a perl hash ( C<return {...}> ).  Again, this same format is
  -used either at the top of this program, or in a configuration file.
  -
  -The examples above place the swish index file(s)
  -in the directory above the C<swish.cgi> CGI script.  If using the example paths above
  -of C</usr/local/apache/cgi-bin> for the CGI bin directory, that means that the index file
  -is in C</usr/local/apache>.  That places the index out of web space (e.g. cannot be accessed
  -via the web server), yet relative to where the C<swish.cgi> script is located.
  +Both of these examples return a reference to a perl hash ( C<return {...}> ).  In the second example,
  +the multiple index files are set as an array reference.
   
  -(If running under mod_perl you will most likely specify absolute paths for your index files.)
  +Note that in the example above the swish-e binary file is relative to the current directory.
  +If running under mod_perl you will typically need to use absolute paths.
   
  -There's more than one way to do it, of course.
  -One option is to place the index in the same directory as the <swish.cgi> script, but
  -then be sure to use your web server's configuration to prohibit access to the index directly.
  +B<Using A Configuration File>
   
  -Another common option is to maintain a separate directory of the all your swish index files.  This decision is
  -up to you.
  -
  -As mentioned above, you can either edit this script directly and modify the configuration settings, or
  +As mentioned above, you can either edit the F<swish.cgi> script directly and modify the configuration settings, or
   use an external configuration file.  The settings in the configuration file are merged with (override)
   the settings defined in the script.
   
  +The advantage of using a configuration script is that you are not editing the swish.cgi script directly, and
  +downloading a new version won't mean re-editing the cgi script.  Also, if running under mod_perl you can use the same
  +script loaded into Apache to manage many different search pages.
  +
   By default, the script will attempt to read from the file F<.swishcgi.conf>.
   For example, you might only wish to change the title used
   in the script.  Simply create a file called F<.swishcgi.conf> in the same directory as the CGI script:
  @@ -1521,62 +1952,19 @@
          title => 'Search Our Mailing List Archive',
       };
   
  -Look at the default configuration settings at the top of this program for information on the available settings.    
  -
  -=item 4 Create your index
  -
  -You must index your web site before you can begin to use the C<swish.cgi> script.
  -Create a configuration file called C<swish.conf> in the directory where you will store
  -the index file.
  -
  -This next example uses the file system to index your web documents.
  -In general, you will probably wish to I<spider> your web site if your web pages do not
  -map exactly to your file system, and to only index files available from links on you web
  -site.
  -
  -See B<Spidering> below for more information.
  -
  -Example C<swish.conf> file:
  -
  -    # Define what to index
  -    IndexDir /usr/local/apache/htdocs
  -    IndexOnly .html .htm
  -
  -    # Tell swish how to parse .html and .html documents
  -    IndexContents HTML .html .htm
  -    # And just in case we have files without an extension
  -    DefaultContents HTML
  -
  -    # Replace the path name with a URL
  -    ReplaceRules replace /usr/local/apache/htdocs/ http://www.myserver.name/
  -
  -    # Allow limiting search to titles and URLs.
  -    MetaNames swishdocpath swishtitle
  -
  -    # Optionally use stemming for "fuzzy" searches
  -    #UseStemming yes
  -
  -Now to index you simply run:
  -
  -    swish-e -c swish.conf
  -
  -The default index file C<index.swish-e> will be placed in the current directory.
  -
  -Note that the above swish-e configuration defines two MetaNames "swishdocpath" and "swishtitle".
  -This allows searching just the document path or the title instead of the document's content.
  +The settings you use will depend on the index you create with swish.  Here's a basic configuration:
   
  -Here's an expanded C<swish.cgi> configuration to make use of the above settings used while indexing:
  -
  -    return {
  +   return {
           title           => 'Search the Apache documentation',
           swish_binary    => './swish-e',
           swish_index     => 'index.swish-e',
           metanames       => [qw/swishdefault swishdocpath swishtitle/],
  -        display_props   => [qw/swishlastmodified swishdocsize swishdocpath/],
  -        title_property  => 'swishtitle',  # Not required, but recommended
  +        display_props   => [qw/swishtitle swishlastmodified swishdocsize swishdocpath/],
  +        title_property  => 'swishdocpath',
  +        prepend_path    => 'http://myhost/apachedocs', 
   
           name_labels => {
  -            swishdefault        => 'Body & Title',
  +            swishdefault        => 'Search All',
               swishtitle          => 'Title',
               swishrank           => 'Rank',
               swishlastmodified   => 'Last Modified Date',
  @@ -1595,56 +1983,365 @@
   The parameter "name_labels" is a hash (reference)
   that is used to give friendly names to the metanames.
   
  -Swish-e can store part of all of the contents of the documents as they are indexed, and this
  -"document description" can be returned with search results.
  +Here's another example.  Say you want to search either (or both) the Apache 1.3 documentation or the
  +Apache 2.0 documentation:
  +
  +    return {
  +       title       => 'Search the Apache Documentation',
  +       date_ranges => 0,
  +       swish_index => [ qw/ index.apache index.apache2 / ],
  +       select_indexes  => {
  +            method  => 'checkbox_group',
  +            labels  => [ '1.3.23 docs', '2.0 docs' ],  # Must match up one-to-one to swish_index
  +            description => 'Select: ',              
  +        },
  +
  +    };
  +
  +Now you can select either or both sets of documentation while searching.    
  +
  +
  +Please refer to the default configuration settings near the top of the script for details on
  +the available settings.
  +
  +=head1 DEBUGGING
  +
  +Most problems with using this script have been a result of improper configuration.  Please
  +get the script working with default settings before adjusting the configuration settings.
  +
  +The key to debugging CGI scripts is to run them from the command line, not with a browser.
  +
  +First, make sure the program compiles correctly:
  +
  +    > perl -c swish.cgi
  +    swish.cgi syntax OK
  +
  +Next, simply try running the program:
  +
  +    > ./swish.cgi | head
  +    Content-Type: text/html; charset=ISO-8859-1
  +
  +    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
  +    <html>
  +        <head>
  +           <title>
  +              Search our site
  +           </title>
  +        </head>
  +        <body>
  +
  +Now, you know that the program compiles and will run from the command line.
  +Next, try accessing the script from a web browser.
  +
  +If you see the contents of the CGI script instead of its output then your web server is
  +not configured to run the script.  You will need to look at settings like ScriptAlias, SetHandler,
  +and Options.
  +
  +If an error is reported (such as Internal Server Error or Forbidden)
  +you need to locate your web server's error_log file
  +and carefully read what the problem is.  Contact your web administrator for help.
  +
  +If you don't have access to the web server's error_log file, you can modify the script to report
  +errors to the browser screen.  Open the script and search for "CGI::Carp".  (Author's suggestion is
  +to debug from the command line -- adding the browser and web server into the equation only complicates
  +debugging.)
  +
  +The script does offer some basic debugging options that allow debugging from the command line.
  +The debugging options are enabled by setting
  +an environment variable "SWISH_DEBUG".  How that is set depends on your operating system and the
  +shell you are using.  These examples are using the "bash" shell syntax.
  +
  +Note:  You can also use the "debug_options" configuration setting, but the recommended method
  +is to set the environment variable.
  +
  +You can list the available debugging options like this:
  +
  +    >SWISH_DEBUG=help ./swish.cgi >outfile
  +    Unknown debug option 'help'.  Must be one of:
  +           basic: Basic debugging
  +         command: Show command used to run swish
  +         headers: Show headers returned from swish
  +          output: Show output from swish
  +         summary: Show summary of results
  +            dump: Show all data available to templates
  +
  +As you work yourself down the list you will get more detail output.  You can combine
  +options like:
  +
  +    >SWISH_DEBUG=command,headers,summary ./swish.cgi >outfile
  +
  +You will be asked for an input query and the max number of results to return.  You can use the defaults
  +in most cases.  It's a good idea to redirect output to a file.  Any error messages are sent to stderr, so
  +those will still be displayed (unless you redirect stderr, too).
  +
  +Here are some examples:
  +
  +    ~/swishtest >SWISH_DEBUG=basic ./swish.cgi >outfile
  +    Debug level set to: 1
  +    Enter a query [all]: 
  +    Using 'not asdfghjklzxcv' to match all records
  +    Enter max results to display [1]: 
  +
  +    ------ Can't use DateRanges feature ------------
  +
  +    Script will run, but you can't use the date range feature
  +    Can't locate Date/Calc.pm in @INC (@INC contains: modules /usr/local/lib/perl5/5.6.0/i586-linux /usr/local/lib/perl5/5.6.0 /usr/local/lib/perl5/site_perl/5.6.0/i586-linux /usr/local/lib/perl5/site_perl/5.6.0 /usr/local/lib/perl5/site_perl/5.005/i586-linux /usr/local/lib/perl5/site_perl/5.005 /usr/local/lib/perl5/site_perl .) at modules/DateRanges.pm line 107, <STDIN> line 2.
  +    BEGIN failed--compilation aborted at modules/DateRanges.pm line 107, <STDIN> line 2.
  +    Compilation failed in require at ./swish.cgi line 971, <STDIN> line 2.
  +
  +    --------------
  +    Can't exec "./swish-e": No such file or directory at ./swish.cgi line 1245, <STDIN> line 2.
  +    Child process Failed to exec './swish-e' Error: No such file or directory at ./swish.cgi line 1246, <STDIN> line 2.
  +    Failed to find any results
  +
  +The above told me about two problems.  First, it's telling me that the Date::Calc module is not installed.
  +The Date::Calc module is needed to use the date limiting feature of the script.
  +
  +The second problem is a bit more serious.  It's saying that the script can't find the swish-e binary file.
  +I simply forgot to copy it.
  +
  +    ~/swishtest >cp ~/swish-e/src/swish-e .
  +    ~/swishtest >cat .swishcgi.conf
  +        return {
  +           title       => 'Search the Apache Documentation',
  +           date_ranges => 0,
  +        };
  +
  +Now, let's try again:
   
  -    # Store the text of the documents within the swish index file
  -    StoreDescription HTML <body> 100000
  +    ~/swishtest >SWISH_DEBUG=basic ./swish.cgi >outfile
  +    Debug level set to: 1
   
  -Adding the above to your C<swish.conf> file tells swish-e to store up to 100,000 characters from the body of each document within the
  -swish-e index.  To display this information in search results, highlighting search terms,
  -use the follow configuration in C<swish.cgi>:
  +    ---------- Read config parameters from '.swishcgi.conf' ------
  +    $VAR1 = {
  +              'date_ranges' => 0,
  +              'title' => 'Search the Apache Documentation'
  +            };
  +    -------------------------
  +    Enter a query [all]: 
  +    Using 'not asdfghjklzxcv' to match all records
  +    Enter max results to display [1]: 
  +    Found 1 results
  +
  +    Can't locate TemplateDefault.pm in @INC (@INC contains: modules /usr/local/lib/perl5/5.6.0/i586-linux /usr/local/lib/perl5/5.6.0 /usr/local/lib/perl5/site_perl/5.6.0/i586-linux /usr/local/lib/perl5/site_perl/5.6.0 /usr/local/lib/perl5/site_perl/5.005/i586-linux /usr/local/lib/perl5/site_perl/5.005 /usr/local/lib/perl5/site_perl .) at ./swish.cgi line 608.
  +
  +Bother.  I fixed the first two problems, but now there's this new error.  Oh, I somehow forgot to
  +copy the modules directory.  The obvious way to fix that is to copy the directory.  But, there may
  +be times where you want to put the module directory in another location.  So, let's modify the
  +F<.swishcgi.conf> file and add a "use lib" setting:
  +
  +    ~/swishtest >cat .swishcgi.conf
  +    use lib '/home/bill/swish-e/example/modules';
   
       return {
  -        title           => 'Search the Apache documentation',
  -        swish_binary    => './swish-e',
  -        swish_index     => 'index.swish-e',
  -        metanames       => [qw/swishdefault swishdocpath swishtitle/],
  -        display_props   => [qw/swishlastmodified swishdocsize swishdocpath/],
  -        title_property  => 'swishtitle',  # Not required, but recommended
  -        description_prop=> 'swishdescription',
  +       title       => 'Search the Apache Documentation',
  +       date_ranges => 0,
  +    };
   
  -        name_labels => {
  -            swishdefault        => 'Body & Title',
  -            swishtitle          => 'Title',
  -            swishrank           => 'Rank',
  -            swishlastmodified   => 'Last Modified Date',
  -            swishdocpath        => 'Document Path',
  -            swishdocsize        => 'Document Size',
  +    ~/swishtest >SWISH_DEBUG=basic ./swish.cgi >outfile
  +    Debug level set to: 1
  +
  +    ---------- Read config parameters from '.swishcgi.conf' ------
  +    $VAR1 = {
  +              'date_ranges' => 0,
  +              'title' => 'Search the Apache Documentation'
  +            };
  +    -------------------------
  +    Enter a query [all]: 
  +    Using 'not asdfghjklzxcv' to match all records
  +    Enter max results to display [1]: 
  +    Found 1 results
  +
  +Now were talking.
  +
  +Here's a common problem.  Everything checks out, but when you run the script you see the message:
  +
  +    Swish returned unknown output
  +
  +Ok, let's find out what output it is returning:
  +
  +    ~/swishtest >SWISH_DEBUG=headers,output ./swish.cgi >outfile
  +    Debug level set to: 13
  +
  +    ---------- Read config parameters from '.swishcgi.conf' ------
  +    $VAR1 = {
  +              'swish_binary' => '/usr/local/bin/swish-e',
  +              'date_ranges' => 0,
  +              'title' => 'Search the Apache Documentation'
  +            };
  +    -------------------------
  +    Enter a query [all]: 
  +    Using 'not asdfghjklzxcv' to match all records
  +    Enter max results to display [1]: 
  +      usage: swish [-i dir file ... ] [-S system] [-c file] [-f file] [-l] [-v (num)]
  +      ...
  +    version: 2.0
  +       docs: http://sunsite.berkeley.edu/SWISH-E/
  +
  +    *** 9872 Failed to run swish: 'Swish returned unknown output' ***
  +    Failed to find any results
  +
  +Oh, looks like /usr/local/bin/swish-e is version 2.0 of swish.  We need 2.1-dev and above!
  +
  +=head1 Frequently Asked Questions
  +
  +Here's some common questions and answers.
  +
  +=head2 How do I change the way the output looks?
  +
  +The script uses a module to generate output.  By default it uses the TemplateDefault.pm module.
  +The module used can be selected in the configuration file.
  +
  +If you want to make simple changes you can edit the TemplatDefault.pm module directly.  If you want to
  +copy a module, you must also change the "package" statement at the top of the module.  For example:
  +
  +    cp TempateDefault.pm MyTemplateDefault.pm
  +
  +Then at the top of the module adjust the "package" line to:
  +
  +    package MyTemplateDefault;
  +
  +To use this modules you need to adjust the configuration settings (either at the top of F<swish.cgi> or in
  +a configuration file:
  +
  +
  +        template => {
  +            package     => 'MyTemplateDefault',
           },
  -        highlight       => {
  -            package         => 'PhraseHighlight',
  -            meta_to_prop_map => {   # this maps search metatags to display properties
  -                swishdefault    => [ qw/swishtitle swishdescription/ ],
  -                swishtitle      => [ qw/swishtitle/ ],
  -                swishdocpath    => [ qw/swishdocpath/ ],
  +
  +
  +=head2 How do I use a templating system with swish.cgi?
  +
  +In addition to the TemplateDefault.pm module, the swish-e distribution includes two other Perl modules for
  +generating output using the templating systems HTML::Template and Template-Toolkit.
  +
  +Templating systems use template files to generate the HTML, and make maintaining the look of a large (or small) site
  +much easier. HTML::Template and Template-Toolkit are separate packages and can be downloaded from the CPAN.
  +See http://search.cpan.org.
  +
  +Two basic templates are provided as examples for generating output using these templating systems.
  +The example templates are located in the F<example> directory.
  +The module F<TemplateHTMLTemplate.pm> uses the file F<swish.tmpl> to generate its output, while the
  +module F<TemplateToolkit.pm> uses the F<search.tt> file.
  +
  +To use either of these modules you will need to adjust the "template" configuration setting.  Examples for
  +both templating systems are provided in the configuration settings near the top of the F<swish.cgi> program.
  +
  +Use of these modules is an advanced usage of F<swish.cgi> and are provided as examples only.
  +
  +All of the output generation modules are passed a hash with the results from the search, plus other data use to create the
  +output page.  You can see this hash by using the debugging option "dump" or by using the TemplateDumper.pm
  +module:
  +
  +    ~/swishtest >cat .swishcgi.conf 
  +        return {
  +           title       => 'Search the Apache Documentation',
  +           template => {
  +                package     => 'TemplateDumper',
               },
  -       }
  +        };
   
  -    };
  +And run a query.  For example:
   
  +    http://localhost:8000/swishtest/swish.cgi?query=install
   
  -Other C<swish.cgi> configuration settings are available, and are listed at the top of the F<swish.cgi>
  -script.
  +=head2 Why are there three different highlighting modules?    
   
  +Three are three highlighting modules included with the swish-e distribution.
  +Each is a trade-off of speed vs. accuracy:
   
  -=back
  +    DefaultHighlight.pm - reasonably fast, but does not highlight phrases
  +    PhraseHighlight.pm  - reasonably slow, but is reasonably accurate
  +    SimpleHighlight.pm  - fast, some phrases, but least accurate
  +
  +Eh, the default is actually "PhraseHighlight.pm".  Oh well.
  +
  +Optimizations to these modules are welcome!
  +
  +=head2 My ISP doesn't provide access to the web server logs
  +
  +There are a number of options.  One way it to use the CGI::Carp module.  Search in the
  +swish.cgi script for:
  +
  +    use Carp;
  +    # Or use this instead -- PLEASE see perldoc CGI::Carp for details
  +    # use CGI::Carp qw(fatalsToBrowser warningsToBrowser);
  +
  +And change it to look like:    
  +
  +    #use Carp;
  +    # Or use this instead -- PLEASE see perldoc CGI::Carp for details
  +    use CGI::Carp qw(fatalsToBrowser warningsToBrowser);
   
  -You should now be ready to run your search engine.  Point your browser to:
  +This should be only for debugging purposes, as if used in production you may end up sending
  +quite ugly and confusing messages to your browsers.
   
  -    http://www.myserver.name/cgi-bin/swish.cgi
  +=head2 Why does the output show (NULL)?
  +
  +The most common reason is that you did not use StoreDescription in your config file while indexing.
  +
  +    StoreDescription HTML <body> 200000
  +
  +That tells swish to store the first 200,000 characters of text extracted from the body of each document parsed
  +by the HTML parser.  The text is stored as property "swishdescription".  Running:
  +
  +    ~/swishtest > ./swish-e -T index_metanames
  +
  +will display the properties defined in your index file.    
  +
  +This can happen with other properties, too.
  +For example, this will happen when you are asking for a property to display that is not defined in swish.
  +
  +    ~/swishtest > ./swish-e -w install -m 1 -p foo   
  +    # SWISH format: 2.1-dev-25
  +    # Search words: install
  +    err: Unknown Display property name "foo"
  +    .
  +
  +    ~/swishtest > ./swish-e -w install -m 1 -x 'Property foo=<foo>\n'
  +    # SWISH format: 2.1-dev-25
  +    # Search words: install
  +    # Number of hits: 14
  +    # Search time: 0.000 seconds
  +    # Run time: 0.038 seconds
  +    Property foo=(NULL)
  +    .
  +
  +To check that a property exists in your index you can run:
  +
  +    ~/swishtest > ./swish-e -w not dkdk -T index_metanames | grep foo
  +            foo : id=10 type=70  META_PROP:STRING(case:ignore) *presorted*
  +
  +Ok, in this case we see that "foo" is really defined as a property.  Now let's make sure F<swish.cgi>
  +is asking for "foo" (sorry for the long lines):
  +
  +    ~/swishtest > SWISH_DEBUG=command ./swish.cgi > /dev/null
  +    Debug level set to: 3
  +    Enter a query [all]: 
  +    Using 'not asdfghjklzxcv' to match all records
  +    Enter max results to display [1]: 
  +    ---- Running swish with the following command and parameters ----
  +    ./swish-e  \
  +    -w  \
  +    'swishdefault=(not asdfghjklzxcv)'  \
  +    -b  \
  +    1  \
  +    -m  \
  +    1  \
  +    -f  \
  +    index.swish-e  \
  +    -s  \
  +    swishrank  \
  +    desc  \
  +    swishlastmodified  \
  +    desc  \
  +    -x  \
  +    '<swishreccount>\t<swishtitle>\t<swishdescription>\t<swishlastmodified>\t<swishdocsize>\t<swishdocpath>\t<fos>\t<swishrank>\t<swishdocpath>\n'  \
  +    -H  \
  +    9
  +
  +If you look carefully you will see that the -x parameter has "fos" instead of "foo", so there's our problem.
   
  -adjusting the server and URL to match your system, of course.    
   
   =head1 MOD_PERL
   
  @@ -1684,41 +2381,6 @@
   Please post to the swish-e discussion list if you have any questions about running this
   script under mod_perl.
   
  -
  -=head1 DEBUGGING
  -
  -The key to debugging CGI scripts is to run them from the command line, not with a browser.
  -
  -First, make sure the program compiles correctly:
  -
  -    > perl -c swish.cgi
  -    swish.cgi syntax OK
  -
  -Next, simply try running the program:
  -
  -    > ./swish.cgi | head
  -    Content-Type: text/html; charset=ISO-8859-1
  -
  -    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
  -    <html>
  -        <head>
  -           <title>
  -              Search our site
  -           </title>
  -        </head>
  -        <body>
  -
  -Now, you know that the program compiles and will run from the command line.
  -Next, try accessing the script from a web browser.
  -
  -If you see the contents of the CGI script instead of its output then your web server is
  -not configured to run the script.  You will need to look at settings like ScriptAlias, SetHandler,
  -and Options.
  -
  -If an error is reported (such as Internal Server Error or Forbidden)
  -you need to locate your web server's error_log file
  -and carefully read what the problem is.  Contact your web administrator for help.
  -    
       
   =head1 Spidering
   
  @@ -1835,6 +2497,8 @@
   
   See http://www.w3.org/Security/Faq/www-security-faq.html
   
  +Security on Windows questionable.
  +
   =head1 SUPPORT
   
   The SWISH-E discussion list is the place to ask for any help regarding SWISH-E or this example
  @@ -1844,11 +2508,11 @@
   
       http://swish-e.org/2.2/docs/INSTALL.html#When_posting_please_provide_the_
   
  -Please do not contact the author directly.
  +Please do not contact the author or any of the swish-e developers directly.
   
   =head1 LICENSE
   
  -swish.cgi $Revision: 1.2 $ Copyright (C) 2001 Bill Moseley search@hank.org
  +swish.cgi $Revision: 1.3 $ Copyright (C) 2001 Bill Moseley search@hank.org
   Example CGI program for searching with SWISH-E
   
   
  
  
  
  1.4       +16 -2     modperl-docs/src/search/swish.conf
  
  Index: swish.conf
  ===================================================================
  RCS file: /home/cvs/modperl-docs/src/search/swish.conf,v
  retrieving revision 1.3
  retrieving revision 1.4
  diff -u -r1.3 -r1.4
  --- swish.conf	4 Feb 2002 09:22:27 -0000	1.3
  +++ swish.conf	3 Mar 2002 11:27:22 -0000	1.4
  @@ -1,5 +1,19 @@
   IndexDir ./spider.pl
   DefaultContents HTML2
   StoreDescription HTML2 <body> 100000
  -MetaNames swishtitle swishdocpath
  -SwishProgParameters default http://localhost/modperl-site/
  +MetaNames swishtitle swishdocpath section
  +
  +# This is to make the URLs shorter in the display.
  +ReplaceRules remove http://perl.apache.org
  +
  +# For example, on my test setup I might do something like:
  +# Need ".." since search is on level down
  +
  +ReplaceRules replace http://mardy:40994/dst_html ..
  +
  +
  +UndefinedMetaTags ignore
  +
  +#BuzzWords in highlighting -- 
  +#How about counting highlighted terms individually in the highlight module
  +#so every term is highlighted at least once, with a total of say five.
  
  
  

---------------------------------------------------------------------
To unsubscribe, e-mail: docs-cvs-unsubscribe@perl.apache.org
For additional commands, e-mail: docs-cvs-help@perl.apache.org


Mime
View raw message