Alternative WWWOFFLE implementation

by Paul Rombouts

WWWOFFLE is a simple proxy server written by Andrew M. Bishop with special features for use with dial-up internet connections.
The official WWWOFFLE homepage by the author can be found at http://www.gedanken.demon.co.uk/wwwoffle/.

I've been using WWWOFFLE on Linux for a number of years now, and I've found it to be very useful in keeping my phone costs down. I have a permanent connection to the Internet now, but I still use WWWOFFLE to share the internet connection on my home LAN, to filter out ads and other undesirable stuff, and to keep backup copies of the webpages I've visited.
I've come to know WWWOFFLE so intimately now that I find it hard to switch to another proxy, even though there are plenty of alternatives.

As with almost any piece of software, WWWOFFLE has its quirks. Because WWWOFFLE is licensed under the GPL and the source is freely available, I was able to fix myself most of the features that I found defective or lacking.
The WWWOFFLE source is nicely structured and quite readable, and I found hacking WWWOFFLE quite enjoyable, even addictive to a certain degree.

During the course of time I made numerous changes to the WWWOFFLE source, fixing bugs, adding a few features, and sometimes just modifying the code to suit my personal taste. Even though I like the overall structure of WWWOFFLE, I disagree with a number of the implementation details and I've changed most of the things I didn't like.

Out of respect for the free-software community, I'm making the modifications I've made available via this webpage as a patch file wwwoffle-2.9d-par.diff.gz. (Previous patches are still available from this page.)

To use this patch file, first untar a fresh copy of the original WWWOFFLE version 2.9d source and cd into the source directory wwwoffle-2.9d. Then apply the patch using the command

gzip -cd <path_to_patch>/wwwoffle-2.9d-par.diff.gz | patch -p2 -N -E

Then ./configure, make and make install just as you would with the original source code. Note: The current patch breaks IPv6 functionality. Because in version 2.9 IPv6 is enabled by default, you will need to configure using the --without-ipv6 option.

Here's a summary of the most important modifications I've made (in reverse chronological order):

• New in version 2.9c-exp: I have implemented support for persistent ("keep-alive") connections, both to the client and to remote servers. This can be significantly more efficient for two reasons:
  • each WWWOFFLE process can handle multiple requests, thus far fewer child processes need to be spawned,
  • fewer TCP connections need to be opened (though this depends to a large degree on the cooperation of the remote server).
  To enable persistent connections, you must set the new configuration option allow-keep-alive = yes. Of course support must also be enabled in the browser, but most browsers have this by default. See README.par for more details.
• New in version 2.8e-9: I have added a new option called keep-cache-if-header-matches. This option is similar to keep-cache-if-not-found, but examines the header lines (instead of the status code) of the reply from the remote server to determine whether the new page is less desirable than the already cached one. I use this new option to prevent a certain site that requires a subscription from overwriting a good copy of an article with a login page. Unfortunately, this new option is not as generally useful as I had hoped, because only few of these types of sites give clues in the reply header which can be used to determine which replies contain the full article text and which contain the login page.
• New in version 2.8e-6: I have added a new option for the ModifyHTML section: insert-file. The insert-file option can be used to add the content of a local file at the end of a webpage, without modifications. One of the more powerful applications of this feature is to add your own Javascript to webpages; this allows you to implement most of the possibilities of Greasemonkey, a popular Firefox extension, even on browsers that do not support Greasemonkey (but do support Javascript).
  Countless things can be done with the addition of your own Javascript; just run a google query on "greasemonkey" to get a list of ideas. My favorite application is to turn the tables on websites that use Javascript to serve irritating ads. I can set disable-script=yes to disable the website's Javascript and use the insert-file option to add my own Javascript to remove any remaining ads that are otherwise difficult to get rid of. Here is an example of an include file that I use to get rid of some of the more annoying ads on linuxtoday.com.
• New in version 2.8e: The URLs of the cached webpages are stored in a single compact database file called urlhashtable instead of the many small U* files. This reduces the number of files in the WWWOFFLE cache by almost 50%. Having many small files is quite inefficient. For example, on my system each U* file occupies 4K of disk space. With 78587 U* files that is 307MB of occupied disk space. My urlhashtable file uses less than 7MB! (The file will appear to be much larger, but actually occupies only a very modest amount of disk space. See README.par for details.)
  My initial motivation for getting rid of the U* files was my irritation with the long time it took WWWOFFLE to generate an index page of a cache directory containing thousands of files. After a while I started to suspect that reading many small files is a very expensive operation. When Marc Boucher explained to me how his wwwoffle-ls2 utility uses a separate lookup table to make pattern matching of URLs significantly more efficient, I got an idea how to implement this directly in WWWOFFLE. My implementation uses a single file which is mmapped to an area of address space that is shared between all WWWOFFLE processes.
  The improvement in the time it takes to generate index pages is dramatic. Also a WWWOFFLE purge operation with "use-url = yes" is now much faster (by as much as a factor of 4 on my system). I also believe that storing the contents of webpages while online has become somewhat faster (because only one file per webpage needs to be written instead of two), but I haven't actually done any measurements to verify this. The reading of webpages stored in the cache will probably not be significantly improved.
• New in version 2.8d-3: I've added a new option to the ModifyHTML section called replacement-meta-refresh-time. See README.par for more information.
• New in version 2.8d: I've added support for connecting via SOCKS v4 proxies. See README.par for an example how to use this. In version 2.8e-4 the SOCKS 4A protocol is also supported. This is useful in connection with Tor, for example.
• New in version 2.7h: I've included a patch provided by Marc Boucher. This implements a new configuration option called always-use-etag. Certain server farms spoil the usefulness of Etags by issuing different Etags for the same content, causing WWWOFFLE to needlessly download the same page or image repeatedly, even though the content hasn't changed at all. Setting always-use-etag = no can remedy this problem by forcing WWWOFFLE to base conditional requests on the Last-Modified time only, if it can be considered a strong validator.
  Note: in version 2.8d this option is now called validate-with-etag, but I still use Marc Boucher's implementation of this feature.
• I've fixed a bug (at least I consider it a bug) that caused all references to a webpage to disappear from the lasttime indexes whenever a webserver replies with 304 (Not Modified). My version of the code will not add a new entry to the lasttime directory in this case, but it will not remove existing ones either.
• I've added a new option to the cache index pages that enables you to see the titles of the listed webpages. I find it useful because it generally gives you a better idea what the pages are about.
• I've added a new option called keep-cache-if-not-found. This is useful for preventing old cached versions of pages being overwritten by error messages from a web server. An implementation of this feature is also available as a separate small patch file from this page.
• I've made some optimizations to the way hash values of URLs are computed, so WWWOFFLE should now waste less time re-computing the same hash values.
• I've improved the efficiency of case-insensitive pattern matching by eliminating the need for temporary allocation of strings on the heap in strstrn().
• I've fixed a bug that sometimes caused webpages to be incompletely decompressed.
• More efficient handling of CGI-scripts. Closer adherence to the CGI-specification.
• I've added an option cache-control-no-cache that can be used in the offline section of the configuration file. This option works similarly to "pragma-no-cache" and can be used to reduce the number of outgoing requests that are generated when you hit the reload button of your web-browser while offline.
• URL-specifications are allowed to have a so-called parameters part. This allows you to use something like http://*.com/images*;*sz=120x480*. In the original version of WWWOFFLE a URL-specification containing a ';' in the path part will never match.
• I've split the CensorHeader section into two sections: CensorIncomingHeader and CensorOutgoingHeader. I find this conceptually cleaner.
• The URL Configuration webpages have been debugged and enhanced. Very useful if you want to see how WWWOFFLE is configured for a particular URL.
• I've added an option called session-cookies-only. When enabled, WWWOFFLE strips the expires field from Set-Cookie: server headers. Most browsers will not store such cookies permanently and forget them in between sessions.
• More conservative treatment of headerlines. Headerlines with the same "key" are no longer combined. This prevents problems where a comma can act both as part of a value or as a separator between values (e.g. the infamous "Set-Cookie" problem). Furthermore server-headers are not censored before being stored in the cache, only at the time they are sent to the browser.

A more detailed list of the changes I've made can be found in the README.par file in the source directory after patching. At the end of this file you can also find my email address should you wish to contact me.

PS: If you use pdnsd, you might also be interested in my pdnsd webpage.

Have fun.

Paul Rombouts.

Please sign the petition for a Software Patent Free Europe