Not long after I first started using rn did I discover that extracting the data from multi-part articles by hand was not fun. I wrote a little perl script which would take a file full of a series of articles and decode the data therein. Much nicer for comp.sources.unix postings, etc.
With the advent of the binaries newsgroups, even saving the parts was a bit of the chore. I expanded my little program to have it search for the articles itself. Fire it up and it would leave you with a directory full of all sorts of good stuff.
The big problem was that one couldn't easily refer back to the source articles which often contained useful descriptive text. It was a batch process and each time it was run it would decode stuff you'd already seen.
I took the next step which was a simple newsreader that would present articles to you in groups. Not only could it trudge through junk quite nicely, it made it easily to pick and choose what to decode.
While much more convenient that rn, it was slow. It would find the article groups every time you entered a newsgroup. So I wrote a new version that used off-line processing to discover and decode the article groups. The data generated was then used by the news reader to speed up the process for a user while creating a system that used less CPU overall. It would even generate previews so you could quickly decide if an article group was worth examining.
This was quite nice, and then came the web. I quickly realized that browsers like Mosaic and their inline images would be a perfect vehicle for present the decoded data. Shortly after I wrote the first version of grnhd which works more-or-less like the current version. A new, even smaller preview image was added to the decoding process and more information on each article was generated to make generating the HTML version on the fly easier.
At this point grn was available to the world. In general, the world seemed to like it. Almost too much. Our web server became heavily used and robust as a matter of survival. I then took another developmental step to reduce server load. Instead of generating the HTML pages whenever one was requested, I would pre-constuct them whenever new article groups were found. Even though the decoding process works incrementally, it was never made to incrementally generate the decode information or the grnhd support information. The easy way to get pre-generated HTML pages was to handle it in a batch fashion and literally re-generate them all every time the decoder was run. This is bad and expensive in CPU, but overall it is cheaper at our site as each pre-generated page will be accessed many times thereby saving lots of CPU. Additionally, the pre-generated HTML document tree can be browsed as a local file system for those that have access. Those users largely leave the server alone and bring even more load savings.
The next version continued on the trend to reduce server load. Fully decoded versions of the articles were saved to disk, sparing the server the CPU cost of dynamic decoding. The article storage was rearranged so that massive hourly updates were avoided and additional articles were added in a lightweight fashion. The changes had the side effect of making the server end of things much easier to install. No more fancy CGI scripts, just a plain old document tree.
For quite some time the bulk of GRN's decoding was done from the junk newsgroup. Due to a policy change at our site, many newsgroups that normally flowed through junk were created. While GRN could handle this, crossposting presented a problem. Each article group would be decoded for every group it was posted to. Forseeing the immense strain on disk space I cut back on the newgroups and worked on a new version that would understand crossposting.
I got fairly far along in the attempt but ultimately saw that the changes were too great. Work on re-implementing GRN to deal with the situation more gracefully has begun.