The first video tutorial shows you what yEnc messages look like in your newsreader if your newsreader doesn’t natively support yEnc. This video will help you recognize yEnc messages so you know if you need a yEnc Decoder like yProxy.

The second video walks you through installing and configuring yProxy with your newsreader, but uses Windows Live Mail as the example newsreader.

The third video walks you through running yProxy for the first time and starting yProxy.

The fourth video demonstrates yProxy in action.

The binary groups I frequented started getting a fair amount of yEnc posts.  About 10% of the posts were yEnc encoded.  I couldn’t read yEnc encoded attachments with my existing newsreader, so I simply conceded to the fact that I would miss out on the content.  However, as more and more of the valuable content was yEnc encoded, I either had to find a yEnc decoder, or keep missing out on the yEnc content.

My news reader, Forte Agent, did not decode yEnc attachments.  It only handled MIME and UUE.  I downloaded yEnc32.  In order to decode attachments with yEnc32, I had to first export the messages to text files, then processes those text files with yEnc32.  While this wasn’t very difficult to do, it was time consuming.

I am a software developer, and one of my specialties is developing web and network utilities.  As a network developer, I was able to apply the pipelining and streaming paradigm to this problem, yielding a proxy as a solution.

What is a Proxy?

A proxy is a generic term in networking for a utility that sits between a client and a server and acts on behalf of the client.  Generally, a proxy filters and/or caches some type of traffic that flows between the client and server.

The most common type of proxy related to the Internet is a caching web proxy.  A caching web proxy intercepts all web requests and attempts to fulfill the request from its cache.  For example, one person might request a photo from the Internet.  The web proxy retrieves the photo from the Inernet and delivers it to the client.  The second person that requests the photo from the proxy is sent a copy of the photo that had been saved to memory or disk.  This saves Internet bandwidth.

yEnc Decoder Proxy

A yEnc decoder proxy also sits between the server and the client, sending requests on behalf of the client.  When the client asks for a yEnc encoded attachment, the proxy requests the attachment from the server in yEnc format and converts it to a more accepted format before sending it to the client.

A yEnc Decoder proxy is also considered a plug-in because it’s a third party utility that adds functionality to an a separate program.

I figured a yEnc Decoder proxy would be the perfect solution to my problem.  The only problem, of course, was that there wasn’t such a thing.   I would have to write one.

yProxy: The Original yEnc Decoder Proxy

I wrote the first, beta, version of yProxy over the weekend and released it to the public.  I also alerted Forte so that they could recommend the free utility to their users until they added yEnc support to Agent.  Unfortunately, Forte never responded.  Yet, Forte did add yEnc support to Agent shortly after, and anyone with a recent version of Forte Agent has native yEnc support.

Still, the word about yProxy spread via the newsgroups and quickly became the yEnc Decoder solution for many people who either weren’t entitled to upgrades for their existing newsreaders, or whose newsreaders just plain didn’t support yEnc.  For example, the most popular free newsreaders are still Outlook Express, Windows Mail, and Mozilla Thunderbird.  None of those newsreaders support yEnc.

With a yEnc Decoder proxy between your newsreader client and the news server, you can continue using your existing newsreader and still be able to read yEnc content, even if your newsreader does not support yEnc natively.  Most yProxy users, therefore, are Outlook Express, Windows Mail, and Mozilla Thunderbird users.

Jurgen, the inventor of yProxy, became one of yProxy’s biggest advocates.  People who had previously complained that their newsreaders didn’t support yEnc were directed to use yProxy and stop complaining.

yProxy is the Leading yEnc Decoder Proxy

yProxy Pro’s biggest competition is the free version of yProxy, which is still available on the yProxy website via the FAQ page.

I stopped counting after one million people downloaded yProxy, which occurred within the first year that it was available.  There is even a French version that a considerate user translated.

yEnc has more advantages than simply being smaller in size.  yEnc also ensures that the file has been delivered intact via two methods:

• CRC32 Error Checking
• File Size Checking

If you yEnc decoder doesn’t support the above features of yEnc, not only are you losing out on some of the benefits of yEnc, but your yEnc decoder is not even yEnc compliant.

CRC32 Error Checking

yEnc CRC32 error checking uses a technology similar to what CDs and DVDs use to check for errors.  It uses a mathematical algorithm at creation time to generate a checksum value based on the data and stores that checksum on the data.  At play time, the value is recomputed against the data and compared against the stored value.  If the checksums do not match, then an error has occurred while reading the data.

In order for a yEnc Decoder to be compliant with the latest yEnc standards, a yEnc Decoder must include CRC32 error checking.  All yEnc attachments have the CRC32 checksum, and if your yEnc Decoder isn’t evaluating the checksum, then you won’t know when you’ve downloaded a file that has been corrupted or modified since it was created.

You may download that treasured song that you’ve been looking for, then when you go to play it back, it may not play or it may contain defects.  At that point, it may be too late to try to download the song again from another source.

Embedded yEnc Decoders will not provide CRC32 error alerts.  yProxy notifies you immediately when an attachment with an error has been detected.  yProxy beeps and changes the system notification icon to one with a large red exclamation mark, in addition to logging the error.  None of yProxy’s competing yEnc proxies provide CRC32 error detection.

File Size Checking

The file size is also computed at creation time and stored in every yEnc attachment.  All yEnc compliant yEnc Decoders must check the final file size against the stored value.  If a file has been truncated by the news server or parts of it are missing, the file may be unusable.  It is important for your yEnc decoder to be able to determine the difference between a corrupt file and an incomplete file.

If the file is incomplete, you may simply need to try again later after the file has finished propagating.  An incomplete file may still be usable, depending on the type of file.  An MPEG video file with the last second missing will still be playable, and you may not notice that missing second.  Therefore, it is important for your yEnc decoder to distinguish between a corrupted file and an incomplete file.

yProxy notifies you immediately if the downloaded file size does not match the expected size.  yProxy will beep and place an exclamation mark in the system notification icon, in addition to logging the error.  The error displays the expected file size and the actual file size so that you can determine the extent of the problem.

Embedded yEnc decoders cannot check the file size and none of yProxy’s competing yEnc proxies provide file size checking.

yEnc Decoder Compliance

CRC32 error checking and file size checking are valuable features of yEnc encoding.  yEnc decoders must support both CRC32 error checking and file size checking in order to be compliant with the latest yEnc standards.  yProxy is the only yEnc decoder proxy that is yEnc compliant.

We are also seeing the number of Usenet groups that have fresh content decrease, as supergroups evolve that contain the majority of Usenet’s quality content.

Usually, yEnc will take over an entire news group, and all downloaders of that group end up having to find a yEnc decoder solution.

Most high volume groups transition to yEnc rather quickly.  Therefore, due to the high volume posted to these supergroups, and the decreasing number of groups, the percentage of content that is posted using yEnc is very high.

Interestingly, spam is typically not posted with yEnc.  Spammers want to reach the largest audience, so they don’t want to exclude those without yEnc decoders.

Therefore, yEnc use is also becoming a good indicator of legitimate content.  Content that is not yEnc encoded is more and more likely spam.  New posters, or infrequent posters, also may not have the proper tools to post in yEnc, but the quality of their content is marginal for most categories.

Even the yEnc test newsgroup will sometimes have some surprisingly good content, and it’s mostly barren of spam.

As the ability of news servers to filter out spam increases, and the amount of spam on peer to peer systems increases, Usenet is making gains over other file sharing methods.

I believe that the popularity of Usenet is likely to spike up in the near future, barring unforeseen, new alternatives.  In addition, yEnc usage will continue to increase until it has mostly saturated the binary newsgroups.

Apparently, the lack of streaming support for yEnc encoding was an oversight, probably stemming from lack of knowledge regarding the general design pattern used for streaming data.

Pipelines

Pipelining relates to streaming, especially with yEnc.  In computing, processes often take data as input, perhaps change the data in some way, and then output the resulting data.  A second process may then take the output of the first process as its input, and so on.  When the output of one process is used as input by another process, this is called pipelining.

This pipelining effect is how proxies work.  The output from your news server is piped through yProxy to your newsreader.  This allows the user to create a pipeline of tools that filter or modify the data in multiple steps.  Connecting pipes allows generation of a specific output, often converting one format or protocol to another.

The pipelining concept is also used when hooking up cable television.  You connect the output from the wall into the input of your cable converter and the output from the cable converter into the input on your television.  Your cable converter unscrambles the TV signals as the data travels from the wall, through the cable converter, and into your TV.  Your cable converter may even be converting coaxial cable input into composite cable, S-Video, or HDMI.

Streams

Streaming means that you process data in real time as you receive it, rather than waiting to receive all of the data before you begin processing it.

Streaming movies on the Internet means that you can start watching the movie from the beginning, as soon as it arrives, while the rest of the movie downloads.  You don’t have to wait for the entire movie to download to begin watching it.

Streaming reduces the latency, or wait time, while data is going through the pipeline.  If your cable converter needed to download an entire show before you could begin watching it, you would definitely notice the wait.  However, the cable converter processes the data as it is received and immediately passes it along to the TV, removing the perception of any delay.

While yEnc supports pipelining in both directions (encoding and decoding), yEnc supports streaming only for decoding.

Streaming yEnc

yEnc includes the size of the file in the yEnc header, instead of in the footer.

Imagine if you were the type of person that hated surprises, so you absolutely had to see the ending of a movie before you could watch it.  If that were the case, you could never watch a streaming movie.  You’d have to download the whole movie, skip to the end, then go back to the beginning.  yEnc has a similar problem.  yEnc requires the encoder to describe the end of the data before sending the data.

If yEnc had simply made the “size” information a footer instead of a header, yEnc would support streaming in both directions.  The current process in the pipe doesn’t have the size of the attachment until all of the data has flown through, so the size should be included at the end, not the beginning.

If you piped an attachment from your newsreader to yProxy, in order for yProxy to encode the attachment, yProxy would first have to receive the whole attachment, buffering it in memory or on your hard disk, generate the yEnc header, and then begin uploading the encoded attachment to the news server.  Outlook Express, Windows Mail, Thunderbird, or whichever newsreader you were using would have to wait while yProxy sends the entire attachment to the news server.  The newsreader would very likely timeout while it waits for yProxy to respond with the success message from the news server.  Therefore, yProxy does not support yEnc encoding.

Streaming and pipelining are important paradigms in communications and information technology.  Developers should adhere to both standards when designing protocols or encoding methods.