Archive for July, 2008

Lack of stream support for yEnc encoding

Tuesday, July 29th, 2008

Jurgen, the creator of yEnc, and I have had discussions regarding the lack of stream support for yEnc encoding.  I’ve even mentioned it on the Wikipedia page for yEnc and also in the yProxy FAQ.  I’m going to cover streaming and pipelining in detail here so that you can better understand the limitations of yEnc and yProxy.  This article will also help you understand some important concepts that are applied elsewhere.

Apparently, the lack of streaming support for yEnc encoding was an oversight, probably stemming from lack of knowledge regarding the general design pattern used for streaming data.


Pipelining relates to streaming, especially with yEnc.  In computing, processes often take data as input, perhaps change the data in some way, and then output the resulting data.  A second process may then take the output of the first process as its input, and so on.  When the output of one process is used as input by another process, this is called pipelining.

This pipelining effect is how proxies work.  The output from your news server is piped through yProxy to your newsreader.  This allows the user to create a pipeline of tools that filter or modify the data in multiple steps.  Connecting pipes allows generation of a specific output, often converting one format or protocol to another.

yProxy Diagram

The pipelining concept is also used when hooking up cable television.  You connect the output from the wall into the input of your cable converter and the output from the cable converter into the input on your television.  Your cable converter unscrambles the TV signals as the data travels from the wall, through the cable converter, and into your TV.  Your cable converter may even be converting coaxial cable input into composite cable, S-Video, or HDMI.

Cable box hookup


Streaming means that you process data in real time as you receive it, rather than waiting to receive all of the data before you begin processing it.

Streaming movies on the Internet means that you can start watching the movie from the beginning, as soon as it arrives, while the rest of the movie downloads.  You don’t have to wait for the entire movie to download to begin watching it.

Streaming reduces the latency, or wait time, while data is going through the pipeline.  If your cable converter needed to download an entire show before you could begin watching it, you would definitely notice the wait.  However, the cable converter processes the data as it is received and immediately passes it along to the TV, removing the perception of any delay.

While yEnc supports pipelining in both directions (encoding and decoding), yEnc supports streaming only for decoding.

Streaming yEnc

yEnc includes the size of the file in the yEnc header, instead of in the footer.

Imagine if you were the type of person that hated surprises, so you absolutely had to see the ending of a movie before you could watch it.  If that were the case, you could never watch a streaming movie.  You’d have to download the whole movie, skip to the end, then go back to the beginning.  yEnc has a similar problem.  yEnc requires the encoder to describe the end of the data before sending the data.

If yEnc had simply made the “size” information a footer instead of a header, yEnc would support streaming in both directions.  The current process in the pipe doesn’t have the size of the attachment until all of the data has flown through, so the size should be included at the end, not the beginning.

If you piped an attachment from your newsreader to yProxy, in order for yProxy to encode the attachment, yProxy would first have to receive the whole attachment, buffering it in memory or on your hard disk, generate the yEnc header, and then begin uploading the encoded attachment to the news server.  Outlook Express, Windows Mail, Thunderbird, or whichever newsreader you were using would have to wait while yProxy sends the entire attachment to the news server.  The newsreader would very likely timeout while it waits for yProxy to respond with the success message from the news server.  Therefore, yProxy does not support yEnc encoding.

Streaming and pipelining are important paradigms in communications and information technology.  Developers should adhere to both standards when designing protocols or encoding methods.

Avoid using your ISP’s email services

Friday, July 25th, 2008

While others may dispute the legitimacy of free email accounts like Hotmail and Yahoo, the benefit of such services is that they are not dependent on your ISP.   A free web based email service can follow you around.  Your ISP’s email account is only temporary.

Many users are enslaved when their ISP raises their rates.  The users don’t want to leave and find another provider because they’ll have to change their email address.

If I need to change ISP to chase better prices or better service, I don’t want to worry about my friends, family, and professional contacts not being able to contact me.  Of course, many ISPs are strictly local (people move away), ISPs go out of business, and ISPs gets bought out.  For all of the above reasons, ISP provided email accounts should be considered temporary.

Previously, free email providers didn’t offer services as favorable as a typical ISP’s service.  However, today, most of the free email providers have grown the size of their mailboxes to compete with Google’s gmail (started at 1 Gigabyte and is now at approximately 6 GB).  Many free email providers also provide decent antispam software, yet still allow you to check your junkmail folder for legitimate emails that my have been mistaken as spam.

It’s time for merchants to stop declaring free email accounts as hot spots for thievery and refusing orders or sign-ups using free email accounts.  Free email accounts are perfectly legitimate.  Despite this, an order from is definitely more deserving of scrutiny than an order from  However, an order from still looks pretty good.

When you register yProxy, please remember that if you use your ISP provided email address such as, what happens when you change ISPs down the road and you need me to resend your registration information?  I can send your registration information right to the email address that you used on the order form because I know that your email address is password protected, and only you should have access to it.

If you want me to send your registration information to a new email address, that new email address could belong to anybody.  You’ll need to prove your identity to me by providing a preponderance of data that I can verify.  That’s not fun for you or for me.

You will have similar problems with many other online sites.  If you need to retrieve your password via email later, and you’ve switched ISPs, you may be out of luck if you used your ISP’s email services.

When I was in college, people asked me for two addresses.  There was my address and my permanent address.  My address was the apartment or other location I was staying at the time, most definitely temporary.  Then there was my parent’s house, my permanent address, where mail would always find me.

Everyone should have a permanent email address.  Either use a free email provider that’s going to be around for awhile, like Yahoo, Hotmail, or Gmail, or get your own domain and have an email address like  It’s the 21st century–everyone should own at least one domain name.