Jurgen, the creator of yEnc, and I have had discussions regarding the lack of stream support for yEnc encoding. I’ve even mentioned it on the Wikipedia page for yEnc and also in the yProxy FAQ. I’m going to cover streaming and pipelining in detail here so that you can better understand the limitations of yEnc and yProxy. This article will also help you understand some important concepts that are applied elsewhere.
Apparently, the lack of streaming support for yEnc encoding was an oversight, probably stemming from lack of knowledge regarding the general design pattern used for streaming data.
Pipelines
Pipelining relates to streaming, especially with yEnc. In computing, processes often take data as input, perhaps change the data in some way, and then output the resulting data. A second process may then take the output of the first process as its input, and so on. When the output of one process is used as input by another process, this is called pipelining.
This pipelining effect is how proxies work. The output from your news server is piped through yProxy to your newsreader. This allows the user to create a pipeline of tools that filter or modify the data in multiple steps. Connecting pipes allows generation of a specific output, often converting one format or protocol to another.
The pipelining concept is also used when hooking up cable television. You connect the output from the wall into the input of your cable converter and the output from the cable converter into the input on your television. Your cable converter unscrambles the TV signals as the data travels from the wall, through the cable converter, and into your TV. Your cable converter may even be converting coaxial cable input into composite cable, S-Video, or HDMI.
Streams
Streaming means that you process data in real time as you receive it, rather than waiting to receive all of the data before you begin processing it.
Streaming movies on the Internet means that you can start watching the movie from the beginning, as soon as it arrives, while the rest of the movie downloads. You don’t have to wait for the entire movie to download to begin watching it.
Streaming reduces the latency, or wait time, while data is going through the pipeline. If your cable converter needed to download an entire show before you could begin watching it, you would definitely notice the wait. However, the cable converter processes the data as it is received and immediately passes it along to the TV, removing the perception of any delay.
While yEnc supports pipelining in both directions (encoding and decoding), yEnc supports streaming only for decoding.
Streaming yEnc
yEnc includes the size of the file in the yEnc header, instead of in the footer.
Imagine if you were the type of person that hated surprises, so you absolutely had to see the ending of a movie before you could watch it. If that were the case, you could never watch a streaming movie. You’d have to download the whole movie, skip to the end, then go back to the beginning. yEnc has a similar problem. yEnc requires the encoder to describe the end of the data before sending the data.
If yEnc had simply made the “size” information a footer instead of a header, yEnc would support streaming in both directions. The current process in the pipe doesn’t have the size of the attachment until all of the data has flown through, so the size should be included at the end, not the beginning.
If you piped an attachment from your newsreader to yProxy, in order for yProxy to encode the attachment, yProxy would first have to receive the whole attachment, buffering it in memory or on your hard disk, generate the yEnc header, and then begin uploading the encoded attachment to the news server. Outlook Express, Windows Mail, Thunderbird, or whichever newsreader you were using would have to wait while yProxy sends the entire attachment to the news server. The newsreader would very likely timeout while it waits for yProxy to respond with the success message from the news server. Therefore, yProxy does not support yEnc encoding.
Streaming and pipelining are important paradigms in communications and information technology. Developers should adhere to both standards when designing protocols or encoding methods.
