Alo Sarv
lead developer

Donate via

Latest Builds

version 0.3
tar.gz tar.bz2
Boost 1.33.1 Headers
MLDonkey Downloads Import Module Development
Payment completed
Development in progress.
Developer's Diary

Tuesday, October 11, 2005

Making pigs fly?

I have to admit that I seriously under-estimated the complexity of bittorrent module complexity. Two months of work has already gone to it, and I was hoping to do a release around 15th this month with bittorrent support, but it's getting painfully obvious that that deadline is no longer achievable. There are enough features and optimizations to qualify as new 0.1.x release, but I really don't want do do yet another 0.1.x release.

Frankly, I'm getting the feeling that it'll be easier to make pigs fly than it is to get bittorrent integration working and doing the right thing under all circumstances. Current idea and related problems are like this:

Bittorrent plugin will cache all known torrent files to config/bt folder. On startup, it shall load all of them up, and store the info_hash values for each file. Note: That could later be optimized to only check timestamps on the .torrent files, and have faster reference file for the info to avoid scanning tens, maybe hundreds, of torrent files on startup. Next step is to scan shared/temp lists, and for each file that has a custom metadata field "torrent:****", attach the file to the corresponding Torrent object. This is where it gets complex: based on this design, we will allow torrents (either seeded or downloaded) to be composed of one-to-many of the files it contains - basically the torrent can have any number of gaps in it. The problem is that in BT, hashes can exceed file boundaries, which means if we are seeding one file of a big torrent, we cannot seed the first and last chunks of the file (since we don't have the full chunks for those).

On the other hand, when we'r downloading a single file out of a bigger torrent, in order to complete that file, we must also download more data in order to complete the first and last chunks of the file. Now add canceling, stopping/pausing and the rest of the stuff you can do with downloads to the mix, and we'v got a whole new set of problems. For example, when doing a cooperative download with other (file-based) network, what happens when the file completes? BT plugin will not allow completing the file unless all of the chunks are complete, but that means the file will stop at 100%, technically complete (could be verified via alternative hashes), but BT plugin still needs the first and last chunks to be downloaded (from other files) in order to complete it. But the other files may be already canceled, or completed, but deleted from disk.

In order to compensate against that, I think we need to add an intermediate cache for BT module. Basically, all downloaded (or shared) chunks that cross file boundaries must be duplicated internally by BT module, and stored somewhere INTERNALLY. Later, when we need that data, and are unable to read it from the original file (canceled/deleted), we use the cached data to complete the file. All of that must be of course cleaned up in the end as well - we don't want to store that data indefinately.

How exactly all of that that should be implemented, I have no idea.

Madcat, ZzZz

I realy look forward to the next HN release with ed2k and BT modules.
I follow your develobment now since one year, and I hope you don't "give up" befor the first realy usable, stable version is ready.
Take your time. :-)
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?