Alo Sarv
lead developer

Donate via

Latest Builds

version 0.3
tar.gz tar.bz2
Boost 1.33.1 Headers
MLDonkey Downloads Import Module Development
Payment completed
Development in progress.
Developer's Diary

Thursday, October 27, 2005

Testing day

Feeling somewhat tired, and un-motivated after yesterday's heavy (and complex) coding session, today I'v been mainly chilling, and doing routine testing. I guess the most interesting area is win32 port; I made an optimized build (god that took long - something around 30-40 minutes on high-end system), and downloaded the usual bittorrent testset - a wired creative-commons-licenced mp3'set and the suse dvd, latter of which has 14'500 chunks in it, which is really heavy even by bittorrent standards (for comparison - in ed2k, 700mb file has 72 chunks, 4gb file has below 450 chunks; in bt, normally torrents have below 5000 chunks). Anyway, this exposes some scalability issues.

Namely, in order to choose the rarest chunk, we must update the availability of each chunk, and that's a linear-complexity operation. Furthermore, in order to do chunk requests, we must pass a vector of bool's to PartData::getRange(), which means when we receive a chunkmap (as bitfield) from a client, we must copy that into bool-vector, which is also a linear operation. Now, I did find a way to avoid the latter - namely generalize the PartData::getRange(), passing a functor returning bool, and using direct bit-manipulation. However, the first issue is harder, and will, no matter what, remain a linear-complexity operation - I don't see any way around it.

Coupled with the above bitfield-problem, the CPU usage when downloading that torrent seems to be around 10%, spiking up to 15% occasionally (at around 150kb/s transfer rate). We also seem to have rather heavy memory-leakage issue somewhere there - 20-30 minutes runtime resulted in leaking from 15mb (normal hydranode memory-usage) up to 90mb for no apparent reason. It's not yet determined whether the leakage is within to hncore/hnbase libraries, or in bittorrent plugin (ed2k has shown some minor leakage, but nothing of this scale, so I'd guess the leakage happens in BT).

However, it also seems there's some more cpu-hog code around in BT module, since while the bitfield-handling is heavy, it only happens once per each client (bitfield is only sent once); however the cpu-usage stayed at the high levels even when there were no new clients being created. Off top of my head, I'd blame the virtual file wrapper, but then again, that can't be THAT heavy - yes it does a lot of signal/slot-based communication, but most of that should'v been optimized out by compiler, and this torrent had only one file, which means the rangelist lookups can't be the bottleneck either. To make things worse, after a while hydranode/bt just maxes out the CPU and stays that way (still downloading tho, so not a local endless loop)... And while we'r at it, bt downloads sometimes stop at 100%, unable to complete; and on win32, deleting temp files fails sometimes with error "file already in use" :( I guess I'll have some fixing and optimizing to do *sighs*.

I ran a profile build two nights ago a whole 12 hours, but accidentally destroyed the profiling data when I woke up in the morning (programmers really shouldn't be allowed near computer before they'v had their second cup of coffee ...). I'll do a more thorough performance analysis again in a few days... however, when it comes to memory leakage, I'm bit stuck - due to sophisticated memory-management of C++, normal memory-leak-check tools (e.g. valgrind etc) are no use, since on shutdown, everything is cleaned up (all big data is stored in std::string's for example, and those get cleaned up properly). So I need some other kind of leak-checking mechanism that is able to tell me on RUNTIME where all the memory is allocated. If anyone knows a tool that can do that, please leave a comment :)

Madcat, ZzZz

This post has been removed by a blog administrator.
Don't post ed2k links here. Your comment was deleted for that reason.

Ah well, you have the link now, that was the intention :D
I think the name of the program would have been enough...
Meh, just saving him some time
also checkout microtorrent, a 90kb BT client for windows, works very well, and has most of the useful azureus features except distributed DB.
No marketing, please ;)

ĀµTorrent is closed source, ok? Why do you mention it? Closed source means no possibility of analyze...
Not necessarily, its GUI can be analysed, as can the way it handles uploads etc...
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?