Alo Sarv
lead developer

Donate via

Latest Builds

version 0.3
tar.gz tar.bz2
Boost 1.33.1 Headers
MLDonkey Downloads Import Module Development
Payment completed
Development in progress.
Developer's Diary

Saturday, October 08, 2005

More fixes

Hate to say it, but I spent another day profiling and bugfixing. But then again, I dislike building new stuff on top of things that I can't be sure they work 100% good, so I guess the time spent ensuring the base is good is well spent. The major fix today was finally closing bug #97, that had plagued us since the first implementation of ClientManager - it caused rather odd crashes every few hours; apparently, the bug was that when merging clients in ed2k, the client extension objects were re-parented, but since they kept parent Client pointer internally, this led to crash. This bug didn't surface earlier, since before the m_parent pointer wasn't used for so important stuff - just some logging calls, and such. To be totally honest, it's not really good design there either - having the extension objects keep reference to their parent - it was originally added to allow log messages coming from extensions to print parent object info (ip/port - for debugging purposes); later it was used to simplify some handling of ED2K::DownloadList management; and now it's used to simplify handling of ClientManager integration. Technically, both of the last two could be implemented "externally", e.g. handled in Client class, instead of the extensions, however that would be more error-prone - lot of stuff needs to be done whenever the extension is constructed, or destroyed, and the most maintainance-safe place to do it is in their constructors/destructors (since they are created/destroyed in many places).

gprof and strace-based profiling only resulted in limited success. I was able to reduce time spent in gettimeofday() syscall (on Linux) from 65% down to 16.5%, and inlined a number of functions in different places (Scheduler, ED2K::Client, Partdata), however personally I didn't record any noticable CPU usage drop - still at around 3-5% at 25kb/s upstream, 185kb/s downstream rates (maxing out my link limits) [release build, 2.4ghz p4/ht]

When I first brought in ClientManager, I was expecting to see a lot of mess being shown about ed2k upload-manager, because I considered that part rather weak; however, as it turns out, it's handling it pretty well - keeping slots at low counts (3-5 usually), and splitting bandwidth between those slots correctly (current implementation uses "slot-focus" I think it's called in emule - single slot gets as much as possible, other slots get the leftovers).

Now the big question here is how to handle upload slots across multiple plugins. The base idea detailed in original Hydranode design documents, dating back August/2004, indicated that upload bandwidth should be distributed between plugins based on how useful the plugin has been to use from downloading perspective; hence, if 80% of incoming traffic has come from edonkey plugin, then 80% of upstream bandwidth should go to edonkey plugin. Now, slots don't measure bandwidth, slots are merely used to "fill" the available bandwidth. Plugins have support for priorities right now - adding per-plugin bandwidth limits would be simple.

Basically, I'd like to end up with two upload handling options: dynamic and static. Dynamic way would then, as described above, automatically adjust per-plugin upload limits based on how much download data has been received (per session or overall - that's still open); static way would allow user to specify per-plugin limits by hand, e.g. 5kb/s ed2k, 5kb/s bt. Now, some networks have minimum upstream requirements (e.g. ed2k), so question is how to handle that properly. One way would be to apply the ed2k-specific limiting only when using static bandwidth limiting. If user sets ed2k upstream limit <10kb/s nice behaviour I don't know yet... suggestions?

Madcat, ZzZz

I really liked the original idea for UL management. Seemed pretty fair to me - more than anything else, especially the artificial ed2k limits which are only there for show. If I get some from a network I should give at least that much back but at the same time not at the expense of another network.

It was good also because it could be specialized. For example ratio trackers - if I owe some particular tracker I'd prefer that tracker be paid back to keep my ratio. We could consider a tracker almost a separate network because in BT it kind of is.

Still a weighed queue on plugins is not perhaps the best solution when it comes to stuff like releasing. An alternative could be specializing on files rather than networks since that's what we're after. Files themselves can have priorities to properly handle the user's wishes, but not the networks themselves.

Other limiting, in particular static, can be abused and seem unfair to me as ultimately they leech this or that network as a whole.
Why don't you implement a session-based ratio of 1:3 when upload is < 10kB/s (or more) for eD2K? This is already included in some eMule mods (e.g. Xtreme). So, if you upload more than 10kB/s (or in Xtreme 11kB/s), you have unlimited download. But if you get below this 10kB/s (maybe when you are uploading more to BT or other plugins), your download will be unlimited as long as you are below the 1:3 ratio.
To make it more fair, this ratio could be changed to 1:2, or even 1:1.

Another thing: Kademlia plugin should be done as soon as possible (maybe after the GUI), because there are a lot of fake-eD2K-servers around. :(
Please add automatic guarding_full.p2p updating:

Personally I think Kademlia support must be done earlier than the GUI, it will can be a good "heavy test" for HN and doing a lot better some networks (ed2k, bt...).

About CPU usage drop, please test it on a slow CPU ;)

What about a subtitute of "m_parent pointer used to simplify some handling of ED2K::DownloadList management and ClientManager integration"? Running stuff is ok, but designing is very important in really well developed projects ;)

About UL/DL management, what about adding exceptions? By example: when you add a friend in ed2k network and you want him to getting the max bw, or you are uploading to a ftp and want the max upload to it because you want to transfer it fast, or want get a file by http/ftp at max speed...

About searching stuff... what about using websites like You specify search using one of the supported pages and then getting directly the results (using http plugin for accesing to the page, sending http post (like cURL does).

For http/ftp functionalities, I think about see what wget and cURL does and implement it on those modules. Because you can use http plugin for a lot more than only dl/up files (p2p pages...), some other stuff must be added (and please not do like other p2p clients that gives you a link for see a search/fake finder/... on the browser, getting the results and showing correctly on hn without the bs, and maybe using it "directly" if needed...)
How is Kad support going to be implemented anyway? A verbatm copy from emule or something totally new but compatible. Because emule's implementation is pretty simple and straightforward for the basic operations.
ed2k_kad module is being developed by cyberz for some time already. It's a new implementation, and also provides a generic kademlia algorithm implementation, which can then be reused for other kad-based protocols. There's no expected release date for it yet.

Can we get one of the famous RFC for Kad as well (one of these days)? I'd love to help if the design is appealing (not that I doubt it). :)
Can't promise anything, but I'll talk to cyberz when I see him.

Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?