Alo Sarv
lead developer

Donate via
MoneyBookers

Latest Builds

version 0.3
tar.gz tar.bz2
Boost 1.33.1 Headers
MLDonkey Downloads Import Module Development
Payment completed
Development in progress.
Developer's Diary
irc.hydranode.com/#hydranode

Tuesday, December 27, 2005

Extending vacation

I realized you can't really have a vacation / rest with just three days, so I'll be on vacation until past new year. Will be returning around 3rd/4th January or so.

Meanwhile, I put up new builds of the last big changeset; actually I wanted to put them up before xmas, but was just too exhausted and xmas arrived too fast :). Anyway, enjoy the new builds, and happy end-of-the-year / new-year's beginning :)

Madcat


Saturday, December 24, 2005

Merry Christmas

... to everyone celebrating this holiday :)

Madcat

PS: I'll be gone until 27th for obvious reasons :)


Lots of new stuff checked in

Been about 25 checkins to the repository today, most of which was the result of yesterday's long coding session with final touchups today; also checked in the prior work on cgcomm done cpl days ago. Here's the complete [human-readable] listing of new stuff:
Madcat, ZzZz


Friday, December 23, 2005

When all you have is a hammer...

I'v always been fascinated about template metaprogramming - a modern C++ paradigm that allows doing things at compile time. Boost libraries rely heavily on this, and have quite a lot of support libraries for the topic as well - Boost.MPL being the most famous, but is backed up by Boost.TypeTraits, Boost.Tuple and many more.

When doing template (e.g. generic) programming, you do not know at the code writing time what types you may be manipulating. A trivial, but motivating example:
template<typename T> void process(T t) { t.foo(); }
Here, the function process() has no knowledge about the type being passed to it, but it calls the foo() member function on the type. If the type doesn't have the function, you get a compile-time error.

However, there's one limitation - the function can only take arguments by value or reference, it cannot handle pointers - it would need to do t->foo() when a pointer is passed to it. A naive approach to the problem would simply provide additional specializations for the function:
template<typename T> void process(T *t) { process(*t); }
However, this has the problem of requiring the same set of overloads for every function that needs this functionality. A more generic solution is needed, something that is capable of handling pointers, pointers-to-pointers, smart pointers and so on. What we'r looking for is syntax like this, and have everything work behind the scenes:
template<typename T> void process(T t) { unchainPtr(t).foo(); }
In order to "unchain" the pointer, we start out by providing a functor that is capable of recursivly "de-pointering" a given type. We cannot solve this with simple overloaded function templates, due to ambiguouties with const overloads, so we need actually two different functors, which guarantee const-correctness:
template<typename Class>
struct UnchainPtrImpl {
// non-const overloads
Class& operator()(Class &x) const { return x; }

template<typename ChainedPtr>
Class& operator()(ChainedPtr &x) const {
return operator()(*x);
}

Class& operator()(boost::reference_wrapper<Class> &x) const {
return operator()(x.get());
}
};

template<typename Class>
struct UnchainConstPtrImpl {
// const overloads
const Class& operator()(const Class &x) const { return x; }

template<typename ChainedPtr>
const Class& operator()(const ChainedPtr &x) const {
return operator()(*x);
}

const Class& operator()(
const boost::reference_wrapper<const Class> &x
) const {
return operator()(x.get());
}
};
However, now we have another problem - we need to make a selection between these two functors, at compile-time, and choose one of them, depending on whether we want const or non-const variant. MPL to the rescue:
template<typename Class, typename ChainedPtr>
Class unchainPtr(ChainedPtr ptr) {
return boost::mpl::if_<
boost::is_const<ChainedPtr>,
UnchainConstPtrImpl<Class>, UnchainPtrImpl<Class>
>::type()(ptr);
}
template<typename Class, typename ChainedPtr>
void process(ChainedPtr ptr) {
unchainPtr<Class>(ptr).foo();
}
process<X>(new X);
Looks good, with one problem - it's inconvenient to always specify the class type when calling the function - we should be able to do better. In fact, all we really need to do is deduce the raw class name, e.g. X, from a type name like X*, X& or similar. Bring out the heavy guns and let's go. I'm not going to walk you through the entire process, but suffice to say, it took several hours of testing and tweaking before finally arriving at a solution that is capable of doing what we need. The resulting code is:
// main template of the loop type isn't implemented. One of 
// the specializations below are chosen always.
template<
typename T,
bool Final = boost::mpl::or_<
boost::is_pointer<T>, boost::is_reference<T>
>::type::value
> struct Next;

// this is selected when Final is true, e.g. T is a pointer or reference type.
// we recurse by "calling" the getType metafunction again, but after removing
// reference and/or pointer-ness from the type.
template<typename T>
struct Next<T, true> {
typedef typename getType<
typename boost::remove_reference<
typename boost::remove_pointer<T>::type
>::type
>::type type;
};

// this ends the Next recursion - the passed type isn't a pointer or reference,
// so just typedef it as itself and don't recurse into getType anymore.
template<typename T>
struct Next<T, false> {
typedef T type;
};

// if the passed is pointer or reference, we call Next, which removes
// the offending part of the type, and calls us back, so we loop
// around until the passed type isn't a pointer/ref type anymore,
// and the resulting type is available as the 'type' typedef.
template<typename T>
struct getType {
typedef typename boost::mpl::if_c<
!boost::mpl::or_<boost::is_pointer<T>,
boost::is_reference<T> >::type::value,
T, typename Next<T>::type
>::type type;
};
// these specializations can be user-defined, but here we show how boost
// smart-pointers can be added to the mix.
template<typename T>
struct getType<boost::shared_ptr<T> > {
typedef typename Next<T>::type type;
};
template<typename T>
struct getType<boost::weak_ptr<T> > {
typedef typename Next<T>::type type;
};
template<typename T>
struct getType<boost::intrusive_ptr<T> > {
typedef typename Next<T>::type type;
};
// the final function template - added getType metafunction calls to the mix
template<typename ChainedPtr>
typename getType<ChainedPtr>::type&
unchainPtr(ChainedPtr ptr) {
return typename boost::mpl::if_<
boost::is_const<ChainedPtr>,
UnchainConstPtrImpl<typename getType<ChainedPtr>::type>,
UnchainPtrImpl<typename getType<ChainedPtr>::type>
>::type()(ptr);
}
This solution works flawlessly. It can handle even rather obscure types, such as const int **const&, shared_ptr<int*&>& etc. The only offset with this is that it crashes gcc4 (works on earlier versions of the compiler), in the third line of the sample (bool Final handling).

However, then it hit me - there's actually a LOT simpler way of doing all this. If we replace the getType mechanism with this:
template<typename T>
struct getType {
typedef T type;
};
template<typename T>
struct getType<T*> {
typedef typename getType<T>::type type;
};
template<typename T>
struct getType<T *const> {
typedef typename getType<T>::type type;
};
template<typename T>
struct getType<T&> {
typedef typename getType<T>::type type;
};// add smart_ptr specializatios as needed
Everything works even better. The getType metafunctions recurse into themselves, just as the Next template above, and the recursion is stopped when no specialization can be chosen, and thus the primary template is chosen, which ends the recursion. Flawless, clean, extendable and simple. Talk about having a hammer and everything looking like a nail - the first hit on getType was major overkill (granted, it was pretty fun 3 hours fighting with it).

The underlying reason for writing this was actually that I'm generalizing WorkThread API into a template class that can handle arbitary types of jobs, and that we can create more than one WorkThread, for different purposes. IOThread simply becomes a typedef (or a derived class), and we'll be having other worker threads for other things - for example DNS resolution will be done in such a worker thread as well. Given we'll have a simple easy-to-use and safe API for doing jobs in other threads, we bring hydranode more multi-threading, thus gaining response times and speed on dual- and multi-core systems, which are not far from becoming a standard.

Since cyberz wanted to use the final code in "other places" as well, I decided to licence this header under Boost software licence, which is more permissive. The header can be viewed in complete here, and the test-app here if anyone is interested. Frankly I'm surprised I didn't find something like this in Boost anywhere - maybe they have it stashed somewhere in the back room, but all I saw was few libraries using local hacks to do this (actually the UnchainPtrImpl template was copied from multi_index library header), so I'll take this up with Boost ppl - maybe even submit this code to them :)

Hope I didn't bore you guys to death with highly-specific code-talk. Will be more human-readable post next time :)

Madcat


Thursday, December 22, 2005

Cgcomm, Win2k3 issues and The Resolver

First off, great thanks to everyone who has donated so far. It's one thing to get thank-you mails once in a while, but it's completely different topic to actually see physical evidence that what I'm doing here is truly worth doing. So once again, thanks.

In that spirit, I'm thinking more and more towards user interface-related code, since this IS the single-most-wanted feature of Hydranode that's pending. After reviewing and tuning cgcomm code somewhat, I realized that we'r actually lot closer to workable UI samples than I originally thought. The side-project I did a while ago, called launcher or console, while by itself being a failure, it boosted cgcomm quite far before I stopped working on it. Now, going on a second run, I'v been tweaking the cgcomm, adding some features, and doing quite a bit of testing on it, and am actually quite pleased with where it's going.

Some things I realized - code reuse. GUI will be using quite a bit of core data structures internally, such as RangeList API, few Utils functions (bytesToString for example), and so on, so I'm thinking about making GUI depend on hnbase library, which contains those classes. That way I can transmit, say, RangeList objects directly over cgcomm, and load them on GUI side directly into the same type, which re-uses the same code (all such types support both saving and loading from stream). Naturally we won't be using events or sockets from hnbase, but it doesn't cause any bloat either, since hnbase dll is shipped with the app anyway. There's some issues right now tho related to that, since core uses msvc-compiled hnbase (and boost) libs, while GUI uses mingw-gcc compiler. Of-course we have the option of using mingw-gcc for both, but I don't really want to use that for core, since it's slow and creates huge binaries.

Disclaimer: working on GUI-related topics doesn't mean in any way that work on BT or other things is stopped. I'm merely opening up more development directions, to avoid having to work at same topic for long periods of time (gets really boring). Furthermore, GUI development requires quite a bit of features at core side on a regular basis (for example search page instantly requires global search capabilities, and I already discovered that FilesList's internal data structure's two indexes actually get broken on file completition), so it's all good.

Another topic that I'v been turning my attention towards is Enig123, who's been complaining about lot of issues with Hydranode on Windows 2003 server. Namely, the issues he's experiencing are Hydranode not being able to fill 51kb/s, or even 35kb/s upstream out of his 512kbit upload pipe (while other p2p clients can). Furthermore, after about 15-20 hours runtime, the entire networking becomes unusable, until a restart is made. Personally I cannot reproduce the symtoms either on WinXP/SP2 system nor a Linux system, on both of which Hydranode can easily use up 50kb/s upstream pipe and can run for many days without problems (the current official Hydranode uptime record belongs to frop, at over 1 month). Has anyone else experienced similar symtoms, and on what platforms?

Yet another thing that's becoming more and more inconvenient is the Resolver subsystem, written by cyberz a while ago. Originally inconveniences were simple lack of timeout handling, but the issues seem to be escalating, now it sometimes fails to resolve addresses that actually are resolvable. Which got me thinking - in the original ticket for the feature, I recommended thread + blocking API calls-based implementation, but it was later implemented by manually implementing the DNS protocol, resulting in about 2500-line code. Now I'm thinking maybe my original choice was still the right one - simpler implementation (I predict under 500 lines), meaning better maintainable, and has the benefit of using OS DNS cache transparently. So I'm thinking about a complete rewrite of the resolver subsystem in hnbase library. There hasn't been a single subsystem in core that hasn't been rewritten at least once, and rewriting/refactoring is a normal part of software-engineering process, so... "Prepare to throw one away; you will, anyhow." - Eric. S. Raymond.

Madcat, ZzZz


Wednesday, December 21, 2005

CGComm thoughts

Few days ago chemical requested a small app that would display the contents of hydranode internal metadatabase (info, e.g. meta data about files, such as size, hashes, known names etc), so I whopped up a small up in 2 hrs that displays the contents. The code is here, if anyone is interested. Basically it just links against hnbase/hncore libs, has the hncore library load the metadatabase, and then converts the contents into GUI objects; pretty straight-forward.

However, that got me thinking. In the hncgcomm library implementation, I was strictly avoiding the usage of Boost.Signals library, since the namespace name signals broke with QT's pre-defined name signals. And the lack of Boost.Signals made hncgcomm library really boring, and inconvenient to use. However, there are actually workarounds for this issue both on QT (4.1 beta) and Boost.Signals library. And using Boost.Signals would allow us to use Boost.Lambda unnamed functors, which can result in rather interesting code, like this:
data->onUpdated.connect(
boost::bind(&DownloadListItem::onUpdated, this)
);
data->onDeleted.connect(
boost::lambda::bind(boost::lambda::delete_ptr(), this)
);
m_downloadList->onAdded.connect(
boost::lambda::bind(
boost::lambda::new_ptr(), m_ui->allList, __1
)
);
The first two lines could go to a GUI download list item, first of which enables automatic display updating as soon as the hncgcomm library reports updates, second removes element from the list when the hncgcomm object is destroyed (download removed). The last line creates an unnamed lambda functor which constructs new DownloadListItem object, passing m_ui->allList parameter (the parent list for the object) as argument, so whenever a new download is detected by hncgcomm library, a GUI object is automatically created. This results in a LOT more interesting and compact code.

Anyway, the GUI topic became bit more interesting with this finding, so I'v been fiddling around with QT and our existing gui-related code pieces (quite a bit of those - thousands of lines of code across various places in the code tree), and testing some of the concepts that have been designed / developed over the past months (the GUI design has been an active topic for a long time already behind the scenes). Don't have anything that I can show you yet, but I might check in some hncgcomm updates in the next few days. I'v got few new ideas for the protocol improvement, such as allowing GUI to select which fields of bigger objects it's interested in (as optimization), and variable-rate updating (a constant update-rate gives bit strange feeling in the GUI, so I'm thinking variable-rate, say, 500ms to 1500ms, would create more smooth user experience.

Madcat, ZzZz


Monday, December 19, 2005

Small changes

A new command has been added to shell, links <num>, which displays plugin-specific links to downloads (e.g. ed2k:// links, location to .torrent file and such). Original idea and feature request by chemical.

Some settings locations have been changed / renamed as well. MailNotify was using "MailNotify" section (capital letters), but the current standard is all lower-cased, so that was changed to conform with the standard. HNShell module wasn't following the standard location for it's settings either, it's settings were stored at top-level, and with long-and-ugly names. Now HNShell settings are stored in [hnsh] section, and have been renamed somewhat to be easier to change / remember.

Shell Command Reference is now somewhat more complete, with few added commands and some rephrasing of descriptions here and there. Also, there's a new page - Configuring Hydranode - in the devcenter, which lists all config.ini settings along with descriptions.

I also dropped the second UDP port used in ed2k module - at hardcoded TCP + 3 port - which was used for server queries. Now both client and server queries are handled by the same UDP port (4673 by default), and it's no longer needed to forward two UDP ports from router / firewall.

New builds for Windows and Linux have been uploaded.

Madcat, ZzZz


Friday, December 16, 2005

Strangenessness

Been a day full of really wierd things, none of which make any sense. Chemical has been complaining that Hydranode does extremely bad upload performance recently, like constantly failing to start uploads, or uploading only few hundred kbytes and then stopping and so on. Naturally, I took a deep look at the trace output, and looked at code, but couldn't determine any possible cause for this behaviour, and I was unable to reproduce it either. Even went as far as to test and run his Hydranode build on his box, and it worked perfectly when I was testing it. Oddness.

Another thing that's bothering me is that iptraf (a linux network traffic monitoring tool) actually reports about 10kb/s higher transfer rates (both for upstream and downstream) than Hydranode internally. At 10kb/s upload, 80kb/s download according to Hydranode, we seem to be doing 24kb/s upload and 95kb/s download rates. Naturally my first thought was that something else is using bandwidth, but as soon as I stopped Hydranode, traffic dropped to 1-2kb/s, so nothing else is using the traffic for sure. I tried to determine if our speed-o-meters are mis-calculating, but everything looks correct there as well. Even the 10s averages, which are calculated independantly from speed-o-meters agree with the speed-o-meters, but not with iptraf. Strangeness.

As for bounty/goal-based donations, as many have requested, I'll think about it. In general, I think it's a good idea (and has been under discussion before), but the actual implementation might be bit trickier. For now, I think it's sufficient to just drop me a mail or do a forum post a'la "Fix this bug, offering $xxx". Other people could also support the same bounty, adding offers. Actual money transfer happens when the bounty is completed, to whoever completes it. This will be based on the assumption that the bounty creator is a honest person and actually intends to keep the offer/promise. This also results in automatic "trust" system - someone who sets a fake bounty once won't be taken seriously the next time, so there's not much point in making false bounties. It's not that I'd think any of you around here would do something like that, but there's an evil world out there... Also those who have a reputation of setting and keeping bounties will most likely be taken seriously and bounties completed faster.

Anyway, been feeling kinda half-asleep the entire day (really crappy weather outside is probably the cause); did numerous attempts at various features/optimizations - automatic inter-module dependancy tracking (for example if BT module requires HTTP module, BT module loading would automatically trigger http module being loaded - similar to how Linux kernel does it), ed2k parser optimizations (lowering the amount of exceptions thrown); however for some reason, all those got screwed up around half-way when I realized I went completely wrong with the code - good idea but the implementation just sucked. So I guess I should get some rest instead.

Madcat, ZzZz


Thursday, December 15, 2005

Donations via Moneybrookers, and misc fixes

While I'm looking for options to make Paypal donations work, I set up an intermediate solution via Moneybrookers, which works in more countries than Paypal. Their website is at https://www.moneybrookers.com, where you need to register (if not already registered). To send me money, just use the email address shown on the right side of this blog post.

On code side, I did some profiling, but didn't turn up anything useful. There's some rare case where BT module starts maxing out CPU usage, which I was hoping to catch within the profiler to determine the cause, but, as was expected, the issue never happened when running in profiler. Still, the following miscellaneous things got fixed today:
On GUI design note, another few pages of sketches were generated during a brainstorming session with our designer, and I believe we have a pretty solid layout finished for search, transfer and media library pages. Initial drafts were also done for Home / Statistics pages, but currently there are more questions than answers on those pages.

Madcat, ZzZz


Tuesday, December 13, 2005

Shell usability fixes

Wow. I figured my last post was discussion-provoking, but I didn't expect THAT many responses. Thanks for all the pep-talk, it helps a lot to know how many people out there are interested in this project and it's success. During the cold and dark winter months, especially in December, it's easy to become depressed and start to question whether what one's doing is actually worth doing it. Facing the risk of having net/power connections being cut at any moment due to unpaid bills doesn't help much either. Still, such feedback gives some will and strength to continue, even when all else fails. So once again, thanks.

So, to address the main concern of the previous post, namely hnshell usability issue, I introduced the following syntax for calling module-specific commands: <module> <command> <args>. Thus, to list ed2k servers, you can now type ed2k list, to connect to a specific server, use ed2k co Raz or ed2k connect 195. Current status can also be seen via ed2k stat command. Thus:
madcat@hnsh:/$ ed2k l
[ 62.241.53.2:4242] 62.241.53.2:4242 Users: 444599 Files: 68192278
[ 64.34.162.148:5661] 64.34.162.148:5661 Users: 464923 Files: 36730543
[ 64.34.165.203:5306] Byte Devils Users: 532885 Files: 42099162
[ 195.245.244.243:4661] Razorback 2.0 Users: 1149180 Files: 164707448
madcat@hnsh/$ ed2k co Raz
eDonkey2000: Connecting to 195.245.244.243:4661
madcat@hnsh/$ ed2k s
eDonkey2000 network: 21648098 users online, sharing 2336502348 files.
Connected to server: [195.245.244.243:4661] Razorback 2.0 with High ID (3357049682)
1149210 users and 164961582 files online on current server.
madcat@hnsh/$ q
Connection closed by foreign host.
This system is internally still based on the object hierarchy (and the old commands still work as before), but I added some wrapper-functions on top of it to get the better syntax. For example, ftp module (if it was finished) could be used with the following commands:
$ ftp co user:password@hydranode.com
$ ftp list
$ ftp get latest-binary.exe
$ ftp disc
All sub-commands, as well as the module-name, can be abbreviated as long as there is no ambiguity. Hence ed c R and ed s also work (the latter is short-hand for "ed2k stat").

Core library also used that context-sensitive stuff for one thing - sharing folders, so now I added a separate 'share' command to replace that. The syntax is same as the former 'adddir' command in files/ subdir, e.g. share <dir> (-r).

As a side-note, commands 'ed2k' and 'share' introduced ambiguity with one-character shorthands of 'exit' and 'shutdown' commands. While with the former, you can use the 'q' shorthand (of 'quit'), 'shutdown' command now requires at least 'sh' being entered. Can't have the cake and eat it too :)

Madcat, ZzZz


Sunday, December 11, 2005

A lot of useless ideas?

A few days ago, chemical pointed out to me (again) that the context-sensitive commands in hnshell sounded like a good idea at first, but is completely unusable in reality. The fact that you have to type four commands in order to change a server is simply stupid. For those readers who actually haven't used hnshell and/or haven't figured out how this works, here's what you'd need to do to change a server:
$ cd modules
$ cd ed2k
$ cd serverlist
$ connect Razorback
Similar command sequence is required to view current server connection status. Yet initially, everyone was so excited about the innovative and cool idea of context-sensitive commands. Which brings up the point - many ideas sound cool and useful as ideas, but fail miserably in real world and/or have very limited usability, so affect only a very small amount of userbase. Unfortunately, Hydranode is full of such ideas. Basically Hydranode is a huge collection of such ideas, all stuffed together into a single application.

When big corporations start considering a feature or idea for a software, they spend a lot of money on research before starting any development, simply because it's a lot cheaper to find out an idea isn't worth the time/effort/money during research rather than spending millions on a feature that only 0.1% of userbase actually uses. Yet in open source world, this is not done, since the general direction of thinking is that software should include features for everybody. This is actually inherent from the fundamenatl idea of open source - everyone can submit patches - and developers rarely say no, resulting in feature/idea bloat.

Hydranode was started based on a lot of ideas - actually all of Hydranode's ideas were developed / designed during first 2-3 months into the project (summer 2004), since then no ideological changes have made, and very little, if any, new concepts have been introduced. Looking back, Hydranode was actually started because at the time, xMule (where I was maintainer at the time) was an awful codebase, lacked core/gui separation which was, again at the time, the "holy grail", and only supported one network. ShareDaemon project was created to address those concerns, and after the failure of that, Hydranode, based on the same ideas, and quite a lot of new ideas. Yet all those ideas are niche features which add very little value to the final software, while each one of them exponentially increases the development time.

Taking one by one - cooperative multi-network downloads is perhaps the most fundamental idea of Hydranode. Or, in a wider scale, multi-network downloads in general. Yet how many successful multinetwork applications do you know? Both mldonkey and shareaza have shown extremely bad network behaviour on all networks they support. It isn't because they such at coding or lack the skills - ShareAza code, for example, is very high quality. It's because the fundamental idea of multi-network client is flawed at it's roots. When you develop a single-network client, you spend all your effort to support that net, and it usually takes 4-6 months to become a considerably-good client for ONE network, about a year to win the market. In multi-network environment (leaving aside the added complexity of multi-network handling itself), the effort is automatically split between multiple tasks, leading inherently worse performance on all supported networks. It's not something the developers intend, but it's inevitable.

To add to that, the idea of true cooperative multinetwork downloads is also a very weak one. With the addition of BT module recently, Hydranode is capable (under certain conditions) of downloading files cooperatively from ed2k and BT. Yet the amount of torrents that actually have hashes for both networks, or even if there was a DHT backend which supplied those hashes, the actual use for such a feature is very limited - all networks are self-contained, and the fact that you can leech off two networks can actually be harmful for both networks being involved, while giving very little speed / reliability increase. Yet another useless feature that sounded good as idea but is useless in practice.

Another idea - core/gui separation. It has very little use for local usage (shutting down GUI and leaving core running - maybe some 10-15 ppl might find some use for it). And for remote usage, it only makes sense for people who actually have TWO computers at home, possibly one running some unix variant. Imagine how large amount of users actually have that? Considering that 5% of the userbase actually use linux, 0.5% use both linux and windows in a multi-computer environment... and the remaining 99.5% of users suffer from the added performance penalty of inter-process communication. It is also a major hit on overall development time (the cgcomm code isn't a trivial task), the added bugs (more code == more bugs), and as practice has shown, it means that it takes longer for the project to attract user/tester base due to lack of user interfaces in early/middle stages of the project. Such large penalties for a feature that a VERY small group needs.

So based on that, isn't Hydranode just a big bucket of ideas that have little or no value at all?

Madcat.


Wednesday, December 07, 2005

New features in Hydranode Shell

Developing experience on Windows is completely different from the development experience on Linux. On Linux, I feel a strong focus towards creating really tidy and nifty code, worry about how the code looks like and so on. Windows platform, however, changes the way I look at the app and development, and focus changes - towards features and usability improvements. Given my background of several years of Linux development, naturally I cannot drop code tidyness/niftyness levels, but my focus still is more towards features / usability rather than "more cool code". I don't really know what causes this (besides the environment change), but that's how it is.

Towards that end, I started adding some long overdue features that you, the users, have been missing longer than you should have. The first two of the feature set that I intend to continue over the next few days, are multi-selection of objects, and download destination directory overriding on download start.

Basically, the syntax is command 1-3,7,8,12, which will cause the command to affect objects 1, 2, 3, 7, 8, and 12. This support has been added to download, pause, stop, resume and cancel commands. Hence, you can now download search results #100-110 by simply typing do 100-110.

The other improvement that was introduced today - changing destination directory of downloads - is even simpler. Syntax: do 100 /data/video will start download from search-result #100, and once the download is completed, it will be placed in /data/video folder, instead of the default incoming directory. Naturally, you can also use do 10-15 /data/series to download all 6 of those results into the specified folder. Note that this feature is not currently available when starting downloads from links, and it's not possible to change destinations of existing downloads. These issues will be addressed over the next few days.

Pondering about our upload-slot-management issues - namely, I discovered that we fail to use up all bandwidth when upload limit is set higher than 25kb, which I never tested - I realized that we must change the upload-slot-opening code to take upload-limit into account when determining the default/recommended slot-count. We need that in any case, since in BT, we must also ensure about 4kb/s per slot. However, that cannot happen when upload speed limit is set to 0 / unlimited. After some thought, I realized that unlimited upload-limit is a really bad idea in any case, since vast majority of users are on DSL links, on which networking slows down to crawl when you max out your links uploadrate. The bottom line is that Hydranode now rejects unlimited upload-rate setting completely. If you really-really want to set unlimited upload limit, you can set an insanely high value, but then again, why would you want to set higher value than your link speed?

Madcat, ZzZz

PS: Builds from r2473 are now available. Main fix is that now the win32 builds should run on any modern Windows system - the previous ones were rather broken actually due to redistribution-related errors.


Tuesday, December 06, 2005

Boost 1.33.1 update, settings changes and new builds

Apparently, the change of config file parser to Spirit wasn't such a good idea after all. The new parser was broken, and completely broke config.ini (and I didn't notice it until testers started reporting it), so quite a few testers lost their settings files due to that. In the end, I decided to revert that change completely, since it caused more problems than it solved.

However, that breakage brought two issues to my attention that I had actually been aware of for a while, but was never motivated enough to fix. Namely, ed2k userhash, and modules statistics data, were also stored in config.ini, but it's completely wrong place for them. So now we have two new files - statistics.ini, which stores all statistics-related things, and ed2k/userhash.dat, which stores your ed2k user hash. Settings are copied automatically from the old locations, but the old values are also left in place (for backwards-compatibility reasons).

Wubbla (known for his work on the http module) came up with an interesting idea today - secure sockets. The idea is to provide SecSocket class, which transparently encrypts the connection session, to simplify implementation of things that use it (https for example). After some discussion we agreed upon the base design (wrapper class with forwarding functions around SocketClient class [pimpl]), using Botan crypto library. However, Botan library doesn't provide SSL itself, it just provides means to implement this. So it's estimated that it'll take at least several weeks to properly implement SSL - but we have a reference implementation, namely Ajisai library, which started to implement the same thing a while ago, but now seems to be dead project.

Early today morning, Boost version 1.33.1 was released, being a bugfix release. Hydranode's bundled Boost sources were updated, and Hydranode compilation now requires Boost 1.33.1 headers. I made a stripped-down version of the headers, available at hydranode.bytez.org in two formats - tar.bz2 and zip. SVN revisions 2459 and newer require those headers to be present during compilation. Of-course, you can use the headers from the official tarball as well (they'r identical).

The reason I was waiting so anxiosly for this upgrade was that Boost.Signals library had major issues when used across DLL boundaries, and Hydranode uses a LOT of dlls, and some of the base components (events) are implemented using Boost.Signals. This update should clear up a lot of wierd crashes when having multiple modules loaded.

New builds (r2461) are available now, built against Boost 1.33.1. Windows binaries are now built with MSVC 8.0 compiler. Note that this is the first time I'm publishing MSVC-compiled binaries, so let me know if the archive is missing some dll's, or similar (I need to bundle C and C++ runtime DLL's, since we cannot link against them statically due to the modular nature of Hydranode - same reason why I'm bundling libstdc++.so in Linux builds).

An interesting thing though - the very first MSVC build I made yesterday resulted in almost 20% smaller binaries than anything I'v managed to create after that. A complete mystery ... the first binaries bundle was 940kb, the r2461 snapshot is 1.2MB. Strangeness ...

Madcat, ZzZz


Monday, December 05, 2005

Further improving MSVC compatibility

Programmer (n): An organism that can turn caffeine into code.
- Unknown source

Another 15 hour dev-session, mainly focused still on improving MSVC compatibility. Support for MSVC 2005 Express Edition is now complete, altough the debug version of it's runtime library does rather aggressive checking, so assertion failures (inherent from Boost code) in debug builds still happen. One of the biggest surprises was code like this:
std::string tmp;
logMsg(boost::format("%s") % tmp);
This caused assertion, since Boost.Format library called std::string.append(0, 0), which, while technically valid operation (doesn't do anything), caused assertion by C++ runtime library. Boost people are aware of this, and hopefully a fix will be available at some point, but currently we have two options - patching boost headers, or disabling those checks completely. That doesn't seem to be an isolated incident either, additional assertion failures happened in different places, but I was unable to track those further yet.

Yesterday I also mentioned the problems with manifest files that MSVC 2005 introduced. What I found out is that this is part of Microsoft Fusion technology, something that will resolve the DLL hell problems once and for all (at least we'd like to think that). In any case, as I discovered, Boost.Build v2 (which we'r using) actually supports it, but it used it only for DLL's. However, as it turned out, it's required to merge a manifest into executables as well in the final versions of the compiler, so the build system was patched to do that.

Http module had some really strange problems when compiled with MSVC 8.0. Namely, it complained, during linking, about duplicate definition of Range::begin(), Range::end(), and Range::length() methods. Now, Range is a template class, with all-inline methods, defined in hnbase library, which made the issue even more odd. We spent practically two nights with wubbla scratching our heads at it, until tonight I managed to ask the right question in google, and stumbled upon this page. As it turns out, it's a rare linker bug (present since 2002 already), that triggers sometimes when a dll imports a class that's derived from a template class - in this case, it was probably LockedRange/UsedRange objects, which are derived from Range64. This was fixed by somewhat surprising code - explicitly import the template class, e.g.:
template class IMPORT Range;
I also installed and tested Hydranode using Intel C++ Compiler for Linux, which is free for open source developers. It detected two or three minor issues, mostly related to pointless comparisons of unsigned integers against zero (one of the checks could'v potentially caused problems). However, from the looks of it, intel linux compiler is more than twice slower in release build than GCC, so we won't be using that in long-term. Still, nice to know Hydranode now compiles cleanly with three different compilers (also shows that the Boost.Build system is indeed portable - there's almost zero compiler-specific things in the Jamfiles).

Enig123 discovered a rather critical issue with credits database handling in Hydranode. Namely, we never updated the lastSeen value of credits, which means eMule dropped ALL entries in our clients.met when it was loaded, since eMule cleans entries that are more than 5 months old. This was now fixed, the lastSeen value is set to current time if it's not set. Hydranode now also removes credits that are more than 5 months old (untested code - we'll see if it works in 5 months :)).

While debugging the assertion failures of MSVC 8.0, the first code that triggered them on startup was configuration file parser (due to a logTrace call in there); at the time I didn't know yet why it was causing it, so due to lack of better ideas, I rewrote the config.ini parser code to use Boost.Spirit parser engine, instead of the former hand-crafted string manipulation. This was actually something I had wanted to do for a long time, but never had time for it, so now it's finally done.

Madcat, ZzZz


Sunday, December 04, 2005

Bittorrent things and MSVC support

10 years it was - "Every piece of software evolves until it can read e-mail."
Today it is - "Every P2P client evolves until it can connect to IRC."
- Madcat

Seriously though - looking how many P2P clients out there have IRC support, how many of them actually have a decent IRC support? Or, while we'r at the topic, how many multi-network IM clients have IRC module, and how many of them actually work? Or, while we'r on the topic, how many BT clients are out there, and how many of them actually work?

I was hanging in #azureus channel today, getting some intel [sic] about Bittorrent protocol and such. In particular, I was looking for info on the 'key' parameter in tracker communication, which was apparently needed by some trackers. It's used to keep track of clients after IP changes, and works somewhat similarly to ed2k userhash, except that it should be re-generated every startup. This is now properly supported in Hydranode. Another thing that came up during discussion was that it's required by Bittorrent netiquette to cap download-rate at 2xuploadrate when upload speed limit is less than 5kb. This is quite similar to ed2k netiquette, so implementing this was trivial.

Yet another interesting thing that came up during discussion was that there are actually more bad BT clients out there than there are good ones. And of-course, it was mentioned that BT support in multi-network applications is notoriously bad - mldonkey and shareaza being examples of that. Ohwell, yet another challenge for Hydranode to prove - that it's actually possible to have good BT implementation in multi-net client. Personally, I think it's doable, but it won't be easy. We still have quite a bit of issues to be resolved before Hydranode's BT implementation can be considered anywhere near good.

I'v been playing around with various versions of MSVC past few days, attempting to restore full MSVC compatibility in Hydranode code. The actual problems were rather trivial, some missing DLLEXPORTS here and some overloaded functions that confused the crap out of MSVC; at the end of the day, I can conclude that we have full compatibility with MSVC.NET 2003, mostly-compatible with MSVC Toolkit 2003 (free commandline tools), and mostly-compatible with MSVC 2005. The problem with MSVC Toolkit is that it lacks some combinations of runtime libraries, namely multi-threaded debug dll, and some more, so we must use explicit runtime-link=static runtime-debugging=off when building with that. As for MSVC 2005, it has introduced DLL versioning, and enforces it quite aggressivly, but I don't know enough yet about the topic to be able to implement the support. Btw, MSVC 2005 is also doing rather aggressive security checking on the code, but that triggers on a lot of valid Hydranode and Boost code as well, and as Boost people generally recommend turning that stuff off, so I did.

There were several reasons for restoring MSVC compatibility. For one, windows people expect to use MSVC for plugin-development and such; secondly, MSVC generates, on average, 4 times smaller code than GCC, and compile times are considerably faster (even without using precompiled headers). The main issue with mingw-gcc is link times, linking hydranode dlls or modules (average 100mb object files and 70mb final dll in debug build) took about a minute with mingw-gcc, but 7 seconds with MSVC. On Linux, it also takes roughly 7 seconds to link that, so it's clearly a mingw-gcc-specific issue. Another thing that MSVC seems good about is release-compilation - 14 minutes, which is roughly same as in debug build; release builds on gcc/linux are well over 20 minutes. In any case, getting Hydranode installed size from 15.0MB to 4.4MB, and installer size from 3MB to 960kb is pretty nice. Memory usage in release build dropped about ten times, from 13MB to 1.3MB, and from the looks of it, it's no longer using any CPU at all. This is what you get for developing on a slow compiler/platform for 1.5 years, and then recompiling the thing with a "real" compiler :)

Madcat, ZzZz

Edit: The memory usage didn't drop ten times, but only about two times. Looking at task-manager while half-asleep isn't such a good idea after-all. Sorry for the confusion.


Thursday, December 01, 2005

Smart Data Structures - Revisited

You all know how much I love smart data structures [and dumb code], and how much I love using Boost MultiIndex library to solve problems. ClientManager and ChunkSelector are just a few of many places MultiIndex is being used in Hydranode. Tonight, I'v managed to take the concept even further, by re-structuring the ChunkSelector (which, as the name says, selects chunks for downloading) to be even dumber. Bear with me as I walk through the concept.

The first thing about ChunkSelector is to define what exectly do we want it to do. The logical ordering of things would be:
In addition to that, whatever the ChunkSelector chooses, must be compared against the peer's available chunks listing, since we cannot requset chunks the peer doesn't have.

Now, formerly this logic was split into two functions, and the container had three different indexes - one based on availability, one based on partial status, and one based on use-count. The first function would thus get the list of partial chunks (partial == true) from the first index, iterate on that, looking at usecounts and availabilities, selecting lowest use-count/availability combination. If that fails, the second function would then do a similar operation, this time walking on availability index (which, btw, also contained completed chunks), skip past completed chunks, and look at use-counts, selecting least-available chunk with usecount 0, or non-zero if no 0-usecount chunk was found. All this occupied actually three functions originally, over 200 lines of code, but was reduced to about 120 lines of code and two functions few weeks ago.

But that data structure logic was actually written about a year ago, when we were just starting out with MultiIndex, and weren't aware of additional features of the library. Now, with the knowledge I'v gathered during that time, I managed to come up with a better solution.

The first logical step is to add support for selecting all chunks that aren't completed:
boost::multi_index_container<
Chunk,
indexed_by<
ordered_non_unique<
composite_key<
Chunk,
member<Chunk, bool, &Chunk::m_complete>
>
>
>
> container;
// returns all chunks with m_complete=false
container.equal_range(boost::make_tuple(false));
Now, we also don't want to select any chunks that have zero availability, since they aren't available at all from any client. However, the m_avail member is an integer, so we need to convert that to bool, using bool Chunk::hasAvail() { return m_avail; } helper method. Thus, the container becomes:
boost::multi_index_container<
Chunk,
indexed_by<
ordered_non_unique<
composite_key<
Chunk,
member<Chunk, bool, &Chunk::m_complete>
const_mem_fun<Chunk, bool, &Chunk::hasAvail>
>
>
>
> container;
container.equal_range(boost::make_tuple(false, true));
Now that we have a listing of chunks that are incomplete, and available, we want to make sure that chunks with lower use-count are selected over chunks that are already being used (e.g. downloaded) by other peers. We also want to select partial chunks over not-at-all-downloaded chunks, AND we want to select rare chunks first, so the next three indexes logically become:
  member<Chunk, uint32_t, &Chunk::m_useCnt>,
member<Chunk, bool, &Chunk::m_partial>,
member<Chunk, bool, &Chunk::m_avail>
However, since the default ordering of STL (and thus MI) containers is std::less, it means false values are in the listing BEFORE true values, which is not what we want here. Thankfully, all STL containers (and thus also MI) allow overriding the comparison predicate, but with composite_key, we must then specify comparison predicates for ALL sub-keys. The final container thus becomes:
boost::multi_index_container<
Chunk,
indexed_by<
ordered_non_unique<
composite_key<
Chunk,
member<Chunk, bool, &Chunk::m_complete>,
const_mem_fun<Chunk, bool, &Chunk::hasAvail>,
member<Chunk, uint32_t, &Chunk::m_useCnt>,
member<Chunk, bool, &Chunk::m_partial>,
member<Chunk, uint32_t, &Chunk::m_avail>
>,
composite_key_compare<
std::less<bool>,
std::less<bool>,
std::less<uint32_t>,
std::greater<bool>,
std::less<uint32_t>
>
>
>
> container;
The result is that when we do:
std::pair ret = container.equal_range(boost::make_tuple(false, true));
We get a listing of chunks that match all required criteria (incomplete and nonzero availability), and they are already sorted by all predicates that we need:

First come partial chunks with zero usecount, ordered by availability. Then come impartial chunks with zero usecount, ordered by availability. Then come partial chunks with usecount 1, ordered by availability, then impartial chunks with usecount=1 and ordered by availability, and so on. And the old two functions totaling 120+ lines of code and doing very clumsy looping now simply becomes:
  CMSelectIndex &idx = m_chunks->get<ID_Selector>();
std::pair r = idx.equal_range(
boost::make_tuple(false, true)
);
for (SIter i = r.first; i != r.second; ++i) {
if (pred(*i)) {
UsedRangePtr ret(new UsedRange(this, i));
if (ret->getLock(1)) {
return ret;
}
}
}
return UsedRangePtr();
On average, that loop runs less than 10 times (on a file with 1500 chunks), very often the chunk is selected in first 1-2 hops through the loop. Dumb code and smart data structures DO work a lot better than the other way around :)

An interesting side-effect of this is that the very first and very last chunks are always selected first (all other factors being equal). I'm unsure why this happens; while I could explain the first chunk being selected by the additional positional index in the actual container, which places the chunks in an ordered fashion, I have no explanation why the very last chunk is selected as second choice. The final code is in hncore/partdata.cpp, if anyone wants to take a look.

Special thanks to Joaquín M López Muñoz for writing the wonderful library in the first place, and for the idea of using composite_key in the ChunkSelector :)

Madcat, ZzZz


Archives: December 2004 January 2005 February 2005 March 2005 April 2005 May 2005 June 2005 July 2005 August 2005 September 2005 October 2005 November 2005 December 2005 January 2006 February 2006 March 2006 April 2006 May 2006 June 2006 July 2006 August 2006 September 2006 Current Posts

This page is powered by Blogger. Isn't yours?