Alo Sarv
lead developer

Donate via
MoneyBookers

Latest Builds

version 0.3
tar.gz tar.bz2
Boost 1.33.1 Headers
MLDonkey Downloads Import Module Development
Payment completed
Development in progress.
Developer's Diary
irc.hydranode.com/#hydranode

Monday, January 31, 2005

Finalizing and regress-testing new Hasher

Caught cold yesterday (stupid winter), so been rather ill (explaining last night's missing blog post), however, today things started looking better, so got some active development done.

The major topic of today was the new hasher - yes, the 130-line one. After lot of testing, fixing and more testing, the thing seems stable now. Had some trouble with getting all the locks properly handled, even found a bug in main event loop which caused some events handling to be delayed until next event was submitted - something that made little difference in main app runtime, but broke my hasher test-app.

After adding some statistics code (again) to hasher, I realized that my original idea of sleeping + reading was still fundamentally flawed - the performance drop from sleeping is significent, even on 2.6 kernel systems - 5MB/s vs 15MB/s. So, now hasher will be running at max speed at all times, and we'll just have to patch Boost.Thread lib at some point to get thread priorities support (which it doesn't support right now, which in turn was the reason why I wanted to test out the sleeping system in the first place). However, it's highly unlikely I'll get around to do that in near future - still have work to do on PartData and friends.

Another interesting question is wtf to do with enums. See - I'v been using enums for events since the very beginning, because they are simply most convenient data types for that. Example:

enum MyEvent { EVT_DESTROY = 0; };
void onEvent(Source *src, MyEvent evt);


However, the enums come with one major problem - enum's can't be forward declared. Yes, I'm aware MSVC, as well as several other compiles support it, however, ISO C++ standard says they cannot be forward declared - simply some compilers implement it anyway as an extension. GCC, however, being most standard-compliant compiler around the block, does not implement it as an extension, so using enums for events creates a damned big header inter-dependancy problem. This only affects compile-time, but it's still not very nice to need to include tons of crap just because enums can't be forward-declared :(

Madcat, ZzZz


Saturday, January 29, 2005

Finalizing PartData, upgrading hasher

Contrary to last nights blog post, namely regarding UsedRange concept's dropping, I realized I still can't drop it out completely, since I can't really track UseCount on chunks w/o it, and alternative solutions to do so proved causing even more problems. So, UsedRange is here to stay, at least with our current knowledge base I see now way around it. Anyway, it's not very big syntactical burden.

I got the locking and writing system somewhat operation, albeit it's not tested at all yet - was about to go there, but then realized I needed to upgrade hasher code too.

Hasher code used it's internal threading code, and owned a separate thread of its own. However, since the recent addition of WorkThread API to the codebase, it should start using that one instead. So basically I just threw away the 800-line hasher.cpp and modified HashWork object somewhat to also perform the work in question. The result: new hasher.cpp is 139 lines of code, and technically capable of doing everything the old one was (again, untested code).

This seems to be the trend lately - during rewrites, things get lot more compact, robust and simpler. I just happened to compare my current local code tree (on which I'v been working for about a week or so already) to what I have in CVS, and to my surprise, realized that my local source tree is 3500 lines of code SMALLER than what's in CVS. That's 3500 lines of code simply thrown away - current codebase (excluding testsuite) is 26'000 sloc, while the one in CVS is 29'500+. So - as authorities have said - "If your code is becoming both simpler and smaller, you'r on the right track" - seems it's a good sign. From what I can see right now, I see at least 1000-2000-line more code that can be rewritten into much more compact, cleaner and faster form (MetaDb/MetaData, Object, and DynObj Subsystem come to mind).

Madcat, ZzZz


Friday, January 28, 2005

UsedRange? Why?!?!

Could someone please tell me what purpose does UsedRange have at all in PartData API ? From what I understand, it's merely additional noise to solve our own internal problems, but gives nothing but trouble to users of the API. Those that aren't quite following what I'm talking about:

PartData *pd;
UsedRange *ur = pd->getRange(partsize);
LockedRange *lr = ur->getLock(size);
lr->write(begin, data);
lr = ur->getLock(size);
lr->write(begin, data);

What purpose does UsedRange have here ? None - UsedRange must still forward the getLock() calls to PartData, since those locks must be shared between multiple UsedRanges, ... why don't we just request locks from PartData directly? Indeed - it would greatly simplify the API ... bye-bye UsedRange.

Just means the internals of PartData just got somewhat more complex. But - better have complex internals, and easy API, instead of vice versa.

Madcat, ZzZz



Thursday, January 27, 2005

More design @ PartData

Can't say I'v made much progress tonight on the PartData stuff, but I am beginning to understand the problems more clearly. For one thing, I realized I need to visualize the internal PartData systems to get a better overview of how things are supposed to run - resulting in this UML diagram. It's still only work-in-progress though ..

While the getRange() concept (e.g. SmartChunkSelector(tm)) is working, the ranges locking mechanism is still somewhat gray area. I originally intended to have the locking handled within UsedRange, and later in Chunk (after I realized I need to allow multiple UsedRange objects per Chunk - originally planned to use shared-ownership single-UsedRange object), however, I realized it'd break as soon as multi-net downloads with different chunksizes are involved, so the lock handling needs to be done in PartData object instead.

How exactly should that be implemented is not clear yet though - basically we'r back at the multi-rangelist-lookup issue that I was trying to avoid during this design (it was the central part of original PartData design), since it's error-prone...

After the locking system, we still need internal data buffers, chunks verifying (easy), and corruption recovery (tricky). There are some interesting aspects in corruption recovery that are somewhat tricky to handle - namely, when we have a checksum fail for a range, we must check if we have higher-resolution hashes for the data, and check those ... after that, we can mark the rest of the range as corrupt, and cry out for help (post notification event which plugins can handle). Upon that, plugins-to-the-rescue (if capable) - submit even higher-res hashes for checking (ed2k AICH for example). So that'll also take some time to figure out.

What's good news tho is that PartData (along with some support code around hasher) is the last of the engine subsystem upgrades series, which started out mid-december. So, once this is done, we can get back to, say, more "real" problems, like network communications and so on.

ETA for PartData completition? 4-5 days ? Smth like that, if all goes well.

Madcat, ZzZz


Wednesday, January 26, 2005

Moving CVS, hnshell updates

First part of the night went to various stuff regarding hnshell with Hell_Fire - namely, now char-mode telnet is enabled automagically, which will (hopefully) soon lead to tab-completition support in hnshell, as well as other nifty features.

Other part of the night was spent on moving CVS to our dedicated hydranode server - due to berlios downtimes, it seems better to run CVS on our own server. Most stuff seems working, e.g. mail notifications (currently still using the same berlios mailing list), CIA-bot updates and developer access, but anonymous access, statistics, webcvs and so on still need some work. Hopefully it won't take more than few days for the rest of the transfer stuff.

Madcat, ZzZz


Tuesday, January 25, 2005

Implementing HydraNode SmartChunkSelector(tm)

As discussed last night, the current topic at development is HydraNode SmartChunkSelector(tm), which needs to select the most important chunk for downloading. One idea for implementation was suggested by SimonMoon - to introduce some kind of scores/ratings system for chunks.

It would mean, for each ChunkSet, we'd have to calculate a score, tweak those scores until they do what we want, keep them updated etc. The development overhead of that just doesn't seem to justify it. Besides, in this approach, we'd have to convince the scores system of what we want to achieve, instead of implementing the neccesery algorithms ourselves, using objects. (Score-based approach would fit well into a non-object-oriented environment, but C++ is OO language, and thus things should be done ... in OO way).

As such, here's my idea of ChunkSelector, based on Boost.MultiIndex.
include/hn/partdata.h
src/partdata.cpp
The base idea is that we construct a ChunkMap, which is indexed by a number of variables we'r interested in - partial/verified status, availability, usecount etc. Using that, we can now perform the neccesery lookups, in the order we prefer, thus have complete control over the situation and tweaking it is much easier than with some score system.

An interesting find during building this was that I realized I can have chunks of different sizes (let's name them ChunkSets) in a ChunkMap - it is NOT limited to a single ChunkSize. As such, for example, BT plugin submits ChunkSet with SHA-1 hashes and ChunkSize=2MB, while ED2K submits ChunkSet with 9500KB ChunkSize and MD4 hashes, and ChunkMap handles all those chunks uniformly. When a chunk is being requested, the chunk is chosen from all ChunkSets, taking all chunks availability/partial/verified status into account.

At this point I realized that the requested chunk size (as passed to getRange() method), just become irrelevant, since a chunk is given out as one of the existing chunks, or a sub-range of that, leaving little control to the requester over the size of the chunk. While I initially started to get worried where it'd lead, I tend to think it's ok - at least from ed2k point-of-view, I see no problems in having PartData return smaller chunks, since those chunks are merely "Used", actual protocol stuff still works on smaller chunks.

Initial testing on the system shows it's working, albeit needs more work - namely, chunk multi-usage (which seems broken atm), and some tweaking here and there.

On other note, the compile time just skyrocketed - takes 18s to compile partdata as is right now, and only the header already adds 10s compile time on my 2.4ghz P4 system. Boost.MultiIndex uses Boost.MPL (template metaprogramming framework) heavily, hence the compile time. But adding 10s compile time to each file including partdata.h isn't my idea of fast development, so I guess I'll have to introduce Bridge pattern to PartData soon to separate the implementation and reduce the compile time significently. My first attempt to do that failed tho (god bless Kate's unlimited undo steps), one thing led to another and almost broke the entire thing. However, sooner or later it'll need to be done to keep compile times at reasonable speeds.

Anyway, ppl are waking up, so I better get to sleep - can't work with ppl trying to talk to me all the time :o

Madcat, ZzZz


Monday, January 24, 2005

Designing HydraNode SmartChunkSelector(tm)

New PartData API design is slowly, but surely evolving. An updated version can be seen here. Most notable changes are also making LockRange an object, and bringing in HashRange (which will be the base of chunkhash maps implementation). As can be seen, the new PartData API is moving more towards nice object-oriented approach, compared to the old, rather clumsy mess.

From the implementation side, adding chunk masks works, and now I'm trying to figure out how should chunkselection done. We basically have a numbe of criterias to be considered when choosing a chunk (most likely in this order):
While the second and third are relativly easy to handle (the least-available-chunk selection system was designed a few days ago already), the first and last are rather complex to handle, so need more time to figure those out.

For those sitting around, watching and wondering "why is he wasting time on stuff like this when he should get ed2k downloading working instead?" - well, ed2k downloading has been working for quite some time - but the ranges get too fragmented near the end of the file, so the entire thing gets confused and it's hard to complete downloads. Hence the need for "complete any partial chunks" for example. As for the least-available chunk selecting, it should enhance the overall download handling significently, and also benefit the networks. So - worth spending time on.

Madcat, ZzZz


Sunday, January 23, 2005

Working on new PartData

As is known, the new PartData API is the biggest/hardest part of current short-term roadmap, and it is also the only reason downloading with hydranode is well... less than satisfactory performance. There are a bunch of problems the new PartData API needs to address, so it's taking a while to figure out all the details.

One possible interface candidate can be seen here. It addresses the following problems:
One current problem in the new API is the issue of keeping back references to given-out used ranges. As seen from the above link, the current idea is to use boost::weak_ptr, which is a weak reference to boost::shared_ptr. This way we do not interfere with the automatic destruction of UsedRange objects when they are no longer referenced to by anyone, while still having access to the data (the weak_ptr's expire when the corresponding shared_ptr is destroyed). However, this most likely needs a specialization of RangeList template which handles it properly. Only part of RangeList API needs to be specialized - namely, merge() and erase() methods will be omitted, since those cannot be handled really when dealing with pointer types.

Madcat, ZzZz


Friday, January 21, 2005

Goodbye Callback lib, hello fast ipfilter loading

Finally I got rid of the Callback functors library, which was the original base of Event subsystem, and used also in Log and DSH (Data Structures Hierarchy, otherwise known as Object) systems. 1500 lines less code to be parsed during each file compilation :) The Log subsystem was changed to use Boost.Signals library, The Object was changed to use Event subsystem for it's work.

Moving on, IpFilter can now also load emule-format ipfilter.dat file (format is detected automagically). Due to some syntactical differences in ipfilter.dat format, inet_addr() was incapable of converting the addresses found in there (it has leading zeros e.g. 004.063.128.006, which inet_addr() considers as octal numbers). So, I decided it's great time to bring out the big guns - Boost.Spirit. 'lo and behold - IpFilter parsing times just got reduced 50% - now I can load ipfilters (both formats) in 0.17s (in release build). Boost.Spirit seems to be highly optimizable, since in debug build, the load times were nearly 10 times slower. However, what matters is release build anyway, so ...

In other news, I finally made a breakthrough on new PartData engine - namely the ChooseRarestIncompleteChunk concept which had been bothering me for ages. Now I have a basic design and proof-of-concept on how it should work, which is a big step towards actually implementing the new PartData we'v all been waiting for. More on this in the coming days/weeks.

On other news... er wait, I said that already. *sighs*. Well, anyway, Boost.ProgramOptions library doesn't seem to quite cut it - at least I can't find any reason why we should drop my current Config/Prefs engine in favour of that one, so guess that's one item off my short-term roadmap.

Madcat, ZzZz



Thursday, January 20, 2005

RangeList optimized, IpFilter engine implemented and tested

Optimizing RangeList logic was, as expected, the most time-consuming operation today. At the end, the key was to modify the sorting algorithm for the underlying rbtree to use the center-point of a range for comparing, e.g. (begin+end)/2. Based on that, I was able to optimize all of the remaining RangeList operations to take logarithmic time instead of former linear time.

Based on the new RangeList, I quickly threw together IPFilter API, as described in previous blog post. Currently, it's capable of loading only mldonkey format ipfilter file, I ran into some odd problems with parsing/loading emule-styled ipfilter.dat right now - hoping to get those fixed tomorrow.

Speed testing (46306 ranges)
Load time: 0.46s (0.33s in optimized build)
isAllowed() method lookup time: estimated 0.00005s per call
100'000 calls take 0.24s (0.17s in optimized build)
Memory usage: 1720 kb for 46306 entries

Madcat, ZzZz



Wednesday, January 19, 2005

Misc fixes, designing ipfilter engine

The comment on previos blog entry raised some thoughts. While ipfilter engine was planned to be implemented some time in the mysterious future, I realized that we can implement it here and now - we have all the pre-requisites in place, and I don't see any obstacles to it right now. While it's not in current development roadmap, it shouldn't take long to implement it, so we can do it right away and be done with it.

The important thing about it is keeping it generic and extendible. It should be pluginnable to the existing Scheduler API, which simply asks currently loaded ipfilter object to allow (or deny) request to a specific IP address. As such, it would keep both the Scheduler and the IpFilter objects separated, which is a Good Thing. What I'v been pondering about is the exact design of the ipfilter engine.

One idea is this. We create an abstract base class called IpFilterBase, which declares single pure virtual function - bool isAllowed(uint32). That function shall return true if the given ip address is allowed to be connected to (or incoming connection accepted from that ip), false otherwise. Derived classes would then override that method and implement it.

The actual IpFilter implementation would rely on Range API v2, which needs some lookup optimizations before it can provide the speeds we need there.

Using this design, it's possible for any1 to later write a custom ipfilter plugin to override whatever I'm providing by default, simply by deriving a custom class from IpFilterBase, and replacing the IpFilter object in Scheduler with the new one.

My only concern right now is that using the aforementioned virtual function mechanism might become a performance bottleneck (virtual function calls have slight runtime overhead compared to normal functions), because that function shall be called during EVERY incoming and outgoing connection. However, I tend to think this goes into the microoptimizations area, and eventually there are most likely other areas to be optimized instead of some virtual function call.

Hopefully I can get this thing implemented in a few days - I'd say the only hard part is optimizing RangeList lookup - currently it provides only linear complexity lookups. However, I already have few thoughts on how to do this, more on this on tomorrows blog post.

Madcat, ZzZz


Tuesday, January 18, 2005

WorkThread API, hydranode log analyzer

Implemented the new generic WorkThread API, which can be used to submit any jobs to secondary thread for processing. The main reason for this API is to localize multi-threading, and - more importantly - serialize disk I/O. For example, when hashing a file and starting to move some other file to incoming dir at same time (in case temp/incoming are on different disks), this would mean duplicate disk load, which slows everything signficently. Also, any arbitary long-running tasks can now be performed in secondary thread, with API open to modules for usage also. The API needs some more touches, but generally should stay as is. The API is implemented in workthread.h / workthread.cpp, and corresponding regress-test is located at tests/test-workthread directory. (Sorry for no links, too tired to re-generate doxygen docs and upload 'em right now).

On other news, I was wondering why we'r losing a lot of sources from ed2k. The main problem in investigating this is that there isn't any useful UI, and it's hard to track where/why sources are dropped, so eventually I had to resort to writing a small shell script to parse hydranode.log (which quickly gets to several MBs of data) and gather some statistics. Here's output from one of my test runs (run the script in same dir as hydranode.log (default ~/.hydranode)):
525 sources received from server(s).
Connection established to 200 HighID clients, dropped 59 (22%)
Connection established to 229 LowID clients, dropped 37 (13%)
Sent StartUploadReq to 352 (67%) sources
Received queue ranking from 313 sources (59%)
Received AcceptUpload from 23 sources (4%)
Received NoFile from 24 sources (4%)
173 (32%) sources lost before sending StartUploadReq
As seen, 32% of sources are still dropped. 22% of direct connection attempts failed (sources gone offline/changed IP?), and 13% of lowid callbacks failed (same reason?). As you might notice, out of 352 requests we got 313 QR's + 23 accepts + 24 nofiles = 360 -> the reason is that later during download process from those sources, they re-sent AcceptUploadReq, which causes this anomaly in statistics.

In any case - <10% CPU usage while downloading from 20 sources / 60kb/s in full debug/trace build - could use some more optimizations later on, but generally doesn't sound too bad. Now if I could only get the new PartData API going, we'd be set :)

Madcat, ZzZz


Monday, January 17, 2005

Red Elephant

*sigh* Exporting HTML from MS Word document isn't very good idea... had to rewrite the short-term roadmap by hand in html to be able to edit it on linux. Anyway, it should be more overviewable now.

Speaking of roads and maps ... well, actually, not speaking (nor even writing) about them - did some regress-testing on networking handling - 3 wide-spread files with hundreds sources. Found a bunch of bugs in Socket* <-> SocketWatcher relationships, which caused abnormal program termination (so not to say crashes :P). Few hours later, the networking handling seems stable - at least I couldn't make it crash in there anymore. There were several issues and race conditions in there, that should be fixed now.

Note to hydranode-cvs list subscribers - due to berlios downtime, some message(s) got lost, namely one of the sockets fix updates. We'r considering alternatives to berlios CVS, having 6+h entire site downtime (everything from web to cvs on berlios was down, only thing up was pserver cvs) isn't very nice. While SourceForge is slow, it's at least up more, and even if CVS goes down, only pserver goes down, not dev access.

Madcat, ZzZz

PS: Thanks for the DevBlogTitleGenerator(tm) - it's working great :)




Saturday, January 15, 2005

Zzzzzz

Been a long, but not very interesting day + night. Sadly, due to several distractions by real life, development didn't go as actively as I had planned, but nonetheless, here's what I managed to get done:
Madcat, ZzZz

PS: If any1 happens to know a good dev-blog-title-generator, let me know :P


Thursday, January 13, 2005

Some updates

Well, got my systems up and running again, so got some development done too. Nothing big and fancy yet tho - added minimal module example to aid future module writers, and enabled -pedantic compile flag, which enforces strict ISO C++ standard compliance from all code. Interesting thing about the latter is that it also detects (and gives errors) on extronous semicolons (;) in places where they shouldn't be (e.g. after function definitions). Generally you don't put semicolons there, e.g.

void myfunc() {
...
};

However, since MSVC parser couldn't handle function-level try .. catch blocks properly and gave parse errors in some cases such as this:

void myfunc() try {
...
} catch (myerr&) {
...
} catch (...) {
..
}

I had to add ; to the last line to keep the parser happy. However, ISO C++ requires no ; there. Ohwell, guess I'll have to implement MSVC_ONLY() macro to compile code only on MSVC in such cases (writing #ifdefs at all such places is tiresome).

On other news, as you might notice, there's google ads on the right side of this blog. While there aren't many visitors right now on the blog, it's still worth a try. Since this is a development blog, adsense will most likely pick up software development-related ads, which should be rather interesting (personally I already found one interesting piece of software through that ad - some MSVC extension called IncrediBuild, that seems interesting and worth a trial run). The money coming from clicks on those ads will be used for additional hardware for better/faster development.

Madcat, ZzZz



Monday, January 10, 2005

Delays, delays delays :(

Just figured I'll drop in an explain why there hasn't been much updates past days. Thing is I had set my system up before xmas as purely windows based, with NTFS partitions and all ... and now I'm trying to convert all the stuff to dual-boot win/lin system, which means conversions and linux setups and all ... Bah. This is what you get for having a single box for everything :(

Due to the storms around here, there's been lot of power failures and net outages (30% of estonia was w/o electiricty for a while), so I delayed those partition conversions (losing data because power failure during conversion wouldn't be very nice) ... So anyway, things are stabilizing around here, so I really hope I can get my systems set up properly in a few days tops.

Sorry for all these delays :(

Madcat.



Saturday, January 08, 2005

OSX port improvements

Well, not much to say - spent some quality time behind my iMac, here's the list of things that got fixed related to HydraNode on Mac OS X:
HydraNode should compile and run out-of-the-box on darwin now. There are still some subtle bugs left (server.met saving was one I noticed), but the complex issues on OSX port are resolved now.

Madcat, ZzZz


Friday, January 07, 2005

Considering development directions

The reason why I haven't delved into full-speed coding yet after the vacation is that I'v been trying to figure out which development direction to take. There has been only trivial updates in CVS - x86-64 support (thanks to sca for testing), precompiled headers support on MSVC, and minor OSX improvements (no, it still crashes on startup), but nothing major.

The thing is, it feels the existing directions (e.g. develop the engine until it works, then attach UI) isn't really working anymore, since there is too much information needed to be monitored for simple trace output to work out. One idea to solve that was the Launcher application (that would need to be written sooner or later), however, that would solve only part of the problem. Yet another way of doing it would be to write the Launcher application in a very generic manner, so that it would be capable of displaying almost any kind of data (not just log output) - have it display lists, settings and so on for example. It's not very hard to design it in a generic manner, since wxWidgets allows very simple runtime UI generation (as opposed to many toolkits mainly relying on pre-built dialogs). But I'm still not 100% certain whether it's the right road to take or not.

What I'd like to avoid regarding the Launcher application is that it would be considered as full user interface - that's not the topic I want to go here. The idea of Launcher application is to provide a shell for the engine to be run in in release versions, and in development versions, also aid the engine developer. As such, it doesn't have to be very user-friendly, but instead must be a good developer aid, exposing the internals of the engine to the developer.

The good side of this approach is also the fact that the Launcher would interact with the engine using Core/GUI comm protocol, which gives us an early testing platform / overview of the protocol and the errors in the protocol design (which are much easier to fix in case only the Launcher depends on the protocol, but would be much more complex when we built a full UI on top of it).

Madcat, ZzZz

PS: If you have experience in OSX C/C++ plugin development, contact me - there are problems regarding module linkage resolving symbols in bundle_loader - Lazy Singleton pattern symbols somehow resolve to 0x0. Probably I'm exporting (or not exporting) the symbols from the bundle_loader properly ...


Thursday, January 06, 2005

Ahh, the wonderful microsoft compiler

You'v heard me cursing the microsoft compiler for quite some time now, and have probably thought "bah, what does this guy know. MSVC is a wonderful compiler. I love it." Well, let me tell you a story.

On Dec 17th, 2004, your favourite kitty developer deployed new event handling subsystem, which was based on Boost.Function and Boost.Signals libraries, being significently cleaner and more flexible. It only had one problem - it caused access violation on startup on win32/MSVC.

Madcat spent two days debugging it, with little or no progress, and finally got so frustrated, and so tired that decided to take a longer break. The following two weeks, Madcat could not help but remember the seemingly unfixable problem that he had no leads to track on or nothing. Returning to development on Jan 3rd, 2005, after some initial comeback things had been done, Madcat returned to the problem, and spent another two long sleepless nights debugging it, using various tools and utilities, with still no success.

Until, on the 16th hour of the second long development session after the vacation, suddenly he discovered the problem. Here's the details:

In function Scheduler::addSocket():

boost::signals::connection c = Impl::getEventTable().addHandler(
s, &EventHandler::template onEvent<scheduler>);

EventTable::addHandler function takes boost::function type argument, which is constructed here on the fly from EventHandler::onEvent function (which is a static member function of EventHandler class). Debugger showed that one member object of the boost::function object, namely, manager member, had invalid address 0x0000e460, causing the access violation. As last resort, Madcat attempted to change the call:

Impl::HandlerType hfunc(&EventHandler::template onEvent<scheduler>);
boost::signals::connection c = Impl::getEventTable().addHandler(s, hfunc);

Technically, there was no change, except the fact that now the boost::function object (typedefed in Impl::HandlerType) is constructed as temporary object, and passed to the addHandler() function as copy. Lo' and behold - suddenly there is no bug anymore, no access violation, no crash, everything working as intended.

Who is to blame? Apparently, the wonderful Microsoft compiler generates wrong code for the first version of those lines, and thus broke it. There is no explanation to my knowledge that would explain this behaviour.

4 full development days and lot of frustration just because compiler generates wrong code. Ain't that just fun?

Madcat.



Wednesday, January 05, 2005

Ahh, the pleasures of deving on win32

Let's see how to put it nicely. Some UI designer once said "The more time user spends using your application is less time user doing whatever he actually wants to do.". The idea is that if your application gets in the way of the user, that's bad. I'm sure you can think of many examples of this (I'd bring few examples of my own, but I'm too zombie-state right now to be able to). So, for past two days, I'v been using M$ crap-Studio.

I could as well bang my head against the wall and still be more productive than trying to develop with that thing. I'v rambled about the crappyness of it before, so I won't go into details again, but bottom line is I decided the only way to be able to productivly develop is do it on linux - an OS that feels like it was designed from ground up for software development. So, I'm trying to push maximum amount of development to linux, to minimize the amount of time I need to spend with that crapstudio.

Enough ranting, here's what's going on. First - the roadmap. I reviewed the code per subsystem, and put together the list of things needed to be done - view roadmap. As you can notice, a large number of engine subsystems need improvements or complete replacements - that's pretty normal in current project state. First implementations rarely survive for longterm - "Plan to throw one away. You will, anyhow." - Fred Brooks, The Mythical Man-Month, Chapter 11. Or - to put it other way - "You don't fully understand the problem until the first time you implement a solution" - Eric Steven Raymond.

Starting from the top, the biggest issue right now is the partially upgraded event engine. While the new engine solved the first problem - multi-handler calling - by using Boost.Signals backend, the added syntactical improvements and de-coupling features of the new engine, while being a nice design touch, cannot be used afterall, due to the problems with win32 linker/dynamic loader. After numerous attempts to get past those issues, I can finally conclude that it is unfeasible to get the exports/imports right, at least with my current knowledge base, and thus I'll just have to go back to the original coupled system, and tie the EventTable with EventSource, most likely using macros, just as in original implementation. We still have the implementation benefits of the new engine, which is simpler and less error-prone, but we must drop the syntactical improvements. Today I'm too stoned to get that finished, but it should take a night or so to get it fully operational again.

Madcat, ZzZz



Monday, January 03, 2005

Returning to development

Today is Jan 3rd, the scheduled date for continueing the development. The break gave me the chance to look back at the work done, as well as review and rethink the short-term and long-term goals and objectives of the project. But first and foremost, I'd like to thank all the commenters on the last blog post - frankly I was somewhat afraid of how the watchers of this project would react to 2-week break, but it all turned out pretty well. And those comments rightly reminded me that christmas was coming (which I had forgotten), and so on. So once again - thanks for that.

Now, returning to the topic at hand - HydraNode. There are several things that I'v given a lot of thought over the vacation, so let's see if I can summarize them up.

We can safely say that the first phase of hydranode development was completed in december - most of the low-level things. Also, the main focus during the first phase was well-formatted, well-documented code, high code quality standards (oh the countless hours of valgrinding), and so on. Also, the development was purely linux-oriented, with little or no support for Mac and Win32 platforms.

In the second phase, starting today, and expected to last 81 days (no, you don't need to know why 81 days - those who need to know know, those who don't, don't need to know), will be oriented towards slightly different objectives. First and foremost, the development will take place on MS Windows, which will also be the primary target platform. Linux and Mac support will be at nice to have, but not 100% required state. And before you start yelling at me - I probably can't leave linux port too much tagging behind, since the majority of my testers (and friends) are using linux. Which brings us to the next topic.

Whenever a developer writes a software, it will inevitable be affected by the surrounding atmosphere - linux-based developers design/develop software that looks, acts and feels like linux software, windows-based developers design/develop software that looks, acts and feels like windows software. The most complex aspect of cross-platform software engineering isn't the differences in the platform itself - it's the differences in the users, their attitudes and their expectations. An application oriented at Windows users doesn't really cut it at linux users, and vice versa. They are simply two different user-groups, with completely different expectations. As for Mac users - I haven't used Mac long enough and I don't have any Mac-user friends, so quite frankly, I have no idea what are their expectations.

In larger development teams, there are different people for different platforms. In case of HydraNode, however, as we all know it's an one-man-show, which makes it more time-consuming and complex than it would if a single developer had to target a single audience. HydraNode has several important basic design elements that appeal to Linux users - namely complete engine/user-interface separation. What must happen now is also make it appealing to Windows users - that is graphical user interface elements.

Starting with those, the first thing that'll need to be done is a Launcher application, which will act as shell for hydranode engine. It will start HydraNode engine as sub-process, taking over its stdin, stdout and stderr (directing them to appropriate GUI controls), provides minimal UI features (like displaying data-rates etc) and so on. This will still be a developer-UI, however, for end-user, it'll act as several modern Linux distros startup - show nice splashscreen + "Press ESC for verbose mode", optionally thus displaying the console output of HydraNode engine. This development road will eventually lead up to full user interface.

In addition to that, as mentioned previously, some components of the engine need upgrades or replacements - which will also need to be performend during this 81-day development period.

All of this leads up to the need for new development roadmap. However, while in the beginning we could clearly show a stream-lined roadmap, with clear goals moving forward, in this phase we cannot, since there are several development directions already, some of which need to be performed simultaneously, and so on. The new roadmap is still being worked on, and should be up for your viewing pleasure within a few days. The first UI design concepts of the launcher application have been proposed, but still need more work.

As for mac - hydranode is portable to Mac, but the actual porting will probably be delayed when the app is finished - it just takes too much time/energy to make it work / keep it working with my 400mhz iMac (unless ofcourse some Mac ppl volunteer to do some coding/debugging - I gladly help/introduce the codebase to whoever is interested). Personally, I can barely keep two ports operational simulataneously ... Mac port tends to be"the last feather which broke the camels back" kind of topic - and kitties with broken backs don't make good developers, now do they?

Another topic that I wanted to address in this blog post was the first HydraNode release. The roadmap put in place in June 2004 states that we should release in Jan 2005 - which is now. Today, it is clear that we will not be releasing anything in January. Furthermore, declaring HydraNode 1.0 isn't that simple after reviewing some things. While the 7th commandment declares "Release early. Release often." - it is biased towards Linux users. Windows users do not expect, nor even like software that is released pre-maturely, nor software that brings out new release every 2 days - they just think the software is buggy (which pre-maturely released software is), form first oppinion of the software based on that pre-mature version and often never come to check back at later time - been there, done that myself. Also, frequent updates make the same impression - the need for constant upgrades irritates the user, and implies the software is buggy (explaining the need for frequent updates).

For hydranode, I would consider it release-ready when it has at least 3 network support, full user interface as well as some minor side-plugins. Because if we release it as pure ed2k client, it'll be always thought of as ed2k client. Same applies to only ED2k+BT support. With three networks, we can show the power of hydranode engine, multi-network downloads and so on - only then hydranode can form a truely good first impression, shaking off all the accusations a'la "Bah, yet another mldonkey?" or similar. The new roadmap will reflect this decision, and propose new final release date - my current assumtions are around spring (making hydranode a one developer-year project), altough usable beta versions should be out far before that - e.g. in a month, or 2 at max.

Stay tuned for more details on the new development directions and roadmap over the next few days, as well as general development news over the coming weeks/months :)

Madcat


PS: No, I won't apologize for loooong blog post - you can't really expect a short post after 2 weeks pause :)


Archives: December 2004 January 2005 February 2005 March 2005 April 2005 May 2005 June 2005 July 2005 August 2005 September 2005 October 2005 November 2005 December 2005 January 2006 February 2006 March 2006 April 2006 May 2006 June 2006 July 2006 August 2006 September 2006 Current Posts

This page is powered by Blogger. Isn't yours?