Been a nice and progressive night, with nearly 900 lines of new code; while the main target of this patchset was new features, some bugs got accidentally killed along the way as well.

• Fixed files of size <= 9500kb completition (ticket #50)
• Config now emits signals when values are changed - this allows changing configuration on runtime.
• Added 'config' command to hnshell and Hydranode Console
• Added 'help' command to Hydranode Console
• Added 'search' command to Hydranode Console
• Added Config Subsystem support to Core/GUI communication module and library
• Fixed a number of bugs in Core/GUI communication socket handling (in cmod_cgcomm), which caused lot of wierd behaviour, packets being lost, and 100% CPU usage at times.
• Scheduler now handles changes to UpSpeedLimit, DownSpeedLimit and ConnectionLimit configuration variables, and reacts accordingly.

In english, this means that you can now do 'config set UpSpeedLimit 10240' from either hnshell or Console, and everything happens as one would expect - within 100-1000ms timeframe, the real speed is adjusted to the new level.

ConnectionLimit changing is somewhat more tricky - if you lower ConnectionLimit below the current open connections count, existing connections are kept open, but new connections aren't accepted until the real ConnectionCount drops below the limit again - this means it might take longer time to completely adjust to the new limit - while speed limits adjust almost instantly, this one can take rather long time to adjust; however, I do not think instantly dropping bunch of random existing connections would be such a good idea either ...

Madcat, ZzZz

Improved - now with toolbar and basic lists.

Just playing around with ideas ... in any case, we have Core/GUI comm support in console now (thanks to libcgcomm it's easy to add Core/GUI comm support to any C++ app). I'm not yet sure what to do with the toolbar idea, two lines is obviously too much, and the buttons need some improvements ...

Oh and btw, I set a release date for 0.1.2, to Aug 10th. Refer to 0.1.2 Roadmap page to see what needs to be done for that. Most important requirements are upgrade to Boost 1.33 (will be released on Aug 4th), and finalizing the console. Async DNS support is bit questionable, but I'm hoping it can be finished and tested for that time.

Anyway, it's 7am, been nice 12h+ dev-session, so -> off to sleep.

Madcat, ZzZz

Many people wonder why I'm heading off towards GUI development instead of bittorrent, gnutella and whatever new networks. To be quite honest, the major reason behind that is that a change is needed - I'm pretty fed up with the core development - it's been what? 14 months now? - and QT development is basically like starting from scratch - all prior knowledge is void, except for base C++ knowledge - in QT, you have nearly no use for STL, Boost or whatever API's we'v written for Hydranode, so it's a clean slate to all prior knowledge base - very nice change. Not to mention that I'v started making really stupid mistakes in core coding already, for example a week ago I checked in a 3-line change into PartData w/o proper testing nor looking at the code, which caused a ton of crashes on file completition (fixed now in r1573). This clearly shows that I should stay off the existing, working code for a while.

So, after many hours, I finally managed to get the first Launcher version up and running:

Hydranode Console with Win2K theme
Hydranode Console with WinXP theme
Hydranode Console with WinXP theme (transparent)
Hydranode Console on Linux

Granted, it's still very early version, so much can change, but I figured I'll drop these screenshots, since I haven't been very active on the blog recently ...

Anyway, the Console will have support for at least partial Core/GUI communication, so one can perform most operations directly from it, without launching full GUI. Rationale: several of my testers on win32 have said "oh, this interface [hydranode shell] is so cool, you HAVE to leave it in!" - so Hydranode Console will become a usable command-line interface, with the addition of some nifty features that we cannot possibly do in telnet-based interface (transparency, for example :P).

Madcat, ZzZz

Past few days have been rather interesting - we'r getting close to finalizing the user interface design. I cannot disclose any design screenshots at this time, but we'v solved large amounts of problems, including dropping several rather tricky-to-implement, but as it turned out, un-necessary features from the design. It shouldn't take long now before I can jump to implementing this thing.

What I can say at this time though, is that basically we'll have a Launcher application (also previously mentioned some months ago in blog posts), that will launch the core as child process, and then bring up user interface as well as child process; which user interface is brought up is determined by command-line arguments. Since the Launcher will be run from a desktop icon, we can, for example, create different icons for different user interfaces, should there be need for this. The Launcher itself will take care of initial startup, splash-screen, but will have no visible windows once the GUI is launched. If the GUI is not launched directly from Launcher (background-mode, for example), the launcher can (optionally) display a tray icon. The launcher will also provide access to a console window of core output, along with an input box for entering commands (technically equivalent to current Hydranode Shell module, except this one would work over Core/GUI communication (hence much more powerful)).

Since the Launcher will have both Core and GUI running as sub-processes, it can monitor the process's state's, and re-start either of them should they crash - while we do not plan on having either of them crashing, in real-world, software still does crash sometimes, so it's nice to have some automated thing behind it, that can, for example, provide automatic bug-data collection for submitting to bugtracker's.

Madcat, ZzZz

There has been nearly 60 repository checkins during past few days; I'll try to summarize it now before falling off my chair :)

Most importantly, the new build system was deployed. Converting the BBv1-based Jamfiles to BBv2 only took some 5 hours, and it's very nice and clean now. Currently it has been tested and builds cleanly on Windows/msvc, Windows/mingw and Linux/gcc. Note: there is a small issue open on win32 builds - library paths aren't hardcoded, so manually moving *.dll from lib/ to top-dir is needed; this will be addressed shortly. I'm aware it currently fails on darwin, this will be looked into shortly as well.

All in all, I believe this was the right choice - dumping GNU auto-tools and going for Boost.Build - it's much more manageable and scalable, and it should be easier for end-user (whether a person compiling a piece of software can be considered an end-user is a different question) to handle. The compilation instructions page on wiki got lot shorter as well.

Other miscellaneus changes/fixes that were done during past few days:

• [1523] Core/gui comm sockets are now properly buffered - no more data getting lost.
• [1526] Porting cgcomm testsuite to qt4 (still needs more work)
  
• [1535] Link Boost.Filesystem statically on win32 (breaks on mingw in dll build)
• [1536] Enabled SIGINT/SIGQUIT handlers on Win32 - now ctrl+c causes clean shutdown on win32 as well.
• [1539] Added WorkThread::pauser object, that is an exception-safe wrapper around WorkThread::pause()/WorkThread::resume methods.
• [1539] WorkThread got suspended/broken when Hydranode was started with -b option (backgrounding after startup), resulting in no hashing being done. This is fixed now.
• [1540] Fixed endless loops when shared files were deleted (tickets #28, #30 and #31)
• [1541] Fixed unhandled exceptions in eDonkey ServerUdp handler (ticket #12)
• [1542] Added current server info to modules/ed2k/serverlist/stat command in hnsh.

Madcat, ZzZz

The good news is that the entire Hydranode (4 libraries, 8 plugins and 1 executable) compile cleanly on MSVC, Mingw and linux/gcc now, using the new build system.

The bad news is that I think I discovered a bug in Boost.Filesystem - when compiled with mingw, some paths get broken. This needs bit deeper investigation and a test-app...

Porting this new build system to also work on mac will take a long time, considering some 50-min recompile times on my mac (and you do a LOT of full recompiles when debugging build system), so that's bit of a problem ....

Anyway, this completes phase 1 of the build system upgrade - get-to-know-chit-chat with Boost.Build and the Jam language. The Jamfiles are a rather big mess right now, since I had no prior experience with BB, so one rewrite of all the Jamfiles (using the knowledge gained from first run) is scheduled - this time writing directly to BBv2 syntax, which should be bit simpler as well.

Build configurations need bit working, need to figure out a bunch of command-line args to configure the build (e.g. how to build plugins, which ones to built-in, how to build static binaries, how to build documentation, how to build release tarballs et al) ...

Madcat, ZzZz

Since until now, we'v only used MSVC on win32 to compile hydranode, some additional delays were introduced due to mingw-specific issues.

First of all, linking with boost_program_options library failed due to some odd reasons; even boost regression tests are failing in the case (and have been failing for a year already apparently), namely in dll configuration - static linkage works fine, but that's not an option for us. The developers have said it fails "due to unresearched issues". Anyway, I did some research, and found that we need -Wl,--enable-runtime-pseudo-reloc linker flag to fix it. An email to the library author is on it's way.

Other issue that seems to come up is related to paths handling (using boost_filesystem library). For some reason, boost::filesystem::path outputs proper win32 paths (e.g. \ as separator), but refuses such paths as input, when compiled with mingw. This issue still needs some research.

On Linux, it builds properly (hncore and hnbase library), but running directly from source tree isn't possible yet, since linux doesn't search for libraries in current dir (I recall there were some hacks to get around it, but I'v lost the links - anyone happen to know them?).

Overall, one might argue that having the entire codebase in single hierarchy binds the components too closely together - the current system is much more loose. However, the difference is that when building/developing the app, we want everything to be closely together, to help with maintainance; but when packaging for releases, we prefer to create separate packages - and that's also easier to do from single code tree. So it's a win-win situation.

Madcat, ZzZz

Hydranode current codebase, including all plugins, test-suite and so on consists of over 200 files, containing over 2MB of code, and frankly, it's starting to get a bit over my head. So, in order to counter that, I believe it's time for another build system upgrade (if you remember, last one was like 8 months ago, to GNU autoconf system).

This time, the plan is to upgrade to Boost.Build v1 (maybe version2 in the future). It's very flexible, highly portable and powerful, which suits our needs. The main problem with build systems is that they often fail to handle large number of sub-projects gracefully, but this one seems to handle it very nicely and cleanly.

So first off, I set out a new layout of the code tree, which can be seen here. The biggest change is that hncore and hnbase are now two separate entities - hnbase providing cross-platform API and bunch of useful, general-purpose classes, and hncore implementing the p2p-related things, e.g. FilesList, MetaDb, and so on.

The other change that I'm planning to bring in is get rid of the pesky /testing hierarchy we have in SVN right now, it doesn't seem to justify it's existance. That code shall be moved to the main tree for better overview and maintainance simplification.

GUI code will be in the same code hierarchy as the engine code, and additional user interfaces can be added there as well. With this setup, we get a better overview of things, and make it easier to manage - consider that very soon we'll also have GUI along with it's own testsuite and plugins there as well.

For those ppl out there that are crying out "but I want my configure/make!" - no problem, I will provide a short configure script, as well as a standard Makefile, in order to keep things "standard"; that Makefile/configure will simply wrap around the Boost.Build-based system.

ETA: Few days - Jamfiles (Boost.Build equivalent to Makefiles) are really powerful, but that means a rather steep learning curve, but currently I'm progressing nicely - hncore compiles/links correctly on win32 at least.

In the longer future, I'm thinking about rolling a small light-weight GUI front-end to it, to further simplify configuration / building of the codebase and related libraries. The rationale is that since QT only works with GCC on win32, we basically lost the only IDE that has proper support for many projects (at least I haven't found another one that does it reasonbly good), and command.com + editor just doesn't cut it in long-term on win32 (which is the main target platform, afterall). So a lightweight bjam frontend, which does compiler-output highlighting, acts as gdb frontend and some other minor things - shouldn't be more than cpl days to roll together.

Madcat, ZzZz

Was busy in the morning with some real-life things that needed my attention, and had a long interface-related discussion with our designer in the evening, which left little time for spitting out code. None-the-less, some minor (?) updates:

• r1459: Added support for guarding.p2p ipfilter format
• r1460: Downloadlist in shell is now sorted by name; fixed list displaying with entries having 100% completed state.
• r1464: Added email notification plugin, which currently can send email notifications on completed downloads.

The email notifier plugin (not compiled by default) is configured from the global config.ini, with the following values:

[MailNotify]
From=santa.claus@north.pole
To=little.elves@north.pole
Server=123.546.223.22

Pretty straight-forward, however, as you probably noticed, we need to use the IP address of the mailserver currently, since we lack DNS query support. cyberz is working on DNS queries support, and I'v seen initial patch for that (really cool code, can't wait to integrate it :)), but he said it's not ready for integration, and now he's on vacation for a month, so guess that'll have to wait a bit.

In the future, the mailnotifier plugin will support notifications on all kinds of user-defined events, with user-defined message formats/templates.

One might argue that the mailnotifier is useless, and I tend to agree, however, it was requested by Tobias, and since it was a small plugin (215 lines total), I figured I'll just throw it together and be done with it. And due to the modular architecture of hydranode, more plugins != more bloat, so no losses.

Madcat, ZzZz

Been a fairly interesting 18-hour dev-session, with updates in various areas. So, without further delay, he's the listing:

• It's now possible to enable/disable trace masks during runtime or as command-line argument.
• We'r now using single random number generator (Utils::getRandom) at all places, to achieve best randomness (patch by theos)
• It's no longer possible to resize MetaData object (since it would break a ton of things if one did that)
• Searching with minsize and maxsize implemented both at ed2k protocol level, as well as internal filter (for networks which don't support such options).
• Added 'vr' (view results) command to shell, which optionally allows sorting keys (name, size, srccount, and reverse orderings of them), for displaying search results. By default, search results are displayed with 'rsrc' ordering, which means the most widely available files are displayed at the bottom of the listing (no more 5-pages scrolling after every search). Default sort ordering can be customized from config, SearchSortOrder value.
• Importing of eMule temp files implemented; currently, all files will need to be completely rehashed. The original .met file is not touched, we simply create our .dat file right next to it. As a side-effect, hydranode is now capable of loading temp files from multiple folders. TempCnt key in config specifies the count of dirs, and Temp_# keys list the dirs. Imported dirs (using import command in shell) are automatically added to temp dir listing.

Importing code still needs some work, while it already does it's job, I believe we can take this a lot further. For one thing, I'd like to update the .met files as we progress with the downloads, so the files can be easily imported back to eMule (currently hydranode-downloaded data would be lost, since eMule won't know more data has been downloaded). Additionally, full rehash of the file can be avoided as well, as soon as I figure out the gap-system eMule uses to write completed-ranges to the .met file.

Regarding the possible bug I mentioned in last blog post, seems it was a ghost-bug - when I woke up next morning, it didn't appear anymore, and right now everything seems to work properly, so I'd blame either my broken windows installation, or my ISP.

Oh, and to explain the missing blog posts since 8th - I spent several days re-installing my boxes; for a while there I thought my vidcard had died, but it's Ati's bad drivers to blame again ... anyway, testing the damn vidcard on multiple boxes and multiple drivers, along with like 6 re-installs took quite some time, but now everything seems to be working properly again.

Madcat, ZzZz

While one could easily view each day's progress at the DevCenter Timeline section, if everyone would be doing that, then I could just drop this blog. So here we go again:

• SPARC, Alpha and MIPS platforms are now properly detected as big endian
• [patch by cyberz] New getVal/putVal engine (used for all binary IO) that converts from host endianess to stream endianess automatically (old system only converted from host to little-endian).
• [patch by theos] Better ed2k userhash generation
• Added 'stat' command to modules/ed2k/serverlist (displays #users/files on net)
• Added support for IPFilters. config.ini value IPFilter can point to the file's location (either relative from config dir, or absolute). Both eMule and mldonkey formats are supported and autodetected.

Now, all this sounds good and so on, however, the new IO patch seems to have introduced some strange bug in networking, which causes ~70% of outgoing connections to fail. It is not known at this time how could that happen, but it was proved that in r1425, we succeed in ~90% of outgoing connections [ed2k], while in r1429 (the next fully working revision), we only succeed in ~40% of outgoing connections. *doh*

Madcat, ZzZz

Sorry for some missing blog posts, there were some real-life things that needed to be taken care of; but I'm back now.

So today, I went over the current pending bugreports in DevCenter, and managed to fix up 4 of them. In addition, cyberz (while implementing DNS resolution API for Hydranode) discovered a rather bad bug in sockets, where "highest" socket number wasn't updated by UDP sockets, so sometimes, UDP sockets weren't properly polled. smiith has been doing some testing of Hydranode on Solaris/Sparc, with nice progress - most things work, but it segfaults in Hello packet parser right now (ed2k module).

The bugs that were closed today were:

• Incorrect percentages in "view downloads" output in shell
• Some missing, non-standard tags in SearchResult packets
• Use inttypes.h instead of stdint.h on some platforms (e.g. solaris)
• WorkThread (IO thread) can now be paused; it is paused during recursive shared-dirs scans to speed up the scans.

There's also a new configure option [on posix platforms], --enable-hexdumps, which enables ALL hexdumps from eDonkey module, should you be interested in the packets contents (helps in debugging endianess, for example). Developer note: per-packet hexdumps can be enabled by hand from packets.cpp anyway, this is just a shortcut to enable all at once.

Madcat, ZzZz