Alo Sarv
lead developer

Donate via

Latest Builds

version 0.3
tar.gz tar.bz2
Boost 1.33.1 Headers
MLDonkey Downloads Import Module Development
Payment completed
Development in progress.
Developer's Diary

Friday, April 01, 2005

Credits; lower mem usage on compilation; working on SrcExch

As you'v probably noticed, I'v been skipping "daily" blog posts a bit recently. It does not, however, mean that I didn't work on HydraNode that day - I pretty much work on hydranode 24/7. Missing blog posts simply mean I was either too tired, too bored, or simply didn't want to say anything. Often, by the time I decide that now it's time to get to sleep, I'm so tired, so frustrated at some bugs, that writing a blog post simply isn't something I want to do. Now, I know that you are expecting blog post every morning, to read before going to work/school etc, but hey - I'm only a kitty, sometimes I need room to breathe too :)

So anyway, to make up for last nights missing blog post, here's the update for last night's entries:
Today was a slightly different day tho. Today, the following things got done:
The MultiIndex stuff might be of interest in detail - we know from past experience that MI can get heavy on compilers, but we found workarounds to that. However, as I'm using MI at many classes already, the memory usage issue became obvious, so the following helped a lot:
// in header
namespace Detail {
struct MyIndex;
class MyClass {
boost::scoped_ptr<Detail::MyIndex> m_list;
// in source file
namespace Detail {
struct MyIndex : boost::multi_index_container<...> {};
MyClass::MyClass() : m_list(new Detail::MyIndex) {}
Simple, stupid, yet effective. Similar tricks had to be done with all iterators we needed in headers (PartData headers needed some, as well as ServerList), and that's that.

The rest of the day/night went towards implementing SourceExchange support. The problem is that some of our data structures didn't really support this idea, so I had to implement DownloadList class in ED2K module, which acts as a thin wrapper around PartData objects. The good side is that it localizes several formerly duplicate code regarding downloads scanning, and handling. Furthermore, it ties several objects more strictly together, acting as a stabilizator between two very dynamic entities - PartData and Client. The implementation is 90% finished, and passed initial testing, but needs more work - AnswerSources packet parser isn't fully working yet (altough I'm already getting responses), and some source counters / lists aren't properly updated yet. So bottom line is the system should be fully functional tomorrow evening.

This means we have 3 entries left on 0.1 roadmap - zstream, buildsystem and shell stuff. Well, in reality, it seems we still have one more thing - scheduler. It's scheduled for post-0.1 fixing, but I'v had some time to think about it, and it's really only cpl hours work. The problem is, Scheduler, in it's current state, isn't very scalable - with 100+ sockets, cpu usage raises to 15% region already - that's bad. The source of the problem is that Scheduler handles events from all sockets, which means map lookups during EVERY socket event. Even though we are talking about only pointer comparisons, it's still heavy. Scheduler got a tad over-designed when I wrote it few months back, so now it's time to simplify it. Namely, I believe we can scrap the ENTIRE scheduler.h file (900 lines of code), and implement the neccesery things in SSocket class, with few callback functions and few shared pointers. It should definately speed up networking a lot, and have positive effect on compile times also (which seems to becoming an issue more and more).

On other notes, a comment mentioned precompiled headers. HydraNode supports precompiled headers, but only on MSVC right now, since I haven't yet managed to convince the build system to do it with GCC (maybe I can implement it with new build system, which is planned for 0.1). However, hydranode relies heavily on templates and template-based libraries, so precompiled headers have very little effect - on MSVC, they only lower compile time by 30% (compared to non-template code, for which pch can have up to 90% speedup effect).

About the athlon64 - I'm doing regular testing on amd64 myself, in 64bit mode, so hydranode is, and continues to be, fully 64-bit aware and supported.

Thanks for the link on Filesystem Hierarchy stuff - useful stuff. I must admit I haven't had time to read it thoroughly, but it seems /opt hierarchy is the right place for hydranode.

Madcat, ZzZz

Comments: Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?