Alo Sarv
lead developer

Donate via
MoneyBookers

Latest Builds

version 0.3
tar.gz tar.bz2
Boost 1.33.1 Headers
MLDonkey Downloads Import Module Development
Payment completed
Development in progress.
Developer's Diary
irc.hydranode.com/#hydranode

Wednesday, August 31, 2005

First bytes downloaded with BT module

Bah, I had already forgotten how much time it takes to write a new module for Hydranode - let's face it, it's been a while since I started a new network module. IIRC, the previous one was FTP, and I only spent one night on it (got data structures up, but got stuck at some parsing stuff). FTP module can be finished w/o issues now, as we have the power of Boost.Spirit around, which is ideal for that parsing (actually it's been bugging me ever since I learned Boost.Spirit - how perfect it is for the FTP module parsing problem

Anyway, it took two weeks to get BT module to the state where it is now - that is, downloading the first bytes. Let me rephrase that: I just downloaded the first bytes using BT plugin. Proof? Here's a trace log. Okey okey, not a big thing, but a milestone non-the-less. For the record, that's a free, creative-commons-licenced mp3 collection.

So, two weeks of work to get this far? But on the other hand, look what the time has been spent on:
Right now, BT module code is ~2700 lines of code. As comparison, eDonkey module is 12'600 lines of code. I estimate BT module to top at around 5000-7000 lines of code, maybe more with trackerless implementations. So - in 2-3 more weeks, and we should have a working BT module...

Anyway, as an observant reader might'v noticed, the bytes downloaded were 3x 9 bytes from 3 different chunks. Not very fancy - seems PartData chunk/lock-selector isn't doing it's job right. But not to worry, Madcat has a plan. More on this tomorrow :)

Madcat, ZzZz

PS: The module code is now properly documented; view it, as usual, in the DevCenter SVN browser.


Tuesday, August 30, 2005

Progressing ....

First of all, thanks for the comments on last post, it gave me several ideas, which I hope the implement in near future. Simple, yet powerful GUI - got it covered. The more I think about it, the more I tend to fall towards the idea of releasing standalone BT client as soon as the BT code is working. Whether or not it'll also have ed2k support is an open topic, but I have some ideas on that area as well.

Anyway, I'v been testing BT myself some now (using Azureus) - hadn't really used BT for a while myself - and am now starting to get the right "feeling" about the protocol / network / however you call it. There are things still that aren't mentioned in any protocol docs that I'v seen, for example upload management - how many slots to keep open? Based on what I see in Azureus, it seems to keep 3x6kb/s slots open with 20kb/s limit... the connection-management is still a mystery to me, w/o queue handling... anyway, from what I can see, uploading is done to the same clients from where we download (fair enough), but many connections seem to stay "0 down, 0 up" state, others "xxx down, 0 up" state ... ohwell.

Anyway, our implementation got some updates today, altough nothing fancy yet - we have initial client-name detection (useless, but better than deciphering the cryptic peer-id), and we'r moving towards chunk-requests, but ran into some trouble in there - PartData doesn't seem to want to give me any ranges to download, keeps bailing out on PartData::doGetRange() method... the wierdest thing is that it works for ed2k (I nearly copy/pasted some of the code from ed2k module), but not for BT... perhaps our customized PartData class in BT module is to blame?

Madcat, ZzZz


Monday, August 29, 2005

What makes Bittorrent tick?

When implementing new networks, there are two factors that I consider most important:

a) How does the protocol work?
b) How does the user use the network?

Two simple questions, but wait, there's more to it. When I think "protocol", I don't just mean the raw implementation of it. I mean understanding the underlying logic, the flow, the concepts behind that, the concepts that resulted in the protocol implementation in the first place. Because if you understand the concept, the logic, then the specific implementation doesn't matter - it's just numbers, in different sequence, but the logic doesn't change. DHT-based systems are like that - for example, if you implement, say, Kademlia, by copy/pasting the code from eMule, w/o understanding HOW it works, then yes, you end up having Kademlia support, but you'v gained no knowledge on DHT's, which means next time you need to implement another DHT-protocol (either 3rd-party, or your own), you'r helpless.

And, let's face it, knowledge and understanding of how P2P systems work, finding out what logic makes something tick, is one of the things that I love about this project - you get to work with many different areas, and learn from them.

The second question was - how does a user interact with a protocol? ed2k is kinda "generic" type network - from users point-of-view, the style is shared between a number of networks, including FastTrack, G2, Ares ... you search, you queue up a lengthy list of downloads, you get your files, that's it.

However, other networks cannot be so easily thrown into same pot of tea. Bittorrent, for instance, doesn't have search, which means users go around in forums and websites, looking for torrents. Bittorrent also seems to be filled with "50x15mb-rar-torrents", something you only rarely see in ed2k. Seeding is an important part of Bittorrent, much closer to user than uploading in ed2k.

Other networks that I hope to implement in longer future also have very specific characteristics. DC++, for example, is chat-based. IRC (xdcc or smth) is a completely different beast as well. Newsgroups - whole new story - 3000-byte-messages, read by binary newsreader, merged together and hashed by some 3rd application - intrigueing topic :)

But, we'r getting off-topic. Bittorrent is the current plugin at hand, and my interest is in understanding what makes it tick, how do the users interact with the protocol, what are the problems users are having, what are the problems in the existing clients, what can we do to releave those problems... what I'm trying to avoid here is simply throwing out a ton of code, saying "ok, we have BT support", and moving on, w/o ever stopping to ask "did we solve the problem?". Heh, we don't even know yet what the "problem" is we'r trying to solve - and don't say "because HN is multi-net client and BT is a must-have-thing".

Rather, think of it like this: If I were to release a standalone Bittorrent client, what features should it have, what would it do better than the existing clients, how would it win the "market"? The rationale is that multi-net client ONLY justifies itself if EACH and EVERY ONE of it's plugins / supported networks is AT LEAST equally as strong as it would be as a standalone client. And thanks to the modular architecture of Hydranode, we actually CAN release standalone clients for different networks, as we see fit.

So what say you, long-time Bittorrent users?

Madcat, ZzZz


Sunday, August 28, 2005

[BitTorrent] Implementing Client management

I hate client management. Seriously. In ed2k module, it took literally months to get it working in a stable manner. Let's hope this won't be the case in Bittorrent module.

To get us started in that area, I drafted another short design doc, which among other things, addresses the client / torrent management issues. Read it here.

There are also some interesting new files in svn, for example:
hncore/bt/client.h
hncore/bt/client.cpp
hncore/bt/manager.h
hncore/bt/manager.cpp
Yeye, I know, documentation is missing - I'll do that once the code stabilizes a bit.

Basically, I implemented all of the protocol Client <-> Client packets, along with client management and messages passing. Next up, we need to start adding protocol flow logic - and that's tricky at best. Bittorrent protocol, while being very small itself (as you can see in client.cpp - that's all of the protocol, implemented in under 300 lines of code), the packet flow logic on the other hand is very complex. We'll be testing, tuning and improving that one for weeks to come.

However, the most current issue is that I can't get my handshake packet right. In all protocol specs (official, unofficial), it says "1 byte of value "19", string "BitTorrent Protocol", 8 reserved bytes, 20 bytes info_hash and 20 bytes peer-id". I event went to look inside libtorrent code, and it's same in there. I'm sending this:
00000000  13 42 69 74 54 6f 72 72  65 6e 74 20 70 72 6f 74  |.BitTorrent.prot|
00000010 6f 63 6f 6c 00 00 00 00 00 00 00 00 00 00 e8 17 |ocol............|
00000020 b8 80 24 a2 58 b9 77 7f c1 98 fc 0e 75 a1 49 60 |..$.X.w.....u.I`|
00000030 0d c2 48 4e 30 31 32 30 39 38 33 34 33 38 30 37 |..HN012098343807|
00000040 30 32 35 31 39 37 |025197|
And all clients just drop the connection. Got some extra info from Azureus - it basically seems to recognize the handshake, and even sends response, but then bails out and cuts the connection with error "unexpected end of stream". Am I supposed to send something else right after handshake? *duh*.

Madcat, ZzZz


Friday, August 26, 2005

Tracker communication implemented

Well, we have basic client <-> tracker communication up and running. Basically, it all comes down to one HTTP GET request, and the response to that. This request / response is done every now and then, and that's about all the client <-> tracker communication there is, as far as I know.

The files, for your viewing pleasure (sorry, undocumented):
hncore/bt/torrent.cpp
hncore/bt/torrent.h

Now, from that we got the peer listing; next logical step is set up the neccesery data structures for handling the peers. I'm thinking along the lines of multi_map<Client*, Torrent*, PtrLess<Client> >-like container. In english - a non-unique map of clients (sorted by ClientId), allowing cross-referncing Client and Torrent. Why non-unique? Because same Client can be related to multiple torrents (contrary to my erronous statement last night).

Now, when clients contact us, we can wait for the handshake, and then look up the client based on the received PeerId, and "merge" with the existing object, pretty much similarly as we do in ed2k. That merging busyness is damn tricky (took quite some time to get it right in ed2k module), but it's unavoidable as far as I know. I just hope we'v learned from the mistakes made at ed2k module.

Speaking of listeners - an interesting topic raised today on our IRC channel - namely, it was pointed out to me that ShareAza somehow manages to work multiple protocols with only TWO ports - one TCP and and UDP. Now, I haven't confirmed this behaviour personally yet, but, it's interesting find indeed. The interesting part is that then you only need to open two ports total in firewall, instead of a ton, as it tends to be the case in multi-net clients.

After some thought, I figured out a way how to accomplish this (I don't know how they do it, but I presume in a similar way). While I don't have free time right now to implement it, this could indeed be a valuable feature, if done right. However, I'm a bit skeptic about the possible performance issues with this approach, since it would require client-level checks / parsing for all incoming connections, something that would otherwise be done at OS level, in much more compact way.

Anyway, I intend to implement this support when I have some free time (possibly after BT), and implement it as optional feature, so as to not break any existing stuff, and to test possible performance issues.

Oh, btw, regarding Madcat(tm) Bittorrent Protocol Spec, I'm not so sure it's neccesery - there's enough _good_ BT protocol documentation out there already, I don't think it's neccesery to create yet another one, with nothing interesting to add. For eDonkey2000 network, it was worth it, since there isn't that much docs on it on the net, but BT protocol specs are available everywhere.

Madcat, ZzZz


Thursday, August 25, 2005

Thoughts, design and what about trackerless?

Mkay, now that we have the wrappers around SharedFile / PartData, the next thing we need is some kind of object to govern those two, and contain all the networking stuff for this torrent. As much as I know about bittorrent protocol, same client cannot offer multiple files (at least I'm not aware of any bt client supporting it - any1?). Hence, this would simplify the module design somewhat.

Namely, we'll create a Torrent class (could probably use a better name), which owns the TorrentFile/PartialTorrent objects, as well as handles all the networking. Each peer would go to Client class (yes, I know the BT terminology calls them "peers", but I prefer keeping the concepts consistent across modules). Clients would be created and owned by the Torrent class. This means we won't have to implement a separate ClientList class, as in ed2k module.

Now, based on that, and from my (still very limited) knowledge of the BT protocol, Torrent will communicate with the tracker, receive clients, construct and maintain the clients, and perform the seeding/downloading. This does put the majority of the module implementation into a single class, but then again, same happened in ed2k module, where all of the protocol logic was handled by single class - Client; actually I did move a ton of stuff into helper, created-on-demand classes in ed2k module, so if the Torrent class gets too big, we can consider that option as well.

What has me worried a bit here is how trackerless torrents fit into the scheme. You could say "dont worry about trackerless right now" - but you can't, we have to consider this right now, since we just don't have the resources to rewrite large parts of the code when, say, we discover that the trackerless stuff doesn't fit into our design at all.

Another interesting thing I discovered today while going over the official protocol specs was this:
"d4:spaml1:a1:bee corresponds to {'spam': ['a', 'b']}"
Hum? How am I supposed to parse that in a generic way? There's no way for a parser to know wether what it's reading is next map key, or a subvalue. What if you encounter input such as:
"d4:spam1:a1:bee:4:span1:c1:de". While in the first case, the parser could "assume" that the last value is also a value to the preceeding key (since the total number of arguments to "dict" was odd number (3), however in the next (home-made) sample, the parser can't make that assumption, and would incorrectly parse {'spam':['a'], 'b':['span'], 'c':['d']}.

In reality, I doubt such situation happens in the protocol, and even if it does, my current protocol handling can easily be customized to handle the situation, however things such as these destroy the possibility of creating generic parser engine which doesn't care about the content being parsed, and you know I'm a fan of those :).

Madcat, ZzZz


Wednesday, August 24, 2005

Progressing on BT data structures

As mentioned, it's critical to get the data structures integration right here, and thus I'm taking time to think things through properly. Also, I'd like to re-use as much of existing codebase as possible, for obvious reasons.

Right now, I have the basics up - wrappers for SharedFile and PartData, which basically mirror their children's state, and forward calls to proper children, all achieved via cpl of virtual functions. Additionally, I had to create a custom HashWork and override some functions there as well to get the hashing stuff working across files. Now, all this is only initial code (it compiles, but hasn't been tested), so bugs are still there, but seems we'r getting somewhere.

So far, changes to the core have mainly been moving some functions into protected from private and changed to virtuals; some operations were also moved to separate functions to allow overriding by derived classes (PartData::corruption, HashWork::getNext come to mind). Other than that, no changes have been neccesery to accomodate these custom data structures. Thus, from the looks of it, the design decisions made about a year ago seem to be holding up well.

The files, for your viewing pleasure:
hncore/bt/files.h
hncore/bt/files.cpp
hncore/bt/bget.cpp

The last of them is a small driver application, that I intend to turn into a wget-like command-line utility, with the difference that it'll just download torrents. The purpose is mainly simplifying the development of the module, but it can be released separately for unix platforms as a standalone tool later, if there's interest in that.

I'd like to get to initial networking stuff tomorrow already, but there are still some open ends at the data structures... on the other hand, I can fix them on the fly... we'll see.

Madcat, ZzZz


Monday, August 22, 2005

Designing Bt module integration

Whenever starting a module-writing for Hydranode, there are two main areas that need work - the actual protocol implementation, and the integration with Hydranode and the other modules. For some modules, the protocol implementation is the harder part, for some, the integration. Ed2k, while being the first "big" module, is hard to qualify properly, but it tends to fall under the "hard protocol implementation" category. Bittorrent, however, falls under the other category, mainly due to it's different way of handling files.

Multi-network downloads support is only as strong as the weakest party in the mix; it's not achieved by some magical code in the core; rather, it's achieved by each and every module explicitly supporting it. For example, Http module can in theory support multi-network downloads as well, by getting the required hashes from the server (often named MD5SUMS or <filename>.md5 for example).

With Bittorrent module, this concept is taken much further. As we know, Bittorrent protocol handles the entire torrent as one big file, which is broken up to real files on client-side. Also, as already noted before, in Hydranode case, this means we will have two views for a torrent - a "flat" view, which other modules see, and the "torrent" view, that BT module itself will operate upon. The latter view is created from a "virtual" file, derived classes from PartData / SharedFile which don't have a corresponding physical file, but rather wrap around a set of actual PartData/SharedFile objects.

What does this mean from users point of view? For one thing, in the User Interface, one can choose "flat-view", to display each and every file in the torrent as "separate" download. Furthermore, the user can operate with each such download as one would with normal downloads - you could pause, resume or cancel files from within a torrent. Even further, when starting a torrent download, the user interface could bring up the list of files in the torrent, and user could choose which files from within the torrent he/she would like to download.

What does this mean from the implementors point of view? A ton of problems, to be honest. Hydranode Core API merely makes it possible to create such a thing, but it's still complex. After two days of heavy thinking on the topic, I have resolved a number of issues, but still more keep popping up.

First and foremost, we need some place to store the .torrent data. While I could store this data in MetaDb (using some custom fields et al), I don't like the idea, since torrents often have hundreds of hashes, and it would unneccerely grow the metadb.dat too large. Instead, I'm thinking of copying the .torrent files into $(configdir)/bt/ dir, and keep them there. Now, we'll sha1sum the .torrent file, and create a MetaData entry with that. It could even be a fully-qualified SharedFile object if wanted, so .torrent files could be shared on other p2p networks (exeem comes to mind), however that's of little importance right now.

What, however, is of importance, is that after we have the sha1sum of the .torrent file, we will create the PartData objects corresponding to the files in the torrent, and attach the sha1sum to the customdata field in THEIR MetaData object. Now, the "normal" downloads are loaded on startup normally by FilesList. When Bt module initializes, it scans the .torrent files location, and either sha1sums the .torrent files again, or perhaps - faster - looks up the sha1sums simply in MetaDb, based on filename / modification date, as we do with normal shared files. Now, Bt module knows which sha1sum goes to which .torrent file; based on this information, it can scan the download / shared files lists (FilesList class), and check for files which have a matching sha1sum in their customdata, and thus generate the list of currently pending .torrent downloads, associating the files with the .torrent data. Based on that information, it can then create the virtual parent objects (which cannot be constructed by FilesList, since they are derived classes, implemented in Bt module).

One part around here that I don't have a solution yet is how to keep track of the order of files in the .torrent (because the order matters). Perhaps also store the # of the file in the torrent also in the customdata field along with the hash, e.g. "btorrent:<hashdata>:12335" would indicate that this file starts at offset 12335 in the .torrent (just saying 5th file wouldn't help, because if we'r missing some files in the middle, we'd be in trouble).

What does this lead us to? It theoretically allows cooperative downloads, in both ways. Consider this: if the .torrent file contained md5sums of the files (rather common nowadays), or even ed2ksums (also seen occasionally), ed2k plugin could start downloading the files right away, alongside with bt module. Vice versa is a bit trickier, and would (for now) work only under specific conditions: namely, if we have a MetaDb entry of a file that was previously downloaded from Bittorrent, we _know_ the .torrent file sha1sum in which this file belongs to. Based on that, it's only a matter of finding the .torrent file (hopefully we still have it), and we can bring in Bt module to the download, which was initiated from ed2k module.

Doesn't sound very useful? But what if you consider that the MetaData recordset, along with the .torrent file, could be requested and transmitted from a p2p network? You have ed2k hash, you look up (on the net) the sha1sum of the .torrent, then you simply download the .torrent file (from the network), and there you go.

I mentioned "problems" in the beginning of this tad long post, so here are a few:
  1. What happens if you start a .torrent download, but it turns out you already have few of the files found in the .torrent?
  2. What happens if you start two torrents, which share few files?
  3. What about seeding? Queueing? How do we make sure we act "nice" on the Bittorrent network?
  4. How do decentralized trackers fit into the module design?
  5. How does Exeem fit into the module design (what hash algorithm does Exeem use? How can we take advantage of an existing p2p network that has exclusivly .torrent files [thinking about the ed2k -> bt integration topic]?)
  6. How deep does the rabbit hole go? (no, seriously - what else can we do/implement here?)
Madcat, ZzZz


Saturday, August 20, 2005

BTParser proof-of-concept finished

Today I finally managed to finish the .torrent file parser. You can see the implementation here. This small <200-liner requires 260mb memory during compilation, but results in 65k binary in release build - behold the power of C++ templates :)

Anyway, this brings us to the second stage of BT plugin - data structures. Naturally, this calls for proper design of the module before moving on. While we do have an initial RFC on the topic, that's not sufficient, because many topics are still open - internal BT module data structures for one thing, and some generic questions, a'la how/where do we store the .torrent info, considering the amount of data needing being stored, as well as the non-file-specific nature of the data (metadb tends to operate on files, while .torrent operates on a group of files). "Virtual" files, as described in the RFC, are most likely the solution, but it needs some thought before jumping to implementation.

Madcat, ZzZz


Friday, August 19, 2005

Getting the feeling...

As I mentioned in yesterday's blog post, Boost.Spirit is a completely new topic, and requires a new kind of thinking / looking at it to make it work to our advantage. Today, I believe I'm starting to understand it, at least a bit better. The key is that you can create small, light-weight, in-place parsers with it. For example, parsing an ed2k:// link using Boost.Spirit looks like this:
uint64_t filesize;
std::string filename;
std::string hash;
std::string link("ed2k://|file|slackware.iso|35623456|def4256fba|/");
rule<> parseLink = "ed2k://|file|"
>> *(ch_p[push_back_a(filename)])
>> '|' >> uint_p[assign_a(filesize)]
>> '|' >> *(ch_p[push_back_a(hash)])
>> "|/"
;
parse(link.c_str(), parseLink);
And that's it, the variables contain the contents. Similar "in-place" parses can be created for all kinds of parsing, for example config file, or in current active topic, .torrent file.

Now, some might argue that Spirit is too complex, and the time spent learning it is not worth it, but I disagree - Spirit gives us yet another very valuable tool for current and future tasks. Just as Boost.MultiIndex, Boost.Signals, and many other Boost libraries, which were initially complex to master have now become our daily tools, without which we'd be helpless, will Boost.Spirit be soon enough.

Anyway, the .torrent parser saw progress today, but the parsing stops mid-way in the chunk-hashes for some reason. Also, as I discovered it doesn't handle unrecognized data very well yet; I'm thinking about re-structuring the parser, have the base as recursive parser which just handles the structure (nested dictionaries and lists), and call sub-parsers for actual values extraction.

On other news, however, I have some news on the GUI design topic. Below's a teaser, see this thread for more information and discussion.

* Notice: this is just an image created in photoshop, it's not a real screenshot :). Also, as you noticed some parts are still borrowed (e.g. the progressbars, some icons) - those will be replaced shortly as well.

Madcat, ZzZz


Thursday, August 18, 2005

The spririt within...

Duh, spirit is unlike anything I'v worked so far before. C++ knowledge is of little use there, when you end up writing constructs like this:

btorrent =
'd'>> *( ("8:announce" >> bstr[var(self.res.announceUrl) = tmpStr])
| ("13:creation date" >> bint[assign_a(self.res.creationDate)])
)
>> 'e'
;
bint = 'i' >> int_p >> 'e';
bstr =
uint_p[assign_a(tmp)]
>> ':'
>> repeat_p(boost::ref(tmp))[anychar_p[push_back_a(tmpStr)]]
;

Sadly, the above doesn't work (yet). Spirit library is a strange beast, and requires completely different way of thinking. Yes, there's the grammar specification stuff (which seriously abuses expression templates and operator overloading), but the logic behind the parsing and accessing the parsed data is tricky at best. In the above sample, you saw three different ways to get data from parser - assignment operator, assign_a() function, and push_back_a() function. Those are called "actions" - and there are a ton of possible ones for that, but never enough. The question is where to we attach the actions, and how hardly we couple the parsing and the data handling together.

What I liked about my parser at ed2k module was that I was able to completely decouple the packet parsing and the packet handling, which clearly separated the protocol logic from the protocol format, allowing much simpler coding at both ends. However, I'm not yet sure how to accomplish such a thing with spirit, and whether it's wise to pursue such goal at all with BT / Spirit.

On other news, something I forgot from yesterday - I re-organized the source code structure again, it's now much more editor-friendly - namely, all components files are now in same directory - e.g. hncore headers and source files are in hncore/, ed2k headers and sourcse are in hncore/ed2k (notice I dropped the "plugins" subdir as well). Rationale: it got really tedious to switch between include/hncore/ed2k/ and hncore/plugins/ed2k/src dirs all the time, and there was actually no reason why it should be like that - the original intent was to allow include paths like , but that was now accomplished with much more maintainable directory structure. And keeping this thing maintainable is a continued effort, considering that Hydranode currently consists of roughly 187 source files, totaling somewhere around 40-45'000 lines of code - quite a bit :)

Madcat, off to wonder what does a spirit of a hurricane look like ...


Wednesday, August 17, 2005

Moving on

As usual, there was few days wind-down time after the release, but now we'r getting back on track. Yesterday, I was working on adding UPnP support into Hydranode, and it went well - we have a basic router-detection, ports adding and removal support. A manual test application is included (called "pfwd"), that is capable of adding/removing ports. Further integration into Hydranode systems will be something for later.

Today I was working on upgrading the internal boost sources and tools to Boost 1.33. That went seamlessly, everything seems to be working as before. The highlights in Boost 1.33 that affect Hydranode most are Boost.Signals major performance upgrade (Hydranode event engine relies on signals), hashed indices to multi_index and compressed iostreams support in Boost.IOStreams library. It is yet to be determined how much does the Signals upgrade affect Hydranode's performance, but based on what I'v seen in Boost mailing lists, the performance boost was said to be major. Hashed indices will require manual code updating to take advantage, but will speed up a lot of lookups in containers. As for compressed streams, these will be put to use in ed2k module, where we (still) lack proper support for compressed downloads, due to lack of stream-based decompression - I was waiting for Boost 1.33, instead of rolling my own there.

Anyway, after some discussions, we came to the conclusion that the most important thing to continue working on right now is Bittorrent. The reason is simple - Hydranode is supposed to be multi-network client. Hydranode is built from ground up with multi-network in mind. All major components in Hydranode are multi-net-aware, and took several times longer to develop just because of that; and now we'v been sitting on this piece of code for countless months already, only using it for single-net, and this is unacceptable. The GUI is irrelevant at this stage, because as long as we don't have multiple networks, we'r simply throwing away countless months of work that went into generic engine components such as Range API, PartData, Hasher API etc. There are people still working on GUI designs, but I'll be at hncore/bt/ dir for a while now.

The first target is to successfully parse (and store information) from .torrent file. Considerining that the .torrent file contains the most complex BT protocol structure (a map with nested strings, integers, lists and submaps), if I can successfully parse that, the rest of the protocol is a piece of cake.

Now, as mentioned before, I want to parse BT protocol using Boost.Spirit, because it's really fast and very powerful parser engine. However, with power, comes complexity. I'v done some studying of the library in the past, and I spent a measly 4 hours on it today as well, with little progress - those guys who wrote that thing are insane :). Right now, it feels like banging head against the wall. Hopefully, the morning brings clarity and progress towards that end.

Madcat, ZzZz

PS: Apparently, I forgot to enable support for files >2GB in the 0.1.2 release (it's fixed in SVN), so I might need to do a 0.1.3[hotfix] release sometime soon....


Thursday, August 11, 2005

Slight delay

Hum, seems I got the release date mis-matched. For past two days, I was feeling that there's still time until the release deadline, which I thought was on 12th, until today I suddenly found out that it was supposed to be on 10th instead. *Doh*. Blame "12" looking so similar to "0.1.2" :)

Anyway, here's the deal. The launcher/console and Core/GUI comm isn't release-ready yet, so that has to wait until 0.2.0. However, I will do a generic update release on the engine tomorrow, without the launcher. I compiled a changelog out of roughly 350 revisions (r1350-1700), viewable here.

The Console and Core/GUI communication still needs about 2-3 weeks work, so expect 0.2.0 to be released before the end of this month, featuring full Console and Core/GUI communication support.

Speaking of the Console, we finally figured out how to create 16x16px icons properly. While this is still a new topic for me and our designer, you can see the first icons created in the following screenshot:


So anyway, an update to the command-line version will be released tomorrow. Note though that darwin binaries will not be available this time, since there are some pending issues in the build system regarding modules compilation on darwin, and I don't have the hardware resources right now to work on that (1.5h debug build times are simply too much). With some hacking, I believe it would be possible to get it compiled with XCode though (the code is 100%-darwin-compatible).

Madcat, ZzZz


Monday, August 08, 2005

Portable GUI programming - on Linux?

Ahh, the pleasures of Linux GUI programming. When you'r writing a "portable" command-line application, generally "portability" boils down to 3 platforms - Windows, Linux and Mac OS. When you'r writing a GUI application, you'r dealing with 8 platforms - Windows, Mac OS and 6 major linux distros, each and every one of which do things differently.

Take a simple example - we'r using a specific, commonly-available fixed-width font in the Console interface. On Windows, it's called "Terminal" (the same one used in command prompt), and must be used at size 10. On Ubuntu Linux, it's called "Console", and must be used at size 16. On Fedora Core 4, it's called "console8x16", and can be used in any size. On Mandrake 10.1, it's called "Console" and must be used at size 10. On SuSE 9.3, it's called "Misc Console", and must be used at size 10.

I'm not even going to the QT vs GTK war that we will have very soon around here anyway. "But I want it native in Gnome/XFCE4/etc". Yes, sure, but then it wouldn't be native in KDE :).

Seriously, the only way to provide a consistent look and feel of your application across the myriad of Linux distros and configurations is to have a fully skinned app. And then everyone just shoots you on general purpose - "But I want it native! You'r evil windows-dev! Go away!". Been there, done that, thank you very much :)

So anyway, here's the plan - based on Distrowatch, the top5 distros are roughly SuSE, Ubuntu, Fedora, Mandrake and Debian (not in any particular order). Hence, the plan is to support those completely, and everyone else needs to somehow manage as well - it's not humanly possible to support the gazillion of Linux configurations out there.

Now, since all our GUI stuff is being written using QT4, this puts GTK-based distros into a disadvantage - this includes Ubuntu and Fedora. I saw how it looks like on those distros, and I'm not happy; but then again, we have the same problem with Classic theme on Windows as well. So - either I figure a way to enforce Plastic skin (comes with QT) in those cases, or write some custom skin for that (doubt we have the time for the latter).

---

Anyway, enough ranting. Finally got the core/gui comm stuff moving again tonight, despite the fact that we still have some unresolved problems in the Console (log window is a performance hog when lots of log data is in there already) - namely we have initial downloadlist and updates monitoring capabilities. It's still very buggy, but it's progress.

Hydranode r1697: Windows Linux Source

Madcat, ZzZz

PS: Linux builds now also work on GCC4-based distros (e.g. FC4).


Saturday, August 06, 2005

Nearing the release

Uh oh, no blog post for 4 days ... what's happening? Is madcat gone? Is there no progress to mention? These, and other similar questions might cross reguar blog-readers minds. Nothing could be farther from the truth though - as anyone can see from the timeline page in DevCenter, development activity is normal as usual. However, I'm bit tired of writing blog posts every night (they get rather boring after a while), so I'm doing longer, but more informative posts at few-days intervals nowadays. Anyway, the updates listing, compacted and re-formatted for your viewing pleasure:
Today's snapshot r1685: Windows Linux Source

So, where are we? The release is supposed to happen in 4 days, Boost 1.33 release was supposed to be 2 days ago but they seem to be behind schedule. Since Boost 1.33 is a requirement for this release, this will most likely mean our release will slip as well. However, on the bright side, looking at ticket browser, we can see that we have 4 tickets open for 0.1.2, and 3 for 0.2.0, with one ticket overlapping (CGComm library is required by #43 and #2).

#8 is basically done, #44 as well, #45 is dependant on Boost release, which basically only leaves three things to do before release: extend libcgcomm, global searching, and use WorkThread for allocating disk space in PartData; all three of them are easy to implement. So basically, we'r looking real good already - there aren't any major bugs open to my knowledge, and we only need three easy-to-implement features for release :). And from the looks of it, it won't be 0.1.2 that gets out of the door, it'll be 0.2.0 :)

Madcat, ZzZz


Tuesday, August 02, 2005

Console UI cleanup






As you can see, the UI of the console got cleaned up; the search-bar is completely optional, and is disabled by default. The three buttons at the bottom-right should be intuitive (if not from screenshot, they should be once you'v clicked them once) - representing three most-used operations - Search, Download and Incoming. Since this is a Console, everything is ofcourse keyboard-controllable - some hotkeys are already in place, others are being added.

On other news, did some testing and improving on darwin port, it now compiles, but most likely won't run yet - need some more hacking around the build system.

Madcat, ZzZz


Monday, August 01, 2005

Miscellaneus fixes and cleanups


On other news, we went over the Console GUI design with our designer, and came up with a bunch of ideas and plans for it, which I will be implementing over the next few days. Stay tuned :)

Madcat, ZzZz


Archives: December 2004 January 2005 February 2005 March 2005 April 2005 May 2005 June 2005 July 2005 August 2005 September 2005 October 2005 November 2005 December 2005 January 2006 February 2006 March 2006 April 2006 May 2006 June 2006 July 2006 August 2006 September 2006 Current Posts

This page is powered by Blogger. Isn't yours?