MLDonkey Downloads Import Module Development
Development in progress.
Monday, January 30, 2006
Support for colors on Windows
Windows console has somewhat different handling of colors than POSIX terminals. POSIX terminals understand a wide range of escape sequences which they interpret as colors and other things; win32 console however uses win32 API functions to achieve the same effect, which hinders one's ability to write portable code in this regard. Until now, I had decided not to support colors on win32 at all, simply since I deemed it unnecessery waste of development resources, considering that majority of the win32 userbase would be using GUI anyway. However, as I hear from several sources already, there are people who want to use console-mode applications even under windows - what the reason might be I do not know, but as they say - "The client is the king" - I introduced coloring support for windows. This covers both the command-line window in which Hydranode runs, as well as hnshell output in telnet. Notice users of Windows 2000 and earlier:
the telnet application that ships with those versions of the OS doesn't support colors to my knowledge, so you'r encouraged to use alternative telnet clients, such as PuTTYtel
For anyone interested in how to create colored output on win32 console, here's the function
that transforms POSIX escape sequences to win32 API calls.
And a screenshot for the disbelievers :)
Sunday, January 29, 2006
More shell features and such
Something that's been bugging me (and possibly other users as well) has been the lack of displaying known file names for downloads, since this is one of the main ways to determine whether the file is fake or not. Now, using the new 'vd #num' command, hnshell displays 15 most popular file names of the download along with frequency, as well as all comments that have been found so far. Note that neither of these fields are stored across sessions (to keep them up to date).
Additionally, hnshell now displays ETA value (when file availability is 100%), which shows how many days, months or hours are left to download completition. Here I ran into a small problem with the 78-character line-length limit of hnsh (from win32 console), so I decided to trim the ETA value to two up-most entries. E.g. if it is estimated to take more than, say, 2 months to complete a download, it only displays months and days, while if it is estimated to take under a hour to complete the download, it will display minutes and seconds. This way most relevant information is displayed without cluttering the output.
In bittorrent module, if you can remember we are using cache folder (config/bt) for cross-file data as well as .torrent files. In some cases, this amount of data can span hundreds of megabytes (when torrent consists of many small files), but usually it's only few megabytes. Nonetheless, until now the cache folder was never cleaned up. Now we perform cache folder cleanup - deleting all torrents and cache files for torrents that are no longer being seeded or downloaded - on startup.
There was a bug in BT uploading handling where when attempting to switch upload slots to other clients, we actually switched it back to the client that we took the upload slot from in the first place, which resulted in instant choke + unchoke combination, hurting overall upload speed stability of the module. This no longer happens - when switching upload slots, we determine which client will get the slot, and if the client matches the one that we'r about to take the slot away from, we simply give the client another 30 seconds upload time before checking again.
Related to the last change, in order to avoid many clients being choked + unchoked at same instant, we now use variable-length upload sessions - current implementation uses 30 += 3 seconds timeframe.
Friday, January 27, 2006
Since the bugtracker has started to fill up due to the recent slower development, I decided to do some cleanup there and fix stuff. With some 12-hour devsession, I managed to close 12 tickets total, most of which were fixed, 1 or 2 were duplicates and some were too old or insufficient information to debug further. Anyway, quite a bunch of crashes were fixed, and few wrong behaviours.
I'm also working towards a new feature-set, which includes proper file-names storage (so you can see what name is most popular on network etc), as well as comments storage. This however needed some changes in MetaData / MetaDb classes, which means further testing is required before I can check in that code (currently it breaks stuff badly).
Thursday, January 26, 2006
Secure Ident fixed and new builds available
Got quite a big surprise yesterday when chemical and other testers discovered that secure user identification isn't working. Apparently it was discovered accidentally by some other developer; we were sending incorrect ip_type field in Signature packet (local and remote setting reversed), which caused the ident to fail. I never detected it since I only tested it locally, but in that scenario, the senders and receivers IP addressed matched, so the ident succeeded. The error has now been corrected (IP_LOCAL and IP_REMOTE defines were reversed), and due to this being a critical issue, I'v uploaded new builds.
These new builds also include the updated hnsh download-list view as described in previous post, as well as some miscellaneus bugfixes.
Tuesday, January 24, 2006
New downloadlist formatting in hnsh
Some comments: the availability percentage is calculated from how large amount of the file is available at all, e.g. if one 9500kb chunk of 19MB file has no sources, it will display 50%. Likewise, if one 9500kb chunk of 100MB file is missing on network, the percentage will be around 10%. As a rule, if the percentage is displayed (it's only displayed for running downloads with <100 value), start worrying.
Also note that colors are not available in win32 builds, and there are no plans on adding that support, the logic being that win32 users will be using graphical interfaces soon.
Feedback, thoughts and comments are welcome.
Madcat.Edit: The typo (transferring missing an 'r') was fixed in svn.
Sunday, January 22, 2006
Small code updates
Since it's staying at below -25 degree margin here (altough there's hope for bit warmth in the beginning of week, after which it's expected to get really
cold), and I'm bit pre-occupied with the other things
, it's hard to write larger blocks of code or do major updates at same time. However, I managed to push through few smaller updates:
- Added functions to Socket API for fine-tuning TCP send/recv window size, based on information alex posted earlier. Technically, this should allow Hydranode to be able to transfer at rates exceeding 2.2MB/s, however experiments with HTTP module didn't yield the expected results. I believe the issue is that in default configuration, ubuntu linux limits the tcp window size to 110kb globally (need root permission to override). But how does wget manage to get 40+MB/s transfer rates then?
- On windows, we now correctly handle console window close events as well as some common OS events, such as shutdown and logoff. Thus, Hydranode performs a clean shutdown now when closing the console window or shutting down your computer (previously, only clean shutdown methods were either 'shutdown' command in hnsh or ctrl+c in console window).
- Fixed guarding.p2p ipfilter parsing when some unexpected lines were encountered (thanks to dani_555 for discovering this).
- Fixed duplicate deletion of mailnotify plugin sockets (thanks for tobiasthecommie for discovering and reporting this one).
Thursday, January 19, 2006
"You can't win if you don't risk losing."
Throughout the last 1.5 years of this project, this blog has served one major purpose - to give insight about what's going on behind the scenes of an open source project, down to the gory details of every-day matters; questions, doubts and thoughts about past work, present objectives and future directions; to share the joy (and occasional sadness) of the life of an open source developer. Throughout this time, I have followed a strict rule of honesty and openness, even at times where it might not be the best option - everyone makes mistakes, but I believe that acknowledging that you made a mistake and correcting it is still better than hide your mistakes in the first place.
Throughout my life, I have always had hobbies of some kind - be it collecting stamps, playing online RPG's or coding free software. One constant that has followed from hobby to hobby is that I don't tend to stay with one thing longer than 6 months at a time. More often than not, I return to the topic (or similar topic) at a later time, usually after another 4-5 months of doing other things. Hydranode project has been a major exception to that rule, with a whooping 18 months of activity with only two vacations (2 weeks during xmas/2005 and 5 weeks during spring/2006). It was the complexity, the satisfaction of creating something that many deemed impossible, the constant chance to learn something new, the friends and blog-readers, that kept me going for all this time.
Doing such a large-scale project alone means 12-hour workdays without free weekends; it's a constant pressure to deliver quality code, user-support, and daily progress updates, which leaves very little, if any, time to spend time in unrelated activities. I have always liked to spend maximum amount of time with my hobbies, be it coding or playing online RPGs, and ignore the world outside computers altogether for extended periods of time. But ignorance can only go so far.
"The light at the end of the tunnel is a train."
We'd like to pretend that the world outside doesn't exist; that us, free software engineers, live outside the rules of the world; that we can provide a service that anyone in the real world considers expensive, for free. It's a fun game, but ignorance has the habit of coming to bite you at the end of the day; this isn't the first time this has happened [to me], and most likely won't be the last. Whether it's getting expelled from school, losing a job, or simply waking up one day, looking in the mirror and going "what the ...?", you suddenly realize all that you've been ignoring for a long time suddenly comes at you at frightening speeds.
"Ignorance can be cured; stupidity is forever."
From past experience, I can say that it's best to handle the situation, solve the issues, play by the rules of the real world, rather than bury one's head into the sand and pretend it doesn't exist. We all make mistakes, but it's those who recognize the mistakes and correct them that stand out as winners at the end of the day. I'm not saying that Hydranode is a mistake - definitely not; but there have been some mistakes made somewhere along the way, which must be corrected before moving on.
"When all your wishes have been granted, many of your dreams will be destroyed."
Everyone wants to get somewhere in life; everyone has a dream of what his/her life should be in a perfect world. Good things happen to those who wait, but on the flip-side, a sleeping cat can't catch a mouse, and left to themselves, things tend to go from bad to worse. In the end, it's all about choices. To pursue one's dream, or to hope that the dream comes to you. Reminds me a scene from Collateral (2004)
, where a cab-driver has a dream of starting a limu-company on a tropical island, and is only driving cab temporarily to raise money... for twelve years already.
Is there a point in this post? Let's find a point. There are very few areas where one can work at one's own schedule, be as challenging, rewarding and intriguing as this project. However, despite what free software enthusiasts would like to believe, development costs money, and a hungry, unmotivated programmer is a bad programmer. Thus, instead of radically lowering the quality of development that has been the key ingredient in Hydranode development, I prefer to not develop when I do not feel motivated; it's like an artist drawing a picture - you can quickly see if the artist was motivated and interested or not.
Contrary to what many of you are thinking right now, this doesn't mean the end of the project. Knowing myself, it's highly unlikely that I can find other areas of work which provide this amount of freedom, flexibility and challenge as Hydranode and similar projects. However, developing merely for the sake of developing and rushing the product out of the door will only hurt the quality and final outcome of the project, jeopardizing the very essence upon which Hydranode was built on.
"The darkest hour is just before dawn."
The bottom line is that I've hit the bottom; both in the financial, and energetical sense; it's -28°C outside, deep into the long, dark and cold Estonian winter, with 4 months left before there's hope for warmth. But in all that darkness, there's a beam of light; a beginning of a path. I do not know if it's the right path, but the only way to find out is to get on it and see where it takes me.
To finish this lengthy posting on a more positive tone, there is one thing that I know for certain; I'v given it a lot of thought, and reached a conclusion that Hydranode is something that I truly want to do; but in order to do so, I must guarantee myself certain level of income and other pre-requisites; Hydranode itself was never designed to be a revenue-generating project, and attempts to attach such methods to it will only hurt it. But I have a plan, and if all goes well, within less than 6 months I hope to fully execute the plan, which will provide me with enough resources to do what I enjoy most - cutting-edge programming :)
Wednesday, January 11, 2006
Been busy with the side-project I mentioned in previous post - seems I managed to convince the client to go with only php+mysql-based solution and avoid Access completely - I'm not really a big fan of VBA. Will hear final feedback on friday/monday, hopefully they'r satisfied. Contract-work seems really so much easier than OSS project management/coding - you have your deadline, expected result, and free schedule, and a LOT less work than that of an OSS project - not to mention getting payed. Developing on web platform is also quite a lot more fun than C++, because you can see the results of your work in a matter of hours/days, rather than in months/years.
The most frustrating part of C++ coding is that progress is lot slower than that of alternate platforms (such as web), and it's a LOT harder to land a contract-work with that. Everyone needs web/db stuff, and it's easy for a freelancer to do few projects w/o prior experience (I started on this db project by reading through the php4 manual, followed by mysql manual, just to learn the topics), while landing a contract-work for custom software solution written in C++ is next to impossible - most of that job area is done by companies, and more often than not, C++ programmers are hired full-time - something which I'm seriously trying to avoid. Full-time work just takes away my freedom, and usually pays considerably less than that of a contract-work. Furthermore, a custom software solution usually takes many months to complete (at least), while a website or db frontend can be completed under a week of dev-time.
Makes one wonder - if I can land hundreds of euros with 1-2 weeks of work on web platform, what on earth am I doing coding C++ then? Only place that I'm aware of that pays good for C++ code is day-job at some respectable company, and dayjob is something I'm trying to avoid at all costs... and speaking of which - C#/.NET seems to be much more "in" nowadays than C++ when it comes to corporate software engineering (where development speed and fast learning are valued), it seems C++ is only used at high-end software market (large-scale applications [office, photoshop] and games [due to performance reasons]).
On other news, I'v had some thoughts on why Hydranode starts having serious networking issues after around 10 days uptime. I think the issue is that we run out of sockets the OS provides. The thing is - Hydranode creates and destroys a LOT of sockets, if you start to think about it. Given a TCP reask every 50 minutes (in case where UDP is disabled), and 10'000 sources, that means around 200'000 sockets created/destroyed after 24h period. In 10 days, that means around 2'000'000 sockets. Now, socket descriptors are 32bit signed integers, and >0 values are used, which should give us roughly 2'000'000'000 available numbers (assuming they aren't being reused). Yes, I notice now that the calculation is off by 3 digits (I didn't notice it during my original calculation), but still it could be a theory worth investigating. Does anyone have experience with similar matters?
Thursday, January 05, 2006
Back from vacation
I'm mostly back and online after the two-week vacation now. I'v given quite a bit of thought even during vacation to various hydranode-related topics, mainly on GUI-design-related areas. I also realized a bottleneck in our event engine which accounts for 8% of overall CPU usage (inherent from Boost.Signals library), so I hope to work around that at some point - 8% overall CPU usage drop would be really nice.
Dani discovered an interesting project - QTWin
, which offers scripts for building QT 4.1 free edition with MSVC - something that I'v been looking for quite some time now. From the looks of it, it's working (build still in progress at the time of this writing), so I'll be able to compile GUI things with MSVC as well.
There are two other places where I believe optimizations can be made - low-level IO functions (putVal/getVal) - 7% can be gained from an alternative implementation; and ipfilter loading can also be sped up via better inlining of loading function.
However, I don't expect to be able to code much in Hydranode before 15th, since a side-project that I agreed to do already in october became active now, so I'm brushing up my SQL/PHP/Access/VBA skills and coding some db-related things.
September 2006 Current Posts