MLDonkey Downloads Import Module Development
Development in progress.
Friday, April 28, 2006
Hydranode 0.3 released
Three months have passed since the last release, and time has come again for a release. When the previous release's focus was on Bittorrent support, this time the focus was on Graphical User Interface. As such, engine updates have been minimal, but still include several important fixes, mostly on the Windows platform.
Important engine updates include support for more than 64 open connections on Windows, support for files larger than 2GB on Windows, and full support for Windows XP Service Pack 2 (half-open connections limiting). eDonkey module upgrades include improved UDP protocol handling, better A4AF handling and other performance and protocol improvements. Bittorrent got some updates as well, most importantly multi-tracker support.
Since this is the first release of the user interface, not all things we planned made it into the release. However, the currently existing features should be sufficient for daily usage. The interface was developed to be ergonomic, easy to use and visually appealing without sacrificing functionality. We hope you will enjoy using the interface as much as we enjoyed creating it.Special thanks to:
- Joakim Kuusemaa aka Arlekiin, for the design and graphics of the user interface
- The testing team - chemical, dani_555, TobiasTheCommie, frop, aLaTaR, oka, CruX, ak and others who have reported bugs or submitted feature requests.
So without further ado, here are the download links.Download for WindowsDownload for LinuxDownload source code (tar.bz2)Download source code (tar.gz)Download source code (zip)
Ed2k serverlist in GUI
After the required 4 hours of sleep last night, and the today's 19-hour dev-session, here's the stuff that got implemented:
- Filtering and clearing completed works properly with categories now.
- Fixed bunch of crashes in various areas inherent from the recent new code (~3000+ lines of code during this week)
- Finalized ed2k page in the interface - status information is displayed at the bottom, double-clicking on servers connects to them.
- Fixed adding/removing shared directories from the interface.
- Fixed some bugs regarding files renaming from comment window reported by dani_555.
- Full support for Object updating over cgcomm.
- Fixed some final issues with the new skin.
Tomorrow (well, today - Friday) evening 0.3 final will be released, barring some disaster. I still need to write the feeds caching code (custom feeds won't make it), and making a release takes many hours as well. Gonna be close. Frankly, I'd be very surprised if this release won't have some bad bugs in it, considering the development speeds and lack of testing recently. At these speeds, you have to take shortcuts and cut corners wherever possible, but due to lack of testing, you can't allow bugs, so balancing between those two is a tricky busyness.
Madcat, falls off the chair.
Thursday, April 27, 2006
Experimental support for passing Objects through cgcomm
Since it's 4am and I'm dead tired, I'll be short today. The entire day was spent on implementing Object hierharchy support for cgcomm. For those who haven't followed this blog for longer time, Hydranode has an internal Object hierarchy, through which it's possible for modules to make their data available for user interfaces and other modules in a generic way. Objects can have data members, child objects and even operations that can be performed with them.
Getting objects working over cgcomm is quite tricky busyness. We need to send objects in right order so that children are always sent before parents, send updates at specified intervals (object updates are propagated up the hierarchy), so user interface will 'monitor', say, ed2k server list, and updates to server objects need to be sent to the gui. And so on.
Reconstructing the hierarchy on libcgcomm side is a tricky busyness as well, but should work now properly. Basically, all the basic stuff is up, and I'm able to view ed2k server list from the user interface. Operations and such aren't supported yet tho.
Wednesday, April 26, 2006
New skin integrated; fine-tuning in other areas
When I woke up today at 9am, the most pressing issue was file names/comments window causing endless loops. So few hours later, the things were fixed and several other optimizations done regarding that window. Basically, the names are updated every 5 seconds for active file (if the comment window is open); less frequently if the file has many names/comments. It's also possible to rename the download to one of the names in the list by double-clicking on the list item.
Another thing that had bothered me for a while was that the config and skin directories were dependant on application working directory. So if the application was launched with a different working directory, the skin wouldn't load. This problem reached it's critical state when I got report yesterday that while browsing folders (for opening file or such), the skin disappears partially and it creates config/ subdirs in each dir visited. To fix it, we'r now saving the application binary dir on application startup and use that instead (absolute path).
Canceling 'add file' operation in transfer window no longer causes error message to be shown either. On related note on core side, when we re-hash a file we previously "knew", we now copy over the existing 'custom metadata' from the previous entry before dropping the old one. This helps with modules using custom metadata (bitttorrent, cgcomm [stable ids]).
At 9pm, I met up with Arlekin to finish up the design side of the GUI for the 0.3 release. The new skin was integrated, replacing the existing one; details box got dropped from the interface for the time being (we like the concept, and it will return, but currently we just don't have enough input data to justify it's existance in 0.3 release). The yellowish question marks got removed from the 'actionbar' as well (meant to be help buttons, but help won't be available before later this summer either). Lot of other things got fine-tuned and tweaked as well on the gui/design side. Some final updates (few icons and such) will be added thursday, but generally it should be finished from the visual point of view.
Which only leaves me with still quite a ton of functionality to code in two days. Serverlist, custom feeds, removing shared dirs (implemented today partially, needs fixing), and some smaller things. Doable, but quite heavy still.
No screenshots of the new skin today yet - it's nearing 8am and I want to get to sleep after the 23h dev-session :)
Tuesday, April 25, 2006
Categories and file names/comments box
The trouble with implementing categories was that I didn't really want to implement them on core-side, to keep the hncore library flexible; besides, categories as a concept tends to be more of a gui-side feature anyway. So the problem was how to keep track of the download/shared files across sessions; generally, we use Object class identifiers for that, but those are not saved anywhere. The solution was to add a customdata
field to MetaData (each download and shared file is tied to one MetaData object), which stores those stable identifiers
, as I like to call them. The user interface stores a list of categories, and file identifiers in it's config file, and is thus capable of re-creating categories on either core or gui restarts. Categories are thus per-user (incase of multi-user setups), so each user connecting to the core can have it's own category settings without affecting other people's category setup. At least that's the theory.
Another thing that got implemented today was file names/comments area in the interface. The cgcomm side support for this was already implemented few days ago, which meant I only had to implement the gui side of it. So to avoid lengthy description of the thing, here's a screenshot.
At the download area you can also see the categories feature - items marked with the blue PLUS sign are categories and open when you click on them. Torrents (and eMulecollections, in the future) behave exactly the same, so from both visual and functional point of view, there's no difference between a category or a torrent.
The file names/comments area opens from either "File Names" button in details box, or from "Show Comments" entry in context menu. Default is turned off, of course.
On other news, you can now connect to Hydranode irc server via irc.hydranode.com
. The channel is the same, #hydranode. The new domain points to the same irc server as before, but if you had problems connecting earlier to the irc.respectp2p.org address, they are fixed now (there's a non-working DNS entry for irc.respectp2p.org which broke some irc clients).
Monday, April 24, 2006
Multi-tracker support; fixed toms-hardware feed
The main attraction today is full multi-tracker support for bittorrent module. This means the 'announce-list' field in torrent files, which can contain any number of additional trackers. As far as I understand the protocol, clients are supposed to connect to all of the known trackers, so that's what hydranode is doing now. What made implementing this take whole 4 hours was that I had to completely reorganize the Torrent class, and introduce new Tracker class into bittorrent module to handle this properly. Anyway, this should considerably improve bittorrent usage experience, since there are more trackers from which to get sources from now.
Additional fixes in bittorrent module were done as well, such as properly emitting PD_DL_COMPLETE event upon torrent completition, fixing few more crashes during torrent completition (a TODO list entry), and fixing up/down ratio calculation (the calculation was done at integer precision, but the expected result was floating-point value). Also fixed 'known torrents' handling when restarting the app before any files of the torrent were downloaded - the previous behaviour completely forgot the finished files of the torrent, which meant they were never seeded anymore. The bug came from a precondition that checked for the torrent's incoming directory's existance during startup, but that directory isn't created until after the first file of the torrent is completed (created-on-demand).
However, this isn't the end of the known torrents db problems - the thing was a quickhack a while ago, and the format isn't extendable, and now I need to store more stuff in there (remember uploaded/downloaded values over sessions), which means I basically have to kill the old format and start fresh, but that can potentially break upgrade path to 0.3 with existing torrent downloads/seeds. Unfortunately, currently I don't see any smooth upgrade paths, except for using a completely new file, but the 'known.dat' name is good for this stuff so ...
On the GUI side, I decided that the third line in the transfer details box (currently containing 'Artist', 'Album' etc fields - data that we won't have before 0.4, if even then), will display 'file status' information, which can be set by modules. E.g. bittorrent could set tracker connection information there. Hopefully I have time to implement it for 0.3, but it's gonna be close.
Additionally, I did some more fixes on GUI side regarding the 'column hiding' code implemented few days ago; it should behave properly now (some columns were restored at 10px widths before).
Sunday, April 23, 2006
FindServers option; Remote torrent downloads
6 days to release, and working down the TODO list. One thing that I forgot to mention in last blog post was the new 'FindServers' configuration option (in 'ed2k' section), which defaults to '1', but can be set to '0' to disable receiving servers from other servers and clients. This might be useful if you have your own server.met and don't want to have Hydranode filling it with fake
servers all the time. Thanks go to dani_555 for requesting this feature.
A lot of time today went to getting 'double-click to open file' working properly from the interface. While it had partially worked for a while already, now it was time to get it working fully. There were a number of problems regarding opening completed downloads - the last update of the file that was sent to GUI still had the old location field set, and we couldn't modify it at hncore library side, since the PartData internally uses it to clean up it's mess on destruction. Instead, I opted to update the 'destination' field of PartData after the file is moved to incoming, and conditionally send that field as 'location' to user interface when the file is in 'completed' state.
That didn't fully resolve the issues though yet, since often core uses relative paths (the default incoming dir is config\incoming, for example); that didn't help the user interface much, since the gui can run at a different physical location than the core. The solution was to send all paths as absolute paths to the user interface, to ensure that the gui finds the files properly. Now it's possible to double-click on 'completed' files in download list to open them with the default application; same applies in library page, altough incomplete downloads (currently also shown on the library page) won't open properly, unless you have an application set to handle .tmp files (personally I use mplayer for that...). In the future, I hope to add an option to download & install mplayer with one click when attempting to preview temp files; alternative external players could be used upon user's decision as well.
Another thing that took roughly 3 hours was getting 'start torrent downloads from gui' feature implemented. This required changes at almost all levels of the Hydranode system - proper function at hncore/search.h, handler/changes in bittorrent module, support for the packets in cgcomm module and library, and finally ofcourse gui-side implementation. In the final implementation, the user interface opens up the requested file, loads all it's contents to memory and sends it to core over cgcomm. Core (cgcomm module) reads the data into memory, passes it to Search::downloadFile signal, to which Bittorrent module has a slot connected. Bittorrent module puts the std::string-contained data into std::istringstream and passes it to a slightly altered createTorrent() method, which then continues normally. When Bittorrent module loads torrents from local file, it merely passes the std::ifstream object to the same method.
Yet another thing that I spent several hours on was tracking, testing and debugging the notorious file allocation freezes on windows (and fat32/ntfs partitions in general). After extensive testing, it finally turned out that while I was seeking through the file, writing \0 characters (either one at the end, or one every 10mb's), that didn't cause proper space allocation at all, and the actual allocation was done still when PartData attempted to write data to somewhere in the middle of the file, causing the main thread to block. The thing is that Windows uses \0 character to fill the empty space of the file itself, and I was writing \0's as well, which windows simply ignored. As soon as I changed it to write '1' character instead, it allocated properly.
It should be noted as well that Hydranode uses single-go allocation now, which means the file is initially created at 0 bytes, and before the first buffer flush, it's resized to it's final size, which allows the operating system to optimize the file position on disk and reduce fragmentation. While it's impossible to truly avoid fragmentation when dealing with files, using this scheme, the operating system should place the file in a free continous block if it can find one, resulting in minimum amount of disk fragmentation.
Friday, April 21, 2006
Filename cleanup; columns hiding; runtime ports updating
9 days to release, and several items got checked from my TODO list.
Changes to listening ports (ed2k and bittorrent) now take effect on runtime. Simply type 'config set ed2k/TCP\\ Port 4662' in hnshell and the ed2k TCP listener will be restarted on the specified port. Reconnecting to server is forced in this situation to detect our new ID (high or low). This functionality will be available from GUI as well eventually.
Core/GUI communication module and library now have support for some more search fields, such as bitrate, codec and length. Also, additional messages were added to download subsystem, such as GetNames, GetLinks, GetComments, and SetName and SetDestination. GUI also supports now file name cleanup and changing file destinations (note that in Hydranode, it's possible to set destination per file, rather than per category, as in eMule). GUI also features details box now on Search page, and both details boxes were reduced to three lines of text to save screen space.
MSVC builds of user interface should work now, as I reverted changeset 2856 (support for static builds - but it never worked).
User interface now also supports hiding/showing list columns, and saving column widths. The entire column resizing code was reorganized and should be less intrusive now. Hidden/Shown columns, as well as column widths, are now saved across sessions as well. All this functionality is accessible via right-click menu on the list header, similar to many other applications. I also added 'Restore Defaults' option to the menu, just in case.
Wednesday, April 19, 2006
10 days to release - TODO list
It's been super-busy 10 days since the last blog post, during which everything else had higher priority than this blog; for that, I apologize. The next ten days leading up to the 0.3 final release on 28th will be even busier, since I'v got a months load of work to be done in under two weeks. However, due to numerous requests, I'll try to keep this blog updated at least semi-daily as the release draws nearer.
So what's happened since the last post? One thing that took several days was Y2006 hardware upgrade; tons of organizing, data transfers and testing. For those interested, my new dev-box is Asus A6J laptop
with 2GB ram and 1.83ghz dual-core CPU. Gotta love those ~5-min compile-times (down from ~8 min).
On the code side, I have a pretty hefty TODO list for this release, as you probably can imagine if you've seen the beta1 version. So far, I've implemented downloads renaming / file name cleanups / incoming dir changes into the cgcomm protocol; media info (codec/bitrate) support for search results and details box to search page. Skinning code got quite a bit of optimizations as well - the primary hog there was images scaling, which was done during every repaint; now we're storing the scaled image and only re-scaling if the size changes.
My complete TODO list currently is as follows:
- Optionally disable receiving servers from other servers (req by dani_555)
- eDonkey serverlist management from user interface (requires cgcomm support)
- Torrent pausing/stopping problems
- Torrent completed/canceled crashes (might be already fixed in my local copy)
- List columns hiding/showing; improved list columns resizing management
- Memory leak @ category creation box
- Remote connection interface
- Donate button and donate page (website)
- Custom RSS feeds interface; caching RSS feeds between sessions
- Removing shared directories (under right-click from library->dir drop-down menu)
- Saving category settings (problematic - can only do it on GUI side, but hard to keep status over sessions due to lack of stable identifiers of objects)
- Interface for modifying listening ports; runtime rebinding of listening ports (core side)
- Tray icon on windows
- Start with windows startup; command-line options for disabling splashscreen and minimizing after startup
- File names / comments window / interface
- (maybe) investigate UPnP support (got the code, just not in use currently)
Arlekin (the designer) has quite a lot to do during that time as well, and some things we must syncronize together. The server list and file names pages / windows need cooperation since they involve design; a new skin is in development ("prepare to throw one away; you will, anyhow"); some 8 file type icons (48x48px) need to be created; some other icons need fine-tuning; and probably other things that I can't remember currently.
As you can see, I wasn't joking about the month's work for 10 days thingie.
Stay tuned for more news in the coming days.
Monday, April 10, 2006
Post-(beta-)release wind-down / wind-up period
Having done software development for many years already, one thing that doesn't seem to be getting easier is releases. I'v done regular releases for like 3-4 years now on different projects, and it's always one hellowa week of very little sleep and much coding leading up to the release. And it also doesn't seem to matter whether it's a small beta release or a widely-published full release - the effects are the same - 2-3 days wind-down period, and another 2-3 days to get back to development mode.
Despite my efforts over last two months to stay far away from any changes that might compromise the stability of the core, it seems I have introduced a crash-bug into core during that time that crashes after 15-20 hours runtime, regularly. So far, my only lead is that the cause is a timed callback to a non-trackable slot, which gets deleted before the callback is invoked, leading to the crash. So far, no solution has been found.
Based on the feedback on the GUI I'v gotten from testers and other people, some changes / fixes will be done to the GUI; mostly on the usability side. One topic that was raised and not fully resolved yet is the library. Technically, the library as a concept, is a mix of two concepts - 'share management' and 'upload management'. Now, in case of 'share management', the focus would be on shared files, directories, adding/removing them etc. In case of 'upload management', I'm not quite sure what an uploader would like to see there, since I don't have personal experience with that.
Another topic that was raised was what should be the next target after I return from my vacation in June. While I originally announced Kademlia development for the summer, recently I'v started to reconsider that choice, and lean more towards Gnutella. The idea is that Kademlia doesn't truly give anything new to Hydranode - just more stable ed2k operation - while Gnutella would add another major network - this time for small files - to the mix, which would complement the other two implemented networks very nicely. You'd have the resources of ed2k, the speed of BT and the speed and easy-to-get-mp3z of Gnutella. Supplemental networks, such as BT/DHT and Kademlia could be implemented at a later time.
Wednesday, April 05, 2006
Hydranode 0.3 beta 1 released
After the long wait, Hydranode 0.3 beta 1 is now finally available for downloading. The major new features for this version are graphical user interface and global searching support. There have been very little core updates, since main focus was on user interface development.
Remember that this is a beta version, not a release candidate, thus many features are half-implemented. This version is provided for your testing and previewing purposes, altough it should be viable to use it in production system. Also, it hasn't been optimized for performance yet, so expect higher CPU usage and sluggyness on <1ghz systems.Currently existing features:
- RSS feeds from major news sites
- Global searching; searching by type, filtering by min/max size
- Transfer/Library page filtering by keywords and type
- Initial support for categories (incomplete)
- Adding downloads from clipboard (links); importing eMule downloads
- Link / file associations from browser / explorer (ed2k, torrent)
- Nearly fully skinned interface (optional)
Things that are missing / planned:
- List columns hiding/showing
- Extended file information in details boxes (also on Search and Library pages)
- Port configurations on Settings page
- Built-in help support (probably won't make it to 0.3 final either)
- Custom user-defined RSS feeds
- File name cleanup; downloads destination changing
- Graphical frontend for connecting to remote core
- Adding torrent files directly from interface (Transfer - Add - From File)
Windows (includes both core and interface)
Linux (core only)
Linux (user interface only) experimental - tested on Ubuntu 6.04 beta
Source code as .zip, .bz2 and .gz
Credits for the user interface graphics and design go to the "invisible designer", Joakim Kuusemaa
.Note: remote cores are supported by the user interface. To connect to a remotely running core, pass the ip and port as arguments to the interface command-line. Easiest way to do it is create a shortcut and change the path to "hydranode-gui.exe 192.168.0.1 9990" for example. Make sure the remote core has cgcomm module loaded and can accept incoming connections on port 9990 (ListenIP setting in [cgcomm] section of config.ini must be modified for this, same as with hnsh). By default, connections are accepted only from localhost.Note to Linux users: You might want to pass -style plastique option to the interface to avoid the ugly win95-look (Qt4-related problem).
Madcat.HOTFIX: If you had problems starting downloads with Hydranode, please re-download Hydranode from the above link - the problem was fixed.
Tuesday, April 04, 2006
One of the promises I made when I started this project was that it would be usable for everyone. At the time, I did not quite know what that meant, but then again, neither could I do template metaprogramming back then. Today, I feel I have a understanding of what are the three main user-groups Hydranode targets, and how to reach them.
The trouble with dividing people into user-groups is that there is always a margin of error; there are always people who fall into multiple categories, or none; every individual is unique, and such generalizations can only serve as general guideline, rather than hard rules. You can rarely find a perfect example of a user-group in a real world, most people have characteristics from multiple groups.
Group A is a computer enthusiast; he can use right mouse button; likes to configure and fine-tune each of his applications to his liking; has used computers since childhood; has mastered at least three scripting languages and is a fan of regular expressions; generally dislikes skinned / non-native applications and applications that try to do things for him. A Linux enthusiast is an example of this user-group.
Group B has used computers for quite a while and finds his way around things. When forced, is capable of figuring things out by himself, altough he prefers asking his son/nephew for advice on computer-related topics; can use right mouse click when needed, altough doesn't usually enjoy it. A middle-aged, male, casual p2p-user is an example of this user-group.
Group C likes beautiful shiny applications; doesn't like applications which display lot of things she doesn't understand; generally wants things just to work without needing any extra input from her; "Settings" and "Options" pages in programs are a complete mystery to her. A 16-year-old girl is an example of this user-group.
So how am I going to target all these three user-groups? Configurability and carefully-tuned default settings. The idea is that in order to target group C, the application must look and feel exactly according to their idea of a good application (skinned, simple); however, as they cannot configure the application, merely providing skinning capability and means to hide extra information is not an option - only users in group B and A can do that. Hence, the default settings of the application must be fully skinned and all extra information hidden, or explained in-place with a short phrase / sentence. Users in group B can enable sufficient amount of things in order to feel comfortable (including disabling skinning, if they don't like it). Users in group A are computer specialists, so they can find even the most obscurely hidden settings and configuration options, enable all extra information displaying and disable all skinning.
The data / options hiding follows the same logic. By default, anything user in group C wouldn't understand is hidden away. Things that users in group B could potentially want to use are easily accessible, from menus, buttons and such. Things that users in group A would potentially want to use are harder to reach, generally under right mouse click, or deeper into settings dialogs or even changeable only from configuration files.
September 2006 Current Posts