MLDonkey Downloads Import Module Development
Development in progress.
Friday, March 31, 2006
More GUI design and updates
The lack of posts during past few days is rather an indication of me being too pre-occupied with other things rather than lack of development. In fact, we did a three-night extreme programming
session with our designer, with him on Photoshop, me in code and discussions around gui design. A lot of things got cleared up, reviewed and lot of progress made in many areas; some of which I can show now, some of which has to wait a few more days before it can be displayed publically. The GUI codebase is nearing 5000 lines of code already (+1000 lines with three nights).
So here's a screenshot of the current version; explanations / discussion follows.
One of the main things that got changed was the dropping of tabs from all pages of the interface. The tabs were actually an implementation artifact, and never existed in any of our sketches. The functionality that they were intended to represent - filtering / categories / in-page navigation - will be implemented differently or dropped completely, where applicaple.
For example, the concept of 'categories' in transfer / library will be implemented in a tree-like way, pretty much how currently torrents work in the interface - you click on the + sign in front of the name, and the list of files in the torrent opens up, allowing you to manipulate each file of the torrent separately, as you'd do with any other download. Torrent is technically a category of downloads. Same applies to emulecollection (currently not supported by core). User-defined categories will behave the same way - the 'category' object in the list merely becomes a 'virtual download', just like a 'torrent' is a virtual download, composed of multiple 'actual' downloads. You'll be able to drag & drop files into categories. Whether or not you'll be able to drop any kind of files into torrents (or other core-created categories) is being discussed, but from technical point of view, there's no reason why you shouldn't. You start downloading a movie torrent of two files, and then find subtitles (from some other network) for it - logically, you could drop the subtitle downloads into the torrent category, so that they'd be moved to the same destination dir once downloaded, as the movie itself.
Another thing that was added was the details box to the bottom of the window, which will display extended information as well as provide access to some additional data - such as known file names, comments and so on. The details-box can be hidden to free up more space in the interface. It will be available on all pages, displaying relevant content about the page items.
To the right side of the window, we'v planned a help bar, which is 'hide/show' as well (from the ? button at top-right). We have some ideas on what the help should contain, but the actual content is still rather open, since other things have higher priority currently. It's not expected to have the help functionality operational by 0.3 Beta1 (expected to be available next week).
Monday, March 27, 2006
Progress bars and such
Added availability-o-meter and progress-bar to search and transfer pages. The text color should probably be changed to white, and/or the bar colors adjusted, since currently the text isn't very readable (black text on partially-black bar). Also finally wrote the backend to the search controls (type, min/max size). On search page, results with no complete sources are displayed with yellow bar; on transfer page, part of the bar becomes red when part of the file is completely unavailable.
Saturday, March 25, 2006
Busy with real life; compiler troubles
Been super-busy with real-life things the entire week, which, unfortunately, has left little or no time for Hydranode development.
One of the main problems I'm facing related to Hydranode right now is compiler(s). As reported earlier, VS2005 has serious memory leakage bug in iostreams library, and rebuilding the library, while fixing the problem, is only a partial solution, since it introduces several problems of it's own related to development and packaging. MinGW doesn't have that leak, and seems to produce lowest-memory-usage version of Hydranode, but has considerably longer compile/link times, as well as nearly twice as large final binaries. Currently I'm testing VS2003, which produces smallest final binaries (I don't fully know yet, but it appears I don't have to ship runtime dll's when compiling with VS2003, since at least XP has those DLL's installed by default), but it also seems to have at least some memory leakage (hydranode uses 50-60mb memory instead of the expected <30mb). I'd really love to use VS2003 though, since ~3mb final installer size (including user interface and qt dlls) is pretty appealing, compared to ~5mb with VS2005 and ~7-8mb with MinGW.
Another topic has been various progress/availability-o-meters in user interface. I have the bars from the designer but haven't gotten around to implement them. Basically, we'll have availability-o-meter (both in search and transfer), and progress-o-meter. The progress-meter is planned to display only progress, not detailed chunk status, as in emule, and most probably will be composed of three differently-colored parts - green for completed, blue for available, but not downloaded, and red for unavailable. Extended chunk-status-meter might be available from download details section (not implemented yet). On the progress-bar, we'll display the progress percentage, and I'm also considering merging the 'completed' column into the same progress column as well, to save space. So the text on the bar would become '78% / 467 MB' for example.
Monday, March 20, 2006
As mentioned several times before, the top-right corner of the My Hydranode box shall become a small statistics box. So without further wait, here's a screenshot.
The graph uses dynamic scaling based on various factors, and only download-rate is displayed. We initially had both download and uploadrate there, and were planning to even add connection-count, but we suddenly realized that that data is redundant. Upload-rate is a stable value (if it's not, it's a programming error), and thus isn't justified to be displayed in a graph, which's entire purpose is to show variable data. As for connection-count, that data gives nearly no useful information to the user.
On the transfer page, we decided to drop the top-right corner buttons idea for the time being. Testers indicated that they became too small and too far away, and we tend to agree. Having a full 22px-height bar for such controls has other bonuses as well, such as the space and ability to add various custom controls (such as the filter-box for example).
A lot of people seem to be trying to compile the code from SVN. I repeat - the gui isn't ready/available for public testing before beginning of april
. That means support for compilation-related questions is minimum - I didn't check in the code to spend time explaining everyone (or writing up a lengthy instruction) on how to compile it. As I said in last blog post, it's in svn because it's more convenient for me to work on it that way, NOT for public testing.
Sunday, March 19, 2006
GUI code in svn
Experience has shown at once corebase starts to reach 4000 lines of code, it becomes too large to maintain locally, and version control system is required. This becomes even further important when doing cross-platform development, which involves copying codebase around different systems. Thus, I checked in the current GUI codebase to svn yesterday; not because it's ready for public testing, but because it's easier for me to maintain it that way. Nonetheless, if you're feeling adventorous, you can check out the codebase and compile it yourself. Or you could just browse it
Today I also made the first full bundle of Hydranode and submitted to the testing team. The final rar package turned out to be roughly 6 MB of size, but it will decrease slightly, since in this package, the engine was compiled with mingw (~50% larger binaries on average). That version runs Hydranode as child process, eliminating the console window completely. Also, I introduced the top-right-corner statistics panel, as discussed before; no screenshot yet though, since it's merely for testing purposes only, and doesn't look too pretty.
Overall, we seem to be on schedule, with less than two weeks left before public testing, and most functionality is in place. Main focus over the next weeks will be improving the existing functionality, such as getting the RSS support user-customizable, adding progress graphs, re-doing graphics and so on. As I mentioned in a comment of previous post, the icons/graphics are in their first incarnation, and those that have followed this project longer know that no component stays in it's first version - nearly all important components of Hydranode have been rewritten at least once, since you don't truly understand the problem until the first time you implement a solution; on the second time, you can hope to get it right.
Friday, March 17, 2006
New builds, networking statistics support in cgcomm, and more
Interestingly, postings which include screenshots seem to gather several times more comments than those without. So here's another screenshot for you screenshot-hungry people :)
As discussed before, the top-right corner will become statistics graph (hopefully should have implementation of that by designer within few days). The bottom part will display module info/overview, but the backend is currently missing, since the time went to implement the network-statistics related core/gui comm protocol parts.
Statistics-fans shouldn't be alarmed about the small space the statistics got in the interface. Thanks to the tab-based approach, we can introduce a separate tab on the Home page which opens on-demand by double-clicking on the graph, or some button, and can display pagefuls of statistics data. This approach avoids cluttering the interface with data the average user isn't interested in, but provides the data for the "mythical power user"
Due to the VS2005 iostreams library memory leak issue, previously mentioned here
, the r2805 win32 builds are built with mingw compiler. We'll have to see what to do with the VS2005 problems; one option would be to use STLPort library instead of the VS2005 STL implementation; other would be to patch the issue and rebuild the dlls; frankly, I don't like either approach, but it seems there are no other options.
I've been experimenting somewhat with a layout change, which will affect transfer and library pages. Namely, I'm trying to get rid of the 'action toolbar' (clear/pause/resume/stop buttons), and move those buttons to top-right corner, on same row as the tabs. This has several reasons - gain of ~20px in height (less scrolling and cleaner look), move less-used functionality away from primary location (most p2p users are more accustomed to using right-click menu for those operations), and balance the interface visually (left/right). Initially, it seemed this would require a lot of manual layouting, including taking apart the QTabWidgets and recomposing them manually from QTabBar and whatever they use additionally. However, it turned out QTabWidget has a function, setCornerWidget(QWidget *w)
, which allows doing exactly what we need. The trouble began when I attempted to put there a larger widget (actually, composite widget) - namely, the space seemed to be limited to ~50px width. So far my attempts of tweaking it have been only partially successful, as can be seen from screenshots below.
New builds from r2805 are up for Windows and Linux, as well as source snapshot in three formats.
Tuesday, March 14, 2006
Type icons and home page
The weekend went mostly to designing and implementing file type icons for the interface. They are currently mainly black&white, further coloring and tuning will be applied later. We figured we'll do the first full set of icons as they come out now, and re-work them on the second round.
Another thing is the home page. Basically, the layout and concept was discussed again and it seems to be holding. However, we won't be having p2p news on the home page; instead, we'r considering loading some movie and/or games news, or some similar mixed-content news there. The rationale is that several testers commented that they wouldn't want to see bad news every time they open up the app, and let's face it, 95% of any kind of generic news is bad. For example, look at what happens if we load Slyck feed by default (also, here's a first screenshot of the home page for you - the empty area on the right will become Hydranode overview, such as transfer rates, small speed graph and module status information - not implemented yet).
The bottom-left box will most likely become multi-purpose (for those not interested in hydranode development news), capable of displaying recently downloaded files list, hydranode log or such.
Thursday, March 09, 2006
RSS reading capabilities
I finally started to implement the "Home" page of the interface today, based on my last reference design. While it hasn't been reviewed and approved by our designer yet, I figured I can at least write up the rss-parsing code and test my skills at Qt painting APIs.
The hardest part was to get the background picture properly displayed on the home page. On all other pages, the backgrounds are drawn on QTreeWidget, which is a single widget, so merely overriding paintEvent() did it. However, on the "Home" page there are multiple widgets (some QHtmlBrowsers, some panels, labels and such), some of which draw their own background, so painting on the lowest-level widget doesn't work, since the child widgets draw on top of the parent.
After numerous attempts and surfing around the net and Qt manual, the solution turned out to be rather simple ("everything is simple, once someone has told you how"):
- Set the background color role in child widget's palette to transparent.
- Install event filter for child widgets which draw their background.
- When the child widget draws it's background, repaint the parent widgets background on top of the child widgets background (remember - the background image is painted at ~30% transparency level).
- Let the child widget paint itself now.
This results in child widget's data being drawn clearly, but their background becomes transparent, showing the background image beneath.
Once that was in place, I used QXmlSimpleReader and QXmlDefaultHandler-derived parser to implement a simple rss reader. As it turns out, every rss feed I got my hands on has slightly different format and completely different date/time formatting system - I implemented support for three (this blog, RespectP2P and Slyck), and had to use different code for each of them. Oh well.
Since custom feeds will be possible in the future as well, I already implemented the stuff as generic classes, so we can easily add user-customizability later on.
Wednesday, March 08, 2006
Been rather slow development past few days due to real life responsibilities taking priority. Perhaps the most notable change was proper support for global searching in cgcomm protocol level, as well as in user interface.
A lot of thought has been given to the front page of the user interface, which is still missing. Basically, so far the plans for it include a statistics/overview-related information bar (~100px wide) on the right side, where there'll be a small statistics graph, plugin status(es) and current/overall transfer statistics. On the left side, there is planned RSS feed from yet-to-be-determined websites. One small block of the front page will also be dedicated to Hydranode-related news feed (probably the developer's diary, at least initially).
I was also playing around a bit with Qt skinning (styling, as Qt calls it) support, and the results have been very positive. With just a few lines of code, I was able to add support for Plastique, Windows and WindowsXP theme under WindowsXP; the theme changes happen instantly, there's no delays whatsoever.
A user asked in previous blog post comment whether there'll be OSX support in next release, so I did a test on my Mac. Took 10.5 hours to compile Qt, plus another 1.5 hours to compile Hydranode, the latter of which failed due to some build system issues. At that point, I gave up for the time being - those compile times aren't really workable.
On Windows, I was wondering why on earth are we leaking ~100mb/day memory during heavy usage. If you remember, I mentioned a memory leak in VS2005 C++ runtime library, however I believed that we had worked around it. It seems there are still some cases which aren't handled. In any case, recompiling Hydranode with mingw on Windows resulted in ~30mb memory usage after 24 hours runtime and nearly two times lower CPU usage (note: CPU usage is somewhat related to memory usage - higher memory usage directly results in higher CPU usage). If you remember some of my past posts when I switched to VS2005 from Mingw, I documented lowered memory and CPU usage, so VS still produces faster code, but it could really use SP1 (and we can't use VS2003, since there's no free version available - VS2003 Toolkit lacks some important components).
Monday, March 06, 2006
Timeline for next 10 months
Based on my past experience with user interface development for a P2P client (remember ShareDaemon?), I estimated Hydranode user interface development time to three months. Some persons close to the project's development indicated that it should be possible to do it in two months, and three months is too much, but I stand by my original assessment. Last 20 month of development has proved that my working speed is around +3000 lines of code/month (note: that's additional code, all maintenance coding doesn't show up in the +lines/month rating really). Considering that ShareDaemon interface was ~15'000 lines of code, for Hydranode user interface I've chosen more advanced toolkit which reduces both development time and code size, I estimated the initial code size of Hydranode interface <8'000 lines of code, which equals under three months of development.
Today, after roughly a month of development, we'r looking at 2506 lines of c++ code plus 3333 lines of layout XML, plus quite a lot of work on core-side and protocol-side. With this data, I was able to prognoze and set next ten month's development timeline in place, and will present it as it is currently planned.
- March 2006 - user interface closed testing and feedback integration
- April 2006 - user interface public testing and feedback integration
- April 30th, 2006 - Hydranode 0.3.0 launch with user interface and new website
- May 2006 - Vacation for me; designer will be designing first skin for user interface
- June-August 2006 - Integration of user feedback into the user interface; skinning engine and initial skin for user interface; Kademlia network support
- September 5th, 2006 - Hydranode 0.4.0 launch with skinning engine and Kademlia
- October-December 2006 - Gnutella plugin development; maintenance and bugfixing for existing codebase (at this point exceeding 100'000 lines of code already)
- Christmas 2006 - Hydranode 0.5.0 launch with Gnutella plugin and new website
As you can see, we're scheduling roughly four-month development cycles between releases, and all major releases include some major features. You can also notice that I increased my vacation time (twice a year) from two weeks to full month - it has become apparent that two week vacation every six months just doesn't cut it, and considering that I work without weekends or public holidays, I believe it's justified to take two months a year off. Furthermore, some time off the project is required to keep the project floating financially, as well as to give me some perspective on the project as a whole - you can't do the latter when you're deep in development mode.
Gnutella plugin was chosen as the next major network component (besides Kademlia, which is merely support component) since it gives something that Hydranode lacks at this point in time - fast single small file downloads - neither eDonkey2000 nor Bittorrent provide that.
The release dates (beginning of September and Christmas) were also chosen for publicity reasons - people return from summer vacations at the beginning of September, and Christmas is also a very good time for a release.
Note: Actual release dates for 0.3 and 0.4 may vary slightly depending on different factors - to either earlier nor later; they are provided in this timeline as general guidelines, rather than hard requirements.
Saturday, March 04, 2006
Lots of bugfixes and several new features
Been pretty active development during last ~36 hours; here's the annotated list of changes.
- Fixed couple of race conditions in networking scheduler that happened due to underlying socket destruction being a delayed process, so events could still arrive while we had deleted all frontend data, which caused crashes / assertion failures.
- eDonkey2000 parser is now safer, and does more strict protocol / data checking. As it turned out, it was possible to send malicious data that caused Hydranode to attempt to allocate 4gb of memory (and get killed by the OS kernel as a result).
- Recusive usage of WorkThread::Pauser object works properly now, the thread is resumed when last Pauser is destroyed. This shold speed up startup times when hashing is requested during startup process, since hashing is now properly delayed until startup has finished.
- Faster disk space allocation on Windows.
- Added 'alloc' command to hnsh which can be used to explicitly request disk space allocation for a download (which hasn't auto-allocated space for itself yet).
- Fixed crashes when search result handler objects are destroyed (e.g. closing hnsh or disconnecting UI during global searching).
- Properly updates file's modification date in MetaDb after move to incoming is finished. No idea how this got missed before, but it was causing a lot of needless rehashing and duplicate entries in metadb.
- It's now possible to pass full paths to files (e.g. .torrents and such) to hlink application - as it turned out I had to double-escape all backslashes, so the resulting code looked like this: boost::algorithm::replace(string, "\\", "\\\\\\\\"). Fun eh? :)
- TCP server source requests are now split into 15-file chunks, ordered by rarity (rarest first), and sent with 4-minute intervals (up to a maximum of 75 files per 20 minutes). This is required by eDonkey2000 netiquette, and keeps your server credits reasonable; furthermore, servers don't respond to more than 15 source requests at a time in any case, so this considerably improves source aquisition from local server.
- UDP server source requests are now done based on file's rarity, the rarest 20-30 files are asked, rather than random rotation. This improves UDP queries usefullness, since it considerably raises the chances of getting few sources for file(s) we don't have any sources for, and then SourceExchange can kick in.
- Handles tracker responses that use \n newline instead of the standard \r\n.
- Closed/fixed bugtracker tickets 188, 228, 196, 229, 199, 213 235 and 215.
- CGComm listener can now be bound to custom ip address, just like hnsh. Configuration value /cgcomm/ListenIp, argument dot-separated ip address.
- Added experimental code for 'known torrent db' handling - namely, until now if the downloaded torrent contained subdirs, after restart we "lost" those files, since those directories were not shared. This torrentdb thingie scans those directories and loads the shared files from there during module init. It's not as robust as I'd like yet, so it'll most likely see a second incarnation soon, but for now it'll do.
- Cleans up torrent cache folder on download completition and download canceling.
- PartialTorrent and TorrentFile objects reparent their children to null when being destroyed, since they don't actually own their children, but the Object hierarhy logic dictates otherwise. This fixes a bunch of crashes during torrent completition/canceling.
- PartialTorrent and TorrentFile properly emit PD_ADDED and SF_ADDED events, respectivly, when constructed.
- HTTP module properly urlencodes requested file names.
The user interface has gotten several improvements as well. The biggest and most time-consuming was link/file associations. After patching QT library in order to access HKEY_CLASSES_ROOT section of the registry using QSettings API and thus avoid writing low-level win32 code, there are now two checkboxes in settings page which associate hlink (the link forwarder app) with ed2k and torrent links/files. No idea yet how to do the same on Linux, since there's no standard way of doing it there.
Friday, March 03, 2006
GUI screenshot on Linux with icons/background
As promised yesterday, here's a screenshot of current version of the GUI, with prelimiary icons and background image. Apparently I had taken the wrong icons from the photoshop file (hey, I got lost between those 80+ layers in the file), the bewel & emboss effect wasn't supposed to be there. Also, the background image is merely a proof-of-concept and full set of backgrounds (one for each page) is in the making. However, this should show you the general direction where we'r going with it.
I also realized that I haven't posted any screenshots of the GUI on Linux platform, so this time, it's done on Ubuntu 6.04 Beta, running Gnome 2.13.92, Qt 4.1.1 with Plastique theme. Enjoy :)
Thursday, March 02, 2006
Icons, backgrounds and global searching
Yesterday and large part of tonight was spent developing icons and backgrounds for the GUI. I can't show screenshot yet due to some technical difficulties, but things are looking rather promising. Expect the missing screenshot within few days.
On core-side, I implemented global searching capabilities tonight. Total devtime ~6 hours. The diff can be viewed here
, if anyone is interested. Enjoy, and let me know if if there are any bugs in it.
September 2006 Current Posts