Alo Sarv
lead developer

Donate via

Latest Builds

version 0.3
tar.gz tar.bz2
Boost 1.33.1 Headers
MLDonkey Downloads Import Module Development
Payment completed
Development in progress.
Developer's Diary

Thursday, January 27, 2005

More design @ PartData

Can't say I'v made much progress tonight on the PartData stuff, but I am beginning to understand the problems more clearly. For one thing, I realized I need to visualize the internal PartData systems to get a better overview of how things are supposed to run - resulting in this UML diagram. It's still only work-in-progress though ..

While the getRange() concept (e.g. SmartChunkSelector(tm)) is working, the ranges locking mechanism is still somewhat gray area. I originally intended to have the locking handled within UsedRange, and later in Chunk (after I realized I need to allow multiple UsedRange objects per Chunk - originally planned to use shared-ownership single-UsedRange object), however, I realized it'd break as soon as multi-net downloads with different chunksizes are involved, so the lock handling needs to be done in PartData object instead.

How exactly should that be implemented is not clear yet though - basically we'r back at the multi-rangelist-lookup issue that I was trying to avoid during this design (it was the central part of original PartData design), since it's error-prone...

After the locking system, we still need internal data buffers, chunks verifying (easy), and corruption recovery (tricky). There are some interesting aspects in corruption recovery that are somewhat tricky to handle - namely, when we have a checksum fail for a range, we must check if we have higher-resolution hashes for the data, and check those ... after that, we can mark the rest of the range as corrupt, and cry out for help (post notification event which plugins can handle). Upon that, plugins-to-the-rescue (if capable) - submit even higher-res hashes for checking (ed2k AICH for example). So that'll also take some time to figure out.

What's good news tho is that PartData (along with some support code around hasher) is the last of the engine subsystem upgrades series, which started out mid-december. So, once this is done, we can get back to, say, more "real" problems, like network communications and so on.

ETA for PartData completition? 4-5 days ? Smth like that, if all goes well.

Madcat, ZzZz

This is maybe off topic...

Have you looked at EBML? It's sort of like Binary XML. I know you were planning on using XML, but finally dropped the idea. Maybe this is useless now, but you never know... it might be useful in the future.

I was kind of sad to see Sharedaemon go. I'm really glad you started a new project and I wish you the best of luck.
No XML (not even in binary form) can compete with raw data in terms of speed and size. And we definately need best possible speed and lowest possible bandwidth usage for Core/GUI communication (former for loop-back connections, latter for remote connections).

Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?