Hydranode Developer's Diary - Developer's Daily Diary: Status update

Alo Sarv
lead developer

Donate via
MoneyBookers

Latest Builds

version 0.3

tar.gz tar.bz2 Boost 1.33.1 Headers

MLDonkey Downloads Import Module Development

Payment completed
Development in progress.

Developer's Diary

irc.hydranode.com/#hydranode

Wednesday, January 11, 2006

Status update

Been busy with the side-project I mentioned in previous post - seems I managed to convince the client to go with only php+mysql-based solution and avoid Access completely - I'm not really a big fan of VBA. Will hear final feedback on friday/monday, hopefully they'r satisfied. Contract-work seems really so much easier than OSS project management/coding - you have your deadline, expected result, and free schedule, and a LOT less work than that of an OSS project - not to mention getting payed. Developing on web platform is also quite a lot more fun than C++, because you can see the results of your work in a matter of hours/days, rather than in months/years.

The most frustrating part of C++ coding is that progress is lot slower than that of alternate platforms (such as web), and it's a LOT harder to land a contract-work with that. Everyone needs web/db stuff, and it's easy for a freelancer to do few projects w/o prior experience (I started on this db project by reading through the php4 manual, followed by mysql manual, just to learn the topics), while landing a contract-work for custom software solution written in C++ is next to impossible - most of that job area is done by companies, and more often than not, C++ programmers are hired full-time - something which I'm seriously trying to avoid. Full-time work just takes away my freedom, and usually pays considerably less than that of a contract-work. Furthermore, a custom software solution usually takes many months to complete (at least), while a website or db frontend can be completed under a week of dev-time.

Makes one wonder - if I can land hundreds of euros with 1-2 weeks of work on web platform, what on earth am I doing coding C++ then? Only place that I'm aware of that pays good for C++ code is day-job at some respectable company, and dayjob is something I'm trying to avoid at all costs... and speaking of which - C#/.NET seems to be much more "in" nowadays than C++ when it comes to corporate software engineering (where development speed and fast learning are valued), it seems C++ is only used at high-end software market (large-scale applications [office, photoshop] and games [due to performance reasons]).

On other news, I'v had some thoughts on why Hydranode starts having serious networking issues after around 10 days uptime. I think the issue is that we run out of sockets the OS provides. The thing is - Hydranode creates and destroys a LOT of sockets, if you start to think about it. Given a TCP reask every 50 minutes (in case where UDP is disabled), and 10'000 sources, that means around 200'000 sockets created/destroyed after 24h period. In 10 days, that means around 2'000'000 sockets. Now, socket descriptors are 32bit signed integers, and >0 values are used, which should give us roughly 2'000'000'000 available numbers (assuming they aren't being reused). Yes, I notice now that the calculation is off by 3 digits (I didn't notice it during my original calculation), but still it could be a theory worth investigating. Does anyone have experience with similar matters?

Madcat

posted by Madcat # 8:45 PM

Comments:

how about "claiming" a whole bunch of sockets at init and reusing those internally during execution?

# posted by Anonymous : 10:05 PM

I think that would be a good idea...

a simpley factory to create and destroy socket i think would do the trick...

but i don't know why to me this is not exactly a good reason...

but we all know that the computer are made of funny spirit that made this kind of things ;)

are this problems that happne both on linux and windows? if so, that could be a good reason!

About C# why is in?

Well i develop all the time in C# the reason why i prefer it above C++

why? i have to take care of less problem..

and about speed... with JIT i have see this kind of problem simply become very small..

ps: the REAL reason? why i fall in love with the IDE O:)
i program so much faster O:)

# posted by TaGoH : 12:32 AM

I wonder why emule doesn't suffers from the "socket out" problem. Does it use another way to handle sockets, where the sockets can be reused?

As I know, other p2p clients (like Azureus) can specify the range of used ports.So doesn't have such kind of problem.

# posted by Enig123 : 5:22 AM

Beacuse , work for oopen source prokjects, you can get famous and a lot of enterprises will look at you !!

# posted by Anonymous : 2:39 PM

"I wonder why emule doesn't suffers from the "socket out" problem. Does it use another way to handle sockets, where the sockets can be reused?"

IME, emule doesn't run long enough to experience this problem, madcat mentioned 10? days? Maybe its just me, but i find that after about 5 eMules resource use grows so huge that my machine becomes unusable, and the harddrive is constantly chugging away at the swapfile. I restart eMule and everything goes fine again.

"As I know, other p2p clients (like Azureus) can specify the range of used ports.So doesn't have such kind of problem."

AFAIK this is not really possible with ed2k. The server expects you to have a single IP and port, which it can send to other clients in search result packets. Also it might not solve the issue, a socket's a socket, no matter what port it's bound to.

# posted by Anonymous : 3:49 AM

Also, You do not mention which OS the 10 days problem occurs on. If it's windows, it may be with looking into using winsock2. AFAIK the winsock 1.1 (BSD-style sockets) DLLs on newer windows versions are simply proxies that interface with the winsock2 API.

# posted by Anonymous : 3:52 AM

If you're under NAT chances are the nat table on the router could be the problem. Another possibility - how many sockets are in TIME_WAIT state?

Finally on windows - i would just blame select() polling degrading over time :)

# posted by Anonymous : 7:17 AM

Yes, if it affects both windows and linux on your machine, I would think about blaming either hydranode itself or your router. Also remember that if you are running XP SP2 you need the event 4226 patch to remove the restriction of 10 half-open TCP connections, and also that there is a fixed, system wide limit of 10 incoming connections. (5 on XP home)

# posted by Anonymous : 11:09 PM

The incoming connections are only limited on the Network Shares (SMB) if I remember correctly - not on any other port. Furthermore, they are not limited by the stack but by the "Microsoft Network Client" handling this protocol.
But this limitation occurs on every Windows Workstation OS and is by design.

chemical

# posted by Anonymous : 1:41 AM

I think the half-open connections affect P2P programs (not just SMB), hence why eMule has an option for max-half-open-connections so SP2 doesn't queue up connections since it has a max of 10 half-open connections. Better explained here: http://www.emule-project.net/home/perl/help.cgi?l=1&topic_id=120&rm=show_topic

Though I've not personally experienced the affects of flooding XP with too many half-open connections since I use a Linux server for my P2P'ing.

More information is probably necessary to try and find a repeatable case for this bug (OS, XP patched or not, NAT?, NAT-router model/type, etc)... Though hopefully its just a simple case of an unpatched XP installation =P.

# posted by Anonymous : 5:56 AM

Oops, some weird text-wrap on that last link here it is again: Link.

# posted by Anonymous : 6:08 AM

the half-open connections limit just limits the speed of aquiring new connections. It can get a problem for clients with a lot of sources, but surely won't magically appear after a runtime of about 10 days.
There has to be a different reason..

# posted by Anonymous : 4:56 PM

It's HN finally dead? Very sad, I think this P2P is the best developed one and soon when getting fully featured, some problems solved, more developers coming and more users using it, it will eat most P2P clients.

Please continue Hydranode and find good developers (training them first for doing HQ code ;)) for the project, this project looks amazing.

# posted by Anonymous : 5:40 PM

Don't worry people,.. OmniP2p beta is coming soon!
Before you balk at the fact it is written in Java, please realize this has only one cost: memory usage. The CPU usage of the core is less than that of jMule, and the Web GUI plug-in renders pages in single digit milliseconds (your browser will be the bottleneck, for example, safari is INSTANT, konqueror is almost instant, and firefox is next).

Project goals: true multiprotocol core, security (a reason Java was chosen), CPU efficiency, 100% stability, and low memory usage is still a priority (i do the best i can with Java)

# posted by Sam Maloney : 8:45 PM

BTW, if you are having that problem under linux, then you only have to look at /proc/<pid>/fd to see that linux (2.6 here) reuses fd numbers.

# posted by Anonymous : 8:55 PM

HN is not dead, Matcat is just involved in some contract work to earn money for survival.

Development process will continue in some days/weeks. It won't get back to the speed it has been, tho. The donations were not enough to live with, so he has to cut some time from hn and move it to contract work.

I'll ping him to do a blog update.

chemical

# posted by Anonymous : 9:29 PM