Tuesday, January 22, 2008

NetBeans Platform autoupdate via BitTorrent

Something I'm pursing nowadays is having my platform applications as decentralized as possible.

And the fist pet peewee I had is that fact that the Update Centers are such, big, centralized, monolithic blocks.

I always assumed that I would need to hack the AutoUpdate module from the NetBeans Platform quite hard in order to get what I wanted all along: BitTorrent downloads for new or updated modules.

So, the first thing we have to notice is that the AutoUpdate Catalog (see DTD) provides for each module in the Update Center a location called distribution. The distribution may be a relative path to the catalog location. That is usually something like ./com-example-mymodule.nbm or it can be a totally new URL.

Now we have a first step towards splitting traffic: we can put the actual NBM file on another URL altogether. Or, if we have the AutoUpdate Catalog location sit behind a servlet we could even try a bit of balancing and return a different distribution link depending on how loaded are the servers. That's a plus...

Ok, that's something but it isn't BitTorrent, you still download the whole file from a single place.

It's alive !

But what the Platform does offer is the possibility to register in the Lookup your own URLStreamHandlerFactory . So, I can register a new handler for the torent:// protocol and the AutoUpdate module will just use my stream handler.

And thus, a few hours later I have a working AutoUpdate infrastructure via BitTorrent. My StramHandler downloads behind the scenes with BitTorrent using Snark and provides a nice InputStream to the AutoUpdate module. It's still not polished but already useable. Install the module from this update center: http://emilianbold.ro/modules-updates.xml or just grab the NBM.

Something else: the destination of the module is no longer pointing now to the NBM, but to a torrent file actually which has the NBM file.

The steps are: place the torrent file on http://example.com/module1.torrent.nbm, edit the catalog to have destination="torrent://example.com/module1.torrent.nbm" and you're good to go. Behind the scenes I'll actually download the http file and then download the torrent.
<!DOCTYPE module_updates PUBLIC "-//NetBeans//DTD Autoupdate Catalog 2.3//EN"
"http://www.netbeans.org/dtds/autoupdate-catalog-2_3.dtd">
<module_updates timestamp="35/26/18/21/01/2008">

<module codenamebase="org.yourorghere.emptymodule"
distribution="torrent://example.com:6881/cc032d0c003b12568c91a0339f88301fa6ca67f5.torrent.nbm"
... >
...

A small remark: note the .nbm extension. It's something the AutoUpdate module needs otherwise it won't be able to install the file as NBM (could be a bug, I'll report it at some point).

The module still needs some extensive testing and different BitTorrent libraries (I'm using Snark, but I would like to have the Azureus core as a different provider in the Lookup maybe) but it does show it is possible.

Using the same technique one could write multiple backend/"protocols" for the AutoUpdate. Drop me a message if you want to know more or want to help me (source code will be online soon).

The case of the different jsch 0.1.54 binaries

As part of the Apache NetBeans IP clearance we are combing through all the code and dependencies. One interesting thing we bumped into wa...