NetBeans Web Toolkit

I'm exploring NetBeans Web Toolkit with the articles here

NetBeans Web Toolkit is the new name I'm trying to give to Jaroslav Tulach's HTML/Java API, a rather impressive library that deserves more use.

Guards in Java

Haskell functions have this nice concept called 'guards' which allow you to define a condition and return a value when that condition is true.

For example:

abs n
  | n < 0     = -n
  | otherwise =  n

This makes the code rather readable, especially when you have more guards.

Guards build one upon another since you know that if your guard condition is checked, all the other failed:

something n
  | n < -2 = 10
  -- bellow we know that n > =-2
  | n < 0 = 8
  | otherwise = n

Back in Java land, where I get paid, I sometimes wondered if I should write a method as:

X method(Y param) {
  if (!param.isSomething()) {
    return null;
  } else {
    return param.getX();
  }
}

or if I should write it as

X method(Y param) {
  if (!param.isSomething()) {
    return null;
  }

  return param.getX();
}

I generally prefer the 2nd variant and now I realised these are a form of function guards!

The case of the different jsch 0.1.54 binaries

As part of the Apache NetBeans IP clearance we are combing through all the code and dependencies.

One interesting thing we bumped into was that the jsch 0.1.54 binary JAR we are using has a different hash (and size) than the binary JAR from Maven Central.

The old hash is 0D7D8ABA0D11E8CD2F775F47CD3A6CFBF2837DA4, the new one is DA3584329A263616E277E15462B387ADDD1B208D.

The binaries are 278,612 bytes vs 280,515 bytes in Maven Central.

Our version is actually the same as the one found on http://www.jcraft.com/jsch/

Also, the Maven JAR is properly signed with the author's CA7FA1F0 key.

This is where it becomes clear that reproducible builds are important. You do not want to have to wonder why a binary differs, especially years later when you are doing a review. And this one is a library doing SSH!

So, why the different binaries?

It seems the original JAR was compiled on Aug 30, 2016 with Java 1.4 (major version 48) while the Maven Central JAR was compiled Sep 3, 2016 with Java 5 (major version 49).

The original JAR also concatenates strings using StringBuffer while the Maven Central JAR uses the newly introduced in 1.5 StringBuilder. Which should also be a bit faster since it's not synchronized.

Next, most of the cypher classes use some reflection via a static static java.lang.Class class$(java.lang.String) method.

What is this? It's just the way class literals worked in Java 1.4. As explained here, in Java 5 the ldc_w instruction was introduced to load a Class object.

In 1.4 the class literal was helped by the compiler by actually introducing the helper Class class$(java.lang.String className) method and replacing the Person.class with a class$("Person") call.

It conclusion, it seems that excluding the Java 1.4 to Java 5 compiler changes, the two JARs are identical. With the Maven Central JAR even a bit better due to StringBuilder being used.

There is no check so far that the sources do produce the specific JAR. This is an exercise left for the reader.

Note: I have also cross-posted this blog post to the Apache NetBeans blog.

Processing a generic Data.Array matrix

I had an interesting Haskell problem the other week: work on columns and rows of a Data.Array i e.

You only have the Ix i, Ord e class constraints, which make sense because the index must be a Data.Ix. The elements also must be Ord to be able to process them.

The thing about Data.Ix is that it's very opaque. It only extends Ord. There is nothing matrix-related in it. One could use Data.Array for a lot of data structures!

But if you do know it's a matrix, although you have no explicit class constraint, there is a nice trick to use: two neighbouring cells will have a Data.Ix.rangeSize of 2!

So, the rows may be extracted by this little function:

byRows :: Ix i => [i] -> [[i]]
byRows indices = incGroupBy isNeighbour $ indices
    where isNeighbour x y = 2 == rangeSize (x, y)

which is called like

process :: (Ix i, Ord e) => Array i e -> Something
process matrix =

    let rows = byRows $ indices matrix
    ...

Note the unknown incGroupBy which is a groupBy that takes pairs incrementally.

That's it! I had a lot of ideas about using rangeSize to figure out the matrix dimensions, but pairing cells this way was really clean.

Retina work

These past months I have done a NetBeans patch for the Apple Retina Display and also made a small Wiki-like site to help me and anyone else interested with finding matching font icons for the NetBeans icons: https://nextbeans.com/retina

The nextbeans.com stack is Angular, Prime NG, nginx, Jetty, Servlets, Spring Framework JDBC, HSQLDB. SSL via Let's Encrypt.  Hosted at Scaleway.

It's a fun project since I got to learn Angular, find a bug and submit a patch for Prime NG, see how Let's Encrypt does free SSL certificates, learn about the EU cookie warning and all the many tiny things that are needed for a site.

Angular in particular and the whole webdev ecosystem was a lot of new information for me. I was changing project configuration as Angular progressed along! Which reminds me: @angular/cli reached 1.0 and I should probably see if I need to tweak something.

Read on JAXenter a longer article about my work.

Machine learning everywhere!

Samsung announced a while back that they used a "Neural Net based predictor" for their CPU branch prediction.

Shortly after that an Intel person claimed it's no big deal because they have also been using a perceptron for some time.

But to me this seemed a rather big discovery! Previously I would have assumed that branch prediction is a super complex algorithm.

Learning that branch prediction is a basic perceptron reduces Intel's perceived strength.

So companies are openly and covertly using machine learning everywhere.

Machine learning is also a perfect fit for companies because there is no moral filter on a neural network and no chance of whistle blowing.

Volkswagen truly missed a golden opportunity here with their diesel scandal.

They should have just trained a neural network on passing the diesel criteria and then have perfect plausible deniability: "the neural network disabled the pollution filters all by itself!"

Migrating the extra large NetBeans Mercurial repository to Git

Note: This article is a living document and will be updated as I learn new useful information (last update 31st December 2016). I will move the helper scripts to a dedicated repository and copy part of this article into the Apache NetBeans wiki.

Introduction

The NetBeans source code has been stored in a Mercurial repository for almost a decade now.

But starting October 2016 NetBeans is preparing to become an Apache project.

And all incubating projects must store their source code on Apache Software Foundation infrastructure which only provides Subversion or Git hosting.

So, NetBeans must migrate its Mercurial repository to Git.

Size concerns

The NetBeans Mercurial repository covers 17 years of history and has grown to over 3GB.

Apache projects are mirrored on GitHub and they have a limit of 2GB or so. As such, any talk of migration started with ways of reducing the size by potentially splitting up the repository or removing some of the history.

Luckily, it turns out the NetBeans Mercurial server was just using a really old Mercurial version. When Gregory Szorc looked into it we learned that with the format.generaldelta=true and format.aggressivemergedeltas=true flags, the repository drops to about 1GB.

This was great news but, of course, we still have to migrate to Git.

Under Git we have to make sure some compression is applied. This is done with the git gc command which reduces the repository to under 1GB too.

With size out of the way, we can do a straight migration and preserve our mono repository and the whole history.

The case of the corrupted repository

The most important NetBeans repository is releases. This repository holds the release branches such as release82 as well as the current state in the default branch. The main-silver default branch is periodically pushed into the releases/ default branch.

A direct conversion of releases/ is impossible though because the repository is corrupt:

$ hg verify
checking changesets
checking manifests                                                                                                                      

crosschecking files in changesets and manifests                                                                                        

checking files
 applemenu/src/org/netbeans/modules/applemenu/layer.xml@?: rev 12 points to unexpected changeset 149753                                  
 (expected 149755)
 defaults/src/org/netbeans/modules/defaults/Eclipse-keybindings-mac.xml@?: rev 0 points to unexpected changeset 149753                  
 (expected 149755)
 defaults/src/org/netbeans/modules/defaults/Eclipse-keybindings.xml@?: rev 25 points to unexpected changeset 149753                      
 (expected 149755)
 defaults/src/org/netbeans/modules/defaults/mf-layer.xml@?: rev 74 points to unexpected changeset 149753                                
 (expected 149755)
192754 files, 313961 changesets, 1122263 total revisions                                                                                

4 warnings encountered!
4 integrity errors encountered!

Luckily, the corruption seems to be in the default branch.

So, we can get a valid releases/ repository by first making a main-silver clone and then pulling the missing changesets from releases:

mkdir releases.fixed
cd releases.fixed
hg init .
hg pull http://hg.netbeans.org/main-silver
hg pull http://hg.netbeans.org/releases
hg out http://hg.netbeans.org/releases
#nothing should be displayed here
hg verify

hg-fast-export all the way

Now that we have a valid repository we just follow the steps in the official documentation about migrating from Mercurial:

git clone http://repo.or.cz/r/fast-export.git /tmp/fast-export
git init ~/git-releases
cd ~/git-releases
/tmp/fast-export/hg-fast-export.sh -r ~/releases.fixed
git gc --aggressive --prune=now

and then wait 48 hours for it to finish!

.. but first: removing the unnamed heads

Once you do start hg-fast-export.sh you'll notice it fails early with

Error: repository has at least one unnamed head: hg rXXXX

This is caused because Git, unlike Mercurial, does not support unnamed branches.

It's not a big problem for the NetBeans repository because there are very few such commits and basically historical mistakes with no relevance.

I have just removed them altogether with hg strip

Incremental push

Although we have world class internet speed in Romania, I happened to be on a slow connection when the conversion finished. And it is no fun to restart a git push after 400MB have already been uploaded and the connection dropped!

I fixed this by uploading incrementally each month:

echo "Incremental git push"

for year in 2012 2013 2014 2015 2016; do

    for month in 1 2 3 4 5 6 7 8 9 10 11 12; do

SHA=`git rev-list -1 --before="$year-$month-1 12:00" master`

echo "$SHA $year-$month"

  echo git push origin "$SHA:master"

git push origin "$SHA:master"

    done;

done;

Syncing with the old Mercurial repository

Right now the GitHub repositories are just experimental. The real work is still done in the Mercurial repository. As such, I still have to convert the new commits from Mercurial to Git.

hg-fast-export seems designed with this in mind. The -r parameter which specifies the source repository is only needed the 1st time. After that it may be skipped and hg-fast-export will incrementally convert the missing changesets.

So, it's just a matter of:

cd ~/releases.fixed

hg pull

cd ~/git-releases

hg-fast-export.sh

git push

Note that a pull on releases/ will bring back the stripped commits...

Saving hg-fast-export state

I did run into a deadlock while incrementally converting with hg-fast-export.

The only solution seemed to be to redo the conversion.

At 48 hours per full repository this doesn't seem like fun, so I recommend periodically saving these files from the .git folder: hg2git-heads, hg2git-mapping, hg2git-marks and hg2git-state.

Make sure not to ignoreCase

I discovered that on macOS core.ignoreCase is true which means that changesets that only change the case of a file name will produce an incorrect git changeset.

So on macOS the option needs to be explicitly set to false:

git config core.ignorecase false

Minimal NetBSD for Raspberry Pi 2

For no reason other than fiddling with the NetBSD codebase and build system plus the desire to reduce the world bandwidth waste, I've published a minimal NetBSD image for the Raspberry Pi 2 containing only the base set.

You only have to download this 76MB compressed image which becomes under 500MB uncompressed so it fits any recent microSD card you have in your drawer.

I have successfully used such a NetBSD system as a customer proxy of sorts this whole year.

Because lots of times you just need a machine with SSH and a shell.

Of course, my patches are minor, it's not like I managed to fit NetBSD onto a floppy disk but I liked doing it none the less!

NBnotify - Native NetBeans Notifications

NBnotify is back with a vengeance: Windows, macOS and Linux notifications from NetBeans plus the very useful build notifications!

NBnotify started 6 years ago as a way to integrate NetBeans with OSX and to provide me with build notifications.

I have almost always used a MacBook Pro as a work laptop for my customers so it was important that NetBeans is hooked up into the Mac ecosystem.

Back then Growl was the rage, providing customizable notifications for a myriad of Mac applications.

NBnotify entered into a short limbo while I was busy doing NetBeans customizations, Javascript type inference, JDBC drivers, ANTLR parsers and all in all Java code for paying customers...

But as soon as I had some free time this winter I had to bring it up to date!

And not only that -- but I've added first Linux and then Windows support too!

Actually, since NBnotify natively binds to libnotify it should work on BSD systems too.

So, if you are using NetBeans, you have to use NBnotify!

NBnotify is closed source, but only because it was an experiment in the market for NetBeans plugins and productization.

As soon as the NetBeans codebase is donated to Apache and we can start work, I plan on slowly integrating these native notifications into NetBeans proper and open-source the plugin.

NBnotify was always a missing core feature of NetBeans.

NetBSD cross compilation is a marvel

While trying to get a patched variant of NetBSD 7.0.2 to run on my Raspberry Pi 3 I had to build NetBSD many times.

And it was pretty amazing how this all worked.

You have a single entry point: ./build.sh And it does everything! It builds the kernel, the userland apps up to the install image you write on the microSD card.

Of course, having a top-level script that does everything is nothing new, it's basically the second question in the Joel Test.

What is amazing is the cross-compilation.

You have an x64 machine and build.sh will happily create the install image for any supported platform.

The Raspberry Pi image is just:

./build.sh -m evbarm -a earmv7hf -u -U release

then you look into the release dir, find arm7.img.gz and you are ready to go.

I ran the builds on a NetBSD virtual machine, but apparently build.sh will happily run with some small preparation even on Linux or macOS.

NetBSD on Raspberry Pi 3

I have used quite successfully a Raspberry Pi 2 running NetBSD 7 as a customer proxy and I assumed 7.0.2 would run on a Raspberry Pi 3.

As it turns out, the Raspberry Pi 3 ARM Cortex-A53 processor is different enough from the previous Cortex A7 processor we have on the Pi 2 that it needs some kernel changes.

And while the bleeding edge NetBSD current- does work on the Raspberry Pi 3, the stable NetBSD 7.0.2 does not.

I even tried to compile my own NetBSD 7.0.2 and backport the patch that added Pi 3 support, plus up-to-date firmware, but it seems it's not sufficient. There are probably more related commits that need backporting.

So, if you have a Raspberry Pi 3 you can only use current- builds. Go on the nightly build server, grab an .img.gz and test it out!

But if you want a stable NetBSD release, you'll have to wait some more or get a Raspberry Pi 2 instead.

Getting the Raspberry Pi 2 is not such a bad thing because the Raspberry Pi 3 wireless will probably not work for a while anyhow.

Time Machine eating its own tail on a FreeNAS ZFS share

I work on a Mac and the OSX Time Machine is a very nice and simple feature to have backups.

While I have some other Apple gear I didn't get an Airport Time Capsule because I already had a tiny Airport Express, the Time Capsule is pricy and I'm not certain it does RAID.

So, what I use instead is the FreeBSD-based FreeNAS on an N54L MicroServer. FreeNAS allows you to easily create a Time Machine Apple Share and it works seamlessly.

The nice thing here is that the FreeNAS share is stored on a ZFS disk! And I have periodic snapshots for those shares.

So, on one side you have Time Machine storing periodic snapshots and only deleting when it needs space.

And on the other side you have the ZFS snapshots that guarantee that nothing is truly deleted!

This proves to be an interesting combination because Time Machine does not expect such a situation when you are low on disk space.

When you are low on disk space OSX's Time Machine will try to remove some data to free some space. But if you have recent snapshots on your ZFS filesystem, deleting files will not create any free space!

So Time Machine will keep on going back in history and delete more and more in order to have enough space to do a backup. I'm pretty sure eventualy it will try to delete the whole history.

Once Time Machine goes free space berserk you have to stop it and fix the problem on the ZFS side.

Either you delete some snapshots, you increase the filesystem quota or move it to a larger array.

In order to prevent the damage Time Machine did, you have to use zfs rollback to go back to a good snapshot. And in order to find a good snapshot it might be worth using zfs diff too.

The Oracle documentation for ZFS Solaris is good enough for this although you might also want to look at the FreeNAS User Guide for your version.

Time Machine doesn't seem to be bothered by a rollback and will gladly continue using the same share and finish the backup!

All in all I believe it's a pretty good combination. In an alternate universe OSX and Time Machine knows ZFS natively but until then FreeNAS gets the job done.

Signing NetBeans modules with a Time Stamping Authority (TSA)

Signing JAR files is a very good practice. And while a proper certificate is not worth the price and effort, self-signing is still a step in the right direction.

Ever since Java 5 jarsigner supported a Time Stamping Authority (TSA) with the --tsa and --tsacert parameters. A Time Stamping Authority is basically an online digital notary that certifies the point in time the jar was signed -- it is designed to prevent signing files after the certificate expired.

It turns out that while you can sign NetBeans modules using the FAQ steps, there is no support in the build harness for a TSA.

I found bug #243213 which also mentions NBM problems and I submitted a patch there.

So, if you want to also add a timestamp to your NBMs, apply this small patch on top of your NetBeans source repository and rebuild NetBeans.

Then, you just have to define in nbproject/project.properties another key with your TSA (I'm using StartSSL's here):

tsaurl=http://tsa.startssl.com/rfc3161

Forcing export of internal API in Java 9 with -XaddExports

I've long been a fan of NetBeans' module system and of OSGi so Java 9's modules are a big improvement to me.

Except modules are really good at enforcing API boundaries and stop allowing one to freely use any public class.

An error such as this is no fun:

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.IllegalAccessError: superclass access check failed: class A$1 (in unnamed module @0x3fb6a447) cannot access class jdk.nashorn.internal.ir.visitor.NodeVisitor (in module jdk.scripting.nashorn) because module jdk.scripting.nashorn does not export jdk.nashorn.internal.ir.visitor to unnamed module @0x3fb6a447

I've assumed that this has to be tweaked at SecurityManager level and played with -Djava.security.manager and -Djava.security.policy and the very handy -Djava.security.debug.

Alas, that doesn't help. (Although I'm still convinced it should, unless there is a bug somewhere).

What one needs to use is the magical -XaddExports flag. This forces an export and allows the code to run:

java -XaddExports:jdk.scripting.nashorn/jdk.nashorn.internal.ir=ALL-UNNAMED -XaddExports:jdk.scripting.nashorn/jdk.nashorn.internal.parser=ALL-UNNAMED -XaddExports:jdk.scripting.nashorn/jdk.nashorn.internal.runtime.options=ALL-UNNAMED -XaddExports:jdk.scripting.nashorn/jdk.nashorn.internal.runtime=ALL-UNNAMED -XaddExports:jdk.scripting.nashorn/jdk.nashorn.internal.ir.visitor=ALL-UNNAMED A

String.trim is an optimization hack

Turns out java.lang.String.trim() has its own unique definition of "whitespace" in order to be able to reuse the same underlying char array.

This blog post from 2008 talks in a lot of detail: https://closingbraces.net/2008/11/11/javastringtrim/

BSD on Raspberry Pi 2

Raspberry Pi 2 is the version that finally seems a good purchase to me with its quad core processor and 1GB of memory. If you need WiFi, Pi 3 is even better since an external WiFi dongle consumes more power.

But I would prefer to use an official release of a known operating system instead of using custom forks like Raspbian.

FreeBSD provides an official image, but to them ARM is a tier 2 platform "not supported by the security officer and release engineering teams". So, no freebsd-update. The existing image is for CURRENT ie. bleeding edge. RELEASE would be nice.

OpenBSD has no plans to support it.

The winner is NetBSD: they have an official image since NetBSD 7!

So now NetBSD and this little Pi 2 will replace a much bigger and nosier machine for a dedicated task. I'm curious how it holds up!

NetBeans - PalmOfMyHand - Dance Demo

I like making NetBeans dance:

NetBeans - PalmOfMyHand - Dance Demo from Emilian Bold on Vimeo.

The above video is a little something I've obsessed about this weekend. It all started from a song actually: Swedish House Mafia - One (Your Name). But I couldn't publish a video with that as a soundtrack so I've used this little Creative Commons gem instead.

It's basically a NetBeans module that plays an mp3 and syncs everything else to the beat.

There are a lot of NetBeans Platform and NetBeans IDE APIs touched in this simple demo. See if you can make a full list: Progress API, StatusLineElementProvider, Editor API, etc.

If people are curious I might write more about this. Turns out on Java one of the harder things is mp3 playback.

nbnotify.com is back up

I forgot to publish this, but it's been a month since I uploaded a new version of nbnotify.com, the OSX notification plugin.

Based on Bootstrap, this new site is as simple as it gets: all static, a stark contract to the old PHP Wordpress.

I brought the old site down in August because the Wordpress I had there was too insecure.

Right now this plugin is looking a bit old. After all, it needs the original Growl, not the new commercial Growl. Also, OSX Mountain Lion has its own notification API which I do not support yet.

Everything takes time

I read somewhere that as you get older you don't necessarily get slower, you just trade speed for quality. This sounds about right.

Here is my time with an UI regression in the new version of a large Swing app:

I start investigating the problem manually and look at the code to see what might be the cause.

I find a quick way to duplicate it.

I deduce from the code what might be the cause but I'm wrong, it could be that, but it isn't, because there is extra code to check for that condition.

I start probing with a BTrace script to see the Swing events.

I finally find the actual cause and I'm stupefied - it's a failure of an underlying system and not of the UI.

I make an unit test that fails for the regression. I also find a possible fix.

Since I'm a bit surprised by how that system worked, I go and check the problem in the previous version of the Swing app. The problem is not there! It is a regression indeed.

Now I'm really surprised - what changed?

I start debugging the unit test on the old version in order to figure out what behavior changed and I finally figure it out when I look at the diff between the versions.

This diff gives me a 2nd way to fix the code. I patch the new code but it fails subtly in another way -- it wasn't changed for nothing, after all.

I decide that the initial fix is the right one and I apply it.

I am finally done!

At this point my fix was a few lines of code and a large unit test.

But besides this I have various byproducts: the BTrace script, the 2nd fix that didn't work as well as my notes file with my remarks, various stacktraces from the BTrace script and the debugger and errors.

Investigating, writing little scripts, running in various configurations and on top of various versions then finally checking the final fix and wrapping up took a lot of time.

A Swing window is just as transparent as a web page

I'm a fan of the web and I admire how powerful the browser has become. Since I'm almost always in a browser (even to type this blog) I would certainly love to see even more useful apps, like, perhaps, an online IDE. So, although I work with desktop apps I like this cloud and web app kool-aid a whole lot.

One of the advantages I see in web apps is how transparent they are. It's all text, with code in an easy scriptable language. It's easy to inspect and change. Easy to deploy.

But I don't work with web apps. I work with large desktop Swing applications, usually on top of the NetBeans Platform.

I routinely debug these Swing apps with the debugger, VisualVM, BTrace and simple apps or unit-tests that reproduce the issue.

And it occurred to me after a debugging session to me that Swing apps are quite transparent too! The fact that Java code is compiled is just a decoy. As a developer you have access to the source code anyhow but, even as an user, you have about the same freedoms as you have with a web app.

Just as I could look into the DOM hierarchy in the browser, I have the Swing hierarchy of components.

Just as I could add my own Javascript event handlers, I have Swing listeners.

Although existing code is compiled, I could make a small BTrace script to see its execution and inspect parameters and values.

Without having to recompile, I can easily use reflection to look into private data and even tweak it.

So, although Swing would seem pure binary, there is nothing you don't have access to. It's an open book!