Updating the root certificates for Java

One usually thinks of SSL in the context of HTTPS, but there are also other protocols which rely on it to provide security. See this link for a short overview of SSL – it only mentions HTTPS, but the same applies for IMAPS, FTPS, etc – SSL is independent of the wrapped protocol. You can have issues with your Java programs in where the party you are communicating with provider changes their certificate and the program rejects it as invalid. The exception is something like:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: 
    PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: 
    unable to find valid certification path to requested target

One cause of the problem can be that the server uses an SSL provider which is based on a root certificate that wasn’t included with the particular version of Java you are using (this is especially true for really old versions like Java 1.5). The issue can be solved by updating to the latest version, but it might be that this isn't an option. Fortunately I found the following article: No more ‘unable to find valid certification path to requested target’

How to use it:

• Compile the program javac InstallCert.java
• Run it with the target host/port. For example in our case it would be: java InstallCert imap.mailprovider.com:993 (993 is the port for IMAPS)
• navigate trough the menus and select which certificate to import
• now you have a file called jssecacerts. You need to copy this to $JAVA_HOME/jre/lib/security/cacerts (back up the existing file first!)
• Now the root certificate is imported (you can confirm this by rerunning InstallCert)

HTH

Vagrant and VirtualBox on Windows

Vagrant is a collection of scripts written in Ruby to manage VirtualBox images in a shared environment (like the QA boxes inside a company): install them, update them, etc. Unfortunately installing it under Windows is not as straight forward as one would want, so here are some useful tips:

• Read the Windows setup page and the Windows x64 setup page (if you are on 64 bit)

If you are on a 64 bit Windows install:

• Check out this post if your JRuby is using the 32 bit JVM on a x64 Windows setup
• You need to use version 4.0 of VirtualBox (rather than the latest). You can get it from here
• You need to use an older version of Vagrant:

  jgem install jruby-openssl jruby-win32ole
  jgem install --version '=0.7.8' vagrant

• If the vagrant box download stops around 4G, check that you have a NTFS filesystem (rather than FAT) and deactivate any "network" scanning capabilities of installed security software (I had problems with NOD32 particularly)

HTH

Another friend blogging

Another friend bit the dust started blogging: Cleonte's GitHub blog - bookmarks at the moment but looking forward to more involved posts :-).

Using Jython from Maven

This blogpost was originally posted to the Transylvania JUG blog.

On the surface it looks simple: just add the dependency and you can run the example code.

However what the jython artifact doesn’t get you are the standard python libraries like re. This means that as soon as you try to do something like the code below, it will error out:

PythonInterpreter interp = new PythonInterpreter();
try {
  interp.exec("import re");
} 
catch (PyException ex) {
  ex.printStackTrace();
}

The solution? Use the jython-standalone artifact which includes the standard libraries. An other advantage is that it has the latest release (2.5.2) while jython lags two minor revisions behind (2.5.0) in Maven Central. A possible downside is the larger size of the jar.

<dependency>
    <groupId>org.python</groupId>
    <artifactId>jython-standalone</artifactId>
    <version>2.5.2</version>
</dependency>

How to post high-quality videos to Google Video

If you have Google Apps for Business, Google Video is still the preferred method of storing videos "in the cloud": it is easier to embed into Google Docs and probably more importantly - it can be truly private (you can only access it if you are logged in with the correct account - as opposed to YouTube where you can "hide" the video, but still anyone with the link can access it).

To post a high quality video to Google Video you only have to do:

1. Upload the video
2. Wait

I kid you not :-). While initially the uploaded video is of poor quality, apparently it gets processed in the background and later on (in my experience: it takes around 30 minutes to process 1 hour of uploaded video) a high quality version will be available.

HTH

Integrating Maven with Ivy

This post was originally published on the Transylvania JUG blog.

The problem: you have some resources in an Ivy repository (and only there) which you would like to use in a project based on Maven. Possible solutions:

• Migrate the repository to Maven (Nexus for example) since Ivy can easily use Maven-style repositories (so your Ivy clients can continue to use Ivy with some slight configuration changes and Maven clients will also work – also the push-to-repo process needs to be changed)
• Try JFrog Artifactory since it reportedly can serve the same resources to both Ivy and Maven (disclaimer: I haven’t tried to use it actually and I don’t know if the Open Source version includes this feature or not)
• or read on…

My goal for the solution (as complex as it may be) was:

• It should be as simple and self-explanatory as possible
• It should respect the DRY principle (Don’t Repeat Yourself)
• It shouldn’t have other dependencies than Maven itself

The solution looks like the following (for the full source check out the code-repo):

Have two Maven profiles: ivy-dependencies activates when the dependencies have already been downloaded and ivy-resolve when there are yet to download. This is based on checking the directory where the dependencies are to be copied ultimately:

...
<id>ivy-dependencies</id>
<activation>
  <activeByDefault>false</activeByDefault>
  <file>
    <exists>${basedir}/ivy-lib</exists>
  </file>
</activation>
...
<id>ivy-resolve</id>
<activation>
  <activeByDefault>false</activeByDefault>
  <file>
    <missing>${basedir}/ivy-lib</missing>
  </file>
</activation>
...

Unfortunately there is a small repetition here, since Maven doesn’t seem to expand user-defined properties like ${ivy.target.lib.dir} in the profile activation section. The profiles also serve an other role: to avoid the consideration of the dependencies until they are actually resolved.

When the build is first run, it creates the target directory, writes the files needed for an Ivy build there (ivy.xml, ivysettings.xml and build.xml – for this example I’ve used some parts from corresponding files of the Red5 repo), runs the build and tries to clean up after itself. It also creates adependencies.txt file containing the chunck of text which needs to be added to the dependencies list. Finally, it bails out (fails) instructing the user to run the command again.

On the second (third, fourth, etc) run the dependencies will already be present, so the resolution process won’t be run repeteadly. This approach was chosen instead of running the resolution at every build because – even though the resolution process is quick quick – it can take tens seconds in some more complicated cases and I didn’t want to slow the build down.

And, Ivy, the Apache BSF framework, etc are fetched from the Maven central repository, so they need not be preinstalled for build to complete successfully.

A couple of words about choosing ${ivy.target.lib.dir}: if you choose it inside your Maven tree (like it was chose in the example), you will receive warnings from Maven that this might not be supported in the future. Also, be sure to add the directory to the ignore mechanism of your VCS (.gitignore, .hgignore, .cvsignore, svn:ignore, etc), as to avoid accidentally committing the libraries to VCS.

If you need to add a new (Ivy) dependency to the project, the steps are as follows:

• Delete the current ${ivy.target.lib.dir} directory
• Update the part of your pom.xml which writes out the ivy.xml file to include the new dependency
• Run a build and watch the new dependency being resolved
• Update the dependencies section of the ivy-dependencies profile to include the new dependency (possibly copying from dependencies.txt)

One drawback of this method is the fact that advanced functionalities of systems based on Maven will not work with these dependencies (for example dependency analisys / graphing plugins, automated downloading of sources / javadocs, etc). A possible workaround (and a good idea in general) is to use this method for the minimal subset – just the jars which can’t be found in Maven central. All the rests (even if they are actually dependencies of the code fetched from Ivy) should be declared as a normal dependency, to be fetched from the Maven repository.

Finally I would like to say that this endeavour once again showed me how flexible both Maven and Ivy/Ant can be and clarified many cornercases (like how we escape ]] inside of CDATA – we split it in two). And it can also be further tweaked (for example: adding a clean target to the ivy-resolve profile, so you can remove the directory with mvn clean -P ivy-resolve or re-jar-ing all the downloaded jars into a single one for example like this, thus avoiding the need to modify the pom file every time the list of Ivy dependencies gets changed – then again signed JARs can’t be re-jarred so it is not an universal solution either).

Upgrading the Options (GlobeTrotter) GI515m

Recently I needed to install an Options (GlobeTrotter) GI515m 3G USB modem on a machine which previously used an older version of the modem (the iCON 225). This seems a pretty common scenario (an existing user getting an update), however the process seems less-than-straight forward:

1. Get a second computer with the same operating system version which didn't have a 3G modem installed (for example if your target system is running Windows 7 64 bit you need a second system with Windows 7 64 bit - different skews like Home vs Ultimate are ok, but the version and "bitness" must coincide - you could also try using a virtual machine for the second machine which supports USB forwarding like VirtualBox or VMWare)
2. Plug in the modem in the second machine. First it will recognize it as an USB stick / CD-ROM. Copy all the files from it to a separate folder (you should see files like "setup.exe").
3. Let the setup complete. Now copy the installed drivers to the same place you've saved setup file. Under Windows 7 you would find them in the location C:\Windows\System32\DriverStore\FileRepository\ in several folders starting with "gth" (like gthsubus_64.inf_amd64_neutral_4810563f34b37ef5), but here is the generic way to identify the folder:
   1. Start Device Manager
   2. Look for one of the devices associated with the modem (you will find actually several, like GlobeTrotter GI515M - Modem Interface, Network Interface and so on)
   3. Properties -> Driver -> Driver Details. Note the name of the driver for which the provider is Option (for example gtuhsser.sys)
   4. Now search your Windows folder for files ending in .inf which contain the name of driver from the previous step. This will point you to the right folders
4. On the first computer (the one you actually want to install the modem on) remove all previous versions of the software using the Add-Remove Programs facility (you will see two-three entries but they can be easily identified by the same orange icon). Restart the computer for good measure.
5. Copy over the setup program and the drivers from the second computer. Plug in the modem to the first computer, install the application (using the setup file captured on the second computer). Go into the device manager and look for "Unknown device"s (you should see four of them). Use the drivers captured on the second computer to resolve these issues.
6. Unplug and replug the modem - it now should work!

A couple more talking points:

• don't use "driver manager" type software - they very rarely (read: never) seem to work
• a symptom that you've hit this problem is when the management interface (dialer / "Internet Everywhere") for the modem starts but it gets stuck in the "Initializing" phase when you connect the modem and consumes CPU (from what I've seen with a debugger it seems to be looking for the installed device in a loop)
• the modem seems to be prone to overheating if the signal-strength is low (around two bars) and in this case it shuts down after ~10 minutes (I assume that this is some kind of thermal protection). You can check if this is the case by putting your hand on the bottom side of the modem. I couldn't find and solution for this, other than looking for a spot which has better signal. Using the modem in EDGE rather than 3G mode also seems to do the trick, but it has lower speeds and I don't know of any reliable method to make the modem use EDGE if 3G is also available.

Getting the most out of your audio recording with Audacity

This article aims to show you some simple techniques to improve the quality of your voice recording quickly and cheaply (for free actually). But first things first:

The best audio is the one you don’t have to improve. Some simple steps you can perform in advance to maximize quality:

• Use quality equipment. Here are some articles about the equipment great-sounding podcasters use. You don’t have to spend a lot of money, but definitely stay away from the built-in laptop microphone
• Eliminate ambient noise as much as possible (close windows, draw the blinds, stop other electronic equipment in the room, etc)
• Record each person on a separate channel - if possible on a computer local to them (avoid recording trough Skype, GoToMeeting or other VoIP solutions)
• Try keeping the recording volume for each microphone at the optimal level – not too low, but also avoiding clipping

After you have the audio recording there is still a lot you can do, but it is preferable to start out with the best source material. For the example below I’ll be using the raw recordings from a recent SE Radio podcast:

The situation with this recording is as follows:

• There are separate audio tracks for the interviewer and interviewee (good)
• There is background noise on the tracks (easily correctable)
• Both persons were picked up by both microphones (correctable)
• The interviewer has some clipping (partially correctable – luckily it’s not the interviewee who has clipping)

The steps to improve the quality of this recording are as follows:

First, install the Noise Gate plugin for Audacity, since it requires program restart (under Windows you have to copy the downloaded noisegate.ny to C:\Program Files (x86)\Audacity 1.3 Beta (Unicode)\Plug-Ins or to a similar location, under Linux you have to place it in /usr/share/audacity). After copying the file you have to close and restart Audacity. To verify that the plugin was properly installed check in the Effect menu – you should see an entry title “Noise gate”.

Now that we have Audacity all set up and the plugin installed, first split the stereo track into mono tracks, since they don’t actually represent left-right channels but rather two speakers which will be mixed together at the end. For this click on the arrow after the filename in the track and select “Split Stereo to Mono”. Sidenote: some people will prefer to mix different speakers in podcasts with different panning (that is to the left or to the right). I would advise against this: it is distracting if you are doing something else while listening to the podcast (like walking / jogging / riding a bike / etc). It can also backfire if for some reason the listening device is missing one of the channels (the “damaged headphone” scenario).

The first thing will be to remove the constant background noise (like AC hum for example). To do this zoom in (Ctrl + 1) and look for low volume zones. Select those zones and go to Effects –> Noise Removal –> Get Noise Profile. Now select a zone where the noise is mixed with speech and test out the settings (Effect –> Noise Removal –> Ok). After the test you can use Undo (Ctrl + Z) to roll back the changes. You should watch for the noise being removed but also the natural sound of the voice being preserved (too aggressive of a noise removal can lead to a “robot voice” effect). If you are satisfied, you can go ahead and apply it to the entire track. Also, since the noise source might change during the recording, you should at least do a quick scroll to check for other low-volume zones which can be a sign of noise. If you find noise from other sources, you can use the same steps to remove it.

Now that you have removed the noise, the next step would be to remove the voices from the channels they don’t belong to. This is where we’ll be using the Noise Gate plugin: since there is a considerable level difference between the wanted audio and the unwanted audio on each channel, we can just declare everything below a certain volume “noise” and use the plugin to silence it. A couple of tips:

• This needs to be done separately for each channel, since the cutoff volume will be different
• You can use the “Analyse Noise Level” function of the plugin to gauge the approximate level of the cutoff volume – this will only give you an estimate and you will have to play around with the settings a little bit to find the optimal volume
• Use a “Level reduction” of –100 dB to completely filter out the sound and an “Attack/Decay” of 1000 milliseconds to avoid false positives
• As with all the steps, you can experiment on a smaller portion of the audio file (since it is much quicker) to fine tune the settings by repeatedly applying the effect with different parameters and undoing (Ctrl+Z) the result after evaluation. When the parameters seem right, just select the entire track and press Ctrl+R (Repeat last effect)

After we’ve finished with both tracks, we have a better situation:

Now we will fix the clipping as much as possible (a perfect fix isn’t possible since clipping means that information got lost and all the plugins can do is to “guess” what the information might have looked like). First we reduce the aplification of the second track (the one which contains the clipping) by 10 dB as the Clip Fix plugin suggests (Effect –> Aplify –> –10 dB) after which we use the Clip Fix plugin. Unfortunately this plugin runs very slowly if we would to apply it to the entire track at once. Fortunately we have a reasonable workaround: select portions of the track and apply the plugin to them individually. After the first application you can use the “Repeat last effect” shortcut (Ctrl+R) to speed up the operation. Sidenote: it is a good habit to use the “Find Zero Crossing” function whenever you do a selection (the shortcut is Z – so whenever you select a portion, just press Z afterwards). This eliminates some weird artifacts when cutting / pasting / silencing part of the audio and it might even help when applying different effects. The fixed audio looks like this:

Now, that all the cleanup steps have been performed, there is one last step which is as important as the cleanup: maximizing the audible volume without introducing clipping. This is very important because all devices can reduce volume but few of them can increase it (some exceptions being: the Linux audio stack and VLC). The easiest way to do this is by using the Levelator (note: while the Levelator is free – as in beer – and does not restrict what you can do with the output, it is not free as in freedom if this is a consideration for you).

To do this, export the audio to WAV (make sure that all tracks are unmuted during export) and run the Levelator on it. The end result will look like the following:

Of course the Levelator isn’t magic pixie dust either, so here are a couple of things to check after it has been run:

• Did it amplify some residual noise which wasn’t available in the initial audio? (if so, you should remove it using the Noise Removal plugin)
• Did it miss segments? (it is rare, but it happens – those segments need to be amplified manually)
• It results in “weird” sounding audio if the recording has been preprocessed by a dynamic compressor – for example GoToMeeting has an option to improve sound quality which uses dynamic compression and thus makes the recording unsuitable for the use with Levelator

That’s it for this rather long article. Don’t be discouraged by the length of the article: after going over the steps a couple of times, it shouldn’t take longer than 15 minutes to process a 2 hour interview (excluding the cutting / pasting / moving parts around) and you will gain listeners because of the higher production value.

A final note on the output formats: while during processing you should always use lossless formats, the final output format I recommend is: MP3 at 64 kbps CBR, Joint Stereo, 22050 MHz sampling rate. I found that this is the best balance between quality, file size and compatibility with the most playback devices out there.