Back to Top

Wednesday, December 04, 2013

Programming advent calendars for 2013

0 comments

Programming advent calendars are posts/articles for a particular topic posted daily between the 1st and 24th of December. They are modeled on the advent calendars received by children on some countries which contain 24 doors for the 24 days of advent and behind each door is a piece of chocolate or other surprise which the child gets on the particular day.

Here is the list of programming related advent calendars for 2013 (if you know of more, leave a comment and I'll update the list):

  • 24 ways - "is the advent calendar for web geeks. Each day throughout December we publish a daily dose of web design and development goodness to bring you all a little Christmas cheer"
  • The Perl Advent Calendar 2013
  • Perl 6 Advent Calendar (if you didn't hear - Perl 6 is a completely different language from Perl 5!)
  • SysAdvent (not only) for sysadmins
  • Performance Calendar Learn all about web performance. (Primarily focused on frontend technologies.)
  • UXmas UX is for everyone. Brush up on the topic this December.
  • Go Advent - the go(lang) advent calendar
  • 24 Pull Requests - "a yearly initiative to encourage developers around the world to send a pull request every day in December up to Xmas"

Tuesday, November 12, 2013

Cleaning up Google AppEngine Mapreduce Jobs

2 comments

Do you use the Google MapReduce library on AppEngine? And do you have a lot of completed tasks which clutter your dashboard? Use the JS below by pasting it into your developer console to clean them up! (use it at your own risk, no warranty is provided :-))

schedule = function() { window.setTimeout(function() { var c = $('a:contains(Cleanup)').first(); if (c.length > 0) { c.click(); } else { $('a:contains(Next page)').click(); schedule(); } }, 300); return true; }; window.confirm = schedule; schedule();

Wednesday, August 14, 2013

Capturing your screen on Ubuntu - with sound

0 comments

Today I have a short script which I cobbled together from Google searches to do screen captures / screen casts with Ubuntu (including audio in so that you can narrate what is going on):

#!/bin/bash
Xaxis=$(xrandr -q | grep '*' | uniq | awk '{print $1}' |  cut -d 'x' -f1)
Yaxis=$(xrandr -q | grep '*' | uniq | awk '{print $1}' |  cut -d 'x' -f2)
avconv -f alsa -i pulse -f x11grab -s $(($Xaxis))x$(($Yaxis)) -i $DISPLAY.0 -r 15 -c:v libx264 -crf 0 -c:a libvo_aacenc -b:a 256k -threads 8 ~/Videos/output.mp4

I found this to work much better than gtk-recordmydesktop, which had lags, especially when "effects" were being drawn on the screen (like bulletpoints sliding in for a presentation or switching between desktops).

Friday, June 28, 2013

Tips for running SonarQube on large / legacy codebases

0 comments

Crossposted from the Transylvania JUG website.

SonarQube (previously Sonar) is a quality management platform aimed mainly at Java (although other programming languages are supported to a varying degree. Here are a couple of tips to get it working on legacy projects:

  • There is an Ant runner and a standalone runner, it is not mandatory to use Maven (although it is a good idea in general to use it)
  • Look into the analysis parameters to customize it for your code.
  • Give it space and time :-). For reference a ~2 million LOC Java project took 77 minutes to be analyzed on my laptop (an Intel i7) with 4G heap.
  • To avoid having a ton of problems reported and to focus only on new problems, look into the Cutoff plugin
  • Test and coverage reports can be reused, no need to run them twice (once for the CI system and then for SonarQube). Look into reusing existing reports. Also, make sure to use the latest version of JaCoCo when generating profile data.
  • Configure your sonar.exclusions property to ignore code you aren't interested in
  • Raise your sonar.findbugs.timeout property (the default of 5 minutes can be low for large projects)
  • Consider disabling source code related plugins (sonar.scm.enabled, sonar.scm-stats.enabled) if the provider for your SCM has an issue (HG has an issue currently for example with username containing spaces)

Keep your code clean!

Sunday, June 23, 2013

Nested fluent builders

0 comments

Crossposted from the Transylvania JUG website.

Builders have become commonplace in current Java code. They have the effect of transforming the following code:

new Foo(1, 5, "abc", false);

Into something like

Foo.builder()
  .count(1)
  .priority(5)
  .name("abc")
  .canonical(true)
  .build();

This has the advantage of being much easier to understand (as a downside we can mention the fact that - depending on the implementation - it can result in the creation of an additional object). The implementation of such builders is very simple - they a list of "setters" which return the current object:

public final class FooBuilder {
  private int count = 1;
  // ...

  public FooBuilder count(int count) {
    this.count = count;
    return this; 
  }

  public Foo build() {
    return new Foo(count, //...
  }
}

Of course writing even this code can become repetitive and annoying, in which case we can use Lombok or other code generation tools. An other possible improvement - which makes builder more useful for testing - is to add methods like random as suggested in this Java Advent Calendar article. We can subclass the builder (into FooTestBuilder for example) and only use the "extended" version in testing.

What can do however if our objects are more complex (they have non-primitive fields)? One approach may look like this:

Foo.builder()
  .a(1)
  .b(2)
  .bar(Bar.builder().c(1).build())
  .buzz(Buzz.builder().build())
  .build();

We can make this a little nicer by overloading the bar / buzz methods to accept instances of BarBuilder / BuzzBuilder, in which case we can omit two build calls. Still, I longed for something like the following:

Foo.builder()
  .a(1)
  .b(2)
  .bar()
     .c(1).build()
  .buzz()
     .build()
  .build();

The idea is that the bar / buzz calls call start a new "context" where we initialize the Bar/Buzz classes. "build" calls end the innermost context, with the last build returning the initialized Foo object itself. How can this be written in a typesafe / compiler verifiable way?

My solution is the following:

  • Each builder is parameterized to return an arbitrary type T from its build method
  • The actual return value is generated from a Sink of T
  • When using the builder at the top level, we use an IdentitySink with just returns the passed in value.
  • When using the builder in a nested context, we use a Sink which stores the value and returns the builder from "one level up".

Some example code to clarify the explanation from above can be found below. Note that this code has been written as an example and could be optimized (like making using a single instance of the IdentitySink, having FooBuilder itself implementing the sink methods, etc).

Implementation of a leaf-level builder:

interface Sink<T> {
  T setBar(Bar bar);
}

final class Bar {
  // ...
  public static BarBuilder<Bar> builder() {
    return new BarBuilder<Bar>(new Sink<Bar>() {
      @Override
      public Bar setBar(Bar bar) { return bar; }
    });
  }
}

class BarBuilder<T> {
  // ...

  protected BarBuilder(Sink<T> sink) {
    this.sink = sink;
  }

  // ...

  public T build() {
    return sink.setBar(new Bar(c, d, fizz));
  }
}
</pre>

<p>Implementation of the root level builder:</p>

<pre lang="java" line="1">
class FooBuilder {
  // ...
  public BarBuilder<FooBuilder> setBar() {
    return new BarBuilder(new Sink<FooBuilder>() {
      @Override
      public Bar setBar(Bar bar) { 
        FooBuilder.this.bar = bar;
        return FooBuilder.this;
      }
    });
  }

  // ...
}

Conclusion: Java has some missing features (liked named parameters or the ease of reuse provided by duck-typing). We can work around them however nicely with some carefully crafted code (and we can put repeating code into code generators to avoid having to write it over and over again). In exchange we get a very versatile and good performing cross-platform runtime.

Wednesday, May 29, 2013

Passing UTF-8 trough HTTP

0 comments

These days we should write every code as if it will be used by international people with a wide variety of personal information (just look at Falsehoods Programmers Believe About Names for some headscratchers). I would like to do add my small contribution to this by showing how UTF-8 encoded strings can be passed into GET/POST parameters.

For this I'll be using the following small PHP script, which can be quickly run by the command line PHP webserver added in PHP 5.4:

<?php header('Content-Type: text/html; charset=utf-8'); ?>
<code><pre>
GETs: <?php print_r($_GET); ?>
POSTs: <?php print_r($_POST); ?>
</pre></code>

We'll test this with the following Python script:

#!/usr/bin/python
# vim: set fileencoding=utf-8 :
import urllib
import urllib2

params = {'name': u'東京'}
params = { k: v.encode('utf-8') for k, v in params.iteritems() }
data = urllib.urlencode(params)

url = 'http://localhost:8000/?' + data
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)

print response.read()

This all works well and nicely, so here are some conclusions:

  • GET and POST variables need to be UTF-8 encoded after which they need to be urlencoded ("% encoded"). See this StackOverflow answer.
  • Based on the same answer: hostnames use Punycode instead (but we are not concerned with hostnames here)

  • You might need to add the following header for POST requests to work: "Content-Type: application/x-www-form-urlencoded; charset=UTF-8"
  • Failing to observe this sequence leads to an UnicodeEncodeError in urllib.urlencode

Connecting to the MtGox market data feed using Perl

0 comments

For a recent project I needed some realistic market data for an electronic exchange. Seeing how MtGox provides free and open access to theirs (thank you!) I chose them. However none of the examples floating around the internet seemed to work, so I whipped one up using Net::Async::WebSocket::Client. Enjoy:

use IO::Async::Loop;
use Net::Async::WebSocket::Client;

my $client = Net::Async::WebSocket::Client->new(
        on_frame => sub {
                my ( $self, $frame ) = @_;
                print "\n", $frame, "\n";
        },
);

my $loop = IO::Async::Loop->new;
$loop->add( $client );

$client->connect(
        host => 'websocket.mtgox.com',
        service => 80,
        url => "ws://websocket.mtgox.com:80/mtgox",
        on_connected => sub {},
        on_connect_error => sub { die "Cannot connect - $_[-1]" },
        on_resolve_error => sub { die "Cannot resolve - $_[-1]" },
);

$loop->loop_forever;

(it is basically the sample program for the module, with the MtGox market data URL hardcoded).

Thursday, February 07, 2013

Converting datetime to UTC in python

0 comments

So you need to convert a python datetime object which has a timezone set ("aware" in the Python nomenclature) to an UTC one with no timezone set ("naive"), for example because NDB on GAE can't store anything else. The solution will look something like this:

date = date.astimezone(tz.tzutc()).replace(tzinfo=None)
For searcheability: the exception thrown by NDB if you fail to do this is "NotImplementedError: DatetimeProperty updated_at can only support UTC. Please derive a new Property to support alternative timezones."

Tuesday, February 05, 2013

Recovering your RoTLD password for domains registered trough Gandi.NET

0 comments

If you need to "recover" your RoTLD password when the .RO domain is registered trough Gandi.NET (I say "recover" because you didn't set it in the first place :-)) - do this:

  • In the Gandi interface go to Account Management -> Update account information and set "Anti-spam system" to No
  • Go to the RoTLD password recovery page and reset your password. Now the password recovery email should have arrived to your inbox.
  • (Optional) go back to the Gandi interface and re-enable the anti-spam system

I needed to do this to enable CloudFlare on IT-Events.RO because RoTLD allows the changing of nameservers only trough their website.

Friday, January 04, 2013

Free software for Windows

0 comments

Inspired by this post I decided to list the software I use/recommend under Windows.

Free/Libre Open Source Software:

  • LibreOffice (for of OpenOffice) - a very capable office solution. Most people don't need anything more than this. If you are installing it for a non-technical user, make sure to set the default file formats to the Microsoft ones (.doc, .xls, ...)
  • Far Manager - a very cool two-panel file manager, for those of us who like text-based things (don't let the looks fool you - it is very modern and very capable). You could also take a look at NDN or DNOSP, but FAR is my personal favorite.
  • VLC - THE video player. It can play 99.999% of the media out there and it won't pollute your system with all kinds of DLLs.
  • ffdshow-tryouts - DirectShow / VFW codecs for all the formats VLC can play (in fact they are both based on the ffmpeg project). Use this to play back videos in programs which are DirectShow based (like WMP or Winamp).
  • 7-zip - for all your zipping and unzipping needs. It supports a lot of other formats too (mainly for extracting) so no need to install the shareware version of WinRar
  • Firefox - 'nuff said. There are also a lot of plugins which one might find useful like Firebug, NoScript, RequestPolicy, etc. As a download manager I would recommend DownThemAll, but I found that with recent increases in Internet access speeds I don't need a download manager.
  • Notepad2 for your small (text) editing needs. There is also Notepad++ and notepad2-mod.
  • PDFForge to create PDFs from any program which can print (it acts as a virtual printer which outputs its results to a PDF file). Sidenote: LibreOffice can natively save to PDF, no need for this if you're using it only for that.
  • Pidgin - you might know it as GAIM, a multi protocol instant messenger. And it is very multi protocol. The only downside is that some advanced protocol features are not always functional (*cough* file-transfer, *cough*), but I find that I rarely use those anyways. If you are installing it for someone else however, make sure to ask them what features they consider essential (like custom background/emoticons) and act accordingly.
  • WinSCP to copy files trough SSH/SCP, and the FileZilla client to do the same over FTP.
  • Various specialized programs: GIMP for photo-editing, Inkscape for vector-based graphics, VirtualDub and Avidemux for (linear) video editing, Wireshark for network analysis, Audacity for audio editing, PuTTY as an SSH client, VirtualBox for running virtual machines and Eclipse for development (it can do more than Java!). There is also the IntelliJ Community Edition for developement which is Open Source Software, but be aware of the limitations.
  • XAMPP for quickly setting up a LAMP environment under Windows

Free (as in beer) - these may try to trick you into installing toolbars / changing your homepage / your search engine so watch out:

  • TeamViewer - a nice remote control solution (also cross platform - although it doesn't run perfectly on other OSs). I especially like the fact that it can run as a service and that it takes care of the NAT traversal problem.
  • CDBurnerXP - for burning optical media. Nothing special, but it works.
  • IrfanView - a very capable image viewer / converter. Don't forget to install the plugins to take full advantage of its features! It is also so lightweight that won't believe how quickly the installation finishes. Watch out though, it tries to install sponsored programs :-(
  • FreeCommander - a two panel graphic file manager. Recommended if you're a Total/Windows Commander fan rather than a Norton Commander one :-)
  • The MS Visual Studio Express series - a good way to get your feet wet with MS specific development (also good for university projects), but be aware that you'll quickly hit a wall with it on professional projects.
  • uTorrent for my downloading needs.
  • Daemon Tools Free for my ISO (CD/DVD/BR) mounting needs. Attention during install! It will try to "upgrade" you several times during install and also try to install additional software / change your home page / search provider if you just click trough next. Don't let it, form a technical point of view it is a great product. There is also the unsupported Virtual CD product from Microsoft and Windows can mount ISOs natively starting from Windows 7 I think.
  • The FoxIt PDF Reader - a lightweight PDF reader, although Adobe Reader X caught up nicely I feel (and they also auto-update to eliminate security vulnerabilities), so you could give it a second go.
  • BB FlashBack Express for screen recording. Little annoying to install (you need to give them an email account, they try to upgrade you to the paid version and you need to "register" afterwards), but after installing it is all good and I found it to be a very capable product (even at the free version level).
  • Chrome and Opera as alternative browsers (and no, Chrome is not Open Source, Chromium is).
  • Paint.NET for advanced but "not photoshop level" image editing.
  • foobar2000 or Winamp (with the classic skin :-)) for music. foobar is very lightweight and quick, but it might lack some features. Winamp is very complete, but tries to make all kinds of changes to your system. You also probably don't need the Winamp Agent to run in the background all the time :-)
  • Dropbox for file synchronization / small-scale file serving. Alternatively there is SkyDrive, but it isn't very Linux friendly.
  • Windows Live Writer - the best blog publishing software I could find. Unfortunately Microsoft ruined it with the Office 2007 look and now seems to want to abandon it completely
  • Skype for video-call / teleconference, although lately I've been dropping it in favor of Google Hangouts

Update: this software looks very promising: Ninite. It purports to auto-install and auto-update a lot of common Windows software. Will take it for a spin the next time I install Windows.