Youtube channel embedding being all borked up?

Update: Google/YouTube seem to have taken the option to generate gadgets showing all the videos from a given user offline, and as such the script below doesn't work anymore. As a workaround I've posted a small script which generates the old embed code for an user-specified YouTube user.

In a previous post I wanted to embed two youtube channel gadgets, but after pasting the official code, I kept getting the following error message: "The URL is not valid and cannot be loaded."

After looking into it a little, it seems at one step the following code is generated: <iframe src="http://?... (the domain is missing). After searching around a little, I came up with the following Perl script which generates (what seems to be) the correct embed code:

use strict;
use warnings;
use LWP::Simple;

my $youtubeCN = get(shift @ARGV);
die("Script tag not found!\n")
 unless $youtubeCN =~ /<script src="(.*?)">/;
my $youtubeScr = $1;
$youtubeScr =~ s/&/&/g;
$youtubeScr =~ s/&/&/g;
my $gmodScr = get($youtubeScr);

$gmodScr =~ s/http:\/\/\?container/http:\/\/www.gmodules.com\/ig\/ifr\?/;
print "<script type=\"text/javascript\">$gmodScr</script>\n"; 

It needs one parameter, the URL of the youtube channel: perl bork_youtube_channel.pl 'http://www.youtube.com/user/pennsays' and it will spit out the code you need to insert. Warning! This is an unofficial method and has the drawback that it may break later on when YouTube changes their embed model. You can download the latest version of the script from my SVN repo.

PS: it seems that playing videos with the channel gadget is more permissive than directly on Youtube, in the sense that I didn't get a single "this can not be played due to age/location restriction" message.

PS2: Does anybody have an idea why do I have to do the replacement twice? When I do it only one, it doesn't seem to "stick". :-(

Update: so I'm an idiot, the URL was double-encoded for some reason...

You've come a long way baby!

In the spirit of Fatboy Slim's Right Here / Right now

Comes the following video about Yahoo Mail:

Communication Evolution @ Yahoo! Video

I got started on Yahoo Mail back in the day and it always served me well. Thank you!

PS: Cat and dog saying Yahoo Mail

Interesting media

Short updates from Penn Jillette of Penn & Teller fame on the pennsays YouTube channel. My only grief is that a few videos are unavailable due to “age or location restrictions”, which really seems just an oversight, since the content most probably isn’t published anywhere else (like TV) to warrant such restrictions. I tried to contact them trough the YouTube messaging system about it, but received no response as of now.

An other interesting (to me, and hopefully to you dear reader) YouTube channel is the one of Weird Al Yankovic. Unfortunately here most of the videos are not viewable from Romania, but the ones which are, are still worth it:

Finally a very “chill-outy” song from a fellow blogger:

Breaking into a process before the TLS gets executed

I found out about this from the SANS blog: you can make Olly break before the TLS get executed. Just Debugging Options –> Events and set “Make first pause at” to “System breakpoint” instead of “WinMain”. Cool! (until now I was patching executables with TLS to avoid them being executed).

Nessus not returning any results?

I mentioned the problem briefly in my GFI Languard review, but just wanted to document it more clearly, maybe it will be useful to somebody: with the default settings my Nessus setup didn’t seem to return any results for a non-firewalled XP host. A quick packet capture seemed to indicate that Nessus was trying to portscan using SNMP and it received an empty response and thus it stopped. The solution I found was to disable all portscanning options, other than the TCP scanner:

Disclaimer: I’m only an n00b Nessus user, so the real problem might be somewhere else, but this seemed to resolve the problem for me.

StackOverflow lightning talk

Below is a short presentation which hopefully I will give later today about Stack Overflow to the Transylvania JUG:

Lightning talk about stackoverflow

View more Microsoft Word documents from cdman83.

Update: the presentation went well (IMHO) and hopefully it will add a few more people to the StackOverflow userbase. Funny thing though: you know the warning in OpenOffice when you save it in PowerPoint format (something along the lines of “this output format will not preserve all the formatting options”) – this is the first time I actually saw it happen – on the second slide I say “For money” with strikethrough – the point I was making is that I’m not affiliated with StackOverflow, however in the PPT version the strikethrough wasn’t present. Funny, and an important lesson: check the, if possible in the environment you will be presenting in.

Hidden Java concurrency bugs

Question: how can the following line of Java code throw the exception shown below?

priv.addAll(common);

Exception in thread "Thread-1" java.lang.ArrayIndexOutOfBoundsException
	at java.lang.System.arraycopy(Native Method)
	at java.util.ArrayList.toArray(Unknown Source)
	at java.util.ArrayList.addAll(Unknown Source)
	at TestConcurrentList$ConsumeThread.run(TestConcurrentList.java:34)

Answer: because of bad synchronization. The scenario is the following: one thread is continuously modifying the list “common” while the second thread tries to perform the “addAll” operation on it. The testcode is shown below:

import java.util.*;

public class TestConcurrentList {
	private static List common = new ArrayList();
	
	private static class GenerateThread extends Thread {
		private List common;
		
		GenerateThread(List common) {
			this.common = common;
		}
		
		@Override
		public void run() {
			while (true) {
				common.add("foo");
				if (common.size() > 1000) common.clear();
			}
		}
	}
	
	private static class ConsumeThread extends Thread {
		private List common;
		
		ConsumeThread(List common) {
			this.common = common;
		}
		
		@Override
		public void run() {
			while (true) {
				List priv = new ArrayList();
				priv.addAll(common);
			}
		}
	}

	public static void main(String[] args) throws Exception {
		Thread gen = new GenerateThread(common),
			consume = new ConsumeThread(common);
		gen.start();
		consume.start();
		System.out.println("Waiting...");
		gen.join();
	}

}

What makes this so hard debug is that (a) the exception doesn’t say anything about concurrency (it’s not like it throws an ConcurrentModificationException), (b) the exception actually occurs in the native Java libraries and (c) the source of the concurrent modifications may not be so obvious as in the reduced test case.

Conclusion? When possible, avoid concurrency or delegate it (to an RDBMS with proper transaction / locking support for example).

PS. This bug is not found by FindBugs (admittedly the support for checking concurrency bugs is fairly low at the moment) and is dependent on the version of the runtime. I reproduced it on 1.5.14, but not on 1.6.07.

Picture taken from yimhafiz's photostream with permission.

GFI LanGuard review

Disclaimer: I never worked as a professional network admin and all my experience was gathered in small testing environments. Also, I received a 12 month evaluation key from GFI for the purpose of writing this review, however aside from the key, I didn’t receive any form of compensation from anyone for writing this review.

As we all know, security is a process, which means that we need metrics and tools to monitor the values of the metrics to have an acceptable amount of security. GFI LanGuard is a tool which is designed to do just that: scan your network periodically and report the “security status” of each network node.

The installation went smoothly and the interface is quite intuitive. (This despite the fact that I used Windows 7 which isn’t officially supported – only Windows Vista is – which further proves that application compatibility between Vista is 7 is very good).

The good:

• Easy to use interface
• When problems appear, a clear indicator is shown, together with an easy fix (along the lines of: “Foo is wrong. Click here to fix it”)
• There are multiple options for customizing workings of the software
• The results are displayed and summarized in an easy to understand way and older reports can be retrieved fast (this is a very nice default compared with Nessus – and many other tools for that matter, where you have to explicitly save your reports)
• There are many “common sense” checks done which are not directly related to exploits / patches. For example things like “user X has never logged in, please consider removing it”.
• The scanning is “agent-less”. This is very nice for many reasons (security – you don’t have to wait for the agent to download, compatibility – the problems with the agent being incompatible with HIPS/AV products is eliminated, etc)
• The scans are multi-threaded. The number of threads seems to be fixed at 3, however this seems to be a good value (although on an extra-beefy machine – think dual quadcore – you might want to use higher values)
• The link to the forum (“Discuss this version...” in the upper right corner) – communities are very useful in resolving problems and can create a safety net for low priority support issues

The bad (to be clear: this is a fine product. the following list is more along the lines of “good to know” stuff – also, it is human nature to observe more things which are wrong, rather than things which are right).

• The scheduling service kept getting stopped with the warning: “Scheduled operations disable due to stopped attendant service”. Given that I didn’t test the scheduled operations this wasn’t a big problem. Also, I was testing on Windows 7, which is officially not supported by GFI LanGuard at this moment.
• Locally (on Windows 7, so it might just be that not all the kinks has been worked out) it found some spurious open UDP ports (spurious because I couldn’t reproduce it) related to “trojans”
• The list of installed software (in the “inventory” part) seems to take only the list of installed software from the registry (ie. things which you see in “Add/Remove programs”). This can be easily circumvented by using any package from Portable Apps for example. Of course scanning the entire harddrive would be be much more time consuming
• The documentation recommends disabling firewalls and AV products during scanning :-(. And it really does matter. Scanning a remote XP system with the firewall took around 7 minutes, while scanning it without the firewall took 2 minutes. The same system got scanned by NMAP in ~1.5 minutes (firewalled, and it identified correctly all the open ports – with the mention that I disabled pinging, because it would have assumed that the host is down). To be fair, NMAP doesn’t do exploit testing on the discovered ports. An other positive part: Nessus was slower than GFI LanGuard – it took 7 minutes to scan the non-firewalled XP machine (note: you need to disable the Nessus SNMP port-scanner for this to work – otherwise it seems to conclude that there are no (!!!) open ports on the remote machine – a nice way to trick Nessus?). Also, it found much fewer problems and the interface is not as friendly (disclaimer: I’m not a Nessus expert and it was using the free home feed). Even though the performance difference is large, there is no easy way to deactivate/activate (or add exceptions) to firewalls on remote systems, which means that the whole process (of turning firewalls on/off) can be rather error prone.
• Some input fields don’t do input validation, it is not entirely clear in what format they expect the input and the feedback that the input was incorrect is given relatively late. Concrete example: When creating a new scan I accidentally typed the username in the format “\\foo\bar” instead of “foo\bar” (I was copy/pasting, can you tell :-)). The feedback came only after the scan ended, in the error log.
• Speaking of the error log, there seems to be no visual differentiation between the log entries associated with different scanning runs and I found no way to clear the error log.
• The checks are mainly geared towards Microsoft products. While Microsoft is a big target, attacks against third-party applications are becoming more common (for example the Adobe products seem to be a big target these days). For similar checks in the free category we have the Microsoft Baseline Security Analyzer. To find outdated versions of software (not just MS software) you should consider the CSI/PSI products from Secunia (the PSI version being free). Be cautious though, the CSI/PSI product does not do configuration validation (either for Microsoft of third-party products)
• GFI LanGuard doesn’t seem to do service detection. For example if I run my webserver on port 80, it gets correctly identified as a webserver and the relevant checks are performed (for example PHP is detected). However if I change the port 12345, it simply gets flagget as the Netbus trojan. This reliance on default port numbers rather than protocol detection can create both false-positive (like a webserver running on port 12345) and false negative (an attack tool running on port 443) issues.

All in all it is a good general tool. It lacks certain capabilities of more specialized tools, but the utility of presenting all the information in one unified interface should not be underestimated. The pricing is per scanned IP / year, which is pretty much the standard for these types of tools. My conclusion is that if you have some budget for such products, GFI LanGuard is a good option. If you have no budget at all, you might want to consider some of the free products listed in the review (then again, the free products are not as well integrated, so in the end they might end up costing you more...) And finally if you have a larger budget, you should consider going with a “best of breed” solution (even though there might be an overlap between the features of the respective tools, it is good to have multiple validation of the same problem area (just in case a tool misses some of them).

Picture taken from kat m research's photostream with permission.

Pidgin/Yahoo issues

It seems that Yahoo has dropped support for legacy way of login, causing some problems with Pidgin. Possible solutions:

• Update to version 2.5.7 (works for me :-))
• Accounts -> Edit account (to yahoo account) -> Advanced -> and to Pager Server erase “scs.msg.yahoo.com” and put instead 66.163.181.189

Further info:

• The bug on Launchpad – also gives a more detailed list of alternative IPs
• The blogpost announcing Pidgin 2.5.7
• Update: even more information abut the issue.

Picture taken from BlueLinden's photostream with permission.

Java tips&tricks

A couple of quick/short Java tips:

• The Collections.unmodifiable* just wrap the original object in a facade class which throws UnsupportedOperationException when you invoke operations that would modify it (like add or remove), but the original collection still remains mutable (and the mutations are reflected in facade object). This might be obvious to some, but not to others. The example below also utilizes the “double brace initialization”.

@SuppressWarnings("serial")
private static void testROSet() {
    Set rw = new HashSet() {{
        add("foo");
        add("bar");
        add("baz");
    }};
    Set ro = Collections.unmodifiableSet(rw);
   
    System.out.println(ro);
   
    try {
        ro.add("barfoo");
    }
    catch (UnsupportedOperationException e) {
        System.err.println("Exception!");
    }
   
    rw.add("barfoo");
    System.out.println(ro);       
}

• The File class refers to file/path names, not actual files/paths. From the documentation (emphasis added):

  An abstract representation of file and directory pathnames.

  User interfaces and operating systems use system-dependent pathname strings to name files and directories. This class presents an abstract, system-independent view of hierarchical pathnames.

  again, this might be obvious to some, but I feel that the class is a little misleadingly named. It would be more appropriate to name it "PathName" for example (of course there is almost no chance that such change will be made, for compatibility reasons).
• Logging with varargs: unfortunately in the current version of log4j (1.2) there is no support for varargs (possibly because it was released before Java 5). Fortunately it is quite easy to roll your own, using a facade class like the following:

class VarargFaccade extends Logger {
    Logger log;
   
    public VarargFaccade(Logger log) {
        super(log.getName());
        this.log = log;
    }
   
    public void infoVA(String message, Object... arguments) {
        if (!log.isInfoEnabled()) return;
        log.info(String.format(message, arguments));
    }
   
    public void debugVA(String message, Object... arguments) {
        if (!log.isDebugEnabled()) return;
        log.debug(String.format(message, arguments));           
    }

    ...
}

It uses the recommended way to check whether it should log, and if so, it passes the message trough String.format. As an alternative to String.format you could use MessageFormat.format (which also has support for replacing error messages from resource bundles). Here are also some quick benchmark numbers using the NullAppender. These are 2 000 000 iterations for each message, so even the worst performer (MessageFormat) only adds an overhead of 0.0000135s per message (!) – meaning that it shouldn’t make you think that logging is a bottleneck in your application, unless you’ve profiled your application and you are 100% convinced that it is.

Simple string message: 0.58
Concated message: 1.74
Formatted message: 13.06
Passed in with varargs: 12.97
Passed in with vargargs/MF: 26.78
Passed in with varargs, no logging: 0.09

• Finally, there is a discussion about Hidden features of Java on Stackoverflow, which I found very useful. One of my favorite features was VisualVM, a monitoring tool which you can attack to a running JVM on-the-fly to gather some stats about it. Below you can see a short video about it:

Picture taken from june29's photostream with permission.

Comparing CVS revision numbers with Perl

Update: see a faster version here. However, make sure that you've nailed the problem down before starting to optimize. (The profiler is your friend)

The code below was only lightly tested and isn’t all that efficient, so use it at your own risk. It returns –1 is the first version is smaller, 1 if it is bigger and 0 if they are equal. I needed it for crunching some CVS logs and I couldn’t find anything on CPAN:

use strict;
use warnings;
use Test::More tests => 8;

is(cmp_cvs_tags('1.1', '1.1'),    0);
is(cmp_cvs_tags('1.1', '1.1.1'), -1);
is(cmp_cvs_tags('1.1', '1.2'),   -1);
is(cmp_cvs_tags('1.2', '1.2'),    0);
is(cmp_cvs_tags('1.2', '1.3'),   -1);

is(cmp_cvs_tags('1.1.1', '1.1'), 1);
is(cmp_cvs_tags('1.2',   '1.1'), 1);
is(cmp_cvs_tags('1.3',   '1.1'), 1);

sub cmp_cvs_tags {
  my ($a, $b) = @_;
  my @a_lst = split /\./, $a;
  my @b_lst = split /\./, $b;
  
  my $i = 0;
  while (1) {
    return 1  if (exists($a_lst[$i])  && !exists($b_lst[$i]));
    return -1 if (!exists($a_lst[$i]) && exists($b_lst[$i]));
    return 0  if (!exists($a_lst[$i]) && !exists($b_lst[$i]));
    if ($a_lst[$i] == $b_lst[$i]) {
      ++$i;
      next;
    }
    return $a_lst[$i] <=> $b_lst[$i];
  } 
}

Quirky Perl tricks

Perl is like a sharp knife: you can do a lot of things with it easily (like carve wood), but you easily hurt yourself. So here are some quirky things which I discovered (credit goes to my friends who came to me with these issues and helped me to grow my knowledge by researching the issues).

• First issue: what is wrong with the following code? (in fact there is nothing wrong, but it can behave in unexpected ways):

my ($input) = <STDIN>;
print "Got: $input";

The unexpected behavior is that this code tries to read multiple lines, instead of a single line. The explanation is hat Perl subroutines know if they are expected to return a single value or an array of values, and in this case it concludes that you want an array, which means all the lines from the file handle (the standard input in this case). However Perl doesn’t have any way (as far as I know :-)) to find out exactly how many elements you want, so it has to give you all and throw away the superfluous elements at the assignment. What is the solution? Either omit the parenthesis around the variable declaration (which implicitly tells Perl that you need a scalar) or explicitly evaluate the read operator in a scalar context. The example below shows both methods (although either one is sufficient):

my $input = scalar(<STDIN>);
print "Got: $input";

• Second issue: is it possible to extract more than one value from a for cycle during one iteration? The code would be something like this:

foreach my($foo, $bar) (qw/a b c d/) {
	print "$foo, $bar\n";
}

From what I know, this is currently not possible, although there is a proposal for this in Perl6.

Picture taken from kaibara87's photostream with permission.