Using Perl to access PostgreSQL under Windows

This appears by a non-intuitive problem for people. Below I assume that you are using some version of ActivePerl for Windows (5.8 or 5.10). First of all:

Under no circumstances (ok, I rephrase: only under extreme circumstances) should you use DBD::PgPP. It is old, not very performant (given that its implemented in Pure Perl) and has some nontrivial bugs. An example for one such bug is the following:

my $sth = $dbh->prepare('SELECT foo FROM bar WHERE baz = ? AND fuzz = ?');
$sth->execute('a?b', 'c');

With PgPP the actual query it will try to execute will be something like: SELECT foo FROM bar WHERE baz = a'c'b AND fuzz = ?. I assume that what it does is that it takes the parameters provided to execute one by one and looks for the first question mark in the query. But given that the first parameter itself contains a question mark, the method fails...

So how do you install DBD::Pg if there aren't in the ActiveState PPM repositories?

1. Go to the dbdpgppm project on pgfoundry
2. Download the correct PPD file for your needs (depending on the Perl version - 5.6, 5.8 or 5.10 - and if you need SSL support or not)
3. Issue the following command:

   ppm install the_downloaded.ppd

Calling variable functions in Perl

What I'm trying to say with this title is the following:

# ... lets say we have a value $foo ....
# ... and we want to call method_1 method_2 ... method_N on it
foreach my $method_name (1..N) {
  $method_name = "method_$method_name";
  my $func = *$method_name;
  my $result = &$func($foo);
  # ...
}

The trick is to look up the given name in the symbol table (done by the * operator), and use the obtained pointer to call the function indirectly. This can be used for example in writing test scripts, where you don't want to copy-paste the same line over and over again.

Working with bitstrings in PostgreSQL

When working with short, fixed width binary data in PostgreSQL (for example: MD5/SHA1 hashes), an option to consider is the bitstring type. It has several advantages:

• Smaller data storage requirements (also - smaller index sizes). Storing an MD5 hash as characters representing hexadecimal numbers for example takes up 32 bytes, while storing it as bit string takes up 16 bytes (exactly the amount of useful data contained).
• There are no canonisation issues. For example deadbeef, DeadBeef, deaDbeeF, etc all represent the same number in hexadecimal form, however, if there isn't a rule enforced on the table (for example: all characters must be lowercase), you can end up different entries representing the same data.

However it also has some disadvantages for which I'm trying to give workarounds (where I know of such):

• It is not standard - not really a problem, since everybody should use PG :-)
• Related to the first problem - database abstraction libraries (like DBI for Perl or PDO for PHP) don't know how to properly interpolate such data (they basically treat it as string) - the workaround - manually build the query string after carefully (!) validating the data. For example "SELECT foo FROM bar WHERE md5 = x'$md5'".
• It doesn't support prefix queries. So you can't do the following: SELECT foo FROM bar WHERE md5 LIKE x'deadbeef%'. The solution lies in the fact that characters to the left are more important in the lexicographical ordering than characters to the right. So you can rewrite the previous query as this (assuming that the md5 field has a width of 128 bits or 16 bytes):

  SELECT foo FROM bar WHERE 
    md5 >= x'deadbeef000000000000000000000000'
    AND md5 <= x'deadbeefffffffffffffffffffffffff'

  A further advantage of this method is that - being a range query - can be accelerated using existing b-tree indexes on the column.
• When data is returned from a query, it is in binary form (ie "10111000011..."). However frequently you want it in hexadecimal form. Below are some code samples for Perl, PHP and Python to accomplish this:

  # Perl
  my $row = unpack('H*', pack('B*', $row));
  

  # PHP - we need to split result because calculation have a limited precision
  # the code also supposes that the length of the bitstring is multiple of 4
  # which it has to be for it to be convertable into hex :-)
  preg_match_all('/[01]{4}/', $row, $nibbles);
  $result = '';
  foreach ($nibbles[0] as $b) {
   $result .= sprintf('%2x', bindec($b));
  }
  

  # python
  result = hex(int(row, 2))
  

Update: You can use the examples shown above to create stored procedures in PG (after installing the appropriate language of course) which would concert a bit string to hex. One note: the perl code can not be used as shown, because the pack operation is not permitted by the safe perl environment. Alternatively you can use plperlu (but be aware of the security implications).

Finding unused indexes in PostgreSQL

First some quick DB/index design tips:

• When you have a 1-1 relationship, you might consider putting the data in one table, instead of several tables and linking them with foreign keys. This will speed up data retrieval considerably. This needs to be balanced with the number of cases when the given values are missing / are duplicated.
• Don't be afraid of multi-column or partial indexes. Creating the right index can speed your queries up considerably.

Now for the main topic: finding (and eliminating) unused indexes is good for at least two reasons: (1) they don't consume disk space any more and (2) they don't need to be updated when the data changes. In PG you can find unusued indexes by issuing the following statement:

SELECT *, pg_size_pretty(pg_relation_size(indexrelname))
 FROM pg_stat_all_indexes 
 WHERE schemaname = 'public' 
 ORDER BY pg_relation_size(indexrelname) DESC, idx_scan ASC

To be able to do this, you have to make sure that the following thigs are configured in your postgresql.conf file:

• The stats collector is on (stats_start_collector = on)
• Collecting of stats is enabled:

  stats_block_level = on
  stats_row_level = on
  

• Track activities and track counts are enabled:

  track_activities = on
  track_counts = on
  

• stats_reset_on_server_start is set to a value you are comfortable with. Setting it to on means that you could loose historical data if the DB is restarted (thing reporting queries executed one a month / once a quarter for example). Setting it to off means that you may be influenced by queries which are no longer being executed.

Now you can start looking at your workload. For even more tips, you can look at the following presentation:

Pro PostgreSQL, OSCon 2008

View SlideShare presentation or Upload your own. (tags: postgres postgresql)

Update: I've just found the post which was the original inspiration for my research and which give much more detailed explanation and a few variations on the theme.

How to verify executable digital signatures under Linux?

The PE executable format (the one used by Windows) supports the use of digital certificates to verify the source of the file. Normally you can verify it using Windows Explorer (by right-clicking on the file and selecting Properties). It also shows up when you try to run an executable downloaded from the Internet with IE or FF3. If you wish to mass-verify it, you can use a tool like sigcheck by Mark Russinovich.

However, great tool as sigcheck may be, there are at least two problems with it:

• It doesn't run on Linux. Last time I've tried under wine, it complained about missing imports from the cryptography libraries (this might have changed in the meantime)
• It wants to connect to the Internet (to verify CLRs I assume). This can lead to some nasty pseudo-hanging processes on systems which (for security reasons) are cut off from the Internet but are still able to resolve domains.

So I looked into verifying the signatures myself. The method I'll describe in the following paragraphs will work on Linux (in fact it will work on any OS which has OpenSSL binaries), but it has at least two limitations you should be aware of:

• It doesn't work with catalog (.CAT) files. The catalog files use an undocumented binary format, but it has been reverse engineered by both the wine and ReactOS folks, so those implementations should point you in the right direction.
• I'm not actually verifying the validty of the signature, although I give a pointer on how you would possibly do this.

Ok, now with disclaimers set aside, here are the steps:

1. Extract the digital signature from the file. This post from Didier Stevens tells you how. What you need is actually the content after the DWORD containing the size and the marker (0x00020200). Just to be clear:

   size DWORD = the size of the signature + 4 (the size DWORD) + 4 (the marker)
   actual signature = location specified by the data directory + 8 (marker + size)
   

   The actual signature is in DER format and is a standard PKCS7 signature, as Didier says (there are two standard file formats for PKCS7: DER - a binary format - and PEM - a text format - you will need this when specifying the input format for OpenSSL)
2. To calculate the MD5/SHA1 hashes which match the one contained in the certificates, I've used the information from this mail by Peter Gutman. Basically, to calculate the matching hashes, you will need to exclude three fields from hashing: the checksum field and the address/size field from the security data directory. The area hashed ends before the location pointed by the security entry in the data directory. The digital signature should be last thing in the file. I didn't test the case when the digital signature is in the middle of the file, but I assume that either an erro is generated, or it is ignored as the afore mentioned three fields (I don't think that there is a security vulnerability which would result in semi-protected executables).
3. The hashes calculated at the previous point should match the one printed out by the command:

   openssl asn1parse -inform DER -in "signature.der"

   The signature can contain either a SHA1 or a MD5 hash. To dump the certificate tree, you can use the command:

   openssl pkcs7 -inform DER -print_certs -text -in "signature.der"

   The first in the list is the one you need to trust (this is usually a big provider like Verisign / Thawte / Comodo / etc) and the last one is the actual signatory of the file.
4. If you want to actually verify the signature, you need to create a copy of the file excluding the parts specified at step 2, and issue the command (taken from here):

   openssl smime -verify -in signature.der -content modified_executable -inform DER -binary

Vista and Dell woes

I've completed my first ever Vista installation on a relative's computer and I'm entirely underwhelmed. First of all, the only hardware it recognized out of the box was the soundcard (LOL). Just to clarify: I'm talking about a recent model of the Dell Inspiron laptop line here. Second of all, it failed to read the CD provided with the drivers (this is more of a Dell problem than a Vista problem).

Now over to Dell: even after specifying the exact model number, they can't show a personalized menu. They can't tell me if the laptop has a Broadcom or an Intel wireless chipset, so I have to download the drivers for both. They offer an option to download all the files at once (very good), but instead of using asynchronous javascript when adding the items (or even better, offer me a download everything that is listed here button), they make me reload the entire page, loose my position in the list and have to endure the painfully slow loading times (probably it was more a latency than throughput issue, because the download itself was pretty fast).

My conclusion is: none of the problems is dealbreaking, but it could be worked on. I will use Linux in the future and from where I stand, they could have called Vista XP SP3 and be done with it (of course they couldn't make any money then, could they?). There just aren't enough practical and visible improvements to make me change.

Changing jobs

I think the proverbial cat is pretty much out of the bag now, so I might as well announce it: as of the first of November I won't be working for BitDefender. It has been a fun three years, but I feel that it is time to go into a new direction. This will probably mean that the security content on the blog will be fewer, but I'm still looking forward to writing posts which convey information in a clear way to help people (in all domains related to IT). Below are a few observations about the companies / interview processes I've experienced.

• Be prepared to answer the question: why are you leaving? In almost every case they will ask that.
• When they are asking about the salary you want, first make it very clear what kind of sum are you talking about (yearly/monthly, net versus gross, in what currency, etc). If they find the sum too high and you really want to work at the given company, you might try to reach an agreement whereby you will work a time period (3-6 months) for lower salary and if they are happy with the quality of your work, then you get the bigger salary.
• Big (huge) companies usually have a predetermined chart for calculating the salary, and it seems that technical know-how isn't that important to them. This might be a good thing if you've just graduated, but a bad thing if you've been working for some time.
• Some companies just plain out ignore you, or even worse: they promise to call you back, but don't. This is just a fact of life.
• When you are discussing with technical people, don't forget to ask them your technical questions (like: what kind of development methodology are you using? do you use source control? do you do testing? what kind of testing?). These might seem natural things that everyone should have, but not everybody does.
• Regarding the previous point: most of the companies I've talked too at least say that they are agile, but there are still exceptions. The most horrendous one was a company doing work on a COBOL system for a large Austrian bank/insurance firm. When I've asked them what kind of testing they do, their answer was something along the lines of (and I'm paraphrasing here): none, we just go ahead and modify the system live on the mainframe.
• Advice for the companies: if you want to hire a technical person, involve a technical person! I understand that I have to talk with HR too, but please, please give me a living technical person to talk too. Paper tests are not that good either, or at least they should have a followup discussion to clarify any misunderstandings (because many times you find yourself wondering: is this a trick question, or did the tester make an error that s/he didn't catch?). If you don't like my price, at least have the courtesy of telling me so right then or later.
• As a candidate you can often get some background information (and you should do this) about the companies using public documents. In the case of Romanian firms a good source would be the Ministry of Finance.

Link list

The "Last in a Loop" Bug - closures are one of those tricky things. This shows a typical example for (mis-)understanding them, and how to fix it.

Via ajaxian: Degrading Script Tags. Very cool, I didn't know about this behavior. Although, I'm not really sure where you would use it? If a part of your hosting infrastructure goes down? Providing failover?

Promise you don't laugh: I've started the one hundred pushups program. On the initial test I've managed a total of five (yes, 5) pushups, so I'm curious if it is really possible to get to 100 pushups by the end of the year (in fact I have a bet with my wife that until the end of the year I will do 20 pushups with clapping in between :-D).

Yet an other (possible) security problem: Cross-site scripting through CSS data. However, in an encouraging move, Microsoft will turn off by default CSS expressions in IE 8.

A funny (but true) answer on stackoverflow to the question How many lines of debugged code do you produce in a day’s work?: I don't have an exact count, but on my best days it's a negative number. In a similar spirit here is the advice given on slashdot to a beginner programmer (emphasis added):

• Plan/Design everything
• Document everything
• Version control everything
• Test everything
• Deny everything

A little entertainment

Viewing colored output with less

When an output is filtered through less, the colored characters are transformed to black and white (or whatever your terminal settings are). This is because less, by default, strips away (filters out) escape sequences. These are special sequences which tell the terminal to do something special (like move the cursor, change the color and so on).

They are stripped away by default, so that they don't interfere with the positioning of the text by less (for example if less counted that it outputted 24 lines, but on the 12th line there was an escape sequence telling the terminal to move to the top of the scree, the output would get mixed up).

If you want to let control characters through, you can use the -r switch. It is however safer to use the -R switch, which only lets control characters through which change the foreground/background color. Even so, your screen might get mixed up, so remember that q exits less and Ctrl+L clears your terminal. If the terminal is very badly messed up, try issuing the tset command.

Autorun malware

There seems to be a lot of confusion out there about this topic, so I'll try to provide here some high-quality technical information to help users / sysadmins out.

What is autorun malware?

Autorun malware is malware which uses the autorun feature present in Microsoft Windows as a way to spread itself. This might or might not be the only spreading method it employs.

What is the autorun feature? Why is it present?

This is a feature whereby you can ask Windows to run an executable when a new disk is inserted / clicked on. Although I have no official sources, I assume that this was implemented to make the installation/use of software from removable media (mainly CDs at the time) easier. Without this feature you would have to insert the CD (DVD), open a file manager (Windows Explorer for example), navigate to the CD drive, select the appropriate file and run it. With this feature however, Windows would sense that a new CD has been inserted, see that it has instructions on which executable to run, and start it automatically. The default executable usually checks if the software is already installed, and if so, launches it from the hard-drive, or, if not, offers to install it.

While until now I've been talking about CD/DVD drives, this feature can be used on any type of drive. Drive types can be divided into two categories from the point of view of this feature:

• Those which notify the OS (Windows) about the fact that the media in them has changed. These include CD/DVD drives and USB drives of all kind. They do not include floppy drives.
• Those which don't. These include hard disks and floppy drives for example.

The autorun facility can be used with both types of drives, what differs is the time when the activation occurs. With drives where Windows is notified in the event of a media change, the new media is automatically scanned for instructions on which file to execute, and if such instructions are found, they are carried out. In the case of disks which don't notify Windows when they change, the presence of the instructions is checked (and if present, carried out), when the root of the drive is accessed (for example by going to My Computer -> C:).

To summarize, the typical flow of events is the following: a disk contains a malware file and instructions for this file to be executed automatically. When the disk is inserted / accessed (depending on the type of disk), windows executes the malware.

A small historical note: this feature was first introduced in Windows 95 / 2000, however problems only appeared recently when flash based storage (sticks) became widely used, because they provide an easily writable solution for malware (as opposed to CDs/DVDs).

Can you give me more technical details?

Of course :-). The instructions are kept in a file named autorun.inf in the root of the drive. Windows checks for the presence of this file whenever it needs to (when a new media is inserted, when the root of a disk is accessed, etc). You can read more details about the different options available on this site for example or from MSDN (Microsoft Developers Network).

The file has a text-based format, so you can inspect the files using a program like Notepad. The best way to inspect a file is to run Notepad (from the Start menu for example), go to File -> Open and type in "F:\autorun.inf" (with the quotes). Of course you have to replace the disk letter from this example (F) with the actual disk letter you are interested in. This method has at least two advantages over navigating with Windows Explorer to the root folder, finding the file and rightclick - editing it:

• Navigating with Windows Explorer might trigger the execution of the program indicated in the autorun file (see the previous point). This can be bad if we are talking about possible malware
• The file might have the hidden attribute set, so that it's not visible by default in Windows Explorer (and the folder settings tab, where you can change the options for displaying hidden files, might be deactivated - see further down)

This file acts as a pointer to the actual executable which is run in case the feature is active. The path of the executable is relative to the root of the drive where the autorun.inf is located. Thus, if you see something like foo\bar.exe in the file F:\autorun.inf, the actual executable file is F:\foo\bar.exe.

What is the difference between Autorun and AutoPlay?

Autoplay is the name of a different technology which aims to achieve the same thing: open up a relevant application when a new media (CD / DVD / USB) is inserted. This is done by searching the media upon insertion to determine the majority content type (music, pictures, video, etc) and presenting a menu based on it.

The presented menu contains programs which are installed on the local computer. This means that no new software is run automatically, making this feature a whole lot less risky than the autorun feature. Autoplay is only activated if autorun is not present or has been disabled (or to put it an other way: autoplay is related to the media change notification - described below - while autorun is one level up).

Can I turn off autorun? What is the disadvantage of turning it off?

The good news is: yes, it is possible. There are several methods actually:

The first (temporary) solution is holding down the shift key while inserting the new device (CD / DVD / USB stick). This will prevent autorun from kicking in, however it has several disadvantages:

• It is only a temporary, one-time solution. You must remember to always do this whenever you insert new media.
• It is prone to error. You might press it to late or release it too early.
• There is no visual or audible feedback that you proceeded correctly (pressed down in time and didn't release it too early)

A second solution is to disable the media change notifications. This isn't recommended and has several disadvantages (the most severe being that it doesn't affect USB devices, the main problem source). If you still want to do this, here is the command you have to execute (adapted from this MS KB article):

REG ADD HKLM\System\CurrentControlSet\Services\CDRom /v Autorun /t REG_DWORD /d 0 /f

This disables MCN for all the CD/DVD drives. There are two other registry keys which can selectively disable MCN for a particular CD/DVD device (if multiple devices are present in the computer). There are documented on Technet, but again, this approach is not recommended.

The third, and recommended, approach would be to use the NoDriveTypeAutoRun registry key (the referred Microsft documentation describes Windows 2000, but the information is relevant to newer versions of Windows too). When this key is set, Media Change Notifications are delivered, but the autorun files are not parsed / acted upon. This key is actually a combination of values, which describe what type of drives (not) to use the autorun.inf feature for. For example, to disable autorun for all types of drives, you would use the 0xFF value:

REG ADD HKCU\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer /v NoDriveTypeAutoRun /t REG_DWORD /d 0xFF /f
REG ADD HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer /v NoDriveTypeAutoRun /t REG_DWORD /d 0xFF /f

As you can see above, there are actually two registry locations: one for system wide (HKLM) and one specific for the local user (HKCU). From the technet documentation:

Autoplay is also disabled on any drive if it is disabled by the value of NoDriveAutoRun (in HKLM or HKCU) or NoDriveTypeAutoRun (in HKLM or HKCU). But if NoDriveAutoRun or NoDriveTypeAutoRun appear in HKEY_LOCAL_MACHINE, the corresponding entries in HKEY_CURRENT_USER are ignored.

This means that it is best to set the flags in both locations. If you would like to disable autorun for all types of drives except CD/DVD drives (a more relaxed, but still mostly secure choice), you should use the value 0xDF.

As for the disadvantages: the impact for turning off autorun for non CD/DVD drives is minimal. Autoplay is still active, meaning that if the user is accustomed to selecting actions from the menu (for example when inserting a memory card with photos s/he can select directly a photo management application), s/he can still use this method. When turning off autoplay for CD/DVD drives, applications from there will not run automatically, which can be a usability hurdle (for example the user might be accustomed to launching an application by inserting its CD) and users need to be retrained. Finally, disabling media change notifications is the most problematic solution (and actually, somewhat misguided, because it only affects CD/DVD drives, not the source of most problems, USB drives) and can result in phantom files appearing (the contents of the old disk being shown, even after a new disk has been inserted into the drive). This method is not recommended at all.

Update: this blog posting from the McAfee blogs seems to indicate that under some circumstances the settings can reset themselves, and you can use the following registry hack as a more permanent solution:

REGEDIT4
[HKEY_Local_Machine\Software\Microsoft\Windows NT\CurrentVersion\IniFileMapping\Autorun.inf]
@="@SYS:DoesNotExist"

Update: It seems that Windows caches the autorun.inf files executed during the current logon session under the HKEY_CURRENT_USER\ Software\ Microsoft\ Windows\ CurrentVersion\ Explorer\ MountPoints2 key. This means that even if you disabled autorun, you may get reinfected if the given autorun.inf file has already been seen during the current logon session. To counter this, be sure to logout/login (or better yet, restart your computer) after making the changes and test them using a bening autorun.inf.

Update: Given the recent problems US-CERT published a document titled "Microsoft Windows Does Not Disable AutoRun Properly", in which they say that Windows might not disabled autorun, even if the proper registry keys are present and they recommend the "@SYS:DoesNotExist" method. Microsoft quickly reacted and it seems that you need to install a hotfix (which doesn't get pushed automatically via Windows Update :-( for XP, 2k and 2k3). The CERT document has also been updated.

I have a small / medium / large network of computers. Can I make these changes automatically on each one?

You have the computers in a Domain, you can use Group Policy to do the changes. If they are not, you can create a batch file with the appropriate commands for example and run it on each computer. You can even use the autorun.inf file to disable the autorun feature: grab an USB stick, create a batch file which executes the commands and an autorun.inf file which points to the batch file. Now go around a put the USB stick in each computer. Ironic, isn't it? :-)

Why do you use the REG command instead of importing .reg files or the registry editor?

Many malware families set a registry key which prevents regedit from functioning. Reg.exe however doesn't verify this registry key and works regardless of its value. It is also easier to use from batch files.

I can't run task manager / registry editor / change folder settings / use "Run" from the start menu. Is this related?

Probably. It is common for this type of malware to disable these tools to make their detection / removal harder. What can you do?

To enable the task manager:

REG ADD HKCU\Software\Microsoft\Windows\CurrentVersion\Policies\System /v DisableTaskMgr /t REG_DWORD /d 0 /f
REG ADD HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System /v DisableTaskMgr /t REG_DWORD /d 0 /f

To enable the registry editor:

REG ADD HKCU\Software\Microsoft\Windows\CurrentVersion\Policies\System /v DisableRegistryTools /t REG_DWORD /d 0 /f
REG ADD HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System /v DisableRegistryTools /t REG_DWORD /d 0 /f

To enable the folder settings page in explorer (to be able to see hidden files):

REG ADD HKCU\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer /v NoFolderOptions /t REG_DWORD /d 0 /f
REG ADD HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer /v NoFolderOptions /t REG_DWORD /d 0 /f

To enable the run menu:

REG ADD HKCU\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer /v NoRun /t REG_DWORD /d 0 /f
REG ADD HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer /v NoRun /t REG_DWORD /d 0 /f

To enable the command prompt:

REG ADD HKCU\Software\Microsoft\Windows\CurrentVersion\Policies\System /v DisableCMD /t REG_DWORD /d 0 /f
REG ADD HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System /v DisableCMD /t REG_DWORD /d 0 /f

It is not uncommon for malware to set these registry keys in a loop (for example once every second), so you should try to remove any infection from the machine before trying to reset the registry settings.

My Anti Virus product stopped working. Is this related?

Possibly. It is common for malware to contain a list of processes/services (related to security products) and try to kill these processes / stop the services (usually they don't succeed if they aren't running with administrative rights, which is an other good reason to be running as a non-privileged user). This can partially or entirely disable security products.

To re-enable your security product(s), after cleaning the machine, it might be enough to restart it. If this isn't effective, try running the original install kit for it. Many of the install kits have a repair option, which you should try. Finally, if this didn't fix the problem either, you should uninstall and then reinstall the product.

I have a server which shares out directories to clients. autorun.inf files keep turning up in the shared directories. Is my server infected?

Probably not. Remember that the autorun.inf file has to be located in the root of the drive to function. The most likely cause is that a client, which mapped the share to a drive letter, is infected (because to it the shared directory is the root of the drive - the network drive that is).

What can you do? When possible, share directories only as read-only. To find out which client is infected, I successfully employed the following method: start capturing the network traffic with Wireshark. Delete the autorun.inf file and wait until it re-appears. Now stop the capture and search in it for the string autorun.inf (without the quotes). This should provide you with the IP address of at least one infected client.

How does this relate to the U3 technology?

U3 compatible devices are a class of USB storage devices with some special hardware in them. A part of the flash storage is reserved and stores a image of a CD. When inserted in a computer, the hardware presents two devices: a CD drive which contains the data from the reserved area and a standard USB storage device, consisting out of the remainder of the storage. This means that we have to think about a U3 device as two separate devices: a CD drive and a USB stick. There isn't anything magical or mysterious about these devices which allows them to auto-execute, the same settings apply to them as to normal CD drives or USB sticks. If you for example disable autorun for all devices, the U3 Launchpad won't execute.

Some advice for USB stick owners

If you have an USB stick which you carry around and put in other people's computers, make sure that you have disabled autorun on your computer and check the stick periodically for an autorun.inf file. Some models have a write-protect switch, which (when activated), makes the stick read-only at a hardware level, protecting it from infection. If possible, get one with this switch and use it whenever you only want to share data.

And finally: remember that USB stick are very tiny and can easily get lost / stolen. Use something like TrueCrypt with a strong password to keep the information safe on it from prying eyes, even after you loose control of the physical device.

Update: Panda released a tool to immunize computers / flash drives. While it is far from perfect, and is effective in 99.99% of the cases (at least until the bad guys catch up and start to put code in the malware to remove the "unremovable" autorun.inf folder).

Update: Via the Microsoft MMPC blog: Autorun will be disabled for USB sticks in Windows 7. This is a very good move, which removes the source of the problems, but there is a long way until Windows 7 becomes the main version of Windows out there (given how it's not even out yet).

Fun post

Via the Ajaxian blog: a Javascript simulation of the Large Hadron Collider. For even more fun check out the webcams of the LHC.

Sending authenticated mail with Net::SMTP

As I've said earlier, one thing about open source is that you can fix fairly easily. In Perl this is especially easy, since it is turtles all the way down - many of the libraries are written in pure Perl, meaning that you can step through them with the Perl debugger.

This helped me recently for example, when I was trying to use Net::SMTP with a server which required authentication, and it kept failing on me. Finally I used the debugger to step into the library, and saw the following code:

eval {
  require MIME::Base64;
  require Authen::SASL;
} or $self->set_status(500, ["Need MIME::Base64 and Authen::SASL todo auth"]), return 0;

So, I installed the Authen::SASL module and everything worked. While I would have preferred for the module to say it more explicitly (dying with a suggestive error or in the documentation), but still, the fact that I could look at the source and understand the problem is great.

Breaking into the debugger programatically with Perl

In some situations you might wish to programatically stop the execution of a script if a debugger is attached. Some use-case scenarios:

• You are debugging an area of the code which gets frequently executed (in a loop for example), but you are only interested to see its status under certain conditions
• You have a central place for error handling (for example a log_error type routine) and you wish to be alerted during debugging if it is called (because it means that possibly there was an error and you would like to examine it live - the stack trace, variable, etc)

This can be done the following way:

$break_debugger = $^P;
# ...
$DB::single = 1 if ($break_debugger);

What does this code do? First, it uses a variable to enable/disable this feature. This can be useful if during the debugging session you decide that you don't want to be alerted any more (you simply eval the expression $break_debugger = ). The $^P predefined variable is set when a debugger is attached, and not set otherwise, providing a good default value for the flag.

One final note: this code breaks at the next line of code after it (actually, as I understand it, it puts the debugger in single step mode, which is checked before executing each line). This means that if this is the last line in a subroutine, the break will occur at the next line after you return from it (at its caller, or at its caller's caller, etc). If you wish to examine the stack / parameters / etc, take care to not to place it on the last line of the subroutine.

What you are willing to pay for

Disclaimer: as always, these are my own opinions, and don't necessarily reflect the opinions of my past or current employers.

To be read with Eminem - Loose Yourself in the background

I've just finished an intervention at a large company. They had a major virus problem and we were brought in to offer expert advice. Broadly, the situation was as follows:

• Somebody brought in an infected USB stick with one of those autorun worms on it (from which a dime is a dozen)
• It infected their computer
• Their working environment consisted out of filservers sharing out programs which the clients mapped out as (writable!) drives. The infection quickly spread to these shares (the worm just saw them as drives and copied itself there).
• As other clients started connecting to those shares, they too got infected. Now different people have to run different applications from different servers, meaning that the infection quickly spread (through infected clients which had mapped drives from more than one server) to around (we were told) 300 servers around the country.
• To make matters worse, this was a slightly advanced version: it had file infecting capabilities (so simply deleting the dropped file from the root of the share didn't solve the problem, because in the meantime it infected the other - writable! - executables from the share), making its spreading even faster. It also injected itself in other processes and killed off security solutions

We offered up several solutions, from which they choose a suboptimal one, but even so, they will manage to eradicate this infection in a couple of days. However, it was a truly eyeopening (in the worst possible sense) experience. Mind you, I'm talking here about a company which has enough cash to buy a large portion of the country!

• They were more interesting in buying a somewhat effective solution than implementing an internal solution which - although would have needed some work from - would protect them from this and many other possible problems.
• The IT guys seemed generally clueless. Their level of knowledge didn't change much as we went up the hierarchy, but they arrogance went up and their willingness to listen declined dramatically...

In general, the situation (and their level of expertise) wasn't much different from what I saw over at a smaller company I've helped out with some IT advice, even though they were bigger by a factor of 2-3.

This go me thinking: do we know what we don't know? How much are people willing to trust summaries and sound-bytes while putting critical thinking on hold? In the area of IT-security (and probably in all other areas, but this is where I have first hand experience) it seems that everyone (and I mean everyone - even more technical people who should know better) lives on the few quotes got from the even fewer researchers. And even they can not be trusted entirely, because those of them who like to give quotes the most are media junkies who will spin (almost) anything just to get in the news.

You just need to apply a little common sense here. Its easy, like 1, 2, 3. Here it goes: police didn't eradicate crime. You pay your police forces (wherever you might be) some amount of money (indirectly, through taxes most probably) some amount of money, which is probably more than 20 EUR / year (the approximate price for an AV license). So why do you think that smaller organizations (security companies - even combined - are smaller than police forces - and this assumes that they cooperate, not compete - which is not the case) for less money can keep you safer in the virtual world than police forces can do it in the physical world? And remember: crime happens even though we have police forces!

Or take an other simple demonstration: what type of security product is the most widely used currently? The known malware scanner (Anti-Virus). What does this mean? It means that the company has to have a sample of the malware (or a variant of it) before it can add detection for that. Where do the companies these samples? From infected customers mostly! Now, it may not always be their customers (they may get it through sample exchange from an other company), but still, somebody has to get infected. So today, it might not have been you who provided the new warning sign of a disease (the outbreak of a malware family), but tomorrow you maybe the one. This is far the shiny reality the security product makers advertise to the public.

So who do we turn to? To experts of course. For example the AV-Comparatives organization is quite a well regarded one. If you look at their results, you should see almost all products scoring above the 90% mark, while most scoring above the 95% mark (the top one scoring between 97% and 99%). Sounds great, doesn't it? But lets apply just a little critical thinking. How many different types of malware is out there? Hard to say (given that there is no universally accepted (or easy for that matter) definition for malware family, but a number 1 000 000 is a good start. So, with one million malware families out there, even the best AV fails to protect you from 10 000 of them! In my opinion there is a co-dependence between testing organizations and AV vendors which prevents them from coming out and giving the straight news: even with AV, you have a good chance of getting infected. Still, you can crunch the numbers yourself.

Or here is an other expert: a pony tailed media junky who is available for phone interviews on virus and security -related questions and speaks English with a funny accent. Of course he works for a great company which has labs all over the world. But take a look at their flagship product. They are licensing most of the technology from Kaspersky Labs (a thing they usually omit to mention). Still, in tests (like the one published by AV-Comparatives), they manage to get smaller scores that the ones of Kaspersky (hey, they supposed to add detections to the already present ones, not remove them!?) and have slower scanning speed. In fact their score was lower than the one of two free products (Avast! and AVG).

Of course, you can manage risk. It's simple, even a forth grader should understand it: risk = probability * loss. But how do you calculate it when your experts have no or almost no idea what the probability is? You can only hope... All you need is love faith, because only blind faith can explain how leaders from large companies all over the world became to believe that their IT security problem can be solved with less money, in less time and fewer people than their physical security for example.

It will be a rude awakening, but until then: sweet dreams. And to the other side: happy hunting (if you can even call this a hunt - it is more like a massacre with the pray lying at your feet without moving)...

The fallout is upon us!

Well, almost. I've just watched the trailers for Fallout 3 and they look very interesting. I have my doubts with regards of the new combat system (they just had to appeal to the FPS fans, didn't they? - but the same time it seems that unless you use the tactical mode, you will suffer a lot of damage) and the viewing angle is also dubious (it felt more like flying than walking from the videos), but still, it is a part of the Fallout series.

Fun post day - save the mouse! petition

Every animal deserves the right to good living conditions. Save the mouse!

PS. I would have liked to embed the people are strange commercial from Animal Planet, but I couldn't find it. It's mind bogging how companies fail to use viral media to their advantage :-(

Blogging from Gnome

I've installed the Gnome Blog widget for some quick blogging, and so here it is - my first quick entry.

Some updates:

• The HTML generated is pretty clean (probably mostly because it doesn't have a whole lot of formatting features)
• It misplaced the title :-( (instead of making it the title of the post, it put the title as the first line of the article)
• It doesn't have support for tags
• On the upside of it: it has a spellchecker

My conclusion is: it still has some way to go before it can be used.

stackoverflow.com

Stackoverflow went into public beta. It is a really fascinating (and addictive!) site. What does it do?

• It provides a place for people to ask and answer programming related questions
• It it a combination of reddit/digg/forums/wiki
• You can vote on questions/answers and also edit them (if you got enough reputation)

A few quick pointers if you decide to try it out:

• To create an account, you need to use OpenID. There was a discussion to add a native authentication system, but I don't know when (and if) this will be done. In the meantime you can check out my tutorial on using OpenID and keeping a centralized presence with Blogger. The login process is wonky sometimes (as in it returns strange errors), but this seem to occur less and less. If it happens to you, try clearing your cookies and restarting your browser.
• At first it can seem very hard to get badges/reputation. Don't despair! There are some simple badges you can get quite easily (look for badges which have been awarded to a large number of people). Getting to a level where you can edit posts (which currently means 2000+ reputation) is much harder however.
• When answering a questions, there currently is a fastest gun phenomenon (the first few answers get the most attention, even if they're not the best ones). The current suggested workaround is to post a short answer quickly, and then edit it to include more detailed information. Be aware however that after a certain number of edits, the post becomes community owned.
• There is an associated blog and podcast you might be interested in, to get an behind the scenes look.

Finally, you can use the RSS feed associated with your user to display the most recent answers contributed by you (as you can see on the lefhand side of my blog). If you are bothered by the Answer by prefix, you can use something like Yahoo! Pipes to transform the feed accordingly: