Speedy hosts blocklists

One of the ways to block your machine from connecting to “unwanted” hosts (however you define this criteria) is to list the DNS name of the host in the hosts file of your system and associate it with a different IP address than the real one. While this solution has some limitations (for example: it requires deployment to each computer, it is rather easy to modify, does not offer wildcard support, etc), in practice it is still very effective. Two frequently updated hosts files which have the purpose of blocking malicious websites that I’ve used in the past are the one from WinHelp2002 and hpHosts (I even written up a short tutorial on how to set up the hosts file under Ubuntu).

Two criteria for selecting the IP address where to redirect the unwanted domains are: (a) it should be different from the “real” address(es) and (b) they should be easily contactable (if they have a long delay, applications trying to contact them will block while trying to open a connection). Currently the loopback address (127.0.0.1) is used in both of the files I’ve mentioned, however this can cause problems in several situations:

• If a webserver is running on the local machine (for development), it will receive hits from the browser (which can interfere with the development process). Alternatively, if a different server is running on port 80 (or an other port the application tries to contact) it may react in an unpredictable manner if it doesn’t understand the protocol (or the client might react incorrectly if the response is in a different format).
• Certain applications try to differentiate between local and remote sites by looking at the IP address and take different decisions. For example I’ve had problems with the ICSI Netalyzr Beta because of my hosts file. More recently Giorgio Maone reported issues with NoScript when using hosts files.

The solution is to use an invalid IP address like 0.0.0.0. This, besides fixing the above issues, has also the added benefit of speed. A quick testing revealed a more than 100 folds of improved by using 0.0.0.0 instead of 127.0.0.1. My testing procedure was as follows:

1. I emptied my DNS cache (ipconfig /flushdns) and set up a hosts file with DNS names mapped to 127.0.0.1 (no server was listening on port 80)
2. I pinged each of the hostnames to get the domain names loaded in the cache
3. I tried to “download” a website from each of names using wget with one try (wget –t1 –i). Trying to contact 100 sites took 1m 44s.
4. I repeated the same process using 0.0.0.0 and wget finished ~1 sec (!)

In conclusion: using 0.0.0.0 to block DNS names with the hosts file has many advantages (both from a functionality and speed point of view) and I’ll be transitioning to this solution. To do this, simple search/replace your hosts file from 127.0.0.1 to 0.0.0.0 (take care to leave the localhost entry alone. That one needs to be mapped to 127.0.0.1).

Picture taken from peregrine blue's photostream with permission.