[How-To] Adblocking using a hostfile

Doomicide · 2012-05-26 18:05:38

First of all, this is the first how-to I write, so any feedback and criticism in this direction is welcome smile . I applied this steps on a up-to-date debian sid install and not in crunchbang, but since both are debian/-based it should work there anyways.

Intro
A lot of users here don't use firefox/iceweasel or chromium, because they tend to be slow and are bloated to the max. One of the biggest problems with smaller and lighter browsers is that there often is no Adblock-Plugin. This how-to is a (safe(?)) step-by-step manual on setting up adblocking, using a host-file and even automating the process of keeping your filter up-to-date.

Stuff you need
- Any hostfile that blocks ads (surprise surprise). I'll use this script for the how-to. Read the thread and download the tarball from the Aur site.

- kwakd A tiny webserver that does nothing but return an empty html-file. We're gonna redirect the blocked ads to this.

- dnsmasq, it's in the repos. Dnsmasq is a lightweight, easy to configure, DNS forwarder and DHCP server. It allows us to use two or more hostsfiles, so we don't have to overwrite /etc/hosts and also reduces resource-usage.

Step 1: Installing dnsmasq
As it's in the repos this is as simple as

sudo apt-get install dnsmasq

Step 2: Installing hostsblock.sh or another hostfile
If you're using hostsblock.sh like me you now want to unpack the tarball and do the following:
# Create a directory for the config and copy the respective files

sudo mkdir /etc/hostsblock
sudo cp white.list black.list rc.conf /etc/hostsblock

Then find the line

/etc/rc.d/dnsmasq

in hostsblock.sh and change it to

/etc/init.d/dnsmasq

for debian-compability.

# Copy hostsblock.sh to your root $PATH

sudo cp hostsblock.sh /usr/sbin

# and execute it

sudo hostsblock.sh

The script will now download several adblocking hostsfiles and combine them in one: /etc/hosts.block by default.

If you're using some other hostsfile download it and save it to /etc/hosts.block or whatever pleases you.

Step 3: Configuring dnsmasq
Open /etc/dnsmasq.conf in an editor of your choice and uncomment/add the following lines:

addn-hosts=/etc/hosts.block
listen-address=127.0.0.1

Change the path of hosts.block according to your file.

If you're using dhcp we'll have to set it up, too.
In /etc/resolv.conf add this line to the start of the file:

nameserver 127.0.0.1

In /etc/dhcp/dhcpclient.conf uncomment this line:

prepend domain-name-servers 127.0.0.1;

Then restart dnsmasq and your network with

 sudo /etc/init.d/dnsmasq restart
sudo /etc/init.d/networking restart

Step 4: Install kwakd
This step is not vital but helps speed up the overall performance a lot. We'll make sure kwakd is started at system boot and the blocked ads get redirected to the empty html-document instead of timing out.
unpack the tarball, cd into the directory and run:

./configure
make
sudo make install

Now that kwakd is installed we'll make sure it is started at boot-time. Save the following script as /etc/init.d/kwakd

#! /bin/sh
# /etc/init.d/kwakd
#
case "$1" in
  start)
    echo "Starting kwakd"
    if [ ! $(pidof kwakd) ]; then
    /usr/local/bin/kwakd -b -p 80
    else
    echo "kwakd is already running!"
    fi 
    ;;
  stop)
    echo "Stopping kwakd"
    if [ "$(pidof kwakd)" ]; then
    killall kwakd
    echo "Stopped."
    else
    echo "kwakd is not running"
    fi 
    ;;
  *)
    echo "Usage: /etc/init.d/kwakd {start|stop}"
    exit 1
    ;;
esac

exit 0

Then run:

sudo update-rc.d kwakd defaults
sudo /etc/init.d/kwakd start

Step 5: Automatically update your hostsfile
To make sure the adblock-filters stay up-to-date we'll use cron.
If you're using hostsblock.sh then simply run

 sudo crontab -e

and add the line

@weekly /usr/sbin/hostsblock.sh

If you're using some other hostsfile you can add this script instead of hostsblock.sh:

cd /etc
# backup the old list
[[ -f hosts.block ]] && cp -p hosts.block hosts.block.old

wget -N --tries=20 --connect-timeout=10 --retry-connrefused --waitretry=5 <link to your file here>

# if download failed move old file back in place
if [[ $? != 0 ]]; then
    [[ -f hosts.block.old ]] && cp -p hosts.block.old hosts.block
    echo "Error: Download Failed!"
    exit 11
else
    [[ -f hosts.block.old ]] && rm hosts.block.old # if all went well remove backup
    echo "All went well. :)"
    exit 0
fi

Sources:
http://www.debian-administration.org/ar … th_dnsmasq
https://wiki.archlinux.org/index.php/Dnsmasq
https://webcache.googleusercontent.com/ … 0Y&ct=clnk

Edit:
- Use kwakd instead of pixelserv, the latter tends to crash... a lot.
- Add link to hostsblock.sh aur page

Last edited by Doomicide (2012-06-02 08:55:00)

nestor · 2012-05-26 18:11:39

...

Last edited by nestor (2012-06-22 04:04:03)

VastOne · 2012-05-26 18:34:54

Very nice How To Doomicide! cool

I have added it to Quick References under Security.

xaos52 · 2012-05-26 18:54:08

Where do I find this 'aur' thingy? big_smile

Doomicide · 2012-05-26 19:04:15

^Second post in the linked thread. Aur is the arch-user reposority, unfortunately the author of hostsblock.sh doesn't provide another way to get the latest version

@all: Thanks for the kind words smile

sunfizz98 · 2012-05-27 05:35:38

Lovely. Will definitely use this for my computers. Thanks for the guide Doomicide!

Awebb · 2012-05-27 07:48:42

How is that solution in terms of performance? Wouldn't every request to the WAN zone result in this thing going through the whitelist first? Does it slow the system down? Does it work differently than I thought?

Doomicide wrote:

unfortunately the author of hostsblock.sh doesn't provide another way to get the latest version

I love those folks. Really. Those people have all the warmth of my heart. Well, they obviously don't have any of their own.

Last edited by Awebb (2012-05-27 07:50:06)

Doomicide · 2012-05-27 11:23:32

Awebb wrote:

How is that solution in terms of performance? Wouldn't every request to the WAN zone result in this thing going through the whitelist first? Does it slow the system down? Does it work differently than I thought?

It works different: You can add an adress to the whitelist file in two ways. If you add 'adress.com' it will allow just this exact adress, if you leave a space in front it will allow all adresses that contain thist string, e.g.: ' adress.com' will also allow 'adress.com/foo/bar.html'. If you added anything to the white- or blacklist you'll have to rerun hostsblock.sh and the generated hostfile just won't include the specified line. I don't notice any slow-down on the system and the author of hostsblock.sh claims it uses less resources than privoxy.

Edit:

gaensrich wrote:

Actually this is a bit off:
'address.com' will whitelist all entries that contain 'address.com', e.g. subdomain.address.com, subdomain2.address.com, subdomain.address.com.com
' address.com' will whitelist all entries that contain ' address.com', e.g. address.com, address.com.com, BUT NOT subdomain.address.com.

/Edit

Awebb wrote:

Doomicide wrote:
unfortunately the author of hostsblock.sh doesn't provide another way to get the latest version
I love those folks. Really. Those people have all the warmth of my heart. Well, they obviously don't have any of their own.

Yeah I'll ask him to set up a git repo, later today or so.

Last edited by Doomicide (2012-05-30 17:30:58)

sadi · 2012-05-27 19:15:43

Thanks a lot for this HOW-TO.
I'm struggling to use dnsmasq and kwakd with this script and your how-to helped me to make considerable progress.
But it seems I need a little more help.

1) The script saved as /etc/init.d/kwakd:
I guess it should be made executable, which I did, but when I ran it on Ubuntu 12.04 I've these messages:

update-rc.d: warning: /etc/init.d/kwakd missing LSB information
update-rc.d: see <http://wiki.debian.org/LSBInitScripts>
Adding system startup for /etc/init.d/kwakd ...
/etc/rc0.d/K20kwakd -> ../init.d/kwakd
/etc/rc1.d/K20kwakd -> ../init.d/kwakd
/etc/rc6.d/K20kwakd -> ../init.d/kwakd
/etc/rc2.d/S20kwakd -> ../init.d/kwakd
/etc/rc3.d/S20kwakd -> ../init.d/kwakd
/etc/rc4.d/S20kwakd -> ../init.d/kwakd
/etc/rc5.d/S20kwakd -> ../init.d/kwakd

And when I run the command "sudo /etc/init.d/kwakd start" I get:

/etc/init.d/kwakd: 25: /etc/init.d/kwakd: Syntax error: end of file unexpected (expecting ";;")

I couldn't solve this problem whatever I tried in the last line.
I've also realized that the command "kwakd -b -p 80" doesn't work unless I remove "-p 80"
But I get kwakd running it doesn't seem to perform its expected function, i.e. replacing error messages with blank areas where ads are blocked.
Maybe this is (also) related with the following issue.

2) At the end of the hostsblock script I get the following error message:

* Restarting DNS forwarder and DHCP server dnsmasq
dnsmasq: failed to create listening socket for port 53: Address already in use
[fail]

As I've said, the hostsblock script works perfectly other than these 2 (dnsmasq & kwakd) issues.

Thanks in advance for any suggestions that can help this inexperienced Linux user solve this puzzle.

Doomicide · 2012-05-28 12:22:18

Hi sadi, nice to hear this how-to was useful to you smile Now to your problems:

sadi wrote:

1) The script saved as /etc/init.d/kwakd:
I guess it should be made executable, which I did, but when I ran it on Ubuntu 12.04 I've these messages:

update-rc.d: warning: /etc/init.d/kwakd missing LSB information
update-rc.d: see <http://wiki.debian.org/LSBInitScripts>
Adding system startup for /etc/init.d/kwakd ...
   /etc/rc0.d/K20kwakd -> ../init.d/kwakd
   /etc/rc1.d/K20kwakd -> ../init.d/kwakd
   /etc/rc6.d/K20kwakd -> ../init.d/kwakd
   /etc/rc2.d/S20kwakd -> ../init.d/kwakd
   /etc/rc3.d/S20kwakd -> ../init.d/kwakd
   /etc/rc4.d/S20kwakd -> ../init.d/kwakd
   /etc/rc5.d/S20kwakd -> ../init.d/kwakd

This shows everything is working correctly, so nothing to worry about here.

sadi wrote:

And when I run the command "sudo /etc/init.d/kwakd start" I get:
/etc/init.d/kwakd: 25: /etc/init.d/kwakd: Syntax error: end of file unexpected (expecting ";;")

This is strange , I now updated the script, but I'm pretty sure there wasn't a syntax-error in it. May be something went wrong while copying? Anyway try the new one, it should definitely work, I double-checked it and it's the exact same script I'm using.

sadi wrote:

I've also realized that the command "kwakd -b -p 80" doesn't work unless I remove "-p 80"

What do you mean doesn't work? You mean kwakd just doesn't start? '-p 80' let's kwakd use port 80 instead of the default 8000, maybe something else is using this port already?. Port 8000 doesn't work for me, too.

sadi wrote:

2) At the end of the hostsblock script I get the following error message:

* Restarting DNS forwarder and DHCP server dnsmasq                             
dnsmasq: failed to create listening socket for port 53: Address already in use
                                                                                                               [fail]

Looks like either dnsmasq is already running, but pidof dnsmasq doesn't find it or some other application is already using port 53. Does dnsmasq start fine for you? Is it already running when you run hostsblock.sh? If so, try 'sudo /etc/init.d/dnsmasq restart'

sadi · 2012-05-28 13:15:14

Thank you very much Doomicide!
It's working now!
It seems "dnsmasq: failed to create listening socket for port 53: Address already in use [fail]" can be ignored.
When I enter the command "nmap localhost" I get "53/tcp open domain" so it seems the script also tries to create something although it already exits, and it shouldn't be described as "failure".
The fact that your revised script for kwakd is now also running in Ubuntu 12.04 ensures getting rid of HTML error messages for blocked ads.

Maybe several people like you and gaenserich can join efforts to put together a nice Linux Hosts Manager soon :-)
I'm also trying to help gaenserich in further improving the excellent script there...
Next step could be a simple GUI for installation, management of lists and scheduling...

Awebb · 2012-05-29 09:52:25

Doomicide wrote:

Yeah I'll ask him to set up a git repo, later today or so.

*is greatful that software has been written, no matter the upload*
*is greatful that...

Doomicide · 2012-05-29 17:10:26

Awebb wrote:

Doomicide wrote:
Yeah I'll ask him to set up a git repo, later today or so.
*is greatful that software has been written, no matter the upload*
*is greatful that...

This is strangely reminiscent of a prayer. I don't see anything wrong with asking, he can still say no and I thanked him ofc for writing the script. If he doesn't want I might set one up, with his permission.

sadi · 2012-09-19 14:34:44

New website for this script:

http://gaenserich.github.com/hostsblock

Many instructions and some configuration details are ArchLinux-specific though.
So I've begun using a script including the following commands for updates:

sudo cat "/etc/hostsblock/black.list" \
          "/etc/hostsblock/black.list.custom" | \
sudo sort -u >> "/etc/hostsblock/black.list"
sudo cat "/etc/hostsblock/white.list" \
          "/etc/hostsblock/white.list.custom" | \
sudo sort -u >> "/etc/hostsblock/white.list"
sudo sed -i 's/\/etc\/rc\.d/\/etc\/init\.d/g' "/etc/hostsblock/hostsblock.sh"
sudo sed -i 's/\/etc\/rc\.d/\/etc\/init\.d/g' "/etc/hostsblock/hostsblock-urlcheck.sh"
sudo sed -i 's/\/etc\/rc\.d/\/etc\/init\.d/g' "/etc/hostsblock/rc.conf"

maniqui · 2012-12-01 17:40:11

Expanding on this great tutorial:

How to run both Apache & kwakd on port 80 and binding them to different interfaces (IPs)

Intro

I wanted to run Apache on my local web-development server, along with kwakd. I also needed kwakd to keep running on port 80, as that's the default port that browsers will ask for content when trying to access a blocked host on 127.0.0.1. So changing kwakd to another port wouldn't have been an option.
Moreover, I needed kwakd to listen on 127.0.0.1, as that's the IP address pointed by all the blocked hosts.
I could have configured Apache to listen on another port (ie. 8000 or 8080), but then, where's the fun? So, I needed Apache to listen on port 80 too, but bound to other interfaces (127.0.1.1, for local access, and 10.0.0.2 for access it on the LAN).

So, my 2 defined tasks were:

1) To bind kwakd to 127.0.0.1:80
2) To bind Apache to 127.0.1.1:80 and to 10.0.0.2:80

Step 1: binding kwakd to 127.0.0.1:80

By default, kwakd binds itself to all interfaces on the specified port (ie. *:80 or 0.0.0.0:80).
Problem is, there is no option on kwakd to override this. I skimmed over the C code and found the relevant lines (related to an INADDR_ANY value) but modifying them was way out of my league.

So, I was about to lose all hopes, but after some quick investigation, I found a post on SuperUser on how to use different network interfaces for different processes, which lead me to this article on binding applications to a specific IP.

Basically, it's about compiling and installing a little C script that lets you bind a process to an arbitrary IP address, thus letting the user override the limitation of, for example, kwakd, which doesn't offer the option to do that.

The script is pretty easy to compile and install, so just follow the instruction on the article.
Once done, it's easy to test if it worked. On the terminal, run:

$ sudo BIND_ADDR="127.0.0.1" LD_PRELOAD=/usr/lib/bind.so kwakd -v -p 80

(remember to stop Apache service if you have it currently running at port 80 on all interfaces. If not, kwakd won't be able to bind itself to 127.0.0.1:80).

To check if it worked, run:

$ sudo netstat -ltnp | grep ':80'

It should output something like this:

tcp        0      0 127.0.0.1:80            0.0.0.0:*               LISTEN      6458/kwakd

As you can see, we have kwakd running on 127.0.0.1:80. Yeah!
It can also be tested using a browser, and trying to access one of the blocked pages (ej. adwords.google.com). It should returng and empty (<html></html>) page.

Finally, to make this permanent, we edit /etc/init.d/kwawd (the script posted by Doomicide on the opening post of this thread) to change this line:

/usr/local/bin/kwakd -b -p 80

To this line:

BIND_ADDR="127.0.0.1" LD_PRELOAD=/usr/lib/bind.so /usr/local/bin/kwakd -b -p 80

Then, we can test if it worked by stopping/starting the service again, and checking it with netstat again.

Step 2: binding Apache to listen on 127.0.1.1:80 and on 10.0.0.2:80

First, note that 10.0.0.2 is the "public" IP address of my computer on my private LAN. Change it accordingly to the IP address of your computer on your LAN.

This one is easy. We have to edit /etc/apache2/ports.conf and change the line:

Listen 80

to:

Listen 127.0.1.1:80
Listen 10.0.0.2:80

Save and restart Apache (usually `sudo service apache2 restart`).
To check if it worked, run:

$ sudo netstat -ltnp | grep ':80'

It should output something like this:

tcp        0      0 127.0.0.1:80            0.0.0.0:*               LISTEN      6458/kwakd      
tcp        0      0 10.0.0.2:80             0.0.0.0:*               LISTEN      6236/apache2    
tcp        0      0 127.0.1.1:80            0.0.0.0:*               LISTEN      6236/apache2

Well, I think that's all.
I've yet to test if VirtualHosts are working properly (I don't have any VH configured yet).

maniqui · 2012-12-07 22:05:06

What about running kwakd also on port 443? There are some blocked hosts that do the request to an https URL (port 443).
It seems it's possible to run two processes of kwakd, one on 127.0.0.1:80 and another on 127.0.0.1:443 (see my post above about binding kwakd to an specific IP address).

I've quickly tried this (running kwakd on port 443) and it seems to somewhat work, although instead of responding with an empty HTML, the browser aborts the mission to connect with kwakd, with the following message (on Chromium 22)

SSL connection error
Unable to make a secure connection to the server. This may be a problem with the server, or it may be requiring a client authentication certificate that you don't have.
Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error.

This is expected, of course.
I'll investigate to check if it's possible to have some local, valid certificate to "cheat" the browser to make a secure connection to kwakd. Of course, I've no idea if what I'm saying makes sense smile
Will try to report back asap.

CrunchBang Linux

SEARCH

#1 2012-05-26 18:05:38

[How-To] Adblocking using a hostfile

#2 2012-05-26 18:11:39

Re: [How-To] Adblocking using a hostfile

#3 2012-05-26 18:34:54

Re: [How-To] Adblocking using a hostfile

#4 2012-05-26 18:54:08

Re: [How-To] Adblocking using a hostfile

#5 2012-05-26 19:04:15

Re: [How-To] Adblocking using a hostfile

#6 2012-05-27 05:35:38

Re: [How-To] Adblocking using a hostfile

#7 2012-05-27 07:48:42

Re: [How-To] Adblocking using a hostfile

#8 2012-05-27 11:23:32

Re: [How-To] Adblocking using a hostfile

#9 2012-05-27 19:15:43

Re: [How-To] Adblocking using a hostfile

#10 2012-05-28 12:22:18

Re: [How-To] Adblocking using a hostfile

#11 2012-05-28 13:15:14

Re: [How-To] Adblocking using a hostfile

#12 2012-05-29 09:52:25

Re: [How-To] Adblocking using a hostfile

#13 2012-05-29 17:10:26

Re: [How-To] Adblocking using a hostfile

#14 2012-09-19 14:34:44

Re: [How-To] Adblocking using a hostfile

#15 2012-12-01 17:40:11

Re: [How-To] Adblocking using a hostfile

#16 2012-12-07 22:05:06

Re: [How-To] Adblocking using a hostfile

Board footer