CrunchBang Linux

hardran3

#! Junkie

From: forest town, lake land

Registered: 2011-02-26

Posts: 359

"Faster" browsing with polipo & pdnsd

I spend a lot of time using a 3G connection, and a lot of the time my connection is slow. I also have to worry about usage, bandwidth isn't cheap. I found this combination, web caching with polipo, and DNS caching with pdnsd. The two proxies make for a much smoother browsing experience, and save me a bit of bandwidth.

First we must install polipo and pdnsd

sudo apt-get install polipo pdnsd

polipo works out of the box, you don't have to mess with anything really. Ad blocking is possible, check the end of this post.

Here is my /etc/polipo/config. It is set for ad-blocking, and also to censor some headers.

# Sample configuration file for Polipo. -*-sh-*-

# You should not need to edit this configuration file; all configuration
# variables have reasonable defaults.

# This file only contains some of the configuration variables; see the
# list given by ``polipo -v'' and the manual for more.


### Basic configuration
### *******************

# Uncomment one of these if you want to allow remote clients to
# connect:

# proxyAddress = "::0"        # both IPv4 and IPv6
# proxyAddress = "0.0.0.0"    # IPv4 only

# If you are enabling 'proxyAddress' above, then you want to enable the
# 'allowedClients' variable to the address of your network, e.g.
# allowedClients = 127.0.0.1, 192.168.42.0/24
                                            
# allowedClients = 127.0.0.1              

# Uncomment this if you want your Polipo to identify itself by
# something else than the host name:

# proxyName = "polipo.example.org"

# Uncomment this if there's only one user using this instance of Polipo:

cacheIsShared = false

# Uncomment this if you want to use a parent proxy:

# parentProxy = "squid.example.org:3128"

# Uncomment this if you want to use a parent SOCKS proxy:

# socksParentProxy = "localhost:9050"
# socksProxyType = socks5


### Memory
### ******

# Uncomment this if you want Polipo to use a ridiculously small amount
# of memory (a hundred C-64 worth or so):

# chunkHighMark = 819200
# objectHighMark = 128

# Uncomment this if you've got plenty of memory:

# chunkHighMark = 50331648
# objectHighMark = 16384


### On-disk data
### ************

# Uncomment this if you want to disable the on-disk cache:

# diskCacheRoot = ""

# Uncomment this if you want to put the on-disk cache in a
# non-standard location:

# diskCacheRoot = "~/.polipo-cache/"

#Needed for ad-blocking, comment if you are not using
localDocumentRoot = "/usr/share/polipo/www"
forbiddenUrl = "http://127.0.0.1:8123/empty.gif"

# Uncomment this if you want to enable the pages under /polipo/index?
# and /polipo/servers?.  This is a serious privacy leak if your proxy
# is shared.

disableIndexing = false
disableServersList = false


### Domain Name System
### ******************

# Uncomment this if you want to contact IPv4 hosts only (and make DNS
# queries somewhat faster):

# dnsQueryIPv6 = no

# Uncomment this if you want Polipo to prefer IPv4 to IPv6 for
# double-stack hosts:

# dnsQueryIPv6 = reluctantly

# Uncomment this to disable Polipo's DNS resolver and use the system's
# default resolver instead.  If you do that, Polipo will freeze during
# every DNS query:

# dnsUseGethostbyname = yes


### HTTP
### ****

# Uncomment this if you want to enable detection of proxy loops.
# This will cause your hostname (or whatever you put into proxyName
# above) to be included in every request:

# disableVia=false

# Uncomment this if you want to slightly reduce the amount of
# information that you leak about yourself:

censoredHeaders = from, accept-language
censorReferer = maybe

# Uncomment this if you're paranoid.  This will break a lot of sites,
# though:

# censoredHeaders = set-cookie, cookie, cookie2, from, accept-language
# censorReferer = true

# Uncomment this if you want to use Poor Man's Multiplexing; increase
# the sizes if you're on a fast line.  They should each amount to a few
# seconds' worth of transfer; if pmmSize is small, you'll want
# pmmFirstSize to be larger.

# Note that PMM is somewhat unreliable.

# pmmFirstSize = 16384
# pmmSize = 8192

# Uncomment this if your user-agent does something reasonable with
# Warning headers (most don't):

# relaxTransparency = maybe

# Uncomment this if you never want to revalidate instances for which
# data is available (this is not a good idea):

# relaxTransparency = yes

# Uncomment this if you have no network:

#proxyOffline = yes

# Uncomment this if you want to avoid revalidating instances with a
# Vary header (this is not a good idea):

# mindlesslyCacheVary = true

pdnsd needs some love. Here is my config, should work for anyone. It is using the Google DNS servers, 8.8.8.8 and 8.8.4.4

Copy the following to /etc/pdnsd.conf

global {
        perm_cache=2048;
        cache_dir="/var/cache/pdnsd";
        max_ttl=604800;
        run_as="pdnsd";
        paranoid=on;
#       next setting allows ppp/ip-up update the name servers -- ABa / 20040213
        status_ctl=on;
        neg_rrs_pol=on;
        par_queries=1;
        server_port=53;
        server_ip="127.0.0.1";
}


server {
        ip="8.8.8.8";
        timeout=60;
        interval=900;
        uptest=ping;
        ping_timeout=500;
        proxy_only=on;
        purge_cache=off;
        caching=on;
}
server {
        ip="8.8.4.4";
        timeout=60;
        interval=900;
        uptest=none;
        ping_timeout=500;
        proxy_only=on;
        purge_cache=off;
        caching=on;
}

server {
    label="resolvconf";
}

source {
        ttl=86400;
        owner="localhost.";
        serve_aliases=on;
        file="/etc/hosts";

Now that we have a DNS proxy running we will want to use it. Edit /etc/dhcp/dhclient.conf and add this line to the end of the file

prepend domain-name-servers 127.0.0.1;

Note that this doesn't work for mobile (3G) connections. You will have to use Network Manager to set the DNS to 127.0.0.1 manually. Is there a way to fix this?

Now we need to point our browser at the web proxy. Most crunchbangers are probably using Chrome or Chromium for web browsing, so I will cover that here.

The Chrome preferences for proxy settings doesn't work in crunchbang, so we have to use a different approach. I ended up editing /etc/profile and adding the following to the end of the file.

http_proxy="127.0.0.1:8123"
export http_proxy

Reboot, and now you should be using DNS and web caching. A manual and some other info for polipo will be available at http://127.0.0.1:8123

Most of this is borrowed and slightly modified from here and here.

You can also use polipo to do some ad-blocking. Check here. This has one flaw. The easylist.txt download and conversion had a line beginning with

+adverts/

The initial + should be escaped, and polipo ignores the entire forbidden list because it fails to parse.
Edit forbidden and change the line to:

\+adverts/

Anyone know of a way to automate this change? Would be nice for the block list to get refreshed with cron.

Ad blocking was borrowed from here and the fix for the forbidden file was found here

This is a great setup, light on resources (I run it on an Atom N270 netbook), and makes browsing seem much faster. I also welcome any suggestions to make it better.

Last edited by hardran3 (2011-02-26 21:57:30)

---

Apt Pinning - Pulseaudio - Nvidia Drivers - Laptop Power Saving - 3G Bandwidth Logging -  obmocmenu

minimalist comrade

#! Member

Registered: 2009-09-13

Posts: 61

Re: "Faster" browsing with polipo & pdnsd

I wonder if this really adds any benefit, in terms of both speed and bandwidth. Browsers should already cache web content and dns results. You can't change how much it caches in chromium. But you can do so with Opera.

tradetaxfree

#! CrunchBanger

Registered: 2011-03-05

Posts: 111

Re: "Faster" browsing with polipo & pdnsd

My 3G connection seems quite a bit faster with this setup for sites I visit regularly. I also turned on the options to disable IPv6 DNS lookups as much as possible. Using the OpenDNS servers didn't seem to work for some sites, using Google's DNS servers works ok for everything. It also works ok with TOR / Vidalia.

I found some userland programs to limit an applications bandwidth. For additional traffic shaping see pyshaper, for a 3G connection some amendments are needed to /usr/local/bin/pyshaper:

change line 2520 to:

tOut, tIn = popen2.popen2("tcpdump -i ppp0 -vnl", 1024)

change line 2596 to:

dev = "ppp0"