Proper Squid Setup and Config + ICS?

[icanhascpu2]

So I'll post from another thread i made that began with me trying to find better methods to lower my net use as I have a 20GB cap on my 3G connection (Its either this, Satellite, or Dialup, so Im short on options)


(Please note, before doing all this, by myself, I had never previously even used Linux terminal, and have very little Linux experience, so If I'm slow, bare with me)

Post From Other Thread:

So somewhow though black magic I:

-> setup ubuntu, installed squid, installed the 3G Wireless USB card, yelled at ubuntu to stop setting the eth0 to default connection (cant disable it, need it on), somehow finally got the wireless connection to set as default so internet would access though it, finally bloody got squid to allow me to load a damn page in the host OS (windows 7)

so broswers work on this computer.

Next problems:
1. Figuring out how to config squid/applications other than browsers to get access (Skype, AIM, Games, etc)
2. How the hell do share this connection over a crossover cable to the computer in the living-room? Buying my way out of the problem is not an option (unless its less than $10 shipped)

So far I have solved connecting Skype, AIM, MSN, (yahoo doesn't want to freakn work with me even when i added allow code in squid and reset the service, but its not a big deal)

The main problems I want to look into today are

1. Setting squid config up properly. Squid seems to be caching somewhat (i know by default its set to 4MB and below) but it seems like its not really being very strict. I want it set so if an image loads once, that image will *NEVER* be downloaded from anything but the squid server again (i.e. not the internet) I want loaded images to ALWAYS load locally after initially being loaded from the net. How do I make sure this happens?​

2. After I make sure #1 is happening, I can move onto figuring out ICS (buying my way out of it isnt an option), but this part isnt relevant until I know squid is really working well enough to take the further effort here.​

 

[kornphlake]

#1 I don't really know, I gave up on trying to force squid to do anything, I don't know enough about networking or proxy servers or internet, once I got it working for what I needed it to do (content filter) I quit tampering with squid.
#2 You'll need to setup a dhcp server (dhcpd) then you'll need to bridge the ethernet connections. I know most hardcore linux guys hate GUI tools, especially for stuff that relates to iptables, but I used firestarter and it actually worked for bridging the connections, I couldn't find a good how-to using the command line, that was a few years ago though, there might be better quality guides out there now.

I setup my proxy server using ubuntu 8.04, I probably spent 100 hours learning Linux, learning enough about networking to be dangerous and blindly following how-to's before everything finally clicked. Now that I know my way aroudn linux and a little bit about networking I could probably get the job done more quickly, but I doubt it would take less than a weekend. I tried to get the same setup going on a different computer running 12.04 and couldn't get it to work, after a few hours I gave up and cloned the drive with 8.04. I was unemployed and had nothing better to do when I setup the first server, but honestly it would probably be worth spending money if I needed to do it again.

 

[MrColin]

I used iptables for a while before changing to the shorewall package to set up NAT etc.

This is part of a script I used. One still needs to make sure the gateway is set up properly on all of the clients iether via static settings or DHCP.

#!/bin/sh


LANIF="eth0"                            # Internal Ethernet card identifier
WANIF="wlan0"                           # External Ethernet card identifier
LAN="192.168.8.0/24"            # Internet network address range
LANBCAST="192.168.8.255"        # Internal Broadcast address


#NAT/Masquerade for ICS
iptables -P FORWARD ACCEPT
iptables --table nat -A POSTROUTING -o $WANIF -j MASQUERADE

#  loopback
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -d 127.0.0.0/8 -j REJECT

#  Port Forwarding
#  RDP to specific host
iptables -t nat -A PREROUTING -p tcp -i $WANIF -d 192.168.1.107 --dport 3389 -j DNAT --to 192.168.8.3:3389
iptables -A FORWARD -p tcp -i $WANIF -d 192.168.8.3 --dport 3389 -j ACCEPT

In my case the 192.168.1.x subnet was my neighbor's wifi and 192.168.8.x was my LAN. I had more rules set up than this but you should be able to get the idea.

 

[icanhascpu2]

Thanks for the replies

Still going at this bit by bit. If I cant get squid to properly and strictly cache what I need it to (all non-dynamic content, video/audio/images) then I cant really move on. Still looking into it

 

[icanhascpu2]

Well so far this is what I have as Im still testing things. Im not sure where to go for more educated squid eyes to look at this. The primary purpose of me doing squid in the first place is #1 to save bandwidth. Everything else is a distant #2 #3 etc.

# 1 week = 10080 mins, 1 month = 43200 mins, 3 months = 129600 mins
refresh_pattern imeem.*\.flv 0 0% 0 override-lastmod override-expire
refresh_pattern ^ftp: 43200 20% 43200 override-expire reload-into-ims store-stale
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern code.googlec.com.*(svn|download) 0 50% 1440 reload-into-ims

# Ads
refresh_pattern ^.*(streamate.doublepimp.com.*\.js\?|utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 129600 20% 129600 ignore-no-cache ignore-no-store ignore-private override-expire ignore-reload ignore-auth ignore-must-revalidate store-stale negative-ttl=40320 max-stale=1440

# Specific-sites
refresh_pattern \.rapidshare.*\/[0-9]*\/.*\/[^\/]* 129600 100% 129600 ignore-reload store-stale
refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?) 129600 100% 129600 override-expire ignore-reload store-stale negative-ttl=0
refresh_pattern ^http://v\.okezone\.com/get_video\/([a-zA-Z0-9]) 129600 100% 129600 ignore-no-cache ignore-no-store reload-into-ims override-expire ignore-must-revalidate store-stale
refresh_pattern \.(ico|video-stats) 129600 100% 129600 override-expire ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth override-lastmod ignore-must-revalidate negative-ttl=10080 store-stale
refresh_pattern \.etology\? 129600 100% 129600 override-expire ignore-reload ignore-no-cache store-stale
refresh_pattern galleries\.video(\?|sz) 129600 100% 129600 override-expire ignore-reload ignore-no-cache store-stale
refresh_pattern \.adtology\? 129600 100% 129600 override-expire ignore-reload ignore-no-cache store-stale
refresh_pattern ^.*safebrowsing.*google 43200 50% 129600 override-expire ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth ignore-must-revalidate negative-ttl=10080 store-stale
refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.id) 43200 50% 129600 override-expire ignore-reload store-stale ignore-private negative-ttl=10080
refresh_pattern ytimg\.com.*\.(jpg|png) 43200 50% 129600 override-expire ignore-reload store-stale
refresh_pattern kaspersky.*\.(avc|xml|dif|klz|kdc)$ 43200 100% 43200 ignore-reload store-stale
refresh_pattern kaspersky 1440 50% 43200 ignore-no-cache store-stale
refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 129600 100% 129600 override-expire ignore-reload store-stale
refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\? 129600 100% 129600 ignore-no-cache override-expire override-lastmod store-stale
refresh_pattern .fbcdn.net.*\.(jpg|gif|png) 43200 90% 129600 ignore-no-cache override-expire ignore-reload store-stale negative-ttl=0
refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\. 129600 100% 129600 ignore-no-cache ignore-no-store ignore-reload override-expire store-stale
refresh_pattern ^http:\/\/www.onemanga.com.*\/ 129600 100% 129600 reload-into-ims override-expire store-stale
refresh_pattern ^http:\/\/\.www[0-9][0-9]\.indowebster\.com\/(.*)(mp3|rar|zip|flv|wmv|3gp|mp(4|3)|exe|msi|zip) 129600 100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store store-stale ignore-auth

# General
refresh_pattern \.(jp(e?g|e|2)|tiff?|bmp|gif|png) 10080 50% 43200 ignore-no-cache ignore-no-store reload-into-ims override-expire ignore-must-revalidate store-stale
refresh_pattern \.(flv|mpeg|ra?m|avi|mp(g|e|4)|mov|divx|asf|wmv|m\dv|rv|vob|asx|ogm|3gp|on2) 10080 50% 43200 ignore-no-cache ignore-no-store reload-into-ims override-expire ignore-must-revalidate store-stale
refresh_pattern \.(z(ip|[0-9]{2})|r(ar|[0-9]{2})|jar|bz2|gz|tar|rpm|vpu) 129600 100% 129600 override-expire reload-into-ims
refresh_pattern \.(mp3|wav|og(g|a)|swf?|flac|midi?|rm|aac|wma|mka|ape) 129600 100% 129600 override-expire reload-into-ims ignore-reload
refresh_pattern \.(exe|msi|dmg|bin|xpi|iso|swf|mar|psf|cab) 129600 100% 129600 override-expire reload-into-ims ignore-no-cache ignore-must-revalidate
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern \.(php|jsp|cgi|asx)\? 0 0% 0
refresh_pattern . 0 50% 43200 store-stale reload-into-ims