Sie sind auf Seite 1von 7

YouTUBE Caching using Squid as a Transparent Proxy server

Umair Junaid

DATED : 09 September 2012

In this tutorial we will be creating a youtube caching server using squid and apache server

We will be using squid proxy server and apache php web server to redirect and store youtubevideos.

We will need

Server Machine with two NIC’s

eth0 (192.168.1.111, 255.255.255.0, gw 192.168.1.1) (eth0 WAN interface ) eth1 (192.168.0.254, 255.255.255.0) ( eth1 LAN interface for local clients) Centos base OS installation ver 5

LAB Setup

STEP 1 (SQUID SERVER) eth0 192.168.1.111

Squid version used squid-3.1.8-1.el5.i386.rpm (at the time of writing it is the latest rpm it can be downloaded from squid website I have not tested on older versions and cannot say they will work or not).

rpm -ivh squid-3.1.8-1.el5.x86_64.rpm

Step 2 (Apache Server with PHP support) eth0 192.168.1.111

NOTE: PREFRABBLY APACHE AND SQUID CAN BE ON DIFFERENT MACHINES THEN YOU HAVE TO CHANGE THE REDIRECTION IP IN PER.PHP TO THE IP OF APACHE SERVER MACHINE.

rpm -Uhv http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.i386.rpm yum install mysql mysql-bench mysql-server mysql-devel mysqlclient10 php-mysql httpd gcc pcre-devel php-gd gd mod_ssl glib2-devel gcc-c++ libpcap-devel php php-pear Note: we will not use mysql directly but some of the libraries will be required for the program so that why we are installing these.

chkconfig httpd on service httpd start

STEP 3 SQUID.CONF

Then replace squid.conf with the attached squid.conf

add place phpredir.php

in /etc/squid/ with permissions 770 owner root and group squid = phpredir.php

STEP 4 APACHE ADDITIONS

Then add per.php in /var/www/html/ add a videos folder in /var/www/html

at the time youtube caching is being done only i am still looking to make metacafe and blip.tv to work and some other after that if possible

with permissions

drwxrwxrwx 2 root root videos

STEP 5 WIN XP CLIENT SETTINGS

User Client Settings

Ip address

Netmask 255.255.255.0 Gateway 192.168.0.254

192.168.0

For Now as Youtube caching is done with the help of apache server Squid access logs will show TCP_MISS when accessing cached videos but don’t worry just ignore this warning You will be able to verify the youtube caching is working properly by replaying already played video in the client’s browser. And no functionality exists for deleting videos that are not accessed often. If someone can guide in this regard it will be very helpful indeed.

per.php

<?php

$file_path="/var/www/html/videos"; $logfile="$file_path/cache.log"; $url=urldecode($_GET['url']); $urlptr=fopen($_GET['url'],"r");

$blocksize=32*1024;

//attempt to get. a 404 shouldn't happen, but if($urlptr===FALSE){

header("Status: 404 Not Found"); die();

}

//echo $_GET['url']; //find content type and length foreach($http_response_header as $line){

//echo $line;

if(substr_compare($line,'Content-Type',0,12,true)==0)

$content_type=$line; else if(substr_compare($line,'Content-

Length',0,14,true)==0){

$content_length=$line;

}

}

/**Youtube will detect if requests are coming form the wrong ip (ie, if only video requests are redirected, so, we must redirect all requests to youtube. As such, we must capture all requests t youtube. Most are unimportant, so we can pass them straight through **/ if(!preg_match("@.*youtube.*videoplayback.*@",$url)){ fpassthru($urlptr); fclose($urlptr);

}

exit(0);

//send content type and length header($content_type); header($content_length);

//find youtube id; $url_exploded=explode('&',$url); $id=""; foreach($url_exploded as $line){

if(substr($line,0,3)==='id=')

$id=substr($line,3);

}

//Get the supposed file size

$length=intval(substr($content_length,16));

file_put_contents($logfile,"\nFound id=$id, content-type:

$content_type content-length=$content_length\n",FILE_APPEND);

//Do we have it? delivar if we do $fname="$file_path/$id-$length"; //Check if we have the file, and it is the correct size. incorrect size implies corruption //print ($fname); //echo $fname; //echo $length;

//echo filesize($fname);

if(file_exists($fname) &&filesize($fname)==$length){ readfile($fname); // echo $fname; logdata("HIT",$url,$fname);

}

exit(0);

//file not in cache? Get it, send it & save it logdata("What the hell",$url,$fname); $fileptr=fopen($fname,"w"); //no validity check, simply don't write the file if we can't open it. prevents noticeable failure/

while(!feof($urlptr)){ $line=fread($urlptr,$blocksize); echo $line; if($fileptr) fwrite($fileptr,$line);

}

fclose($urlptr); if($fileptr) fclose($fileptr);

function logdata($type,$what, $fname){ $file_path="/var/www/html/videos"; $logfile="$file_path/cache.log"; $line="@ ".time()."Cache $type url: $what file: $fname client:".$_SERVER['REMOTE_ADDR']."\n"; file_put_contents($logfile,$line,FILE_APPEND);

?>

}

phpredir.php

#!/usr/bin/php

<?php

while ( $input = fgets(STDIN) ) { // Split the output (space delimited) from squid into an array. $input=explode(" ",$input);

if(preg_match("@youtube@",$input[0])){

$input[0]=urlencode($input[0]);

$input= implode(" ",$input);

echo "http://192.168.1.111/per.php?url=$input"; //URL of my

web server

}else

// change this ip as per apache server

echo ""; // empty line means no re-write by Squid.

}

squid.conf # # Recommended minimum configuration: # acl manager proto cache_object acl localhost src

squid.conf

#

#

Recommended minimum configuration:

#

acl manager proto cache_object acl localhost src 127.0.0.1/32 ::1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1

# Example rule allowing access from your local networks.

# Adapt to list your (internal) IP networks from where browsing

# should be allowed

acl localnet src 10.0.0.0/8 # RFC1918 possible internal network

acl localnet src 172.16.0.0/12

# RFC1918 possible internal network

acl localnet src 192.168.0.0/24 # my internal networks

acl localnet src fc00::/7 acl localnet src fe80::/10 plugged) machines

# RFC 4193 local private network range # RFC 4291 link-local (directly

acl SSL_ports port 443 acl Safe_ports port 80 acl Safe_ports port 21 acl Safe_ports port 443 acl Safe_ports port 70 acl Safe_ports port 210 acl Safe_ports port 1025-65535 acl Safe_ports port 280 acl Safe_ports port 488 acl Safe_ports port 591 acl Safe_ports port 777

acl CONNECT method CONNECT visible_hostname squidpxy.umair.com

# http # ftp # https # gopher # wais # unregistered ports # http-mgmt # gss-http # filemaker # multiling http

#

#

Recommended minimum Access Permission configuration:

#

#

Only allow cachemgr access from localhost

http_access allow manager localhost #http_access deny manager

# Deny requests to certain unsafe ports http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports http_access deny CONNECT !SSL_ports

# We strongly recommend the following be uncommented to protect

innocent

# web applications running on the proxy server who think the only

# one who can access services on "localhost" is a local user

#http_access deny to_localhost

#

INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS

#

 

#

Example rule allowing access from your local networks.

 

#

Adapt localnet in the ACL section to list your (internal) IP

 

networks

 

#

from where browsing should be allowed

 

http_access allow localnet

 

http_access allow localhost

 

#

And finally deny all other access to this proxy

http_access deny all

 

#

Squid normally listens to port 3128

http_port 3128 transparent

 

acl youtube dstdomain .youtube.com .googlevideo.com

cache deny youtube

 

acl youtubeip dst 74.125.15.0/24

 

cache deny youtubeip

 

acl youtubeip2 dst 74.125.236.0/24

 

cache deny youtubeip2

 

maximum_object_size 13107200 KB

 

maximum_object_size_in_memory 102400 KB

cache_mem 100 MB

 

#

pass requests

 

url_rewrite_program /etc/squid/phpredir.php

url_rewrite_access allow youtube

 

#

leave caching up to the local web server

cache deny youtube

 

cache deny youtubeip

 

cache deny youtubeip2

 

# We recommend you to use at least the following line. hierarchy_stoplist cgi-bin ?

# Uncomment and adjust the following to add a disk cache directory. #cache_dir ufs /var/spool/squid 100 16 256

# Leave coredumps in the first cache dir coredump_dir /var/spool/squid

# Add any of your own refresh_pattern entries above these.

refresh_pattern ^ftp:

1440 20%

10080

refresh_pattern ^gopher:

1440 0%

1440

refresh_pattern -i (/cgi-bin/|\?) 0

0%

0

refresh_pattern .

0

20%

4320

STEP 6 IPTABLES RULES

Configure the iptables as shown below. And it will make your server act as a router for clients. Add the following rules in vi /etc/rc.local to make squid act as transparent proxy server on startup.

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE iptables -A FORWARD -i eth1 -j ACCEPT echo 1 > /proc/sys/net/ipv4/ip_forward iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 3128

Note : if you have trouble having to view youtube then try http://www.youtube.com instead of https://www.youtube.com

You can always contact me on umairjunaid137@gmail.com

This tutorial was inspired and taken from http://wiki.squid- cache.org/ConfigExamples/DynamicContent/YouTube