by Marliza Ramly Zurina Saaya Wahidah Md Shah Mohammad

Transcription

version 2.6
by
Marliza Ramly
Zurina Saaya
Wahidah Md Shah
Mohammad Radzi Motsidi
Haniza Nahar
Faculty of Information and Communication Technology
Universiti Teknikal Malaysia Melaka (UTeM)
May 2007
Copyright © 2007 Fakulti Teknologi Maklumat dan Komunikasi, UTeM
TABLE OF CONTENT
1.
PROXY SERVERS......................................................................................... 1
1.2
2.
INTERNET CACHING ................................................................................... 4
2.1
2.2
2.3
2.4
3.
3.4
3.5
3.6
4.2
ACCESS CONTROLS ........................................................................ 25
List of ACL type......................................................................... 26
src .......................................................................................... 27
srcdomain ................................................................................ 28
dst .......................................................................................... 29
dstdomain ................................................................................ 29
srcdom_regex........................................................................... 30
dstdom_regex........................................................................... 30
time ........................................................................................ 31
url_regex ................................................................................. 32
urlpath_regex ........................................................................... 33
port......................................................................................... 34
proto ....................................................................................... 35
method .................................................................................... 36
browser ................................................................................... 36
proxy_auth............................................................................... 37
maxconn.................................................................................. 38
Create custom error page ........................................................... 39
EXERCISES ................................................................................. 40
CACHING.................................................................................................... 42
5.1
5.2
6.
HARDWARE AND SOFTWARE REQUIREMENT ............................................ 10
DIRECTORY STRUCTURE .................................................................. 11
GETTING AND INSTALLING SQUID ........................................................ 11
Custom Configuration for Network ............................................... 11
INSTALL SQUID............................................................................. 16
BASIC SQUID CONFIGURATION .......................................................... 17
Configure SQUID ....................................................................... 17
Basic Configuration.................................................................... 17
Starting Squid Daemon ............................................................. 19
Starting Squid Daemon .............................................................. 20
BASIC CLIENT SOFTWARE CONFIGURATION ............................................ 22
Configuring Internet Browser ...................................................... 22
Using proxy.pac File................................................................... 23
ACL CONFIGURATION............................................................................... 25
4.1
5.
HIERARCHICAL CACHING ...................................................................4
TERMINOLOGY FOR HIERARCHICAL CACHING .............................................5
INTERNET CACHE PROTOCOL ...............................................................7
BASIC NEIGHBOUR SELECTION PROCESS .................................................7
INTRODUCTION TO SQUID ......................................................................... 9
3.1
3.2
3.3
4.
KEY FEATURES OF PROXY SERVERS ........................................................2
Proxy Servers and Caching ...........................................................2
CONCEPTS .................................................................................. 42
CONFIGURING A CACHE FOR PROXY SERVER ............................................ 42
SQUID AND WEBMIN ................................................................................. 47
i
6.1 ABOUT WEBMIN ................................................................................ 47
6.2
OBTAINING AND INSTALLING WEBMIN .................................................. 47
Installing from a tar. gz.............................................................. 48
Installing from an RPM ............................................................... 48
After Installation ....................................................................... 49
6.3
USING SQUID IN WEBMIN ................................................................ 49
6.4
PORTS AND NETWORKING ................................................................ 50
Proxy port ................................................................................ 51
ICP port ................................................................................... 51
Incoming TCP address................................................................ 51
Outgoing TCP address ................................................................ 52
Incoming UDP address ............................................................... 52
Outgoing UDP address ............................................................... 52
Multicast groups........................................................................ 52
TCP receive buffer ..................................................................... 53
6.5
OTHER CACHES ............................................................................ 53
Internet Cache Protocol.............................................................. 53
Parent and Sibling Relationships .................................................. 54
When to Use ICP?...................................................................... 54
6.6
OTHER PROXY CACHE SERVERS ......................................................... 55
Edit Cache Host ........................................................................ 56
Hostname ................................................................................ 56
Type........................................................................................ 57
Proxy port ................................................................................ 57
ICP port ................................................................................... 57
Proxy only? .............................................................................. 58
Send ICP queries? ..................................................................... 58
Default cache ........................................................................... 58
Round-robin cache? ................................................................... 58
ICP time-to-live ........................................................................ 59
Cache weighting........................................................................ 59
Closest only.............................................................................. 59
No digest?................................................................................ 59
No delay?................................................................................. 60
Login to proxy .......................................................................... 60
Multicast responder ................................................................... 60
Query host for domains, Don’t query for domains .......................... 60
Cache Selection Options ............................................................. 61
Directly fetch URLs containing ..................................................... 61
ICP query timeout ..................................................................... 62
Multicast ICP timeout................................................................. 62
Dead peer timeout .................................................................... 62
Memory Usage.......................................................................... 63
Memory usage limit ................................................................... 63
Maximum cached object size....................................................... 64
6.7
LOGGING ................................................................................... 64
Cache metadata file................................................................... 65
Use HTTPD log format ................................................................ 65
Log full hostnames .................................................................... 66
Logging netmask....................................................................... 66
6.8
CACHE OPTIONS ........................................................................... 67
6.9
ACCESS CONTROL ......................................................................... 68
Access Control Lists ................................................................... 69
Edit an ACL .............................................................................. 69
Creating new ACL...................................................................... 70
Available ACL Types................................................................... 71
6.10 ADMINISTRATIVE OPTIONS ............................................................... 75
7.
ii
ANALYZER ................................................................................................ 78
7.1
7.2
7.3
7.4
7.5
STRUCTURE OF LOG FILE .................................................................. 78
Access log ................................................................................ 78
Cache log ................................................................................. 90
Store log.................................................................................. 93
METHODS ................................................................................... 96
Log Analysis Using Grep Command .............................................. 96
Log Analysis Using Sarg-2.2.3.1 .................................................. 96
SETUP SARG-2.2.3.1 .................................................................... 97
REPORT MANAGEMENT USING WEBMIN ................................................. 98
LOG ANALYSIS AND STATISTIC ........................................................ 105
iii
ABBREVIATIONS
Abbreviation
Details
ACL
Access Control List
CARP
Cache Array Routing Protocol
CD
Compact Disk
DNS
Domain Name Service
FTP
File Transfer Protocol
GB
Gigabyte
HTCP
Hyper Text Caching Protocol
HTTP
Hypertext Transfer Protocol
I/O
Input/Output
ICP
Internet Cache Protocol
IP
Internet Protocol
LAN
Local Area Network
MAC
Media Access Control
MB
Megabyte
RAM
Random Access Memory
RPM
Red Hat Package Manager
RTT
Round Trip Time
SNMP
Simple Network Management Protocol
SSL
Secure Socket Layer
UDP
User Datagram Protocol
URL
Uniform Resource Locator
UTeM
Universiti Teknikal Malaysia Melaka
WCCP
Web Cache Coordination Protocol
iv
1
Chapter
1. Proxy Servers
A Proxy Server is an intermediary server between the Internet browser
and the remote server. It acts like a "middleman" between the two
ends of the client/server network connection and also works with
browsers and servers or other application by supporting underlying
network protocols like HTTP. Furthermore, it store and download
documents in its local cache so that the downloading time from the
internet can be faster because the document is store in a local server.
For example, lets imagine when a user want to download documents
from the Internet browser with a specify URL address such as
http://www.yahoo.com, which then the document will be transfer to
workstation. (e.g UTeM to local workstation). In that situation, the
internet browser communicates directly with the proxy server UTem to
get the document.
In addition, a cache is combined with a proxy server which will make it
reliable for quicker transfer. In this matter, Internet browser will no
longer contact the remote server directly but it request document from
the proxy server.
1
Proxy Servers
1.2 Key features of proxy servers
Four main functions provided are:

Firewalling and Filtering (security)

Connection Sharing

Administrative Control

Caching service
Proxy Servers and Caching
Proxy Server with the caching of Web pages may leads to a better
improvement for QoS in network as in Figure 1-1. It can be specified in
three ways:

Caching may preserve bandwidth on the network and proliferate
scalability

Enhancement of response time (e.g: HTTP proxy cache can load
Web Pages more quickly into the browser)

Proxy server caches boost to the availability, where Web pages
or other files in the cache remain accessible even if the original
source or an intermediate network link goes offline.
2
Proxy Servers
client
client
Proxy Server
Internet
client
client
client
Figure 1-1: Generic Diagram for Proxy Server
3
2
Chapter
2. Internet Caching
2.1 Hierarchical Caching
Cache Hierarchies are a logical extension of the caching concept. A
sharing concept might help and give some benefit for a group of Web
caches and a group of Web Clients. Figure 2-1 shows how it works.
However, there are some disadvantages as well. It will depends on the
specific situation discuss below whether the advantages will outweigh
the disadvantages.
4
3
Proxy server caches
returned page
5
client
Proxy server returns the
requested page to the client
Proxy Server
Web server returns
requested URL to
proxy server
Yes
internet
1
Client browser initiates
request to proxy server
for the URL
Is requested page in
proxy server cache?
2
No
Proxy server
requests the page
from the web server
Figure 2-1: Proxy Server Caching Process
4
Web server
Internet Caching
The major advantages are:

Additional cache hits. In general, the cache hits that are
expected from the requested user will be at the neighbor caches.

Request routing. The availability to direct the HTTP traffic along
a certain path can be done by routing requests to specific
caches. (e.g., accessing the Internet with two paths, one of it is
cheap and the other is being expensive, therefore, the user can
send HTTP traffic over the cheapest link using the request
routing.
The disadvantages among the concept:

Configuration hassles. The coordination from both parties are
required to configure neighbors caches. As a result, it will put
some weight to the exacerbates membership

Additional delay for cache misses. There are many factors to
consider due to the delay. For example, delays between peers,
link congestion, and whether or not ICP is used.
2.2 Terminology for Hierarchical Caching
Cache
It is refers to an HTTP proxy that store some requests.
Objects
It is a generic term for any document, image, or other type of data
that available on the Web. Nowadays, the Uniform Resource Locators
(URLs) will identify Web Page with objects (such as images, audio,
video and binary files) rather than documents or pages only from the
data available at HTTP, FTP, Gopher and other types of servers.
5
Internet Caching
Hit and misses
It is a valid copy when a cache hit the requested existing object in a
cache.
If the object does not exist or no longer valid, it is refer to cache miss.
That situation, a cache must forward cache misses toward the origin
server.
Origin Server
It is the authoritative source for an object. For example, the origin
server is the hostname in URL.
Hierarchy vs. Mesh
It is hierarchically arrange when the topology is like a tree structure or
in mesh when the structure is flat. In either case these terms simply
refer to the fact that caches can be ''connected'' to each other. In
squid it can be seen at directory cache after creating it.
Neighbours, Peers, Parents, Siblings
In general, the terms neighbour and peer are the same for caches in a
hierarchy or mesh. While, for parent and sibling will refer to the
relationship between a pair of caches.
Fresh, Stale, Refresh
The status of cached objects can be refer to

A fresh object when a cache hit is returnable.

A stale object and refresh object when the Squid refresh it by
including an IMS request header and forwarding the request on
toward the origin server.
6
Internet Caching
2.3 Internet Cache Protocol
A quick and efficient method of inter-cache communication in ICP's is
by offering a mechanism to establish a complex cache hierarchies. The
advantages by using are;

ICP can be utilized by Squid to provide an indication of network
conditions.

ICP messages are transmitted as UDP packets. It is easier to
implement because each cache needs to maintain only a single
UDP socket.
ICP may convey to some disadvantages as well. One of the failures in
ICP is when the links is highly congested, therefore the ICP become
useless where its caching is needed most. Furthermore, an extra delay
may be a factor in processing request due to the transmission time of
the UDP packet. As a result, ICP is not the appropriate for this delay in
some situation.
2.4 Basic Neighbour Selection Process
Before describing Squid features for hierarchical caching, first lets
briefly explain the neighbor selection process referring to. Squid must
decide where to forward the request when it is unable to satisfy the
request from cache. There are basically three choices can be use:

parent cache

sibling cache

origin server
7
Internet Caching
How ICP can make decision for Squid?

In parent and sibling cache, Squid will send an ICP query
requested URL message to its neighbors. Usually in a UDP
packets and Squid will remembers how many queries it sends for
a given request.

By receiving ICP query in each neighboring, the URL will be
search in its own cache. If a valid copy of the URL exists, then
cache sends ICP_HIT, otherwise an ICP_MISS message.

The querying cache now collects the ICP replies from its peers.

If the cache receives an ICP_HIT reply from a peer, it
immediately forwards the HTTP request to that peer.

If the cache does not receive an ICP_HIT reply, then all replies
will be ICP_MISS.

Squid waits until it receives all replies, up to two seconds.

If one of the ICP_MISS replies comes from a parent, Squid
forwards the request to the parent whose reply was the first to
arrive. We call this reply the FIRST_PARENT_MISS. If there is no
ICP_MISS from a parent cache, Squid forwards the request to
the origin server.
We have described the basic algorithm, to which Squid offers
numerous possible modifications, including mechanisms to:

Send ICP queries to some neighbours and not to others.

Include the origin server in the ICP “pinging” so that if the origin
server reply arrives before any ICP_HITs, the request is
forwarded there directly.

8
Disallow or require the use of some peers for certain requests.
3
Chapter
3. Introduction to Squid
Squid is a high-performance proxy caching server for Web clients,
support FTP, gopher, and HTTP data objects. It has two basic
purposes;

to provide proxy service from machines that must pass Internet
traffic through some form of masquerading firewall

caching
Unlike traditional caching software, Squid handles all requests in a
single, non-blocking, I/O-driven process.
Squid keeps meta data and especially hot objects cached in RAM,
caches
DNS
lookups,
supports
non-blocking
DNS
lookups,
and
implements negative caching of failed requests.
Squid consists of a main server program, a Domain Name System
lookup program (dnsserver), a program for retrieving FTP data (ftpget)
and some management and client tools.
In other words Squid is
1. full featured Web proxy cache
2. free, open-source software
3. the result of many contributions by unpaid (and paid) volunteers
9
Introduction to Squid
Squid Support

proxy and caching of Hypertext Transfer Protocol (HTTP), File
Transfer Protocol (FTP), and other

Uniform Resource Locators (URLs)

Proxiying for Secure Socket Layer (SSL)

cache hierarchies

Internet Cache Protocol (ICP), Hyper Text Caching Protocol
(HTCP), Cache Array Routing Protocol (CARP), Cache Digests

transparent caching

Web Cache Coordination Protocol (WCCP) (Squid v2.3 and
above)

extensive access controls

HTTP server acceleration

Simple Network Management Protocol (SNMP)

caching of DNS lookups
3.1 Hardware and Software Requirement

RAM
Minimum RAM recommended = 128mb (scales by user count and
size of disk cache)

Disk
Small user count = 512MB to 1G
Large user count = 16G to 24G

Most version on UNIX
Also work on AIX, Digital UNIX, FreeBSD, Hp-UX, IRIX, LINUX,
NetBSD, NextStep,SCO, Solaris and SunOS
10
3.2 Directory Structure
Squid normally creates a few directories shown in Table 3-1
Directories
Explaination
/var/cache
Stored the actual data
/etc/squid
Contains the squid.conf file which
it is only squid config file
/var/log
Query each connection (example
if the directory getting larger)
Table 3-1: Squid Directory
3.3 Getting and installing squid
Custom Configuration for Network
There are three configurations for proxy server in the network. The
configuration file will follow to the requirement for the usage in your
network. They are transparency proxy, reverse proxy and web cache
proxy.
11
Configuring squid for transparency
Internet
Transparent Proxy Server
10.1.1.1
10.1.1.1
80
client
client
client
client
client
LAN
client
client
LAN
Figure 3-1: Transparent Proxy
A Transparent proxy (Figure 3-1) is configured when you want to
grab a certain type of traffic at your gateway or router and send it
through a proxy without the knowledge of the user or client. In other
words, router will forward all traffic to port 80 to proxy machine using
a route policy.
By using squid as transparent proxy, it will involve two part of process:
1. squid need to be configured properly to accept non-proxy
requests
2. web traffic gets redirected to the squid port
12
This type of transparency proxy is suitable for

Intercept the network traffic transparently to the browser

Simplified administration- the browser does not need to be
configured to talk to a cache

Central control – the user cannot change the browser to bypass
the cache
The disadvantages of using this type of proxy are

Browser dependency – transparent proxy does not work very
well with certain web-browsers

User control Transparent – caching takes control away from the
user where the user will change ISPs to either avoid it or get it
Configuring squid for reverse proxy
Internet
client
Reverse Proxy Server
Web Server Cluster
Figure 3-2: Reverse Proxy
13
A Reverse Proxy (also known as Web Server Acceleration) (Figure
3-2) is a method of reducing the load on a busy web server by using a
web cache between the server and the internet.
In this case, when a client browser makes a request, the DNS will
route the request to the reverse proxy (this is not the actual web
server). Then the reverse proxy will check its cache to find out
whether the request contains is available to fulfill the client request. If
not, it will contact the real web server and downloads the requested
contains to its disk cache.
Benefits that can be gained are
1. security improvement
2. scalability improvement without increasing the complexity of
maintenance too much.
3. easy burden on a web server that provides both static and
dynamic content. The static content can be cached on the
reverse proxy while the web server will be freed up to better
handle the dynamic content.
To run Squid as an accelerator, you probably want to listen on port 80.
Hence, you have to define the machine you are accelerating for. (not
covered in this chapter).
14
Configuring squid for Web Cache proxy
Internet
Router
Web Cache Proxy Server
Router
client
client
client
client
client
Figure 3-3 Web Cache Proxy
By default, squid is configured as a direct proxy (Figure 3-3). In order
to cache web traffic with squid, the browser must be configured to use
the squid proxy. This needs the following information

proxy server's IP address

port number by which the proxy server accepts connections
15
3.4 Install squid
The Squid proxy caching server software package comes with Fedora
Core V6. Therefore, we do not have to install it. Just manage the
configuration file to make it work.
If no Squid installed in your server you can install it from Squid RPM
file. To do so, you need to download the RPM file from the Internet or
copy it from installation CD. Then run this command
# rpm –i squid-2.6.STABLE4-1.fc6.i386.rpm
NOTE: The RPM file name may be differ depends on the version of
Squid you have downloaded
Alternatively, you can install it from Squid installation script where it
can be downloaded from official Squid Proxy server web site,
http://www.squid-cache.org.
To
do
so,
you
need
to
copy
the
installation folder into your local drive and run the following command.
# ./configure
# make
# make install
NOTE: Make sure all the dependency files are already installed in
your machine before starting to install Squid
16
3.5 Basic Squid Configuration
Configure SQUID
All Squid configuration files are kept in the directory /etc/squid.
The following paragraph of this chapter will works through the options
that may need some further changes to get Squid to run. Most people
will not need to change all of these settings. What usually needs to
change is at least one part of the configuration file though: the default
file in squid.conf, which denies the access to the browser. If you
don't change this, Squid will not be very useful.
Basic Configuration
All of squid configuration goes in one file - squid.conf. This section
details up the configuration of Squid as a caching proxy only, not as
http-accelerator.
Some basic configuration need to be implemented. First, uncomment
and edit the following lines in the configuration file found at default file
/etc/squid/squid.conf
To construct the squid server, do the following tasks
1. log in as root to the machine
2. type the following command
# vi /etc/squid/squid.conf
The above command will open Squid configuration file for editing
17
Then, set the port on which Squid listens. Normally, Squid will listen
on port 3128. While it may convenient to listen on this port, network
administrators often configure the proxy to listen on port 8080 as well.
This is a non-well-known port, while (port 1024 are well-known ports
and are restricted from being used ordinary users processes), and is
therefore not going to be in conflict with other ports such as 80, 443,
22, 23, etc. Squid need not be restricted to one port. It could easily be
started in two or more ports.
At squid.conf file, find out the following sentence for some changes
or leave it as default if its port is 3128.
http_port
Check
http_port 3128 (is a default.)
or
http_port 8080 3128 (for multiple port)
.
18
Additionally, if you have multiple networks cards in your proxy server,
and would like to restrict the proxy to start on port 8080 on the first
network card and port 3128 on the second network card. You can add
the following sentence.
http_port
10.1.5.49:8080
10.0.5.50:3128
http_access
By default http_access is denied. The Access Control Lists (ACL) rules
should be modified to allow access only to the trusted clients. This is
important because it prevents people from stealing your network
resources.
ACL will be discussed in Chapter 4.
cache_dir
This directive specifies the cache directory storage format and its size
as given below.
cache_dir ufs /var/spool/squid 100 16 256
The value 100 denotes 100MB cache size. This can be adjusted to the
required size. (cache will be discuss later in Chapter 5)
cache_effective_user
cache_effective_ group
NOTE: You can edit the squid.conf file by using gedit instead
of command line
19
Starting Squid Daemon
In this chapter, we will learn how to start Squid. Make sure you have
finished editing the configuration file. Then you can start Squid for the
first time.
First, you have to check the error in conf file. Type this command at
your terminal
# squid -k parse
If error detected, for example
# squid –k parse
FATAL: could not determine fully qualified hostname, Please
set ‘visible hostname’
Squid Cache (versio 2.6.STABLE4):Terminated abnormally.
CPU Usage:0.0004 seconds=0.0004 user+0.000 sys
Maximum Resident Size:0KB
Page faults with physical i/o:0
Aborted.
Solution : Add the following sentence in squid.conf file
visible_hostname localhost
If no error detected, continue with the following command to start
squid. (This is temporarily step to start the squid)
# service squid start
If everything is working fine, then your console displays:
Starting squid: .
If you want to stop the service,
# service squid stop
Then your console will display:
20
[OK]
Stopping squid: .
[OK]
You should be a privileged user to start or stop squid.
For permanent step, try this command
# chkconfig –list
# chkconfig –-level 5 squid on
You can restart the squid service by typing
#/etc/init.d/squid restart
While the daemon is running, there are several ways you can run the
squid command to change how the daemon works by using this
options:
# squid –k reconfigure
- causes Squid to read again its configuration file
#squid –k shutdown
- causes Squid to exit after waiting briefly for current connections to
exit
#squid –k interrupt
- shuts down Squid immediately, without waiting for connections to
close
#squid –k kill
– kills Squid immediately, without closing connections or log files. (use
this option only if other methods don’t work)
21
3.6 Basic Client Software Configuration
Basic Configuration
To configure any browser, you need at least two pieces of information:

Proxy server's IP Address

Port number that the proxy server is accepting the requests
Configuring Internet Browser
The following section will explain the steps to configure proxy server in
Internet Explorer, Mozilla Firefox and Opera.
Internet Explorer 7.0
1. Select the Tools menu option
2. Select Internet Options
3. Click on the Connection tab
4. Select LAN settings
5. The Internet using a proxy server
6. Check the box in proxy server Æ Type in the proxy IP address in
the Address field, and the port number in the Port field.
Example:
Address : 10.0.5.10 Port : 3128
Mozilla Firefox
1. Click Tools Æ Options Æ Advanced
2. Click at Network Æ go to connection Æ Settings
3. At the configure proxies to Access Internet
22
4. Choose manual proxy configuration
5. At HTTP Proxy: 10.0.5.10
Port: 3128
6. Check the box to use the proxy server for all protocols
7. Then click OK
8. Now, the client can access the internet.
Opera 9.1
1. Click Tools Æ Preferences Æ Advanced
2. Choose Network
3. Click at Proxy Sever
Check
HTTP
: 10.0.5.10
Port :3128
HTTPs
: 10.0.5.10
Port :3128
FTP
: 10.0.5.10
Port :3128
Gropher
: 10.0.5.10
Port :3128
4. Then, Click OK
Using proxy.pac File
This setting is for the clients when they want to have browsers pick up
proxy setting automatically. The browser can be configured with a
simple proxy.pac file as shown in the example below;
function FindProxyForURL(url, host)
{
if (isInNet(myIpAddress(), "10.0.5.0", "255.255.255.0"))
return "PROXY 10.0.5.10:3128";
else
return "DIRECT";
}
23
proxy.pac needs to be installed in a web server such as Apache, and
the client can configure proxy server using the automatic configuration
script. This script is useful when there is possibility that the proxy
server will change its IP address. To access the script, client needs to
add the URL of proxy.pac in its automatic configuration proxy script
(Figure 3-4).
Figure 3-4: Using automatic configuration script
24
4
Chapter
4. ACL Configuration
4.1 Access controls
Access control lists (ACL) are the most important part in configuring
Squid. The main use of the ACL is to implement simple access control
where it is used to restrict other people from using cache infrastructure
without certain permission. Rules can be written for almost any type of
requirement. It can be very complex for large organisations or just a
simple configuration to home users.
ACL is written in squid.conf file using the following formats
acl name type (string|"filename") [string2] ["filename2"]
name is a variable defined by user and it should be descriptive while
type is defined accordingly and it will be described in the next section
.
25
ACL Configuration
There are two elements in access control: classes and operators.
Classes are defined by the acl, while the name of the operators varies.
The most common operators are http_access and icp_access. The
actions for this operator are allow and deny. allow is used to allow or
enable the ACL while deny used to deny or restrict the ACL
General format for operator
http_access
allow|deny
[!]aclname [!]aclname2 ... ]
List of ACL type
ACL Type
Details
src
client IP address
srcdomain
client domain name
dst
destination’s IP address
dstdomain
destination’s domain name
srcdom_regex
Regular expression describing client domain name
dstdom_regex
Regular expression describing destination domain
name
time
specify the time
url_regex
Regular
expression
describing
whole
URL
of
URL
of
destination (web server)
urlpath_regex
Regular
expression
describing
path
of
destination (not include its domain name)
port
Specify port number
proto
Specify protocol
method
Specify method
browser
Specify browser
proxy_auth
User authentication via external processes
maxconn
Specify number of connection
26
ACL Configuration
src
Description
This ACL allows server to recognize client (the computer which will use
server as proxy to get access to the internet ) using its IP address. The
IP address can be listed using single IP address, range of IP or using
defined IP address in an external file.
Syntax
acl
aclname
src
ip-address/netmask .. (clients IP address)
acl
aclname
src
addr1-addr2/netmask .. (range of addresses)
acl
aclname
src
“filename” ..(client's IP address in external file)
Example 1
acl fullaccess src “/etc/squid/fullaccess.txt”
http_access allow fullaccess
This ACL is using external file named fullaccess.txt where fullaccess.txt
consist of list of IP address of the client.
Example of fullaccess.txt
198.123.56.12
198.123.56.13
198.123.56.34
Example 2
acl office.net src 192.123.56.0/255.255.255.0
http_access allow office.net
This ACL set the source address for office.net in range 192.123.56.x to
access the Internet using http_access allow operator
27
ACL Configuration
srcdomain
Description
This ACL allows server to recognize client using client’s computer
name. To do so, squid needs to reverse DNS lookup (from client ipaddress to client domain-name) before this ACL is interpreted, it can
cause processing delays.
Syntax
acl
aclname
srcdomain domain-name..(reverse lookup client IP)
Example 1
acl staff.net srcdomain staff20 staff21
http_access allow
staff.net
This ACL is for clients with computer name staff20 and staff21. The
operator http_access is allowing the ACL named staff.net to access
the Internet. This option is not really effective since the computer must
do reverse name lookup to determine the source name.
NOTE: Please ensure the DNS server in running in order to use
DNS lookup service
28
ACL Configuration
dst
Description
This is same as src, the difference is only it refers to Server’s IP
address (destination). First, Squid will dns-lookup for IP Address from
the domain-name, which is in request header, and then interpret it
Syntax
acl
aclname
dst
ip_address/netmask .. (URL host's or the site
IP address)
Example 1
acl tunnel dst 209.8.233.0/24
http_access deny tunnel
This ACL deny any node with IP 209.8.233.x
Example 2
acl allow_ip dst 209.8.233.0-209.8.233.100/255.255.0.0
http_access allow allow_ip
This ACL is allowing destination with IP address range
from
209.8.233.0 to 209.8.233.100.
dstdomain
Description
This ACL recognize destination using its domain. This is the effective
method to control specific domain
Syntax
acl
aclname
dstdomain
domain.com
(domain name from the site's URL)
29
ACL Configuration
Example 1
acl banned_domain dstdomain www.terrorist.com
http_access deny banned_domain
This ACL deny destionation with domain www.terrorist.com
srcdom_regex
Description
This ACL is almost similar to srcdomain where the server needs to
reverse DNS lookup (from client ip-address to client domain-name)
before this ACL is interpreted. The difference is this ACL allow the
usage of regular expression in defining the client’s domain.
Syntax
acl
aclname
srcdom_regex -i
source_domain_regex
Example 1
acl staff.net srcdom_regex -i staff
http_access allow staff.net
This ACL allows all the node with the domain contains word staff to
access the internet. Option -i is used to make expression caseinsensitive
dstdom_regex
Description
This ACL allows server to recognize destination using its domain
regular expression.
Syntax
acl
30
aclname
dstdom_regex -i
dst_domain_regex
ACL Configuration
Example 1
acl banned_domain dstdom_regex -i terror porn
http_access deny banned_domain
This ACL denies client to access the destinations that contain word
terrorist or porn in its domain name. For example the access to the
domain www.terrorist.com and www.pornoragphy.net will be denied by
proxy server.
time
Description
This ACL allows server to control the service using time function. The
accessibility to the network can be set according the scheduled time in
ACL
Syntax
acl
aclname
time
day abbrevs h1:m1h2:m2
where h1:m1 must be less than h2:m2 and day will be represented
using abbreviation in Table 4-1
day
abbreviations
S
Sunday
M
Monday
T
Tuesday
W
Wednesday
H
Thursday
F
Friday
A
Saturday
Table 4-1 Abbreviation for Day
31
ACL Configuration
Example 1
acl SABTU time A 9:00-17:00
ACL SABTU refers to day of Saturday from 9:00 to 17:00
Example 2
acl pagi time 9:00-11:00
acl office.net 10.2.3.0/24
http_access deny pagi office.net
pagi refers time from 9:00 to 11:00, while office.net refer to the
clients' IP. This combination of ACLs deny the access for office.net if
the time is between 9.00am to 11.00 am
url_regex
Description
The url_regex means to search the entire URL for the regular
expression you specify. Note that these regular expressions are casesensitive. To make them case-insensitive, use the -i option
Syntax
acl
aclname
url_regex -i
url_regex ..
Example 1
acl banned_url url_regex -i terror porn
http_access deny banned_url
This ACL deny URL that contains word terrorist or porn.
For example, the following destination will be denied by the proxy
server;
http://www.google.com/pornography
http://www.news.com/terrorist.html
http://www.terror.com/
32
ACL Configuration
urlpath_regex
Description
The urlpath_regex is regular expression pattern matching from URL
but excluding protocol and hostname.
If
URL
is
http://www.free.com/latest/games/tetris.exe
then
this
acltype only looks after http://www.free.com/. It will leave out the http
protocol and www.free.com hostname.
Syntax
acl
aclname
urlpath_regex
pattern
Example 1
acl blocked_free urlpath_regex free
http_access deny blocked_free
This ACL will blocked any URL that only containing "free'' not "Free”,
and without referring to protocol and hostname.
These regular expressions are case-sensitive. To make them caseinsensitive, add the –i option.
Example 2
acl blocked_games urlpath_regex –i games
http_access deny blocked_games
blocked_games refers to the URL containing word “games” no matter if
the spelling in upper or lower case.
Example 3
To block several URL.
acl block_site urlpath_regex –i
“/etc/squid/acl/block_site”
http_access deny block_site
33
ACL Configuration
To block several URL, it is recommended to put the lists in one file. As
in Example 3, all block_site list is in /etc/squid/acl/block_site file.
File block_site may containing, for example
\.exe$
\.mp3$
port
Description
Access can be controlled by destination (server) port address
Syntax
acl
aclname
port port-number
Example 1
Deny requests to unknown ports
acl Safe_ports port 80
acl Safe_ports port 21
acl Safe_ports port 443 563
# http
# ftp
# https, snews
http_access deny !Safe_ports
Example 2
Deny to several untrusted ports
acl safeport port “/etc/squid/acl/safeport”
http_access deny safeport
34
ACL Configuration
proto
Description
This specifies the transfer protocol
Syntax
acl
aclname
proto
protocol
Example 1
acl protocol proto HTTP FTP
This refers protocols HTTP and FTP
Example 2
acl manager proto cache_object
http_access allow manager localhost
http_access deny manager
Only allow cachemgr access from localhost.
Example 3
acl ftp proto FTP
http_access deny ftp
http_access allow all
This command should block every ftp request
35
ACL Configuration
method
Description
This specifies the type of the method of the request
Syntax
acl
aclname
method
method-type
Example 1
acl connect method CONNECT
http_access allow localhost
http_access allow allowed_clients
http_access deny connect
the CONNECT method to prevent outside people from trying to connect
to the proxy server
browser
Description
Regular expression pattern matching on the request's user-agent
header. To grep the user-agent header information, squid.conf
should be added this line:
useragent_log /var/log/squid/useragent.log
Then, try to run the Mozilla browser. The user-agent header for Mozilla
should be as in the example.
Syntax
acl
aclname
browser
pattern
Example 1
acl mozilla browser ^Mozilla/5\.0
http_access deny mozilla
This command will deny Mozilla browsers or any other browser related
to it.
36
ACL Configuration
proxy_auth
Description
User authentication via external processes. proxy_auth requires an
EXTERNAL
authentication
program
to
check
username/password
combinations. In this configuration, we use the NCSA authentication
method because it is the easiest method to implement.
Syntax
acl
aclname
proxy_auth
username...
Example 1
To validate a listing of users, we should do the following steps.
Creating passwd file
# touch
# chown
# chmod
/etc/squid/passwd
root.squid /etc/squid/passwd
640 /etc/squid/passwd
Adding users
# htpasswd
/etc/squid/passwd shah
You will be prompted to enter a passwd for that user. In the example is
the passwd for user shah.
Setting rules
auth_param basic program /usr/lib/squid/ncsa_auth
/etc/squid/passwd
auth_param basic children 5
auth_param basic realm Squid proxy-caching web-server
auth_param basic credentialsttl 2 hours
These listings are already in the configuration file but need to be
adjusted to suit your environments.
37
ACL Configuration
Authentication configuration
acl LOGIN proxy_auth REQUIRED
http_access allow LOGIN
This command will only allow user that have been authenticated during
accessing network connection.
CAUTION !! proxy_auth can't be used in a transparent proxy.
maxconn
Description
A limit on the maximum number of connections from a single client IP
address. It is an ACL that will be true if the user has more than
maxconn connections open.
Syntax
acl
aclname
maxconn
number_of_connection
Example 1
acl someuser src 10.0.5.0/24
acl 5conn maxconn 5
http_access deny someuser 5conn
The command will restrict users in 10.0.5.0/24 subnet to have only
five (5) maximum connections at once. If exceed, the error page will
appear. Other users are not restricted to this command by adding the
last line.
CAUTION !! The maxconn ACL requires the client_db feature. If
client_db is disabled (for example with client_db off) then maxconn
ALCs will not work.
38
ACL Configuration
Create custom error page
# vi /etc/squid/error/ERROR_MESSAGE
Append the following
<HTML>
<HEAD>
<TITLE> ERROR : ACCESS DENIED FROM PROXY SERVER </TITLE>
</HEAD>
<BODY>
<H1> The site is blocked due to IT policy</H1>
<p> Please contact helpdesk for more information: </p>
Phone: 06-2333333 (ext 33) <br>
Email: helpdesk@utem.edu.my <br>
CAUTION !!
Do not include HTML close tags </HTML></BODY>
Displaying custom error message
acl blocked_port port 80
deny_info ERROR_MESSAGE block_port
http_access deny block_port
39
ACL Configuration
4.2 Exercises
1.
Why the users still can do the download process with the
following configuration.
acl download urlpath_regex -i \.exe$
acl office_hours time 09:00-17:00
acl GET method GET
acl it_user1 src 192.168.1.88
acl it_user2 src 192.168.1.89
acl nodownload1 src 192.168.1.10
acl nodownload2 src 192.168.1.11
http_access
http_access
http_access
http_access
allow
allow
allow
allow
it_user1
it_user2
nodownload1
nodownload2
http_access deny GET office_hours nodownload1 nodownload2
http_access deny all
The configuration should deny the nodownload1 and nodownload2. the
allow lines should be deleted.
40
ACL Configuration
2.
Why this configuration still bypasses the game.free.com?
acl ban dstdomain free.com
http_access deny ban
3.
The following access control configuration will never work. Why?
acl ME src 10.0.0.1
acl YOU src 10.0.0.2
http_access allow ME YOU
41
Caching
5
Chapter
5. Caching
5.1 Concepts

Caching (a.k.a proxy server) is the process of storing data on the
intermediate system between the Web server and the client.

The proxy server can simply send the content requested by the
client form it copy in cache.

The assumption is that later requests for the same data can be
serviced more quickly by not having to go all the way back to the
original server.

Caching also can reduce demands on network resources and on
the information servers.
5.2 Configuring a cache for proxy server
There are a lot of parameters related to caching in Squid and these
parameters can be divided into three main groups as below:
A. Cache size
B. Cache directories and log file path name
C. Peer cache servers and Squid hierarchy
42
Caching
However, in the following subsection, only the first two groups will be
covered.
A. Cache Size
The following are the common parameters used in cache size.
i. cache_mem
Syntax
cache_mem size(MB)
This parameter specifies the amount of cache memory (RAM)
used to store in-transit object (ones that are currently being
used), hot objects (one that are used often) and negative-cached
object (recent failed request). Default size value is 8MB.
Example:
cache_mem 16 MB
ii. maximum_object_size
Syntax
maximum_object_size
size(MB)
This parameter used if you want not to cache file that are
larger or equal to the size set. Default size value is 4MB.
Example:
maximum_object_size 8 MB
43
Caching
iii. ipcache_size
Syntax
ipcache_size
size(MB)
This parameter used to set how many IP address resolution
values Squid stores. Default value size is 1MB.
Example:
ipcache_size 32MB
iv. ipcache_high
Syntax
ipcache_high percentage
This parameter specifies the percentage that causes Squid to
start clearing out the least-used IP address resolution. Usually
the default value is always used.
Example:
ipcache_high 95
v. ipcache_low
Syntax
ipcache_low
percentage
This parameter specifies the percentage that causes Squid to
stop clearing out the least-used IP address resolution. Usually
the default value is always used.
44
Caching
Example:
ipcache_low 90
B. Cache Directories
i. cache_dir
Syntax
cache_dir
type dir
size(MB)
L1
L2
This parameter specifies the directory/directories in which cache
swap files are stored. The default dir is /var/spool/squid
directory. We can specify how much disk space to use for cache
in megabytes (100 is the default), the default number of firstlevel directories (L1) and second-level directories (L2) is 16 and
256 respectively.
Example:
cache_dir aufs /var/cache01 7000 16 256
NOTE: /var/cache01 is a partition that have been created
during Linux Fedora installation
Formula to calculate the first-level directories (L1):
Given :
x=Size of cache dir in KB (e.g., 6GB = 6,000,000KB)
y=Average object size (e.g, 13KB)
z=Objects per L2 directories (Assuming 256)
calculate:
L1 = number of L1 directories
L2 = number of L2 directories
such that:
L1 x L2 = x / y / z
45
Caching
Example :
x = 6GB
= 6 * 1024 *1024 = 6291456 KB
so ;
x / y / z = 6291456 / 13 / 256
= 1890
and
L1 * L2 = x / y / z
L1 * 256 = 1890
L1
= 7
ii. access_log
Syntax
cache_log dir
This parameter specifies the location where the HTTP and ICP
accesses are stored. The default dir /var/log/squid/access.log is
always used.
Example:
cache_log /var/log/squid/access.log
46
6
Chapter
6. SQUID and Webmin
6.1 About Webmin
Webmin is a graphical user interface for system administration for
Unix. It is a web-based system and can be installed in most of the Unix
system. Webmin is a free software and the installation package can be
downloaded from the Net. Webmin is largely based on Perl, and it is
running as its own process, and web server. It usually uses TCP port
10000 for communicating, and can be configured to use SSL if
OpenSSL is installed.
6.2 Obtaining and Installing Webmin
Webmin installation package is available at the official Webmin site
http://www.webmin.com/download.html.
You can download the latest package and locate it in the local machine.
47
SQUID and Webmin
Installation of Webmin differs slightly depending on which type of
package you choose to install. Note that Webmin requires a relatively
recent Perl for any of these installation methods to work. Nearly all, if
not all, modern UNIX and UNIX-like OS variants now include Perl as a
standard component of the OS, so this should not be an issue.
Installing from a tar. gz
First you must untar and unzip the archive in the directory where you
would like Webmin to be installed. The most common location for
installation from tarballs is /usr/local. Some sites prefer /opt. If
you’re using GNU tar, you can do this all on one command line:
#tar zxvf webmin-1.340.tar.gz
If you have a less capable version of tar, you must unzip the file first
and then untar it:
# gunzip webmin-1.340.tar.gz
# tar xvf webmin-1.340.tar.gz
Next, you need to change to the directory that was created when you
untarred the archive, and execute the setup.sh script, as shown in
the following example. The script will ask several questions about your
system and your preferences for the installation. Generally, accepting
the default values will work. The command for installation as below:
# ./setup.sh
Installing from an RPM
Installing from an RPM is even easier. You only need to run one
command:
# rpm -Uvh webmin-1.340-1.noarch.rpm
48
SQUID and Webmin
This will copy all of the Webmin files to the appropriate locations and
run the install script with appropriate default values. For example, the
Webmin perl files will be installed in /usr/libexec/webmin while the
configuration files will end up in /etc/webmin. Webmin will then be
started on port 10000. You may log in using root as the login name
and your system root password as the password. It's unlikely you will
need to change any of these items from the command line, because
they can all be modified using Webmin. If you do need to make any
changes, you can do so in miniserv.conf in /etc/webmin.
After Installation
After
installation,
your
Webmin
installation
will
behave
nearly
identically, regardless of operating system vendor or version, location
of installation, or method of installation. The only apparent differences
between systems will be that some have more or fewer modules
because some are specific to one OS. Others will feature slightly
different versions of modules to take into account different functioning
of the underlying system. For example, the package manager module
may behave differently, or be missing from the available options
entirely, depending on your OS.
6.3 Using Squid in Webmin
To launch Webmin, open a web browser, such as Netscape or Mozilla
Firefox, on any machine that has network access to the server on
which you wish to log in. Browse to port 10000 on the IP or host name
of the server using http://computername:10000/. Go to menu Squid
Proxy Server (in submenu Server) to open the main panel (Figure 6-1)
49
SQUID and Webmin
Figure 6-1: Squid Proxy Main Page
6.4 Ports and Networking
The Ports and Networking page provides you with the ability to
configure most of the network level options of Squid. Squid has a
number of options to define what ports Squid operates on, what IP
addresses it uses for client traffic and intercache traffic, and multicast
options. Usually, on dedicated caching systems these options will not
be useful. But in some cases you may need to adjust these to prevent
the Squid daemon from interfering with other services on the system
or on your network.
50
SQUID and Webmin
Proxy port
Sets the network port on which Squid operates. This option is usually
3128 by default and can almost always be left on this address, except
when multiple Squids are running on the same system, which is
usually ill-advised. This option corresponds to the http_port option in
squid.conf.
ICP port
This is the port on which Squid listens for Internet Cache Protocol, or
ICP, messages. ICP is a protocol used by web caches to communicate
and share data. Using ICP it is possible for multiple web caches to
share cached entries so that if any one local cache has an object, the
distant origin server will not have to be queried for the object. Further,
cache hierarchies can be constructed of multiple caches at multiple
privately interconnected sites to provide improved hit rates and higherquality web response for all sites. More on this in later sections. This
option correlates to the icp_port directive.
Incoming TCP address
The address on which Squid opens an HTTP socket that listens for
client connections and connections from other caches. By default Squid
does not bind to any particular address and will answer on any address
that is active on the system. This option is not usually used, but can
provide some additional level of security, if you wish to disallow any
outside network users from proxying through your web cache. This
option correlates to the tcp_incoming_address directive.
51
SQUID and Webmin
Outgoing TCP address
Defines the address on which Squid sends out packets via HTTP to
clients and other caches. Again, this option is rarely used. It refers to
the tcp_ outgoing_address directive.
Incoming UDP address
Sets the address on which Squid will listen for ICP packets from other
web caches. This option allows you to restrict which subnets will be
allowed to connect to your cache on a multi-homed, or containing
multiple
subnets,
Squid
host.
This
option
correlates
to
the
udp_incoming_address directive.
Outgoing UDP address
The address on which Squid will send out ICP packets to other web
caches. This option correlates to the udp_outgoing_address.
Multicast groups
The multicast groups that Squid will join to receive multicast ICP
requests. This option should be used with great care, as it is used to
configure your Squid to listen for multicast ICP queries. Clearly if your
server is not on the MBone, this option is useless. And even if it is, this
may not be an ideal choice.
52
SQUID and Webmin
TCP receive buffer
The size of the buffer used for TCP packets being received. By default
Squid uses whatever the default buffer size for your operating system
is. This should probably not be changed unless you know what you’re
doing, and there is little to be gained by changing it in most cases.
This correlates to the tcp_recv_bufsize directive.
6.5 Other Caches
The Other Caches page provides an interface to one of Squid’s most
interesting, but also widely misunderstood, features. Squid is the
reference implementation of ICP, a simple but effective means for
multiple caches to communicate with each other regarding the content
that is available on each. This opens the door for many interesting
possibilities when one is designing a caching infrastructure.
Internet Cache Protocol
It is probably useful to discuss how ICP works and some common
usages for ICP within Squid, in order to quickly make it clear what it is
good for, and perhaps even more importantly, what it is not good for.
The most popular uses for ICP are discussed, and more good ideas will
probably arise in the future as the Internet becomes even more global
in scope and the web-caching infrastructure must grow with it.
53
SQUID and Webmin
Parent and Sibling Relationships
The ICP protocol specifies that a web cache can act as either a parent
or a sibling. A parent cache is simply an ICP capable cache that will
answer both hits and misses for child caches, while a sibling will only
answer hits for other siblings. This subtle distinction means simply that
a parent cache cans proxy for caches that have no direct route to the
Internet. A sibling cache, on the other hand, cannot be relied upon to
answer all requests, and your cache must have another method to
retrieve requests that cannot come from the sibling. This usually
means that in sibling relationships, your cache will also have a direct
connection to the Internet or a parent proxy that can retrieve misses
from the origin servers. ICP is a somewhat chatty protocol, in that an
ICP request will be sent to every neighbor cache each time a cache
miss occurs. By default, whichever cache replies with an ICP hit first
will be the cache used to request the object.
When to Use ICP?
ICP is often used in situations wherein one has multiple Internet
connections, or several types of paths to Internet content. Finally, it is
possible,
though
usually
not
recommended,
to
implement
a
rudimentary form of load balancing through the use of multiple parents
and multiple child web caches.
One of the common uses of ICP is cache meshes. A cache mesh is, in
short, a number of web caches at remote sites interconnected using
ICP. The web caches could be in different cities, or they could be in
different buildings of the same university or different floors in the same
office building. This type of hierarchy allows a large number of caches
to benefit from a larger client population than is directly available to it.
54
SQUID and Webmin
All other things being equal, a cache that is not overloaded will
perform better (with regard to hit ratio) with a larger number of
clients. Simply put, a larger client population leads to a higher quality
of cache content, which in turn leads to higher hit ratios and improved
bandwidth savings. So, whenever it is possible to increase the client
population without overloading the cache, such as in the case of a
cache mesh, it may be worth considering. Again, this type of hierarchy
can be improved upon by the use of Cache Digests, but ICP is usually
simpler to implement and is a widely supported standard, even on
non-Squid caches.
Finally, ICP is also sometimes used for load balancing multiple caches
at the same site. ICP, or even Cache Digests for that matter, are
almost never the best way to implement load balancing. Using ICP for
load balancing can be achieved in a few ways.
•
Through have several local siblings, which can each provide hits
to the others’ clients, while the client load is evenly divided
across the number of caches.
•
Using fast but low-capacity web cache in front of two or more
lower-cost, but higher-capacity, parent web caches. The parents
will then provide the requests in a roughly equal amount.
6.6 Other Proxy Cache Servers
This section of the Other Caches page provides a list of currently
configured sibling and parent caches, and also allows one to add more
neighbor caches. Clicking the name of a neighbor cache will allow you
to edit it. This section also provides the vital information about the
neighbor caches, such as the type (parent, sibling, multicast), the
proxy or HTTP port, and the ICP or UDP port of the caches. Note that
55
SQUID and Webmin
Proxy port is the port where the neighbor cache normally listens for
client traffic, which defaults to 3128.
Edit Cache Host
Clicking a cache peer name or clicking Add another cache on the
primary Other Caches page brings you to this page, which allows you
to edit most of the relevant details about neighbor caches (Figure 6-2)
Figure 6-2: Create cache Host page
Hostname
The name or IP address of the neighbor cache you want your cache to
communicate with. Note that this will be one-way traffic. Access
Control Lists, or ACLs, are used to allow ICP requests from other
caches. ACLs are covered later. This option plus most of the rest of the
options on this page correspond to cache_ peer lines in squid.conf.
56
SQUID and Webmin
Type
The type of relationship you want your cache to have with the neighbor
cache. If the cache is upstream, and you have no control over it, you
will need to consult with the administrator to find out what kind of
relationship you should set up. If it is configured wrong, cache misses
will likely result in errors for your users. The options here are sibling,
parent, and multicast.
Proxy port
The port on which the neighbor cache is listening for standard HTTP
requests. Even though the caches transmit availability data via ICP,
actual web objects are still transmitted via HTTP on the port usually
used for standard client traffic. If your neighbor cache is a Squid-based
cache, then it is likely to be listening on the default port of 3128. Other
common ports used by cache servers include 8000, 8888, 8080, and
even 80 in some circumstances.
ICP port
The port on which the neighbor cache is configured to listen for ICP
traffic. If your neighbor cache is a Squid-based proxy, this value can
be found by checking the icp_port directive in the squid.conf file on
the neighbor cache. Generally, however, the neighbor cache will listen
on the default port 3130.
57
SQUID and Webmin
Proxy only?
A simple yes or no question to tell whether objects fetched from the
neighbor cache should be cached locally. This can be used when all
caches are operating well below their client capacity, but disk space is
at a premium or hit ratio is of prime importance.
Send ICP queries?
Tells your cache whether or not to send ICP queries to a neighbor. The
default is Yes, and it should probably stay that way. ICP queries is the
method by which Squid knows which caches are responding and which
caches are closest or best able to quickly answer a request.
Default cache
This is switched to Yes if this neighbor cache is to be the last-resort
parent cache to be used in the event that no other neighbor cache is
present as determined by ICP queries. Note that this does not prevent
it from being used normally while other caches are responding as
expected. Also, if this neighbor is the sole parent proxy, and no other
route to the Internet exists, this should be enabled.
Round-robin cache?
Choose whether to use round-robin scheduling between multiple
parent caches in the absence of ICP queries. This should be set on all
parents that you would like to schedule in this way.
58
SQUID and Webmin
ICP time-to-live
Defines the multicast TTL for ICP packets. When using multicast ICP, it
is usually wise for security and bandwidth reasons to use the minimum
tty suitable for your network.
Cache weighting
Sets the weight for a parent cache. When using this option it is
possible to set higher numbers for preferred caches. The default value
is 1, and if left unset for all parent caches, whichever cache responds
positively first to an ICP query will be sent a request to fetch that
object.
Closest only
Allows
you
to
specify
that
your
cache
wants
only
CLOSEST_PARENT_MISS replies from parent caches. This allows your
cache to then request the object from the parent cache closest to the
origin server.
No digest?
Chooses whether this neighbor cache should send cache digests. No
NetDB exchange When using ICP, it is possible for Squid to keep a
database of network information about the neighbor caches, including
availability and RTT, or Round Trip Time, information. This usually
allows Squid to choose more wisely which caches to make requests to
when multiple caches have the requested object.
59
SQUID and Webmin
No delay?
Prevents accesses to this neighbor cache from affecting delay pools.
Delay pools, discussed in more detail later, are a means by which
Squid can regulate bandwidth usage. If a neighbor cache is on the
local network, and bandwidth usage between the caches does not need
to be restricted, then this option can be used.
Login to proxy
Select this if you need to send authentication information when
challenged by the neighbor cache. On local networks, this type of
security is unlikely to be necessary.
Multicast responder
Allows Squid to know where to accept multicast ICP replies. Because
multicast is fed on a single IP to many caches, Squid must have some
way of determining which caches to listen to and what options apply to
that particular cache. Selecting Yes here configures Squid to listen for
multicast replies from the IP of this neighbor cache.
Query host for domains, Don’t query for domains
These two options are the only options on this page to configure a
directive other than cache_peer in Squid. In this case it sets the
cache_peer_domain option. This allows you to configure whether
requests for certain domains can be queried via ICP and which should
not. It is often used to configure caches not to query other caches for
content within the local domain. Another common usage, such as in
60
SQUID and Webmin
the national web hierarchies discussed above, is to define which web
cache is used for requests destined for different TLDs. So, for example,
if one has a low cost satellite link to the U. S. backbone from another
country that is preferred for web traffic over the much more expensive
land line, one can configure the satellite-connected cache as the cache
to query for all .com, .edu, .org, net, .us, and .gov addresses.
Cache Selection Options
This
section
provides
configuration
options
for
general
ICP
configuration (Figure 6-3). These options affect all of the other
neighbor caches that you define.
Figure 6-3: Global ICP options
Directly fetch URLs containing
Allows you to configure a match list of items to always fetch directly
rather than query a neighbor cache. The default here is cgi-bin ? and
should continue to be included unless you know what you’re doing.
This helps prevent wasting intercache bandwidth on lots of requests
that are usually never considered cacheable, and so will never return
hits
from
your
neighbor
caches.
This
option
sets
the
hierarchy_stoplist directive.
61
SQUID and Webmin
ICP query timeout
The time in milliseconds that Squid will wait before timing out ICP
requests. The default allows Squid to calculate an optimum value
based on average RTT of the neighbor caches. Usually, it is wise to
leave this unchanged. However, for reference, the default value in the
distant past was 2000, or 2 seconds. This option edits the icp_ query_
timeout directive.
Multicast ICP timeout
Timeout in milliseconds for multicast probes, which are sent out to
discover the number of active multicast peers listening on a given
multicast address. This configures the mcast_icp_query_timeout
directive and defaults to 2000 ms, or 2 seconds.
Dead peer timeout
Controls how long Squid waits to declare a peer cache dead. If there
are no ICP replies received in this amount of time, Squid will declare
the peer dead and will not expect to receive any further ICP replies.
However, it continues to send ICP queries for the peer and will mark it
active again on receipt of a reply. This timeout also affects when Squid
expects to receive ICP replies from peers. If more than this number of
seconds has passed since the last ICP reply was received, Squid will
not expect to receive an ICP reply on the next query. Thus, if your
time between requests is greater than this timeout, your cache will
send more requests DIRECT rather than through the neighbor caches.
62
SQUID and Webmin
Memory Usage
This page provides access to most of the options available for
configuring the way Squid uses memory and disks (Figure 6-4). Most
values on this page can remain unchanged, except in very high load or
low resource environments, where tuning can make a measurable
difference in how well Squid performs.
Gambar memory usage
Figure 6-4: Memory and disk usage
Memory usage limit
The limit on how much memory Squid will use for some parts of its
core data. Note that this does not restrict or limit Squid’s total process
size. What it does do is set aside a portion of RAM for use in storing intransit and hot objects, as well as negative cached objects. Generally,
the default value of 8MB is suitable for most situations, though it is
safe to lower it to 4 or 2MB in extremely low load situations. It can
also be raised significantly on high-memory systems to increase
63
SQUID and Webmin
performance by a small margin. Keep in mind that large cache
directories increase the memory usage of Squid by a large amount,
and even a machine with a lot of memory can run out of memory and
go into swap if cache memory and disk size are not appropriately
balanced. This option edits the cache_mem directive. See the section on
cache directories for more complete discussion of balancing memory
and storage.
Maximum cached object size
The size of the largest object that Squid will attempt to cache. Objects
larger than this will never be written to disk for later use. Refers to the
maximum_object_size directive. IP address cache size, IP cache highwater mark, IP address low-water mark The size of the cache used for
IP addresses and the high and low water marks for the cache,
respectively. This option configures the ipcache_size, ipcache_high,
and ipcache_low directives, which default to 1024 entries, 95%, and
90%.
6.7 Logging
Squid provides a number of logs that can be used when debugging
problems and when measuring the effectiveness and identifying users
and the sites they visit (Figure 6-5). Because Squid can be used to
“snoop” on user’s browsing habits, one should carefully consider
privacy laws in your region and, more importantly, be considerate to
your users. That being said, logs can be very valuable tools in ensuring
that your users get the best service possible from your cache.
64
SQUID and Webmin
Figure 6-5: Logging configuration
Cache metadata file
Filename used in each store directory to store the Web cache
metadata, which is a sort of index for the Web cache object store. This
is not a human readable log, and it is strongly recommended that you
leave it in its default location on each store directory, unless you really
know what you're doing. This option correlates to the cache_swap_log
directive.
Use HTTPD log format
Allows you to specify that Squid should write its access.log in HTTPD
common log file format, such as that used by Apache and many other
Web servers. This allows you to parse the log and generate reports
using a wider array of tools. However, this format does not provide
several types of information specific to caches, and is generally less
65
SQUID and Webmin
useful when tracking cache usage and solving problems. Because there
are several effective tools for parsing and generating reports from the
Squid standard access logs, it is usually preferable to leave this at its
default of being off. This option configures the emulate_httpd_log
directive. The Calamaris cache access log analyzer does not work if
this option is enabled.
Log full hostnames
Configures whether Squid will attempt to resolve the host name, so the
the fully qualified domain name can be logged. This can, in some
cases, increase latency of requests. This option correlates to the
log_fqdn directive.
Logging netmask
Defines what portion of the requesting client IP is logged in the
access.log. For privacy reasons it is often preferred to only log the
network or subnet IP of the client. For example, a netmask of
255.255.255.0 will log the first three octets of the IP, and fill the last
octet with a zero. This option configures the client_netmask directive.
66
SQUID and Webmin
6.8 Cache Options
The Cache Options page provides access to some important parts of
the Squid configuration file. This is where the cache directories are
configured as well as several timeouts and object size options (Figure
6-6).
Figure 6-6: Configuring Squids Cache Directories
The directive is cache_dir while the options are the type of filesystem,
the path to the cache directory, the size allotted to Squid, the number
of top level directories, and finally the number of second level
directories. In the example, I've chosen the filesystem type ufs, which
is a name for all standard UNIX filesystems. This type includes the
standard Linux ext2 filesystem as well. Other possibilities for this
option include aufs and diskd.
The next field is simply the space, in megabytes, of the disk that you
want to allow Squid to use. Finally, the directory fields define the upper
and lower level directories for Squid to use
67
SQUID and Webmin
6.9 Access Control
There are three types of option for configuring ICP access control.
These three types of definition are separated in the Webmin panel into
three sections. The first is labeled Access control lists, which lists
existing ACLs and provides a simple interface for generating and
editing lists of match criteria (Figure 6-7). The second is labeled Proxy
restrictions and lists the current restrictions in place and the ACLs they
effect. Finally, the ICP restrictions section lists the existing access rules
regarding ICP messages from other Web caches.
Figure 6-7: Access Control Lists
68
SQUID and Webmin
Access Control Lists
The first field in the table represents the name of the ACL, which is
simply an assigned name, that can be just about anything the user
chooses. The second field is the type of the ACL, which can be one of a
number of choices that indicates to Squid what part of a request
should be matched against for this ACL. The possible types include the
requesting clients address, the Web server address or host name, a
regular expression matching the URL, and many more. The final field is
the actual string to match. Depending on what the ACL type is, this
may be an IP address, a series of IP addresses, a URL, a host name,
etc.
Edit an ACL
To edit an existing ACL, simply click on the highlighted name. You will
then be presented with a screen containing all relevant information
about the ACL. Depending on the type of the ACL, you will be shown
different data entry fields. The operation of each type is very similar,
so for this example, you'll step through editing of the localhost ACL.
Clicking the localhost button presents the page that's shown in Figure
6-8
Figure 6-8: Edit an ACL
69
SQUID and Webmin
The title of the table is Client Address ACL which means the ACL is of
the Client Address type, and tells Squid to compare the incoming IP
address with the IP address in the ACL. It is possible to select an IP
based on the originating IP or the destination IP. The netmask can also
be used to indicate whether the ACL matches a whole network of
addresses, or only a single IP. It is possible to include a number of
addresses, or ranges of addresses in these fields. Finally, the Failure
URL is the address to send clients to if they have been denied access
due to matching this particular ACL. Note that the ACL by itself does
nothing, there must also be a proxy restriction or ICP restriction rule
that uses the ACL for Squid to use the ACL.
Creating new ACL
Creating a new ACL is equally simple (Figure 6-9). From the ACL page,
in the Access control lists section, select the type of ACL you'd like to
create. Then click Create new ACL. From there, as shown, you can
enter any number of ACLs for the list.
Figure 6-9: Creating an ACL
70
SQUID and Webmin
Available ACL Types
Browser Regexp
A regular expression that matches the client’s browser type based on
the user agent header. This allows for ACL's operating based on the
browser type in use, for example, using this ACL type, one could
create an ACL for Netscape users and another for Internet Explorer
users. This could then be used to redirect Netscape users to a
Navigator enhanced page, and IE users to an Explorer enhanced page.
Probably not the wisest use of an administrators time, but does
indicate the unmatched flexibility of Squid. This ACL type correlates to
the browser ACL type.
Client IP Address
The IP address of the requesting client, or the clients IP address. This
option refers to the src ACL in the Squid configuration file. An IP
address and netmask are expected. Address ranges are also accepted.
Client Hostname
Matches against the client domain name. This option correlates to the
srcdomain ACL, and can be either a single domain name, or a list or
domain names, or the path to a file that contains a list of domain
names. If a path to a file, it must be surrounded parentheses. This ACL
type can increase the latency, and decrease throughput significantly on
a loaded cache, as it must perform an address-to-name lookup for
each request, so it is usually preferable to use the Client IP Address
type.
71
SQUID and Webmin
Client Hostname Regexp
Matches against the client domain name. This option correlates to the
srcdom_regex ACL, and can be either a single domain name, or a list
of domain names, or a path to a file that contains a list of domain
names. If a path to a file, it must be surrounded parentheses
Date and Time
This type is just what it sounds like, providing a means to create ACLs
that are active during certain times of the day or certain days of the
week. This feature is often used to block some types of content or
some sections of the Internet during business or class hours. Many
companies block pornography, entertainment, sports, and other clearly
non-work related sites during business hours, but then unblock them
after hours. This might improve workplace efficiency in some situations
(or it might just offend the employees). This ACL type allows you to
enter days of the week and a time range, or select all hours of the
selected days. This ACL type is the same as the time ACL type
directive.
Ethernet Address
The ethernet or MAC address of the requesting client. This option only
works for clients on the same local subnet, and only for certain
platforms. Linux, Solaris, and some BSD variants are the supported
operating systems for this type of ACL. This ACL can provide a
somewhat secure method of access control, because MAC addresses
are usually harder to spoof than IP addresses, and you can guarantee
that your clients are on the local network (otherwise no ARP resolution
can take place).
72
SQUID and Webmin
External Auth
This ACL type calls an external authenticator process to decide whether
the request will be allowed. Note that authentication cannot work on a
transparent proxy or HTTP accelerator. The HTTP protocol does not
provide for two authentication stages (one local and one on remote
Web sites). So in order to use an authenticator, your proxy must
operate as a traditional proxy, where a client will respond appropriately
to a proxy authentication request as well as external Web server
authentication requests. This correlates to the proxy_auth directive.
External Auth Regex
As above, this ACL calls an external authenticator process, but allows
regex pattern or case insensitive matches. This option correlates to the
proxy_auth_regex directive.
Proxy IP Address
The local IP address on which the client connection exists. This allows
ACLs to be constructed that only match one physical network, if
multiple interfaces are present on the proxy, among other things. This
option configures the myip directive.
Request Method
This ACL type matches on the HTTP method in the request headers.
This includes the methods GET, PUT, etc. This corresponds to the
method ACL type directive.
73
SQUID and Webmin
URL Path Regex
This ACL matches on the URL path minus any protocol, port, and host
name
information.
It
does
not
include,
for
example,
the
"http://www.swelltech.com" portion of a request, leaving only the
actual path to the object. This option correlates to the urlpath_regex
directive.
URL Port
This ACL matches on the destination port for the request, and
configures the port ACL directive.
URL Protocol
This ACL matches on the protocol of the request, such as FTP, HTTP,
ICP, etc.
URL Regexp
Matches using a regular expression on the complete URL. This ACL can
be used to provide access control based on parts of the URL or a case
insensitive match of the URL, and much more. This option is equivalent
to the url_regex ACL type directive.
Web Server Address
This ACL matches based on the destination Web server's IP address.
Squid a single IP, a network IP with netmask, as well as a range of
addresses
in
the
form
"192.168.1.1-192.168.1.25".
This
option
correlates to the dst ACL type directive.
Web Server Hostname
This ACL matches on the host name of the destination Web server.
74
SQUID and Webmin
Web Server Regexp
Matches using a regular expression on the host name of the
destination Web server.
6.10
Administrative Options
Administrative Options provides access to several of the behind the
scenes options of Squid. This page allows you to configure a diverse
set of options, including the user ID and group ID of the Squid process,
cache hierarchy announce settings, and the authentication realm
(Figure 6-10)
Figure 6-10: Administrative Options
Run as Unix user and group
The user name and group name Squid will operate as. Squid is
designed to start as root but very soon after drop to the user/group
specified here. This allows you to restrict, for security reasons, the
permissions that Squid will have when operating. By default, Squid will
operate as either nobody user and the nogroup group, or in the case of
some Squids installed from RPM as squid user and group. These
75
SQUID and Webmin
options
correlate
to
the
cache_effective_user
and
cache_effective_group directives.
Proxy authentication realm
The
realm
that
will
be
reported
to
clients
when
performing
authentication. This option usually defaults to Squid proxy-caching web
server, and correlates to the proxy_auth_realm directive. This name
will likely appear in the browser pop-up window when the client is
asked for authentication information.
Cache manager email address
The email address of the administrator of this cache. This option
corresponds to the cache_mgr directive and defaults to either
webmaster or root on RPM based systems. This address will be added
to any error pages that are displayed to clients.
Visible hostname
The host name that Squid will advertise itself on. This affects the host
name that Squid uses when serving error messages. This option may
need to be configured in cache clusters if you receive IP-Forwarding
errors. This option configures the visible_hostname.
Unique hostname
Configures the unique_hostname directive, and sets a unique host
name for Squid to report in cache clusters in order to allow detection of
forwarding loops. Use this if you have multiple machines in a cluster
with the same Visible Hostname.
Cache announce host, port and file
The host address and port that Squid will use to announce its
availability to participate in a cache hierarchy. The cache announce file
is simply a file containing a message to be sent with announcements.
76
SQUID and Webmin
These options correspond to the announce_host, announce_port, and
announce_file directives.
Announcement period
Configures the announce_period directive, and refers to the frequency
at which Squid will send announcement messages to the announce
host.
Most of the content in Chapter 6 is taken from Unix System Administration with
Webmin by Joe Cooper (2002) available online at
http://www.swelltech.com/support/webminguide/
77
7
Chapter
7. Analyzer
7.1 Structure of log file
In Fedora, the Squid log files are stored in the /var/log/squid directory
by default. It makes 3 log files which are:

Access log

Cache log

Store log
Throughout this section, each log attribute will be discussed including it
content as well as how these logs might help admin debugging
potential problems.
Access log
Location : /var/log/squid/access.log
Description

It contains entries of each time the cache has been hit or missed
when a client requests HTTP content.
78
Analyzer

The identity of the host making the request (IP address) and the
content they are requesting.

It also provides the expected time when content is being used
from cache and when the remote server must be accessed to
obtain the content.

It contains the http transactions made by the users.
Format
Option 1 : This option will be used if the emulate http daemon log is
off.
Native format (emulate_httpd_log off)
Timestamp Elapsed Client Action/Code Size Method URI Ident Hierarchy/From Content
Option 2 : This option will be used if the emulate http daemon log is
on.
Common format (emulate_httpd_log on)
Client Ident - [Timestamp1] "Method URI" Type Size
With:
Timestamp
The time when the request is completed (socket closed). The format is
"Unix time" (seconds since Jan 1, 1970) with millisecond resolution.
Timestamp1
When the request is completed
(Day/Month/CenturyYear:Hour:Minute:Second GMT-Offset)
Elapsed
The elapsed time of the request, in milliseconds. This is the time
between the accept() and close() of the client socket.
79
Analyzer
Client
The IP address of the connecting client, or the FQDN if the 'log_fqdn'
option is enabled in the config file.
Action
The Action describes how the request was treated locally (hit, miss,
etc).
Code
The HTTP reply code taken from the first line of the HTTP reply header.
For ICP requests this is always "000." If the reply code was not given,
it will be logged as "555."
Size
For TCP requests, the amount of data written to the client. For UDP
requests, the size of the request. (in bytes)
Method
The HTTP request method (GET, POST, etc), or ICP_QUERY for ICP
requests.
URI
The requested URI.
Ident
The result of the RFC931/ident lookup of the client username. If
RFC931/ident lookup is disabled (default: `ident_lookup off'), it is
logged as - .
Hierarchy
A description of how and where the requested object was fetched.
80
Analyzer
From
Hostname of the machine where we got the object
Content
Content-type of the Object (from the HTTP reply header).
The example of access.log file.
Figure 7-1 Access.log
From Figure 7-1, we know that the native format has been used. Here,
we try to understand each format fields over the contents of access.log
file. By taking the first line, we found the result as in Table 7-1
Format
Value
Timestamp
1173680297.727
Elapsed
450
Client
10.0.5.10
Action
TCP_MISS
Code
302
Size
786
Method
GET
URI
http://www.google.com/search?
Ident
–
Hierarchy
DIRECT
From
64.233.189.104
Content
text/html
Table 7-1 The format and its value
81
Analyzer
There are some elaborations on:
Timestamp

The timestamp represents in UNIX time with a millisecond
resolution. However, it can be converted into more readable form
by using this short Perl script:
#!
/usr/bin/perl -p
s/^\d+\.\d+/localtime
$&/e;
Action

The TCP_ codes (Table 7-2) refer to requests on the HTTP port
(usually 3128). Meanwhile the UDP_ codes refer to requests on
the ICP port (usually 3130)
Codes
Explanation
TCP_HIT
A valid copy of the requested object
was in the cache
TCP_MISS
The requested object was not in the
cache
TCP_REFRESH_HIT
The requested object was cached but
STALE. The IMS query for the object
resulted in "304 not modified"
TCP_REF_FAIL_HIT
STALE. The IMS query failed and the
stale object was delivered
TCP_REFRESH_MISS
STALE. The IMS query returned the new
content
82
Analyzer
TCP_CLIENT_REFRESH_MISS The client issued a "no-cache" pragma,
or
some
analogous
cache
control
command along with the request. Thus,
the cache has to re-fetch the object
TCP_IMS_HIT
The client issued an IMS request for an
object which was in the cache and fresh
TCP_SWAPFAIL_MISS
The object was believed to be in the
cache, but could not be accessed
TCP_NEGATIVE_HIT
Request for a negatively cached object,
e.g. "404 not found", for which the
cache
believes
inaccessible.
to
Also
know
refer
that
to
it
is
the
explainations for negative_ttl in your
squid.conf file
TCP_SWAPFAIL_MISS
The object was believed to be in the
cache, but could not be accessed
TCP_MEM_HIT
was in the cache and it was in memory,
thus avoiding disk accesses
TCP_DENIED
Access was denied for this request
TCP_OFFLINE_HIT
The requested object was retrieved
from the cache during offline mode. The
offline mode never validates any object,
see offline_mode in squid.conf file.
UDP_HIT
83
Analyzer
was in the cache
UDP_MISS
The requested object is not in this
cache
UDP_DENIED
Access was denied for this request
UDP_INVALID
An invalid request was received
UDP_MISS_NOFETCH
During "-Y" startup, or during frequent
failures, a cache in hit only mode will
return either UDP_HIT or this code.
Neighbours will thus only fetch hits
NONE
Seen
with
errors
and
cachemgr
requests
Table 7-2 TCP codes and Explanation
Code

These codes are taken from RFC 2616 and verified for Squid.
Squid-2 uses almost all codes except 307 (Temporary Redirect),
416 (Request Range Not Satisfiable) and 417 (Expectation
Failed)
Code
84
Explanation
000
Used mostly with UDP traffic
100
Continue
101
Switching Protocols
102
Processing
200
OK
Analyzer
201
Created
202
Accepted
203
Non-Authoritative Information
204
No Content
205
Reset Content
206
Partial Content
207
Multi Status
300
Multiple Choices
301
Moved Permanently
302
Moved Temporarily
303
See Other
304
Not Modified
305
Use Proxy
[307
Temporary Redirect]
400
Bad Request
401
Unauthorized
402
Payment Required
403
Forbidden
404
Not Found
405
Method Not Allowed
406
Not Acceptable
407
Proxy Authentication Required
408
Request Timeout
409
Conflict
410
Gone
411
Length Required
412
Precondition Failed
413
Request Entity Too Large
414
Request URI Too Large
415
Unsupported Media Type
85
Analyzer
[416
Request Range Not Satisfiable]
[417
Expectation Failed]
*424
Locked
*424
Failed Dependency
*433
Unprocessable Entity
500
Internal Server Error
501
Not Implemented
502
Bad Gateway
503
Service Unavailable
504
Gateway Timeout
505
HTTP Version Not Supported
*507
600
Insufficient Storage
Squid header parsing error
Method

Squid recognizes several request methods as defined in RFC
2616. Newer versions of Squid (2.2.STABLE5 and above) also
recognize RFC 2518 ``HTTP Extensions for Distributed Authoring
-- WEBDAV'' extensions (Table 7-3).
method
defined
cachabil. meaning
GET
HTTP/0.9
possibly
object
retrieval
and
simple
searches
HEAD
HTTP/1.0
possibly
POST
HTTP/1.0
CC
metadata retrieval
or submit data (to a program)
Exp.
PUT
HTTP/1.1
never
upload data (e.g. to a file)
DELETE
HTTP/1.1
never
remove resource (e.g. file)
TRACE
HTTP/1.1
never
appl. layer trace of request route
86
Analyzer
OPTIONS
HTTP/1.1
never
CONNECT
HTTP/1.1r3 never
request available comm. options
tunnel SSL connection
ICP_QUERY Squid
never
used for ICP based exchanges
PURGE
Squid
never
remove object from cache.
PROPFIND
rfc2518
?
retrieve properties of an object
PROPATCH
rfc2518
?
change properties of an object
MKCOL
rfc2518
never
create a new collection
COPY
rfc2518
never
create a duplicate of src in dst
MOVE
rfc2518
never
atomically move src to dst
LOCK
rfc2518
never
Lock
an
object
against
modifications
UNLOCK
rfc2518
never
unlock an object
Table 7-3 List of Methods
87
Analyzer
Hierarchy
The following hierarchy codes are used in Squid-2 (Table 7-4):
Codes
Explanation
NONE
For TCP HIT, TCP failures, cachemgr
requests and all UDP requests, there is no
hierarchy information.
DIRECT
The object was fetched from the origin
server.
SIBLING_HIT
The object was fetched from a sibling
cache which replied with UDP_HIT.
PARENT_HIT
The object was requested from a parent
cache which replied with UDP_HIT.
DEFAULT_PARENT
No ICP queries were sent. This parent was
chosen because it was marked ``default''
in the config file.
SINGLE_PARENT
The object was requested from the only
parent appropriate for the given URL.
FIRST_UP_PARENT
The object was fetched from the first
parent in the list of parents.
NO_PARENT_DIRECT
The object was fetched from the origin
server, because no parents existed for the
given URL.
FIRST_PARENT_MISS
The object was fetched from the parent
with the fastest (possibly weighted) round
88
Analyzer
trip time.
CLOSEST_PARENT_MISS
This
parent
was
chosen,
because
it
included the lowest RTT measurement to
the origin server. See also the closestsonly peer configuration option.
CLOSEST_PARENT
The parent selection was based on our
own RTT measurements.
CLOSEST_DIRECT
Our own RTT measurements returned a
shorter time than any parent.
NO_DIRECT_FAIL
The
object
could
not
be
requested
because of a firewall configuration, see
also never_direct and related material,
and no parents were available.
SOURCE_FASTEST
The origin site was chosen, because the
source ping arrived fastest.
ROUNDROBIN_PARENT
No ICP replies were received from any
parent. The parent was chosen, because
it was marked for round robin in the
config file and had the lowest usage
count.
CACHE_DIGEST_HIT
The peer was chosen, because the cache
digest predicted a hit. This option was
later replaced in order to distinguish
between parents and siblings.
CD_PARENT_HIT
The parent was chosen, because the
89
Analyzer
cache digest predicted a hit.
CD_SIBLING_HIT
The sibling was chosen, because the
cache digest predicted a hit.
NO_CACHE_DIGEST_DIR
This output seems to be unused?
ECT
CARP
The peer was selected by CARP.
ANY_PARENT
part of src/peer_select.c:hier_strings[].
INVALID CODE
part of src/peer_select.c:hier_strings[].
Table 7-4 Hierarchy Codes in Squid-2
Cache log
Location : /var/log/squid/cache.log
Description

It contains various messages such as information about Squid
configuration, warnings about possible performance problems
and serious errors.

Error and debugging messages of particular squid modules
Format
[Timestamp1]| Message
With
Timestamp1
When the event occurred (Year/Month/Day Hour:Minute:Second)
90
Analyzer
Message
Errors
Description of the event
ERR_READ_TIMEOUT
The
remote
site
or
network
is
unreachable - may be down.
ERR_LIFETIME_EXP
The remote site or network may be
too slow or down.
ERR_NO_CLIENTS_BIG_OBJ
All
clients
went
away
before
transmission completed and the object
is too big to cache.
ERR_READ_ERROR
The remote site or network may be
down.
ERR_CLIENT_ABORT
Client
dropped
connection
before
transmission completed. Squid fetches
the Object according to its settings for
`quick_abort'.
ERR_CONNECT_FAIL
The remote site or server may be
down.
ERR_INVALID_REQ
Invalid HTTP request
ERR_UNSUP_REQ
Unsupported request
ERR_INVALID_URL
Invalid URL syntax
ERR_NO_FDS
Out of file descriptors
ERR_DNS_FAIL
DNS name lookup failure
91
Analyzer
ERR_NOT_IMPLEMENTED
Protocol Not Supported
ERR_CANNOT_FETCH
The requested URL can not currently
be retrieved.
ERR_NO_RELAY
There is no WAIS relay host defined
for this cache.
ERR_DISK_IO
The system disk is out of space or
failing.
ERR_ZERO_SIZE_ OBJECT
The
remote
server
closed
the
connection before sending any data.
ERR_FTP_DISABLED
This
cache
is
configured
to
NOT
retrieve FTP objects.
ERR_PROXY_DENIED
Access
Denied.
The
user
must
authenticate himself before accessing
this cache.
Table 7-5 List of Error Messages
92
Analyzer
The example of cache.log file (Figure 7-2).
Figure 7-2 Cache.log
Store log
Location : /var/log/squid/store.log
Description

It contains the information and status of [not] stored objects
Format
Timestamp Tag Code Date LM Expire Content Expect/Length Methods Key
With:
Timestamp
The time entry was logged. (Millisecond resolution since 00:00:00 UTC,
January 1, 1970)
Tag
SWAPIN (swapped into memory from disk), SWAPOUT (saved to disk)
or RELEASE (removed from cache)
Code
The HTTP replies code when available. For ICP requests this is always
"0". If the reply code was not given, it will be logged as "555."
93
Analyzer
The following three fields are timestamps parsed from the HTTP reply
headers. All are expressed in Unix time (i.e.(seconds since 00:00:00
UTC, January 1, 1970). A missing header is represented with -2 and an
unparsable header is represented as -1.
Date
The time captures from the HTTP Date reply header. If the Date
header is missing or invalid, the time of the request will be used
instead.
LM
The value of the HTTP Last-Modified: reply header.
Expires
The value of the HTTP Expires: reply header.
Content
The HTTP Content-Type reply header.
Expect
The value of the HTTP Content-Length reply header. The Zero value
will be returned if the Content-Length was missing.
/Length
The number of bytes of content actually read. If the Expect is nonzero, and not equal to the Length, the object will be released from the
cache.
Method
The request method (GET, POST, etc).
94
Analyzer
Key
The cache key. Often this is simply the URL. Cache objects which never
become public will have cache keys that include a unique integer
sequence number, the request method, and then the URL.
( /[post|put|head|connect]/URI )
The example of store.log file (Figure 7-3).
Figure 7-3 Store.log
Based on Figure 7-3, we try to understand each format fields over the
contents of store.log file. By taking the second line, we found that
(Table 7-6):
Format
Value
Timestamp
1173680297.727
Tag
Release
Code
-1
Date
FFFFFFFF
LM
7832CBDDD1604B89D0F75A2437F37AD7
Expire
302
Content
1173680306 -1 -1 text/html
Expect
-1
/Length
/278
Methode
GET
Key
http://www.google.com/search?
Table 7-6 Format in Store.log
95
Analyzer
7.2 Methods
Log Analysis Using Grep Command
The log files also can be analysed using Linux or UNIX command such
as grep. It is used to filter the required information from any log files.
By using a terminal, follow the following commands in order to start
analysis the related log file.
For example:
# cat /var/log/squid/access.log | grep www.google.com
By referring Figure 7-4, the output shows the result of grep command
for the access.log file. The same technique can be applied for cache.log
and store.log files.
Figure 7-4 Analysis the Access.log using Grep command
Log Analysis Using Sarg-2.2.3.1
Basically, the preferred log file for analysis is the access.log file in the
native format. We choose to use Squid Analysis Report Generator
(Sarg) as a tool. It is used to analyze the users pattern concerning the
Internet surfing. It generates reports in html including many fields
such as users, IP addresses, bytes, sites and times.
This tool can be downloaded from:
http://linux.softpedia.com/get/Internet/Log-Analyzers/sarg-102.shtml
96
Analyzer
7.3 Setup Sarg-2.2.3.1
Step:
Download software named Sarg-2.2.3.1.tar.gz for Linux and Unix
environment.
Make a new directory called installer located in the root path.
# mkdir /installer
Copy the downloaded file into the installer directory
# copy sarg-2.2.3.1.tar.gz
/installer
Then, go into the directory and extract file Sarg-2.2.3.1.tar.gz using
the following command.
# tar –zxvf sarg-2.2.3.1.tar.gz
After successfully extracted, go into sar-2.2.3.1 directory and start
configure it. Follow these command:
#
#
#
#
cd /installer/sar-2.2.3.1
./configure
make
make install
NOTE: Make sure the Squid already started before run the following
script.
Go into sarg-2.2.3.1 directory, run the sarg script.
# ./sarg
The generated result will be kept at /var/www/html/squid-reports. It is
recommended to view using GUI enviroment.
97
Analyzer
7.4 Report Management Using Webmin
For managing the report, we choose to use Webmin which is a webbased interface for system administration for Unix. In our case, it helps
admin to set some information such as the location of log source and
report destination, the format of generated report, the size of report
and also the schedule of automatic report to be generated.
Step:
1. Make sure the webmin is already setup in the server. Then, open
the browser and type http://127.0.0.1:10000/ to find the webmin.
After that, login the webmin.
Figure 7-5 Login
2. Choose Server tab, and then click on Squid Analysis Report
Generator. There are four (4) modules being offer such as Log
Source and Report Destination, Report Option, Report Style and
Scheduled Report Generation.
98
Analyzer
Figure 7-6 Sarg Main Modules in Webmin
3. Click on Log Source and Report Destination icon. In this module,
admin allows to set the source of log file and also define the
destination of generated report. For report maintenance, it also
allows admin to set the number of report to keep in certain location
and acknowledgement can be sent to admin’s e-mail.
Note: Please check the sarg.conf file which is located in
/usr/local/sarg/sarg.conf to ensure the correct path for locating
the source of log files.
Figure 7-7 Setting on Source and Destination Report
99
Analyzer
After setting the changes, click on Save button.
4. Click on Report Option icon. In this module, admin can manages the
pattern of generated report including data ordering, size of data
displayed, data format and log file rotation.
There are several types of report can be generates depending on
the implementation of access control list (ACL) that has been set
before.
For log file rotation, it becomes important to ensure enough disk
space to handle log file storage especially when it involves the long
term evaluations. This can covers more in Scheduled Report
Generation.
100
Analyzer
Figure 7-8 Setting on Report Content and Generation Option
101
Analyzer
5. Click on Report Style icon. Here, it allows admin to make the
generated report looks more interesting in terms of language, title
and other common style setting.
Figure 7-9 Setting on HTML Report Style and Colour Option
6. Click on Scheduled Report Generation icon. In this module, admin
allows to define the frequency of generated report by enabling the
selected or default schedule stated.
Regarding to rotate feature in Squid, it is recommended to apply
simple schedule. During a time of some idleness, the log files are
safely transferred to the report destination in one burst. Before
transport, the log files can be compressed during off-peak time. On
the destination, the log file is concatenated into one file. Therefore
one file for selected hour is the yield. However, it is depends on
company’s requirement on how to generate report.
102
Analyzer
Figure 7-10 Setting on Scheduled Reporting Options
7. After setting some information in Scheduled Report Generation, the
following statement will be displayed on the main page.
Figure 7-11 Generate Report Setting
103
Analyzer
There are some considerations to be taken:
1.
Should never delete access.log, store.log, cache.log while
Squid is running. There is no recovery file.
2.
In squid.conf file, the following statements can be applied if
admin wants to disable certain log file. For example:
To disable access.log:
cache_access_log /dev/null
To disable store.log:
cache_store_log none
To disable cache.log:
cache_log /dev/null
However, the cache.log is not suitable to be disabled because it
has
file
messages.
104
contains
many
important
status
and
debugging
Analyzer
7.5 Log Analysis and Statistic
After running the Sarg analyser, the reports will generated for
access.log. This can be found in /var/www/html/squid-reports.
Figure 7-12 Collection of Squid Report for Access.log
From Figure 7-12, throughout this example we found that there are
three (3) reports generated. Basically, the latest version has no
number at the end of the filename. Each time the access log file being
analysed, the filename will renamed and an incremental number will be
placed automatically at the end of the file.
For example 2007Mar22-2007Mar22.2 was the first report had been
generated compared to 2007Mar22-2007Mar22 which indicated as the
latest version report.
105
Analyzer
Based on (Figure 7-13), the index.html file shows the list of reports
that have been generated by Sarg. To get more detail information for a
specific report, we need to click on the selected file name.
Figure 7-13 Summary of Squid reports
For example, a folder named as 2007Mar22-2007Mar22 has been
selected and opened. From (Figure 7-14), there are several standard
files which can be found in all Squid reports. Briefly, there are five (5)
html reports show statistical information regarding to index, denied,
download, siteuser and topsites. Besides, the folder also presents
collection of report for specific user by their IP addressess.
106
Analyzer
Figure 7-14 Contents of 2007Mar22-2007Mar22 as example
The following figure will show the html reports:
1. Index html
2.
Figure 7-15 Index html
107
Analyzer
3. Denied html
Figure 7-16 Denied html
4. Download html
Figure 7-17 Download html
108
Analyzer
5. Sites and Users
Figure 7-18 Siteuser html
109
Analyzer
6. Top 100 Sites
Figure 7-19 Topsites html
If we click on specific IP address, we will view all information as in
Figure 7-20
Figure 7-20 Reports generated for specific user (IP Address)
110