Thursday, November 29, 2012

Apache 2.2 Logging

It wasn't obvious to me which variables were available inside httpd.conf, and the log_config documentation didn't include examples, so I wrote up my own reference.

Here's how I think the logging section of the default http.conf should look. Near line 500, replace the ErrorLog, LogLevel, LogFormat and CustomLog directives with the following.

vi /etc/httpd/conf/httpd.conf
################################################################

# Because we don't specify an ErrorLog directive within our <VirtualHost>
# containers, error messages for all virtual hosts are logged here.
#
ErrorLog logs/error_log

# Only record warn and above in the error_log
# Options: debug, info, notice, warn, error, crit, alert, emerg.
#
LogLevel warn

# Define formatting for use with CustomLog. 
# http://httpd.apache.org/docs/2.2/mod/mod_log_config.html
#
# A client sends a request to apache. Apache provides a response. The entire transaction
# is recorded in a single log line. Note that values logged are not necessarily available
# during the request phase and therefore may not be used for determining redirects.
#
# %t                [29/Nov/2012:10:35:08 -0500]  The time the request was received in default format
# %{%F %T}t         2012-11-29 11:43:31           The time the request was received local to apache in 24hour format
#
# %h                1.1.1.1                       The hostname of the client, or just the IP of the client when: HostnameLookups Off
# %a                1.1.1.1                       The IP of the client
#
# %p                80                            The port on which the request arrived
# %A                2.2.2.2                       The IP of this apache server
# %v                my.apache.com                 The hostname of this apache server as defined by: ServerName
# %V                wild.apache.com               The hostname of this apache server as it was requested by the client
#
# %l                -                             The ident logname from the client, or just a dash when: mod_ident not loaded or IdentityCheck Off
# %u                -                             The auth user from the client, or just a dash when not supplied
#
# %r                GET /page.html?x=1 HTTP/1.1   The first line of the request
# %m                GET                           The method from the request
# %U                /page.html                    The path from the request not including the query string
# %q                ?x=1                          The query string with ? when the query string exists, otherwise an empty string
# %H                HTTP/1.1                      The protocol from the request
# %f                /var/www/html/page.html       The computed file name to be served
# %{USERID}C        JonSmith                      The contents of the "USERID" cookie, or just a dash if no cookie by that name was included in the request
# %{Referer}i       http://you.com/site.html      The contents of the "Referer" request header after apache mods have updated the request, or just a dash if no header by that name exists in the request
# %{Content-Type}o  text/html                     The contents of the "Content-Type" response header, or just a dash if no header by that name exists in the response
#
# %s                200                           The status of the original request
# %>s               200                           The status of the last request after a chain of internal redirects
#
# Common Requst Headers:
#
# %{Referer}i          http://www.yourSite.com/page.html?x=1
# %{User-Agent}i       Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/17.0 Firefox/17.0
# %{Accept-Encoding}i  gzip, deflate
# %{Content-Type}i     application/x-www-form-urlencoded
# %{Content-Length}i   338
#
# Common Response Headers:
#
# %{Transfer-Encoding}o  chunked
# %{Content-Encoding}o   -
# %{Content-Type}o       text/html
# %{Content-Length}o     251
#
LogFormat "%{%F %T}t %a %V:%p %m %U%q %>s" access
LogFormat "%{%F %T}t %a %V:%p %m %U%q %>s \n Request Referer: %{Referer}i\n         User-Agent: %{User-Agent}i\n         Accept-Encoding: %{Accept-Encoding}i\n         Content-Type: %{Content-Type}i\n         Content-Length: %{Content-Length}i\n Response Transfer-Encoding: %{Transfer-Encoding}o\n          Content-Encoding %{Content-Encoding}o\n          Content-Type %{Content-Type}o\n          Content-Length %{Content-Length}o\n" headers

# Because we don't specify a CustomLog directive within our <VirtualHost>
# containers, every request/response pair will cause this loggger to fire.
# Change the format to modify the details logged.
#
CustomLog logs/access_log access

# Enable rewrites so we can redirect incoming requests.
#
RewriteEngine On

# Because we don't specify a RewriteLog directive within our <VirtualHost>
# containers, every request/response pair will cause this loggger to fire.
# These logs explain what the RewriteEngine is doing.
#
RewriteLog logs/rewrite_log

# Define the verbosity of the RewriteLog. Choose a value from 0 - 9.
# 0 being no logging at all and 9 logging all possible information.
# Currently levels 4 - 9 return the same results. 
#
RewriteLogLevel 4

################################################################

Note that the behaviour of %V depends on the ServerName and UseCanonicalName directives. This way you have access to both the hardcoded name and the name used in the request, which allows you to reference your server by its real name or by the dns alias by which it was requested

ServerName your.box.com
UseCanonicalName Off

I prefer for the http and https logs to go to the same file, so I comment out the existing log directives.

vi /etc/httpd/conf.d/ssl.conf
#ErrorLog logs/ssl_error_log
#TransferLog logs/ssl_access_log
#LogLevel warn

#CustomLog logs/ssl_request_log \
#          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

Now you can watch some traffic go by.

tail -f /etc/httpd/logs/access_log

You can also turn on logging of all request headers.

vi /etc/httpd/conf/httpd.conf
# uncomment this line
LoadModule log_forensic_module modules/mod_log_forensic.so

# add this at end of file, or inside a VirtualHost block if you're using that
ForensicLog logs/forensic_log

service httpd restart
tail -f /etc/httpd/logs/forensic_log
+7232:5086aa09:0|GET / HTTP/1.1|Host:box1.yourdomain.com|User-Agent:Mozilla/5.0 (Windows NT 5.1; rv%3a15.0) Gecko/20100101 Firefox/15.0.1|Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8|Accept-Language:en-us,en;q=0.5|Accept-Encoding:gzip, deflate|Connection:keep-alive
-7232:5086aa09:0
aws
{ "loggedin": false, "owner": false, "avatar": "", "render": "nothing", "trackingID": "UA-36983794-1", "description": "", "page": { "blogIds": [ 308 ] }, "domain": "holtstrom.com", "base": "\/michael", "url": "https:\/\/holtstrom.com\/michael\/", "frameworkFiles": "https:\/\/holtstrom.com\/michael\/_framework\/_files.4\/", "commonFiles": "https:\/\/holtstrom.com\/michael\/_common\/_files.3\/", "mediaFiles": "https:\/\/holtstrom.com\/michael\/media\/_files.3\/", "tmdbUrl": "http:\/\/www.themoviedb.org\/", "tmdbPoster": "http:\/\/image.tmdb.org\/t\/p\/w342" }