Manipulating Cisco Firewall Syslogs Using Linux Command Line

Recently, I've been doing a lot of firewall log auditing for various reasons. My Cisco firewalls send their logs to a centralized syslog server and are Gzipped, so the file sizes don't get out of control, once an hour.

I'm doing my analysis on a Linux box (BT4 distro) with limited disk space and many of the commands I like to use aren't installed on the syslog server. With these obstacles in place, here is how I've been dealing with the Gigs of syslog messages to get what I really want.

The first step was simply to grab 24 hours worth of log messages and this was as simple as FTP'ing into the syslog server and performing an "mget" on the date I desire. Now I have a lot of gzipped syslog files waiting for me to play with.

root@bt# ls -la
total 2915188
drwxr-xr-x 2 root root 4096 2010-07-28 14:01 .
drwxr-xr-x 48 root root 4096 2010-07-22 16:18 ..
-rw-r--r-- 1 root root 62478364 2010-07-27 15:25 fwsm.log.20100726.0000.gz
-rw-r--r-- 1 root root 64624441 2010-07-27 15:25 fwsm.log.20100726.0100.gz
-rw-r--r-- 1 root root 58798851 2010-07-27 15:25 fwsm.log.20100726.0200.gz
-rw-r--r-- 1 root root 57448529 2010-07-27 15:25 fwsm.log.20100726.0300.gz
-rw-r--r-- 1 root root 57916715 2010-07-27 15:25 fwsm.log.20100726.0400.gz
-rw-r--r-- 1 root root 58860324 2010-07-27 15:26 fwsm.log.20100726.0500.gz
-rw-r--r-- 1 root root 68676596 2010-07-27 15:26 fwsm.log.20100726.0600.gz
-rw-r--r-- 1 root root 78945716 2010-07-27 15:26 fwsm.log.20100726.0700.gz
-rw-r--r-- 1 root root 112321501 2010-07-27 15:26 fwsm.log.20100726.0800.gz
-rw-r--r-- 1 root root 169440875 2010-07-27 15:27 fwsm.log.20100726.0900.gz
-rw-r--r-- 1 root root 209838484 2010-07-27 15:27 fwsm.log.20100726.1000.gz
-rw-r--r-- 1 root root 227651887 2010-07-27 15:27 fwsm.log.20100726.1100.gz
-rw-r--r-- 1 root root 233153646 2010-07-27 15:28 fwsm.log.20100726.1200.gz
-rw-r--r-- 1 root root 244517081 2010-07-27 15:29 fwsm.log.20100726.1300.gz
(/snip)

Since I don't necessarily want to uncompress all the files just to inspect the contents, I'm going to use the nifty "zcat" command which does the same thing as "cat" but on a zipped file. Here is what the syslog format looks like on a Cisco firewall with no field manipulation.

root@bt# zcat fwsm.log.20100726.0000.gz | more
Jul 25 23:00:00 firewall main %FWSM-6-302013: Built outbound TCP connection 146767145916591362 for inside:1.1.1.1/2359 (1.1.1.1/31921) to outside:66.166.52.133/80 (66.166.52.133/80)

As you can see here, it is simply an outbound web session initiated from one of my internal hosts (IP changed to protect the innocent) destined for a web server on Covad's IP block. In this particular adventure, I'm trending for any outbound telnet sessions leaving the boundary of my network. Obviously, telnet is not a preferred transport mechanism, and that multiplies when leaving the internal network and crossing the Internet. Telnet is not encrypted and any snooping on the network can reveal the contents being transferred including usernames, passwords, etc. The first thing I want to do is find a sample telnet session, although the format will be identical the the syslog message above. My syntax includes a "Built outbound" as I don't want to see all the deny messages...also don't forget the space character after the /23. If you forget to include that then you'll get a lot of port numbers which include "23" such as 2300, 23000, etc.

root@bt# zcat fwsm.log.20100726.0000.gz | grep "Built outbound TCP" | grep "/23 " |more
Jul 25 23:06:17 firewall.example.com main %FWSM-6-302013: Built outbound TCP connection 146569495816626096 for inside:1.1.1.1/1366 (1.1.1.1/1366) to outside:166.111.30.174/23 (166.111.30.174/23)
Jul 25 23:30:55 firewall.example.com main %FWSM-6-302015: Built outbound TCP connection 146634027700247405 for inside:1.1.1.2/35886 (1.1.1.1/25685) to outside:91.213.175.139/23 (91.213.175.139/23)

Great, that worked like a champ as we can see in the example above. At this point it should be worth noting that there are multiple ways to skin this cat. We can combine search terms using grep, or simply use the more powerful sed/awk combination. For the purpose of this example, I'm trying to keep things simple so many of the syntax examples can be shortened and sped up. Perhaps this can be an exercise for a later time...

With my sample telnet violator message format handy, I now am interested in finding any source address participating in a telnet session from within my network. We'll also want to know who they talk to but we'll get to that in a moment. Here is how I manipulated the data fields to accomplish what I wanted. I simply used the built in Linux "cut" command to clean up the syslog format and then followed that with the "sort" "uniq" combination. This left me with exactly what I wanted. Before moving on, I save all these addresses into an Excel document so I can come back to them later. This could also be done a little quicker by first writing your data into a seperate text file before manipulation but, as mentioned earlier, I'm limited on disk space (and will be creating a file in a minute) so this was a fine compromise.

root@bt# zcat fwsm.log* | grep "Built outbound TCP" | grep "/23 " | cut -f5 -d ":" | cut -f1 -d "/" | sort | uniq
1.1.1.1
1.1.1.2
1.1.1.3
1.1.1.4
(/snip)

Okay, for the next phase, I want a text file containing all syslog messages related to built telnet sessions. I will be using this text file when I examine destination addresses connected to from the list of source addresses in my Excel document. To do this is rather easy, and covered earlier...the only difference is we'll be redirecting the output into a file instead of to the screen buffer.

root@bt# zcat fwsm.log* | grep "Built outbound TCP" | grep "/23 " > telnet-list.txt
root@bt# ls -la | grep "telnet"
-rw-r--r-- 1 root root 8785 2010-07-29 12:32 telnet-list.txt

In order to find destination addresses for my source IPs, I write a little bash script which allows me to do this. I call the script "telnet-lookup.sh" and it looks like this.

root@bt# nano telnet-lookup.sh
#!/bin/bash
cat telnet-list.txt | grep $1 | grep "Built outbound TCP" | cut -f6 -d ":" | cut -f1 -d "/" | sort | uniq

root@bt# chmod 755 telnet-lookup.sh
root@bt# ./telnet-lookup.sh 1.1.1.1
74.22.17.99
98.136.48.48
147.55.32.107

Simply repeat this on all source IPs and you now have a working matrix you can use to continue investigating why these flows are occurring...