In an earlier post, Scott did this in a simple shell script. I was curious if I could do it in Python since I am learning Python.

Below is my first attempt. Watch for upcoming updated version.

import sys
 
if not sys.argv[1:]:
   mac = raw_input ("Enter MAC address\n")
   print "Mac is %s\n"; % (mac)
   #sys.exit(0)
else:
   mac = sys.argv[1]
   print "Mac is %s\n" % (mac)
 
##I'm checking to see if someone is giving me in the right format
maclen=len(mac)
if maclen == 12:
    ## converting to list, since strings are immuatable
    maclist=list(mac)
    ## defining positions on list
    positions=2,5,8,11,14
 
    for i in positions:
        maclist.insert(i,':')
    ##joining list at the ','
    newmac=",".join(maclist)
    ## stripping ',' out and converting to lowercase.  
    final12mac=newmac.replace(',','').lower()
 
    print final12mac
 
##assuming that 17 characters will either have - or : in MAC address    
elif maclen == 17:
    ##replacing - with : and converting to lowercase
    mac17final=mac.replace('-',':').lower()
    print mac17final
else:
    ## I'm telling them that if it isn't 12 or 17 characters, it has X chars
    print "There are only %s characters" % (maclen)

TL;DR Some Apache log processing one liners to get IP addresses from the access and error logs that I have found handy.

There may be some log processing questions asked during the course of an interview, so I am going to concentrate on a couple that will get the IP addresses from the apache log files. If you do have log processing quesitons I sincerely hope you get to play at a command line, as off the top of the head can be difficult, unless you’re good at visualizing commands.

Q: How would you get the IP address from the access logs?
A: This one is fairly straight forward, I would:

cat access.log | awk '{print $1}' | uniq

This will output (if you choose not to use uniq you will see multiple of the same ip, I’ll leave to the discretion of the reader):

127.0.0.1
127.0.0.2
127.0.0.3

The breakdown is as follows:

1. read through the contents of the file (cat access.log)
2. pipe output to awk to print the first field ($1)
3. pipe output to only show unique data (uniq)

Another question might be to parse out the IP address from the error.log, while this is a bit more difficult, it is fairly straight forward using readily available system tools.

Q: Can you please extract the IP address from the error.log?
A: Here is my solution (Thanks Malcolm!)

cat error.log | awk '{ if($7 == "[client") {print $8} }' | sed -e 's/]$//g' | uniq

The breakdown is as follows, this one is a bit more complex so I will walk through each step of it:

First part: cat error.log - read through the log file.
 
[Sun Aug 14 13:27:14 2011] [error] [client 96.126.120.254] Invalid method in request \x80e\x01\x03\x01
[Sun Aug 14 13:27:14 2011] [error] [client 96.126.120.254] Invalid method in request \x80e\x01\x03\x01
[Sun Aug 14 13:27:14 2011] [error] [client 96.126.120.254] Invalid method in request \x80e\x01\x03\x01
[Sun Aug 14 13:27:14 2011] [error] [client 96.126.120.254] Invalid method in request \x80e\x01\x03\x01
[Mon Aug 15 15:16:21 2011] [error] [client 194.72.238.62] Invalid method in request \x16\x03\x01
[Mon Aug 15 15:50:27 2011] [notice] caught SIGWINCH, shutting down gracefully
[Mon Aug 15 15:50:37 2011] [notice] mod_python: Creating 8 session mutexes based on 75 max processes and 0 max threads.
[Mon Aug 15 15:50:37 2011] [notice] mod_python: using mutex_directory /tmp 
PHP Warning:  Module 'gd' already loaded in Unknown on line 0
PHP Warning:  Module 'mysql' already loaded in Unknown on line 0
PHP Warning:  Module 'mysqli' already loaded in Unknown on line 0
[Mon Aug 15 15:50:38 2011] [warn] mod_wsgi: Compiled for Python/2.5.1.
[Mon Aug 15 15:50:38 2011] [warn] mod_wsgi: Runtime using Python/2.5.2.
 
2. Second part: cat error.log | awk '{ if ($7 == "[client") {print $8} }' - if the 7th field matches client (this seems to be pretty standard though ymmv) print out eighth field (which should be the IP address, also notice the trailing "]" character).
 
96.126.120.254]
96.126.120.254]
96.126.120.254]
96.126.120.254]
194.72.238.62]
 
3. Third Part: cat error.log | awk '{ if ($7 == "[client") {print $8} }' | sed -e 's/]$//g' - use sed to remove the trailing "]" character.
 
96.126.120.254
96.126.120.254
96.126.120.254
96.126.120.254
194.72.238.62
 
4. Fourth Part: cat error.log | awk '{ if ($7 == "[client") {print $8} }' | sed -e 's/]$//g' | uniq - lets output unique ips and not all of them (multiple matches).
 
96.126.120.254
194.72.238.62

Addendum:
If you would like you can then add another awk on the end (or pipe to any other command you feel like) for instance. The following pipes the output to the host command:

[Edit: for host might want to specify the -W <time> flag just in case, it could try forever on some unless specified]
cat error.log | awk '{ if ($7 == "[client") {print $8} }' | sed -e 's/]$//g' | uniq | awk '{ print | "host -W 3 " $1 }'

I hope this helps, share and enjoy! Thanks for reading!

-Scott.

Disclaimer: I make no claims to the viability of the code/script/commands and make no guarantees that it will work on your system, use at your own risk.

TL;DR
I like interview questions, this is completely subjective, name 5 two letter linux commands and what they do.

Everyone once in a while I’ll go looking for good technical interview questions to challenge myself, to see how far I’ve grown, and it is for me a good way to ground myself and figure out how far I need to grow. I tend to be of the mindset that it’s good to keep the questions somewhat open ended, I think a lot of what shows a persons aptitude is their ability to reason out answers to difficult questions, not whether or not they remember what the theoretical throughput of 8Gig Fibre Channel (about 1600MBps if I remember correctly, but I digress). So what I am going to do is go out find / make up what I think are good interview questions (or ones that just strike my fancy, and why yes this is completely subjective thanks for pointing that out) and answer them from my own experience. I can’t say that my solutions will be the *right* way to do it, but hopefully give the correct results, but I make no promises.

So for the first round I am going to echo a question that I got as an interview question and really liked, and maybe take it a bit further:

Q: Can you name five two letter Linux commands?
A: Yes I can (and did).

Now this is a good question for those starting out to gauge how far they have gone into the system admin side of things, stuff that you can’t necessarily (or as quickly) do from just the gui. So the thing that I thought of to make this more interesting is:

Q: Can you name five two letter linux commands, and what they do? Give an example of each two letter command you give.

I personally think this is interesting because not only does it cause some of the more senior people to have to think (muscle memory is a helluva drug), but it also will show a good basic understanding of what is going with the commands that are being used. In all honesty I don’t remember all of them, and I don’t remember ever single command line switch for them, that is why man pages exist.

Share and enjoy!

-Scott.

So part of what I am doing at my current job is helping one of the Unix admins with DNS and DHCP. For the DHCP portion to setup the reservations we need the MAC addy in a certain format, of which the people requesting never seem to get consistently right, so I wrote a small shell script (it’s still rough) that will normalize the MAC address for what we need (colon separated, alpha characters lower case).

[Edit: the below only works on FreeBSD and Linux, for Solaris swap the awk '{print tolower($0)}' with tr '[:upper:]‘ ‘[:lower:]‘ <- or you could just do that for all of them as well).

#!/usr/bin/env sh
 
mac="$1"
 
#check to see if input is empty
if [ ! -n "$mac" ]
    then
        echo "Please enter a MAC address."
        exit
else 
    # echo the input, strip out dot(.), strip out colon(:), strip out dash(-)
    # add colon(:) every two chars, remove last colon(:)
    # awk to lowercase characters [EDIT: updated the sed for . seps and escape]
    echo $mac | sed -e 's/\.//g' -e 's/\://g' -e 's/\-//g' -e 's/../&:/g' -e 's/:$//g' \
| awk '{print tolower($0)}'
fi

Some of the standard formats we see are:
1A:BC:35:57:33:08
1A-BC-35-57:33:08
1ABC35573308
1abc35573308
1a.bc.35.57.33.08
1A.BC.35.57.33.08

The script will take all of these and make them look like: 1a:bc:35:57:33:08.

Here is the breakdown for those that are curious:
1. if [ ! -n "$mac" ] checks to make sure first input variable is not empty.
2. echo $mac outputs the first argument (does not do any input validation)
3. The sed is in five parts:

[Edited to escape the sequences, just in case they should have special meaning]
    a. 's/\.//g' strips out the period should it be in that format
    b. 's/\://g' strips out the colons should they be there (to avoid extra colons)
    c. 's/\-//g' strips out the dashes should they be in that format
    d. 's/../&:/g' every two characters append a colon
    e. 's/:$//g' remove the last trailing colon

4. awk ‘{print tolower($0)}’ make all of the upper case alpha chars lower case to match our format needs.
4a. tr ‘[:upper:]‘ ‘[:lower:]‘ for Solaris (yes I know tr ‘[A-Z]‘ ‘[a-z]‘ will work as well, but this is easier to read IMHO)

This will work on a file full of MAC addresses, just throw it in a for loop (this is if you name the file macnorm.sh):

for i in `cat filename`; do ./macnorm.sh $i; done

Again it is important to realize that there is no checking of the input, so if you did it over /etc/passwd it would add a colon (:) every two characters. Hope this helps.

-Scott.

Quick, Dirty, Simple. Set the default printer with Powershell. Needed this as a part of a logon script to set default printer for certain users on a terminal server.

## Get the Printer with WMI
$printer = Get-WmiObject -Query "Select * from Win32_Printer Where Name = 'PDFCreator'"
 
## Set printer as default printer
$printer.SetDefaultPrinter()

Being busy with sysadminy type stuff, I haven’t been able to practice my Python as much as I would like, though I am eating my own words and practicing. I was reading somewhere about adding IP’s to a iptables block list, and thought about creating a quick and dirty parser to go through a log file (or any other file for that matter) and grep out any ip addy’s. The caveat here is that it was quick and dirty and only works on ipv4, and only outputs to STDOUT, but that can all be fix0red real quickstyle (which I think I’ll do as I carry on and make it better). I am still learning how to do things the pythonic way, as opposed to just getting things working, so please be gentle, the style comes with time. *Should have mentioned this before, but I am on CentOS 5.6, and the python version is 2.4.3 (system) so that is why there is no with statement.

#!/usr/bin/env python
 
#import the necessary modules
import re #for regular expressions - to match ip's
import sys #for parsing command line opts
 
# I need to probably make this more pythonic but am working on that...
# if file is specified on command line, parse, else ask for file
if sys.argv[1:]:
    print "File: %s" % (sys.argv[1])
    logfile = sys.argv[1]
else:
    logfile = raw_input("Please enter a file to parse, e.g /var/log/secure: ")
 
try:
    # open the file
    file = open(logfile, "r")
    # create an empty list
    ips = []
    # read through the file
    for text in file.readlines():
       #strip off the \n
        text = text.rstrip()
       #this is probably not the best way, but it works for now
        regex = re.findall(r'(?:[\d]{1,3})\.(?:[\d]{1,3})\.(?:[\d]{1,3})\.(?:[\d]{1,3})$', text)
        # if the regex is not empty and is not already in ips list append
        if regex is not None and regex not in ips:
            ips.append(regex)
 
 
    #loop through the list
    for ip in ips:
        #I know there is argument as to whether the string join method is pythonic
        addy = "".join(ip)
        if addy is not '':
            print "IP: %s" % (addy)
    #cleanup and close file
    file.close()
#catch any standard error (we can add more later)
except IOError, (errno, strerror):
    print "I/O Error(%s) : %s" % (errno, strerror)

Output looks like:

File: /var/log/secure
IP: 68.169.40.177
IP: 14.63.253.14
IP: 113.105.159.162
IP: 125.208.5.78
IP: 89.187.142.141
IP: 212.116.152.36

Actually, I’ll fix the output so it writes it to a file for keeping, mind you this is just a quick addition so it won’t double check to see if the ip address is already there, so it will write dupes…also learned that one should not update vim tab options in mid-coding, as python doesn’t like mixing tabs and spaces…if you need to check “python -m tabnanny file.py” will tell you where you went wrong :), then in vi you can type :retab and it should fix it…

original:
#loop through the list
    for ip in ips:
        #I know there is argument as to whether the string join method is pythonic
        addy = "".join(ip)
        if addy is not '':
            print "IP: %s" % (addy)
    #cleanup and close file
    file.close()
 
new (for output to file):
    for ip in ips:
        outfile = open("/tmp/blocked_ips", "a")
        addy = "".join(ip)
        if addy is not '':
           print "IP: %s" % (addy)
           outfile.write(addy)
           outfile.write("\n")
    file.close()
    outfile.close()

It seems to work, though it’s probably not the best way to do it, and is most definitely not pythonic, though again I’ll plead that I am still learning the style portion. So now that we have the ip’s to block here is a small shell script to add them to your iptables configs, if you so choose, you might need to update for your configuration.

BLOCKDB="/tmp/blocked_ips"
IPS=$(grep -Ev "^#" $BLOCKDB) #so you can use # to add comments to and it will grep them out
for i in $IPS
do
    iptables -A INPUT -s $i -j DROP
    iptables -A OUTPUT -d $i -j DROP
done

As noted many times, these are both quick and dirty, neither do any real error checking or sanity checking, but it took me 20 mins, so use at your own risk, I make no claims as to the usability or completeness of said code. As a sidenote if you want to loop through all the log files or a bunch of files, you can do the following “for i in `ls -1 /var/log/`; do ./get_ips.py /var/log/$i; done” – Two things, 1. I named the file get_ips.py, 2. This is extremely hackish.

As mentioned by someone, I fixed the outer try statement:

#!/usr/bin/env python
#fixed the import, just red PEP 8
import sys
import re
 
try:
    if sys.argv[1:]:
        print "File: %s" % (sys.argv[1])
        logfile = sys.argv[1]
    else:
        logfile = raw_input("Please enter a log file to parse, e.g /var/log/secure: ")
    try:
        file = open(logfile, "r")
        ips = []
        for text in file.readlines():
           text = text.rstrip()
           regex = re.findall(r'(?:[\d]{1,3})\.(?:[\d]{1,3})\.(?:[\d]{1,3})\.(?:[\d]{1,3})$',text)
           if regex is not None and regex not in ips:
               ips.append(regex)
 
        for ip in ips:
           outfile = open("/tmp/blocked_ips_test", "a")
           addy = "".join(ip)
           if addy is not '':
              print "IP: %s" % (addy)
              outfile.write(addy)
              outfile.write("\n")
    finally:
        file.close()
        outfile.close()
except IOError, (errno, strerror):
        print "I/O Error(%s) : %s" % (errno, strerror)

Ok, fixed for the last time (hopefully) imabonehead refactored, moved the finally clause inside, and moved the except outside.

This is a subject that is near and dear to most of us, as most of the systems admins that I know have come up through the ranks. I believe as we get more experienced, and hence more responsibility we as systems admins tend to go with the mindset* of “Screw it, I can do this faster myself than show it to, and explain it to someone else” because of project constraints, deadlines** and the myriad of other reasons we collectively don’t have the time. But I think that this can be detrimental to those that are more Jr., who are in need of good mentorship (yes I understand some of the difficulties here), but think that if possible we should take a little extra time and effort to help grow them***. I honestly believe that I missed out a lot on some basic stuff when I was learning because the person that I worked with, who was a very smart systems engineer was under such time constraints that most often he would just do it not giving me the chance to learn. As a sidenote I think it is also important to let them know the scope of their responsibilities, and that in the beginning it’s ok to ask questions.

Things I wish I would have known as a Junior****:

1. Ask questions – ask! ask! ask! But here is the flipside, I don’t want to have to answer the same question 19 times, get a notebook, take notes and go google if you need to, I am not into hand holding or completely supporting (e.g. doing their job for them) attitudes.

2. Knowledge – as more experienced admins we do not expect you to know everything as you are just building the foundation of knowledge you need to succeed in this field, I promise you there will be epiphanous moments, and the layers of abstraction will become clearer as time passes.

3. Expect mistakes – you will make mistakes, plain and simple. Do not be afraid of them, embrace them, learn from them and carry on. It’s not about whether you’re going to fall down, you will, it’s about how you pick yourself up and carry on which matters more.

4. Build up your toolsets – I don’t mean physically (though those help as well), I mean learn as much as you can, don’t limit yourself, read and learn on your own.

5. Listen – in the beginning you don’t know everything, listen to the more senior people, especially the good ones. They will not always be right (dammit you mean we don’t know it all?), but they will generally have been working with the environment long enough to know the common pitfalls.

6. Stand up for yourself – as a caveat to #5, even the most senior or those that have been doing it for years are not always right, they are human and make mistakes.

7. Stay away from poisonous people – this rule applies in life in general as well, but there are a lot of bitter, jaded systems administrators who will spew venom, you’ll be able to spot them a mile away. Try to keep your distance if you can (though I know it’s not always possible).

This is just my opinion, take of it what you will, thank you for reading.

* YMMV
** As Douglas Adams said “I love deadlines, I like the whooshing sound they make as they fly by.”
*** There are of course those that will never learn, and probably shouldn’t be in this business to begin with, but I digress…
**** This applies to programmers, network wonks, security freaks, etc…just s/systems admin/tech job/g

I am a Systems Administrator (mainly Linux and Storage, some Windows) by trade, and there are a couple of rules that I have learned in the time I have been doing this job:

1. “Just because you can, doesn’t mean you should.” There are a ton of places where this is applicable, especially since most of us are responsible for a lot of sensitive data, and have the keys to the proverbial kingdom.

2. “You can do it right, or you can do it again.” <–This is mine, though probably not original. I have seen, in my career a lot of quickly done, rough, overly complex, and not well designed solutions. I do believe that a lot of times it is about simplicity, though there are times when a complex solution is necessary. Now don’t mistake me, I am just as apt to hack something together to get it working, but when it comes to actually designing for a production environment, it is my not so humble opinion that you should take a little extra time upfront and design to the best of your ability.

I use one of these escape sequences all the time as I tend to leave my connections going when I leave from work (and then have to VPN back in) so the ssh session hangs. ~. will disconnect a hung ssh session. Here are some of the other common ssh escape sequences:

~?
Supported escape sequences:
  ~.  - terminate connection (and any multiplexed sessions)
  ~B  - send a BREAK to the remote system
  ~C  - open a command line
  ~R  - Request rekey (SSH protocol 2 only)
  ~^Z - suspend ssh
  ~#  - list forwarded connections
  ~&  - background ssh (when waiting for connections to terminate)
  ~?  - this message
  ~~  - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)
 
Here is what they look like from my command line
(note these have to be done after a newline):
 
~. 
Output: Connection to somehost.somedomain.com closed.
 
~C
Output:
ssh> ?
Commands:
      -L[bind_address:]port:host:hostport    Request local forward
      -R[bind_address:]port:host:hostport    Request remote forward
      -D[bind_address:]port                  Request dynamic forward
      -KR[bind_address:]port                 Cancel remote forward
~^Z
 
Output:
[Wed Sep 22]{snyce@somehost in ~}$ ~^Z [suspend ssh]
 
[1]+  Stopped                 ssh -X somehost.somedomain.com
 
(you can resume by typing fg)
 
~#
 
The following connections are open:
  #0 client-session (t4 r0 i0/0 o0/0 fd 5/6 cfd -1)

Hope this helps.

An acquaintance of mine asked a really good interview question, so I thought I would share. For the linux wonks, describe three ways to translate from lower case to uppercase on the command line (or vice versa). I have to admit I only thought of one (though the other should have been obvious), so I came up with tr, missed sed (which should have been obvious), and last but not least, one I would have never thought of, dd. *Actually thought of another way using awk so I’ll add that as well. Below are the examples:

Created a file called test with the following contents: test test test (to go from upper to lower on the tr flip the options).

Tr:
tr: cat test | tr [:lower:] [:upper:]
Output: cat test | tr [:lower:] [:upper:]
TEST TEST TEST
 
tr: cat test | tr "a-z" "A-Z"
Output: cat test | tr "a-z" "A-Z"
TEST TEST TEST
 
Sed:
*With sed replace U with L if you want to lower (there are other ways as well):
 
sed: cat test | sed 's/\(.*\)/\U\1/'
Output: cat test | sed 's/\(.*\)/\U\1/'
TEST TEST TEST
 
dd: (bonus points if you know where dd got its name**)
 
dd if=test conv=upper
Output: dd if=test conv=ucase
TEST TEST TEST
0+1 records in
0+1 records out
15 bytes (15 B) copied, 0.000155 s, 96.8 kB/s
 
Awk:
cat test | awk '{print toupper($_)}' (use tolower($_) if you want to lcase)
Output: cat test | awk '{print toupper($_)}'
TEST TEST TEST

** From the history files, the name comes from “convert and copy” but some wily compiler designers had already taken the “cc” command.