I originally posted this on my blog http://www.barrymorrison.com/2011/09/04/learning-python-resources/ but thought it’d be a good fit over hear as well. These are simply links to Python resources I’ve collected recently in an effort to tech myself Python. Hopefully others can find it equally valuable.

http://code.google.com/edu/languages/google-python-class/index.html

http://i.imgur.com/YQafj.png

http://code.google.com/appengine/docs/python/gettingstarted/

http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-00-introduction-to-computer-science-and-programming-fall-2008/

http://diveintopython.org/toc/index.html

http://codingbat.com/

http://www.lightbird.net/py-by-example/

http://rgruet.free.fr/PQR26/PQR2.6.html

http://www.udemy.com/learn-python-the-hard-way/

http://learnpythonthehardway.org/

http://www.swaroopch.com/notes/Python

http://www.pythonchallenge.com/

http://www.youtube.com/user/thenewboston#p/search/0/4Mf0h3HphEA

I was working on a script that parsed log files. The file names were log.mmddyy (log.080811). This was a little bit of a learning experience for me (new to shell scripting). I was working on a Solaris 10 box, but I thought I’d do all my dev/testing on Ubuntu. Big mistake. Linux was running bash/gnu tools. Solaris was not. After I figured out how to do it on Ubuntu, I had to figure out how to do it on Solaris (thanks Scott!), then I did it in Powershell (had done something similar already for another script) and since I’m teaching myself Python, I wanted to do it in Python.

Are they the “right” way to do them? They work! Probably not a “right” way to do them, as there is always more than one way to do something

Shell

#!/bin/bash
 
os=`uname`
 
if [ $os == 'Linux' ];
    then
    yesterday="$(date -d 'yesterday' +%m%d%y)"
    echo $yesterday
elif [ $os == 'SunOS' ];
    then
    yesterday=`TZ=GMT+24 date +%m%d%y`
    echo $yesterday
fi

Gnu Date

#!/usr/bin/bash
 
## Get Yesterday
yesterday="$(date -d 'yesterday' +%m%d%y)"

Powershell

$yesterday = (get-date -format mmddyy).adddays(-1)
$yesterday.tostring('MMddyy')

Python

 
import datetime
today = datetime.datetime.today()
one_day = datetime.timedelta(days=1)
yesterday = today - one_day
yesterday.strftime('%m%d%y')

In an earlier post, Scott did this in a simple shell script. I was curious if I could do it in Python since I am learning Python.

Below is my first attempt. Watch for upcoming updated version.

import sys
 
if not sys.argv[1:]:
   mac = raw_input ("Enter MAC address\n")
   print "Mac is %s\n"; % (mac)
   #sys.exit(0)
else:
   mac = sys.argv[1]
   print "Mac is %s\n" % (mac)
 
##I'm checking to see if someone is giving me in the right format
maclen=len(mac)
if maclen == 12:
    ## converting to list, since strings are immuatable
    maclist=list(mac)
    ## defining positions on list
    positions=2,5,8,11,14
 
    for i in positions:
        maclist.insert(i,':')
    ##joining list at the ','
    newmac=",".join(maclist)
    ## stripping ',' out and converting to lowercase.  
    final12mac=newmac.replace(',','').lower()
 
    print final12mac
 
##assuming that 17 characters will either have - or : in MAC address    
elif maclen == 17:
    ##replacing - with : and converting to lowercase
    mac17final=mac.replace('-',':').lower()
    print mac17final
else:
    ## I'm telling them that if it isn't 12 or 17 characters, it has X chars
    print "There are only %s characters" % (maclen)

TL;DR Some Apache log processing one liners to get IP addresses from the access and error logs that I have found handy.

There may be some log processing questions asked during the course of an interview, so I am going to concentrate on a couple that will get the IP addresses from the apache log files. If you do have log processing quesitons I sincerely hope you get to play at a command line, as off the top of the head can be difficult, unless you’re good at visualizing commands.

Q: How would you get the IP address from the access logs?
A: This one is fairly straight forward, I would:

cat access.log | awk '{print $1}' | uniq

This will output (if you choose not to use uniq you will see multiple of the same ip, I’ll leave to the discretion of the reader):

127.0.0.1
127.0.0.2
127.0.0.3

The breakdown is as follows:

1. read through the contents of the file (cat access.log)
2. pipe output to awk to print the first field ($1)
3. pipe output to only show unique data (uniq)

Another question might be to parse out the IP address from the error.log, while this is a bit more difficult, it is fairly straight forward using readily available system tools.

Q: Can you please extract the IP address from the error.log?
A: Here is my solution (Thanks Malcolm!)

cat error.log | awk '{ if($7 == "[client") {print $8} }' | sed -e 's/]$//g' | uniq

The breakdown is as follows, this one is a bit more complex so I will walk through each step of it:

First part: cat error.log - read through the log file.
 
[Sun Aug 14 13:27:14 2011] [error] [client 96.126.120.254] Invalid method in request \x80e\x01\x03\x01
[Sun Aug 14 13:27:14 2011] [error] [client 96.126.120.254] Invalid method in request \x80e\x01\x03\x01
[Sun Aug 14 13:27:14 2011] [error] [client 96.126.120.254] Invalid method in request \x80e\x01\x03\x01
[Sun Aug 14 13:27:14 2011] [error] [client 96.126.120.254] Invalid method in request \x80e\x01\x03\x01
[Mon Aug 15 15:16:21 2011] [error] [client 194.72.238.62] Invalid method in request \x16\x03\x01
[Mon Aug 15 15:50:27 2011] [notice] caught SIGWINCH, shutting down gracefully
[Mon Aug 15 15:50:37 2011] [notice] mod_python: Creating 8 session mutexes based on 75 max processes and 0 max threads.
[Mon Aug 15 15:50:37 2011] [notice] mod_python: using mutex_directory /tmp 
PHP Warning:  Module 'gd' already loaded in Unknown on line 0
PHP Warning:  Module 'mysql' already loaded in Unknown on line 0
PHP Warning:  Module 'mysqli' already loaded in Unknown on line 0
[Mon Aug 15 15:50:38 2011] [warn] mod_wsgi: Compiled for Python/2.5.1.
[Mon Aug 15 15:50:38 2011] [warn] mod_wsgi: Runtime using Python/2.5.2.
 
2. Second part: cat error.log | awk '{ if ($7 == "[client") {print $8} }' - if the 7th field matches client (this seems to be pretty standard though ymmv) print out eighth field (which should be the IP address, also notice the trailing "]" character).
 
96.126.120.254]
96.126.120.254]
96.126.120.254]
96.126.120.254]
194.72.238.62]
 
3. Third Part: cat error.log | awk '{ if ($7 == "[client") {print $8} }' | sed -e 's/]$//g' - use sed to remove the trailing "]" character.
 
96.126.120.254
96.126.120.254
96.126.120.254
96.126.120.254
194.72.238.62
 
4. Fourth Part: cat error.log | awk '{ if ($7 == "[client") {print $8} }' | sed -e 's/]$//g' | uniq - lets output unique ips and not all of them (multiple matches).
 
96.126.120.254
194.72.238.62

Addendum:
If you would like you can then add another awk on the end (or pipe to any other command you feel like) for instance. The following pipes the output to the host command:

[Edit: for host might want to specify the -W <time> flag just in case, it could try forever on some unless specified]
cat error.log | awk '{ if ($7 == "[client") {print $8} }' | sed -e 's/]$//g' | uniq | awk '{ print | "host -W 3 " $1 }'

I hope this helps, share and enjoy! Thanks for reading!

-Scott.

Disclaimer: I make no claims to the viability of the code/script/commands and make no guarantees that it will work on your system, use at your own risk.

TL;DR
I like interview questions, this is completely subjective, name 5 two letter linux commands and what they do.

Everyone once in a while I’ll go looking for good technical interview questions to challenge myself, to see how far I’ve grown, and it is for me a good way to ground myself and figure out how far I need to grow. I tend to be of the mindset that it’s good to keep the questions somewhat open ended, I think a lot of what shows a persons aptitude is their ability to reason out answers to difficult questions, not whether or not they remember what the theoretical throughput of 8Gig Fibre Channel (about 1600MBps if I remember correctly, but I digress). So what I am going to do is go out find / make up what I think are good interview questions (or ones that just strike my fancy, and why yes this is completely subjective thanks for pointing that out) and answer them from my own experience. I can’t say that my solutions will be the *right* way to do it, but hopefully give the correct results, but I make no promises.

So for the first round I am going to echo a question that I got as an interview question and really liked, and maybe take it a bit further:

Q: Can you name five two letter Linux commands?
A: Yes I can (and did).

Now this is a good question for those starting out to gauge how far they have gone into the system admin side of things, stuff that you can’t necessarily (or as quickly) do from just the gui. So the thing that I thought of to make this more interesting is:

Q: Can you name five two letter linux commands, and what they do? Give an example of each two letter command you give.

I personally think this is interesting because not only does it cause some of the more senior people to have to think (muscle memory is a helluva drug), but it also will show a good basic understanding of what is going with the commands that are being used. In all honesty I don’t remember all of them, and I don’t remember ever single command line switch for them, that is why man pages exist.

Share and enjoy!

-Scott.

So part of what I am doing at my current job is helping one of the Unix admins with DNS and DHCP. For the DHCP portion to setup the reservations we need the MAC addy in a certain format, of which the people requesting never seem to get consistently right, so I wrote a small shell script (it’s still rough) that will normalize the MAC address for what we need (colon separated, alpha characters lower case).

[Edit: the below only works on FreeBSD and Linux, for Solaris swap the awk '{print tolower($0)}' with tr '[:upper:]‘ ‘[:lower:]‘ <- or you could just do that for all of them as well).

#!/usr/bin/env sh
 
mac="$1"
 
#check to see if input is empty
if [ ! -n "$mac" ]
    then
        echo "Please enter a MAC address."
        exit
else 
    # echo the input, strip out dot(.), strip out colon(:), strip out dash(-)
    # add colon(:) every two chars, remove last colon(:)
    # awk to lowercase characters [EDIT: updated the sed for . seps and escape]
    echo $mac | sed -e 's/\.//g' -e 's/\://g' -e 's/\-//g' -e 's/../&:/g' -e 's/:$//g' \
| awk '{print tolower($0)}'
fi

Some of the standard formats we see are:
1A:BC:35:57:33:08
1A-BC-35-57:33:08
1ABC35573308
1abc35573308
1a.bc.35.57.33.08
1A.BC.35.57.33.08

The script will take all of these and make them look like: 1a:bc:35:57:33:08.

Here is the breakdown for those that are curious:
1. if [ ! -n "$mac" ] checks to make sure first input variable is not empty.
2. echo $mac outputs the first argument (does not do any input validation)
3. The sed is in five parts:

[Edited to escape the sequences, just in case they should have special meaning]
    a. 's/\.//g' strips out the period should it be in that format
    b. 's/\://g' strips out the colons should they be there (to avoid extra colons)
    c. 's/\-//g' strips out the dashes should they be in that format
    d. 's/../&:/g' every two characters append a colon
    e. 's/:$//g' remove the last trailing colon

4. awk ‘{print tolower($0)}’ make all of the upper case alpha chars lower case to match our format needs.
4a. tr ‘[:upper:]‘ ‘[:lower:]‘ for Solaris (yes I know tr ‘[A-Z]‘ ‘[a-z]‘ will work as well, but this is easier to read IMHO)

This will work on a file full of MAC addresses, just throw it in a for loop (this is if you name the file macnorm.sh):

for i in `cat filename`; do ./macnorm.sh $i; done

Again it is important to realize that there is no checking of the input, so if you did it over /etc/passwd it would add a colon (:) every two characters. Hope this helps.

-Scott.

Because someone I follow on twitter mentioned it (I had read this article before), I did the fizzbuzz in python (quite a while ago), though as someone pointed out not very pythonically, here’s my entry (non-pythonic python):

I’ll maybe work on making it more pythonic later.

#!/usr/bin/env python
 
i = 100
count = 0
while (count < i):
    if (count % 3 == 0 and count % 5 == 0):
        print "FizzBuzz: %d" % (count)
    elif (count % 5 == 0 and count &gt; 0):
        print "Buzz: %d" % (count)
    elif (count % 3 == 0):
        print "Fizz: %d" % (count)
    else:
        print "Count is %d" % (count)
    count += 1

Quick, Dirty, Simple. Set the default printer with Powershell. Needed this as a part of a logon script to set default printer for certain users on a terminal server.

## Get the Printer with WMI
$printer = Get-WmiObject -Query "Select * from Win32_Printer Where Name = 'PDFCreator'"
 
## Set printer as default printer
$printer.SetDefaultPrinter()

Extended version of Hal’s original “dirtoday” script via twitter https://twitter.com/#!/halr9000/status/88211937720156160

## NAME: Dirtoday.ps1
## AUTHOR: Barry Morrison
## LASTEDIT: 07/05/2011 16:18:33
<#
 .Synopsis
 Get's files from today's date. Will also return narrowed results based on keyword
 .Description
 Get's files from today's date. Will also return narrowed results based on keyword
 .Parameter Path
 Will run in current directory. Path argument will allow you to define a path not in current working directory
 .Parameter Keyword
 Keyword argument will allow you to define a keyword to search on to narrow results.
 .Example
 PS C:\scripts > .\dirtoday.ps1
 
 Directory: C:\scripts
 
 Mode LastWriteTime Length Name
 ---- ------------- ------ ----
 -a--- 7/5/2011 3:25 PM 114 cat
 -a--- 7/5/2011 4:11 PM 2252 dirtoday.ps1
 -a--- 7/5/2011 3:46 PM 848 dirtoday2.ps1
 -a--- 7/5/2011 2:55 PM 110 info.txt
 -a--- 7/5/2011 1:52 PM 37 test.foo
 
 .Example
 PS C:\scripts > .\dirtoday.ps1 -path "c:\Users\John Doe\Pictures"
 
 Directory: C:\Users\John Doe\Pictures
 
 Mode LastWriteTime Length Name
 ---- ------------- ------ ----
 -a--- 7/5/2011 11:05 AM 49888 weinerdog.jpg
 
 .Example
 PS C:\scripts > .\dirtoday.ps1 -keyword foo
 
 cat:2:test.foo:1:blah, blah, foo, ice cream,
 test.foo:1:blah, blah, foo, ice cream,
 
#>
 
param(
 [string]$Path = "",
 [string]$keyword = "";
 )
 
## Checking to see if the Keyword argument is passed -- If it is, search for $keyword
if ($keyword) {
   $files = dir -Path $path | Where-Object { $_.lastwritetime -ge (get-date).date } | Select-String $keyword
   ## If there are not results returned, tell me!
   if (!$files) {
     Write-Output "Nothing Here"
   }else{
     ## If Results exist, give them to me!
     $files
   }
 } else {
 ## If not keyword argument is passed, do a simple search
   $files = dir -Path $path | Where-Object { $_.lastwritetime -ge (get-date).date }
   ## Like above, checking to see if no results are found for the query
   if (!$files) {
     ## Tell me if nothing exists
     Write-Output "Nothing Here"
   }else{
     ## Write results if something exists
     $files
  }
}

I figure I should say who I am before I start slinging words out.  My name is Barry Morrison, and I work with Scott as a System’s Administrator.  I’ve been in IT since 2003, I’ve been in my current position since 2007.  My primary background is in Windows desktop/server administration, but my focus more so recently has been on storage (NetApp and Data Domain), backups and whatever else (that isn’t windows) I can get my hands on.  I’m very passionate about technology, combined with my short attention span, I like to play with everything.  I’ve recently taken a liking to Powershell (I hear the cringes now), and will now script everything I possibly can.  It has also helped me branch into other languages (Sh/Bash/Python — which I hope to do a lot more of).  So I’ll take it!

Why am I here?  Scott invited me, and we have a few ideas I think we’re both excited about regarding this particular blog space.  I think most of the code I write will be posted here, and less on my personal site going forward.  Seems more fitting here.  Thanks again Scott for the invite.  I hope I do your site justice!