•

» sed

Penetration Testing : NMAP.XML to TAB

Posted on October 6, 2010 by The Shell Shakespear in Main | 1 Comment

Following up on my last NMAP post, processing port scan data in a meaningful manner is essential to network penetration testing. For those who wish to skip the SQL stage and get quick results, the following one-liner will use xmlstarlet to parse a NMAP XML file:

cat nmap.xml | xmlstarlet sel -T -t -m "//state[@state='open']" -m ../../.. -v address/@addr -m hostnames/hostname -i @name -o ' (' -v @name -o ')' -b -b -b -o "	" -m .. -v @portid -o '/' -v @protocol -o "	" -m service -v @name -i "@tunnel='ssl'" -o 's' -b -o "	" -v @product -o ' ' -v @version -v @extrainfo -b -n -

Into the following tab delimited format:

IP (HOST) \t PORT/PROTOCOL \t SERVICE \t EXTRAINFO

This command sorts ports properly, but does not properly order the hosts. To do this, pipe the above command to the following:

sed 's_^\([^\t ]*\)\( ([^)]*)\)\?\t\([^\t ]*\)_\1.\3\2_' | sort -n -t. -k1,1 -k2,2 -k3,3 -k4,4 -k5,5 | sed 's_^\(\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}\)\.\([^ \t]*\)\( ([^)]*)\)\?_\1\4\t\3_'

This command converts lines that look like IP (HOST) \t PORT to IP.PORT (HOST), sorts it, and then converts them back to IP (HOST) \t PORT. From there, it is simply a matter of grepping for your favorite service. For example, if you wanted to focus on web penetration testing, all you would have to do is pipe the above to:

grep -i -e http

To get a list of services relevant to your testing. Happy Hacking.

Sed, Grep and Awk

Posted on July 8, 2009 by The Shell Shakespear 1 Comment

Sed, Grep and Awk are true *nix tools, known for their awkward names and equally awkward syntax. They represent the most immediate access to Regular Expressions (REs) which are themselves worthy of knowledge. Even their attempted replacement, Perl, is also known producing useful yet unreadable code. Though I acknowledge their awkward natures, their usefulness cannot be ignored, and learning how to use each will aid you in your ascension to line processing supremacy. Each is best used in the following manner:

  • Grep: Matching
  • Sed: Replacing and Line Manipulation
  • Awk: Advanced Line Processing
# Insert 'Beginning' at the start of a file, and 'Ending' at the end
sed "1s/\(.*\)/Beginning\n\1/;\$a\\Ending"
 
#Escape shell metacharacters active within double quotes
sed 's/\([\\/\\`\\"$\\\\^.\\+\\{\\}]\)/\\\1/g'
 
#Replace all literal newlines with their representation '\n'
sed -e :a -e '$!N;s/\n/\\n/;ta'
 
# Filter out URL parameters
sed 's_=[^&]*\(&\|$\)_=\1_g'
 
# Get rid of regular expressions in a variable
sed 's:[]\[\^\$\.\*\/]:\\\\&:g'`
 
#Replace last comma(,) in each line with 'and'
sed 's#\(.*\),\([^,]*\)#\1 and\2#'
 
#Match phone numbers with area code in any given format and output in format: (nnn) nnn-nnnn
# SED DOES NOT RESPECT the shorthand character classes \c\s\S\d\D\w\W
sed -e 's#[^0-9]*\([0-9]\{3\}\)[^0-9]*\([0-9]\{3\}\)[^0-9]*\([0-9]\{4\}\)#(\1) \2-\3#'
grep -o '(\?[0-9]\{3\})\? \?[0-9]\{3\}-\?[0-9]\{4\}'
 
# Match CVE Numbers
grep -o 'CVE-[0-9]\{4\}-[0-9]\{1,5\}'
 
# Match input fields with a hidden input type in an HTML file
grep -io ']*hidden[^>]*>' hidden.csv | sed 's#""#"#g;s#value="[^"]*"#value=""#g' | sort -u | less
 
#Parse IIS Logs for a certain IP ADDRESS (127.0.0.1)
grep 127.0.0.1 *.log | grep -v -e ".gif" -e ".jpg" -e ".ico" -e ".css" -e ".pdf" -e "404" | cut -d' ' -f2,4,5,6,10 | awk '{printf "%s %-04s http://site.com%s?%s  Ref:(%s)\n",$1,$2,$3,$4,$5}' | tr -d '-' | sed 's/Ref:()//g' | sed 's/\? //g' | awk '{printf "%s %-04s %-70s\t%s\n",$1,$2,$3,$4}'
 
#Find all links in a file
egrep -IRo '(((http(s)?|ftp|telnet|news|gopher)://|mailto:)[^\(\)<\"'\''[:space:]]+)'
 
#Pretty printing fields with awk
awk -F':' '{printf "%-16s %-16s\n",$1,$2}'
 
# 'uniq' the file using only the first field
awk '!x[$1]++'
 
# uniq 3rd field in a file
awk '{ if (! third_col[$3]) print $0;  third_col[$3]++; }'
 
# Lists directories where the tree contains one or more files:
find ./ -type f | awk -F/ '{$NF=""} d[$0]++==0' OFS=/
 
# How many lines in a file that do not start with # and are not empty would fit in a tweet (140 characters)?
grep -v '^#\|^$' shell1liners.sh | awk '{if (length<141) {print "Tweet("length"): " $0;}}'
grep -v '^#\|^$' shell1liners.sh | awk '{if (length>140) {print "No Tweet("length"): " $0;}}'