•

» bash

Converting Lots of PDFs to TXTs in Ubuntu/Debian

Posted on April 15, 2010 by David Bailey Leave a comment

For those of you who are struggling to find a way to convert PDF files into TXT files, here is a quick bash script. There are many alternatives out there, but none were reliable for me. You’ll need to have acroread and ghostscript installed for this to work.


#!/bin/bash
mkdir ps txt
FILES="*.pdf"
for f in $FILES
do
echo "Processing $f"
acroread -toPostScript $f ps/
g=`basename $f .pdf`
ps2txt ps/$g.ps > txt/$g.txt
done

You can also change the second to last line to read
ps2txt ps/$g.ps | grep -v "EXCLUDE" > txt/$g.txt
where EXCLUDE is a line that you want to exclude from each PDF. Please let me know if you have any problems.

enjoy,
db

Keeping Current with Skipfish

Posted on April 7, 2010 by The Shell Shakespear Leave a comment

You have followed our posts on how to install skipfish on your Ubuntu/Debian based machine, and read our testing with Mutillidae. With the rapid speed of skipfish releases however, your local copy may start to smell. Now you can automate the delivery of fresh fish to your doorstep. Copy the code below to update_skippy.sh in your skipfish directory, run chmod +x update_skippy.sh and then ./update_skippy.sh to ensure your security tool-bowl is kept well stocked.

#!/bin/bash
# Checks local version of skipfish and update if newer version exists
# Copyright (C) 2010 Paul Haas <phaas AT redspin DOT com>
# Licensed under the GNU Public License version 3.0 or greater

SKIP='https://code.google.com/p/skipfish/downloads/list'
TGZ='"[^"]*.tgz"'
LOCAL=$(head -n1 ChangeLog | grep -o '[0-9.]*')
REMOTE=$(wget -q "$SKIP" -O- | grep -e "version" | cut -d' ' -f4)
#REMOTE=$(curl -s "$SKIP" | grep -e "version" | cut -d' ' -f4) # Curl optional
LM=${LOCAL%.[0-9]*}
Lm=${LOCAL#[0-9]*.}
RM=${REMOTE%.[0-9]*}
Rm=${REMOTE#[0-9]*.}

if [[ "$LM" -lt "$RM" || "$LM" -eq "$RM" && "$Lm" -lt "$Rm" ]]
then
	echo "Updating Skipfish from $LOCAL to $REMOTE."
	wget -q "$SKIP" -O- | grep -o "$TGZ" | tr -d \" | wget -q -i- -O- | tar zxf - --strip=1
	#curl -s "$SKIP" | grep -o "$TGZ" | tr -d \" | xargs curl -s | tar zxf - --strip=1
	make clean
	make
else
	echo "Skipfish $LOCAL ($REMOTE) is up to date."
fi

Checking for SSL Vulnerabilities on the Command Line

Posted on July 3, 2009 by The Shell Shakespear 2 Comments

While Nessus is a wonderful vulnerability scanner, sometimes it is too slow and resource heavy for individual issues. The following 2 equivalent scripts perform checks for the following SSL related Nessus plugins:

  • 20007: SSL Version 2 (v2) Protocol Detection
  • 26928: SSL Weak Cipher Suites Supported
  • 31705: SSL Anonymous Cipher Suites Supported

The first is the curl version:

#!/bin/bash
# phaas at redspin.com: Never us a 'sh when a bash is necessary
# Checks the Equivalent of Nessus Plugin 20007, 26928 and 31705 (10863+21643)
 
if [ $# -lt 1 ]
then
  echo "List SSL Weakness present for a given website"
  echo "Usage: `basename $0` website {port}"
  exit 1
fi
web=${1-'www.redspin.com'}
port=${2-'443'}
 
# Check for the insecure SSLv2 version
curl -m1 -Ik "https://$web:$port" --ciphers sslv2 &amp;&gt; /dev/null
if [[ "$?" -eq 0 ]]; then echo -e "$web:$port: (ssl2) Weak SSLv2 encryption enabled"; fi
 
# Enumerate weak SSL ciphers using curl
IFS=$'\n' # Loop across lines, rather than words
ciphers='LOW:EXP:eNULL:aNULL' # Include EXP (Export Ciphers)
for line in `openssl ciphers -v $ciphers | tr -s ' '`; do
	version=`echo "$line" | cut -d' ' -f2 | tr [:upper:] [:lower:]`
	cipher=`echo "$line" | cut -d' ' -f1`
	auth=`echo "$line" | tr -s ' ' | grep -o "Au=[^ ]*" | cut -d'=' -f2`
	strength=`echo "$line" | sed 's#Kx=[^ ]*##' | grep -o '([0-9]*)' | tr -d '()' | grep -v 'None'`
	if [[ "$auth" == 'None' ]]; then auth="no"; fi
	if [[ -z "$strength" ]]; then strength="without encryption"; else strength="at $strength bit encryption"; fi
 
	#echo "curl -m1 -Ik https://$web:$port --ciphers $cipher -$version &amp;&gt; /dev/null"
	curl -m1 -Ik "https://$web:$port" --ciphers "$cipher" -$version &amp;&gt; /dev/null
	if [[ "$?" -eq 0 ]]; then
		echo -e "$web:$port: ($version) $cipher = Supported $strength with $auth authentication support"
	fi
done

And the following is the openssl version:

#!/bin/bash
# phaas at redspin.com: Never us a 'sh when a bash is necessary
# Checks the Equivalent of Nessus Plugin 20007, 26928 and 31705 (10863+21643)
 
if [ $# -lt 1 ]
then
  echo "List SSL Weakness present for a given website"
  echo "Usage: `basename $0` website {port}"
  exit 1
fi
web=${1-'www.redspin.com'}
port=${2-'443'}
 
# Check for the insecure SSLv2 version
sslv2=`echo -e '' | openssl s_client -connect $web:$port -ssl2 -no_ssl3 -no_tls1 2&gt;/dev/null | grep -i 'SSLv2'`
if [ -n "$sslv2" ]; then echo -e "$web:$port: (ssl2) Weak SSLv2 encryption enabled"; fi
 
# Enumerate weak SSL ciphers using openssl
IFS=$'\n' # Loop across lines, rather than words
ciphers='LOW:EXP:eNULL:aNULL' # Include EXP (Export Ciphers)
for line in `openssl ciphers -v $ciphers | tr -s ' '`; do
	version=`echo "$line" | cut -d' ' -f2 | tr [:upper:] [:lower:] | tr -d 'v'`
	cipher=`echo "$line" | cut -d' ' -f1`
	auth=`echo "$line" | tr -s ' ' | grep -o "Au=[^ ]*" | cut -d'=' -f2`
	strength=`echo "$line" | sed 's#Kx=[^ ]*##' | grep -o '([0-9]*)' | tr -d '()' | grep -v 'None'`
 
	if [[ "$auth" == 'None' ]]; then auth="no"; fi
	if [[ -z "$strength" ]]; then strength="without encryption"; else strength="at $strength bit encryption"; fi
 
	#echo "openssl s_client -connect $web:$port -$version -cipher $cipher"
	supported=`echo "" | openssl s_client -connect $web:$port -$version -cipher $cipher 2&gt;&amp;1 | grep DONE`
	if [[ -n "$supported" ]]; then
		echo -e "$web:$port: ($version) $cipher = Supported $strength with $auth authentication support"
	fi
done

I decided to include both because while openssl is usually included by default on most Linux distributions, curl is easier to obtain on Windows machines.

Handling HTTP and SSL in the Shell

Posted on July 1, 2009 by The Shell Shakespear Leave a comment

The topic of this week’s shell1liners is handling HTTP and SSL in Bash:

#netcat scanner for HTTP servers
for i in $(seq 1 255); do nc -n -v -z "192.168.1.$i" 80 | grep "open"; done | tee webservers.txt
 
# Manually perform a HTTP Get Request
echo -ne "GET / HTTP/1.0\n\n" | nc www.redspin.com 80
# Manually perform a HTTP Get Request on a SSL Port
echo -ne "GET / HTTP/1.0\n\n" | socat – OPENSSL:www.website.com:443,verify=0
# Create a local TCP pipe to a remote SSL port (to allow netcat to probe a SSL service)
socat -vd TCP-LISTEN:8888,fork OPENSSL:www.redspin.com:443,verify=0
 
# Always connect to a given webserver PORT regardless if it is SSL or normal HTTP
(curl -iks -m2 "https://www.redspin.com:PORT" || curl -iks -m2 "www.redspin.com:PORT")
 
# Perform a check on a list of webservers (HTTP or HTTPS): HOST:PORT -&gt; HOST:PORT|WEB SERVER|HTML Title
# Includes a 2 seconds timeout using curl's -m2, and parallelization using xargs's -P10
cat webservers.txt | xargs -P10 -I'{}' bash -c '(curl -Liks -m2 "https://{}" || curl -Liks -m2 "{}") | grep -iao -e "^Server: .*" -e "" | sed "s#Server: \(.*\)#|\1|#i;s###ig" | tr -d "\r\n" | sed "1s/^/{}/;\$a\\" | sed "s/^\([^|]*\)|$/\1||/"' | tee webserver_info.txt
 
# Check if Trace is enabled on a given website
echo -ne "TRACE /something HTTP/1.0\nX-Header: Trace Enabled\n\n" | socat - OPENSSL:www.website.com:443,verify=0
# Check for the insecure SSLv2 protocol on a website
echo -e '' | openssl s_client -connect WEBSITE:PORT -ssl2 -no_ssl3 -no_tls1 2&gt;/dev/null | grep 'SSLv2'
 
# Bruteforce a given numerical webpath, printing the HTTP status code for each request
for ((i=0;i/dev/null | grep HTTP/1.1) | tee webbf.txt ; done
 
# Simple HTTP Listener
python -m SimpleHTTPServer
# Simple HTTPS (SSL) Listener without a server certificate
sudo openssl s_server -accept 443 -nocert
# Simple HTTPS (SSL) Listener with a bad self-signed server certificate
echo -ne "\n\n\n\n\n\n\n" | openssl req -new -newkey rsa:1024 -days 1 -nodes -x509 -keyout out.pem -out out.pem ; openssl s_server -cert out.pem -www

An Introduction to Shell One Liners

Posted on June 25, 2009 by The Shell Shakespear 3 Comments

The knowledge and use of the command line is a powerful tool that can aid in the creation, modification and automation of routine tasks that a security auditor or any computer user may come up against.  The flexibility, simplicity, and leetness of the shell oneliner can replace thousand-line perl code which otherwise would be thrown away after the task is complete.  We have decided to provide share some of our favorite oneliners that we have found useful, either culled from other sources or created by ourselves.  All of these examples should run comfortably from a Linux bash shell or Cygwin-Windows equivalent, with the required applications listed in the oneliner.

The topic of this week is IP address manipulation:

# Sort by IP Addresses
sort -n -t. -k1,1 -k2,2 -k3,3 -k4,4
 
# Sort by IP Addresses and Port like IP:PORT
sed 's#:#.#' | sort -n -t. -k1,1 -k2,2 -k3,3 -k4,4 -k5,5 | sed 's#\(\([0-9]\{1,3\}\.\)\{4\}\)#\1:#;s#\.:#:#'
 
# IP2HOST: IP -&gt; IP (HOST) using 'bind-host' package built into Ubuntu
for i in $(cat ips.txt); do echo "$i ("`host $i | grep -v NXDOMAIN | cut -d' ' -f5`")"; done | sort -n -t. -k1,1 -k2,2 -k3,3 -k4,4 | sed 's#()##' | tee ip_hosts.txt
 
# HOST2IP: HOST -&gt; IP (HOST) using 'host' package available in Ubuntu
for i in $(cat hosts.txt); do host `echo "$i" | tr -d [[:blank:]]` | grep -v -e 'alias' -e 'handled' -e 'timed' | sed 's/Host \(.*\) .*/\1 0.0.0.0/' | sed "s/;;.*/$i - - 0.0.0.0/" | awk -F' ' '{printf "%s (%s)\n",$4,$1}'; done | sort -n -t. -k1,1 -k2,2 -k3,3 -k4,4 | tee ip_hosts.txt
 
#Print IP addresses in a file
egrep -o '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}'
 
# Print IP addresses in a file: Perl edition
perl -nle 'print $&amp; if /(\d{1,3}\.){3}\d{1,3}/'
 
# Print IP address in all files in the current directory tree with some pretty color matching
find . -type f -exec egrep -a -H -n --color=auto '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' {} \;

Thanks to readers for suggesting the color syntax support implemented using wp-syntax. I furthermore learned that the Visual Editor of WP was oppressing my HTML code, stripping tags in the WP-Syntax <pre> tag that should be present, so I’ll be keeping my edits in HTML mode from now on.