» NMAP

Importing and Working with Nmap Scans in Metasploit Framework 4

Posted on by Mark Marshall in Main | Leave a comment

Importing Nmap scans directly into Metasploit is one of the best time-saving tricks you can accomplish while using the Metasploit Framework. Once the full Nmap data is happily in your PostgreSQL database and accessible to Metasploit you can do all kinds of cool things with it that will save you lots of time and frustration on a large penetration test.
For this example I’m assuming you’ve got a fully functional PostgreSQL database already configured and accessible to Metasploit. This is normally the case if you’ve performed a full install of Metasploit 4. I’ll cover the basics of setting up and connecting to a PostgreSQL database in a future post.
Run db_status to determine if your database is set up properly and accessible to Metasploit. If you see the following output you are set:

msf > db_status
[*] postgresql connected to msf_database

To start, you need Nmap output saved to a file. Do this by feeding Nmap the -oA flag when you scan which will save the results in all 3 major file formats: XML, Nmap and Grepable.
From within msfconsole import your scan data:

msf > db_import 192.168_scan.xml
[*] Importing 'Nmap XML' data
[*] Import: Parsing with 'Nokogiri v1.4.3.1'
[*] Importing host 192.168.1.1
[*] Importing host 192.168.1.2
[*] Importing host 192.168.1.3
[*] Importing host 192.168.1.4
[*] Importing host 192.168.1.7
[*] Importing host 192.168.1.9
[*] Importing host 192.168.1.10
[*] Importing host 192.168.1.13
[*] Importing host 192.168.1.15
[*] Importing host 192.168.1.16
[*] Importing host 192.168.1.22
[*] Importing host 192.168.1.100
[*] Successfully imported /home/mark/192.168_scan.xml

Once this completes successfully your Nmap data will be contained in the Postgresql database and fully accessible to Metasploit. This opens up all kinds of flexibility that will really save your bacon on large scans.
If you want to you can also perform Nmap scans directly from within the Metasploit Framework and have it automatically added to the database. To do this use the db_nmap command followed by the flags you wish to use and the hosts or subnets you want to scan. I typically like to do Nmap scanning outside of Metasploit in order to have more flexibility about the types of scans I perform and I may run many different scans and cat them together or otherwise manipulate them prior to feeding them into Metasploit. Obviously, do what makes sense for your situation.
Type ‘hosts’ to get a list of all hosts in the database. Use ‘hosts -u’ to get a list of only hosts that respond to ping and are believed to be up.

msf > hosts -u
Hosts
=====
address        mac  name             os_name  os_flavor  os_sp  purpose  info  comments
-------        ---  ----             -------  ---------  -----  -------  ----  --------
192.168.1.1                          Unknown                    device
192.168.1.10        goro.home        Unknown                    device

You can also query based on services. Execute ‘services’ with no parameters to dump all hosts and all services in the database. This isn’t particularly useful and can be quite huge depending on the scan data that you’re working with. Thankfully you can parse this further before it’s output to the console. Use the -p flag to only list specific ports you’re interested in.

msf > services -p 445 -u 
Services
========
host           port  proto  name          state  info
----           ----  -----  ----          -----  ----
192.168.1.10   445   tcp    microsoft-ds  open   Samba smbd 3.X workgroup: SKYNET
192.168.1.100  445   tcp    microsoft-ds  open
192.168.1.11   445   tcp    netbios-ssn   open
192.168.1.2    445   tcp    microsoft-ds  open
192.168.1.22   445   tcp    microsoft-ds  open
192.168.1.4    445   tcp    microsoft-ds  open   Microsoft Windows 2003 or 2008 microsoft-ds
192.168.1.6    445   tcp    netbios-ssn   open
192.168.1.9    445   tcp    microsoft-ds  open

Here i’m listing only hosts that have 445/tcp open. I’ve also added the -u flag to only show services that are open.
If you’re a narcissist, at this point you’re probably thinking “big whoop, I can do all this via a few grep strings on the Nmap output”. And you’re correct.
Now to do something useful with this.

msf > services -p 445 -R

Services
========

host           port  proto  name          state  info
----           ----  -----  ----          -----  ----
192.168.1.10   445   tcp    microsoft-ds  open   Samba smbd 3.X workgroup: SKYNET
192.168.1.100  445   tcp    microsoft-ds  open
192.168.1.11   445   tcp    netbios-ssn   open
192.168.1.2    445   tcp    microsoft-ds  open
192.168.1.22   445   tcp    microsoft-ds  open
192.168.1.4    445   tcp    microsoft-ds  open   Microsoft Windows 2003 or 2008 microsoft-ds
192.168.1.6    445   tcp    netbios-ssn   open
192.168.1.9    445   tcp    microsoft-ds  open

RHOSTS => file:/tmp/msf-db-rhosts-20110909-32464-oyzbko

Looks the same as before, but by adding the -R flag, you’ve told Metasploit to set the RHOSTS variable to the output of the database query you’ve just performed. This is reflected in the last line of output which is the filename of the hosts that you’ve selected from the database which Metasploit created and populated.
Now select an exploit to use against these hosts

msf > use auxiliary/scanner/smb/smb_enumusers
msf  auxiliary(smb_enumusers) > show options

Module options (auxiliary/scanner/smb/smb_enumusers):

   Name       Current Setting                                Required  Description
   ----       ---------------                                --------  -----------
   RHOSTS     file:/tmp/msf-db-rhosts-20110909-32464-oyzbko  yes       The target address range or CIDR identifier
   SMBDomain  WORKGROUP                                      no        The Windows domain to use for authentication
   SMBPass                                                   no        The password for the specified username
   SMBUser                                                   no        The username to authenticate as
   THREADS    1                                              yes       The number of concurrent threads

As you can see Metapsloit has filled in the RHOSTS variable automatically for this exploit. You don’t need to have a pre-selected exploit in order for Metasploit to do this, and can choose an exploit after you’ve piped the output of a database query to the input of the RHOSTS variable.
Using Metasploit Framework 4 tied to a database is a great way to save time and effort while working with large projects and scans of several hundred to several thousand hosts and many more services.

Penetration Testing : NMAP.XML to TAB

Posted on by The Shell Shakespear in Main | 1 Comment

Following up on my last NMAP post, processing port scan data in a meaningful manner is essential to network penetration testing. For those who wish to skip the SQL stage and get quick results, the following one-liner will use xmlstarlet to parse a NMAP XML file:

cat nmap.xml | xmlstarlet sel -T -t -m "//state[@state='open']" -m ../../.. -v address/@addr -m hostnames/hostname -i @name -o ' (' -v @name -o ')' -b -b -b -o "	" -m .. -v @portid -o '/' -v @protocol -o "	" -m service -v @name -i "@tunnel='ssl'" -o 's' -b -o "	" -v @product -o ' ' -v @version -v @extrainfo -b -n -

Into the following tab delimited format:

IP (HOST) \t PORT/PROTOCOL \t SERVICE \t EXTRAINFO

This command sorts ports properly, but does not properly order the hosts. To do this, pipe the above command to the following:

sed 's_^\([^\t ]*\)\( ([^)]*)\)\?\t\([^\t ]*\)_\1.\3\2_' | sort -n -t. -k1,1 -k2,2 -k3,3 -k4,4 -k5,5 | sed 's_^\(\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}\)\.\([^ \t]*\)\( ([^)]*)\)\?_\1\4\t\3_'

This command converts lines that look like IP (HOST) \t PORT to IP.PORT (HOST), sorts it, and then converts them back to IP (HOST) \t PORT. From there, it is simply a matter of grepping for your favorite service. For example, if you wanted to focus on web penetration testing, all you would have to do is pipe the above to:

grep -i -e http

To get a list of services relevant to your testing. Happy Hacking.

NMAP Database Output : XML TO SQL

Posted on by The Shell Shakespear 4 Comments

SQL support has been a much requested feature of NMAP in the Redspin office. While a number of tools exist to support NMAP SQL output, their database format has left much to be desired. Using SQLite, Perl’s DB and the NMAP Parser module, our tool extracts all supported fields in an NMAP XML file and creates the following database format:

TABLE nmap (
    sid INTEGER PRIMARY KEY AUTOINCREMENT,
    version TEXT,
    xmlversion TEXT,
    args TEXT,
    types TEXT,
    starttime INTEGER,
    startstr TEXT,
    endtime INTEGER,
    endstr TEXT,
    numservices INTEGER) 

TABLE hosts (
    sid INTEGER,
    hid INTEGER PRIMARY KEY AUTOINCREMENT,
    ip4 TEXT,
    ip4num INTEGER,
    hostname TEXT,
    status TEXT,
    tcpcount INTEGER,
    udpcount INTEGER,
    mac TEXT,
    vendor TEXT,
    ip6 TEXT,
    distance INTEGER,
    uptime TEXT,
    upstr TEXT) 

TABLE sequencing (
    hid INTEGER,
    tcpclass TEXT,
    tcpindex TEXT,
    tcpvalues TEXT,
    ipclass TEXT,
    ipvalues TEXT,
    tcptclass TEXT,
    tcptvalues TEXT) 

TABLE ports (
    hid INTEGER,
    port INTEGER,
    type TEXT,
    state TEXT,
    name TEXT,
    tunnel TEXT,
    product TEXT,
    version TEXT,
    extra TEXT,
    confidence INTEGER,
    method TEXT,
    proto TEXT,
    owner TEXT,
    rpcnum TEXT,
    fingerprint TEXT) 

TABLE os (
    hid INTEGER,
    name TEXT,
    family TEXT,
    generation TEXT,
    type TEXT,
    vendor TEXT,
    accuracy INTEGER)

The resulting database can then be queried directly using SQLite in order to extract the relevant information. The tool also prints output in the following format sorted by IP and PORT as show below:

$ nmap -A -T4 scanme.nmap.org -oX scanme >/dev/null
$ nmap_xml2sql.pl scanme | grep -v "^#"
64.13.134.52 (scanme.nmap.org)	53/tcp	domain
64.13.134.52 (scanme.nmap.org)	80/tcp	http

Compare this database structure with the other similar XML2SQL approaches:
NMAP-SQL: http://sourceforge.net/projects/nmapsql/ outdated – only supports NMAP 3.75 and MySQL:

table portstat, table runlist, table targets

NMAP-Parser (nmap2db.pl): http://search.cpan.org/dist/Nmap-Parser/ MySQL and SQLite support:

table hosts(
  ip              VARCHAR(15) PRIMARY KEY NOT NULL,
  mac             VARCHAR(17),
  status          VARCHAR(7) DEFAULT 'down',
  hostname        TEXT,
  open_ports      TEXT,
  filtered_ports  TEXT,
  osname	      TEXT,
  osfamily        TEXT,
  osgen           TEXT,
  last_scanned    TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  UNIQUE (ip))

PBNJ -x option: http://pbnj.sourceforge.net/ MySQL and SQLite support:

table machines(
  mid             PRIMARY KEY AUTOINCREMENT,
  ip             	TEXT,
  host          	TEXT,
  localh        	INTEGER,
  os      		TEXT,
  machine_created TEXT,
  created_on      TEXT)

table services(
  mid             INTEGER,
  service         TEXT,
  state          	TEXT,
  port        	INTEGER,
  protocol      	TEXT,
  version 		TEXT,
  banner      	TEXT,
  machine_updated TEXT,
  updated_on      TEXT)

The tool has been released under a NMAP equivalent license as well as a Fyodor may-use-as-he-pleases license, and can be download here: nmap_xml2sql.pl

Requirements: