Home » hephaestus » Multithreaded function calls in Python

Whilst working on my dissertation project, hephaestus, I had a requirement to multithread an I/O-bound function. The following code is a simplified version, simmering multiple classes into a single module for demonstration purposes. The example output at the end was generated using pexpect, which you can read about in this post.

Overview / Pseudocode:

  • Given a list of network prefixes, generate a list of IP addresses
  • Perform multithreaded function calls, one thread per IP
  • The function call issues an SNMP GET request to each IP and performs an action dependent upon the response (or absence thereof)

Implementation:


#!/usr/bin/env python
import configparser
import futures
import ipaddr
import paramiko
import time

# configparser unpickles a dictionary object from the config file
CONFIG = configparser.read_config("geant.conf")

# If we have called this function and CONFIG is an empty dictionary, inform user
if CONFIG == {}:
  print("Current configuration is empty. First enter configuration mode and load a file or run the wizard.")
# Else we have initialised the CONFIG dictionary, proceed
else:
  print("Using current configuration:\n" + CONFIG['info'])
  print("Scanning for routers in prefixes " + CONFIG['routerprefixes'] + " using SNMP community " + CONFIG['routersnmpcommunity'])
  prefixes = CONFIG['routerprefixes'].split()
  # Add an empty list to the dictionary which will store the IPs contained in the prefixes
  CONFIG['routerips'] = []

  # Iterate over prefixes, use ipaddr library to generate IPs and add to our list
  for prefix in prefixes:
    network = ipaddr.IPNetwork(prefix)
    for ip in network.iterhosts():
      CONFIG['routerips'].append(str(ip))

  # How many IPs have we?
  numscanned = len(CONFIG['routerips'])

Now we define the function which will be multithreaded:

# Worker thread
def snmpWorker(ip):
  # SSH to the jump box host --- IPs are firewalled, snmpget must be issued from a trusted host
  client = paramiko.SSHClient()
  client.load_system_host_keys()
  key = paramiko.DSSKey.from_private_key_file(CONFIG['jumpboxsshkey'])
  print(ip + '\t: connecting...')
  client.connect(CONFIG['jumpboxhost'], username=CONFIG['jumpboxsshuser'], pkey=key)
  # Build the snmpget command string
  command = 'snmpget -v 2c -c ' + CONFIG['routersnmpcommunity'] + ' ' + str(ip) + ' 1.3.6.1.4.1.2636.3.1.2.0'
  print(ip + '\t: executing ' + command)
  # Execute the command
  stdin, stdout, stderr = client.exec_command(str(command))
  print(ip + '\t: waiting...')
  # Grab the output
  output = stdout.read()
  # See what we got
  if "Router" in output:
    print(ip + '\t: is a ' + output.split('"')[1])
  else:
    print(ip + '\t: is not a router')
    # prune from list
    CONFIG['routerips'].remove(ip)

Here’s the ‘magic’ which makes use of the futures library:

# Create a ThreadPoolExecuter object which will dispatch 10 threads concurrently
executor = futures.ThreadPoolExecutor(10)
# Optional: Grab the start time (seconds since UNIX epoch)
start = time.time()
# Submit calls to the executor:
# 1st arg is the function to be called
# 2nd arg is passed to the function
# Each call is derived from a list comprehension; CONFIG['routerips'] object is a list of IP addresses
futures = [executor.submit(snmpWorker, ip) for ip in CONFIG['routerips']]
# Wait for threads to finish
futures.wait(futures)
# Optional: Grab the end time
end = time.time()

# After workers execute, routerips list may be shorter, grab a new count
numfound = len(CONFIG['routerips'])

# Calculate the wall time for scanning 'numscanned' IPs
diff = end - start

# Print a summary
print
print('Scanned\t\t: ' + str(numscanned) + ' IPs')
print('Found\t\t: ' + str(numfound) + ' routers')
print('Wall time\t: ' + str(diff) + ' seconds')

Here’s how it looks in practice:

niall@concrescence ~/Code/CSC3002/HEAD/code $ ./demo.py
Welcome to hephaestus, the stateless firewall configuration parser for JunOS.

Command help with 'help' or '?', auto-completion is available, shell commands available by prefixing with '!', eg: '!ls -lf'

hephaestus> config
You are now in configuration mode, different commands are available.
Command help with 'help' or '?'

hephaestus(config)# load
The following configuration files are available:
0 acme.conf
1 acmenew.conf
2 acmefoo.conf
Enter the number of the file you wish to load:
1
hephaestus(config)# exit

hephaestus> scan
You are now in scan mode, different commands are available.
Command help with 'help' or '?'

hephaestus(scan)> autoscan
Using current configuration:
The wonderful and amazing network known as ACME, AS65000
Scanning for routers in prefixes 172.16.100.0/28 192.168.88.0/28 using SNMP community $nMpC0mmun1ty
172.16.100.2    : connecting...
172.16.100.1    : connecting...
172.16.100.3    : connecting...
172.16.100.4    : connecting...
172.16.100.5    : connecting...
172.16.100.6    : connecting...
172.16.100.8    : connecting...
172.16.100.7    : connecting...
172.16.100.9    : connecting...
172.16.100.10   : connecting...
172.16.100.10   : executing snmpget -v 2c -c $nMpC0mmun1ty 172.16.100.10 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.2    : executing snmpget -v 2c -c $nMpC0mmun1ty 172.16.100.2 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.3    : executing snmpget -v 2c -c $nMpC0mmun1ty 172.16.100.3 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.5    : executing snmpget -v 2c -c $nMpC0mmun1ty 172.16.100.5 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.6    : executing snmpget -v 2c -c $nMpC0mmun1ty 172.16.100.6 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.1    : executing snmpget -v 2c -c $nMpC0mmun1ty 172.16.100.1 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.8    : executing snmpget -v 2c -c $nMpC0mmun1ty 172.16.100.8 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.9    : executing snmpget -v 2c -c $nMpC0mmun1ty 172.16.100.9 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.4    : executing snmpget -v 2c -c $nMpC0mmun1ty 172.16.100.4 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.7    : executing snmpget -v 2c -c $nMpC0mmun1ty 172.16.100.7 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.8    : waiting...
172.16.100.5    : waiting...
172.16.100.4    : waiting...
172.16.100.1    : waiting...
172.16.100.10   : waiting...
172.16.100.2    : waiting...
172.16.100.3    : waiting...
172.16.100.6    : waiting...
172.16.100.9    : waiting...
172.16.100.7    : waiting...
172.16.100.5    : is a Juniper MX960 Internet Backbone Router
172.16.100.11   : connecting...
172.16.100.6    : is a Juniper MX480 Midplane Internet Backbone Router
172.16.100.2    : is a Juniper MX960 Internet Backbone Router
172.16.100.12   : connecting...
172.16.100.4    : is a Juniper MX960 Internet Backbone Router
172.16.100.13   : connecting...
172.16.100.14   : connecting...
172.16.100.3    : is a Juniper MX480 Midplane Internet Backbone Router
192.168.88.1    : connecting...
172.16.100.7    : is a Juniper MX960 Internet Backbone Router
192.168.88.2    : connecting...
172.16.100.9    : is a Juniper MX960 Internet Backbone Router
192.168.88.3    : connecting...
172.16.100.11   : executing snmpget -v 2c -c $nMpC0mmun1ty 172.16.100.11 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.13   : executing snmpget -v 2c -c $nMpC0mmun1ty 172.16.100.13 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.12   : executing snmpget -v 2c -c $nMpC0mmun1ty 172.16.100.12 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.14   : executing snmpget -v 2c -c $nMpC0mmun1ty 172.16.100.14 1.3.6.1.4.1.2636.3.1.2.0
192.168.88.1    : executing snmpget -v 2c -c $nMpC0mmun1ty 192.168.88.1 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.11   : waiting...
172.16.100.13   : waiting...
172.16.100.12   : waiting...
172.16.100.14   : waiting...
192.168.88.2    : executing snmpget -v 2c -c $nMpC0mmun1ty 192.168.88.2 1.3.6.1.4.1.2636.3.1.2.0
192.168.88.1    : waiting...
172.16.100.11   : is a Juniper MX960 Internet Backbone Router
192.168.88.4    : connecting...
172.16.100.10   : is a Juniper MX960 Internet Backbone Router
192.168.88.5    : connecting...
192.168.88.3    : executing snmpget -v 2c -c $nMpC0mmun1ty 192.168.88.3 1.3.6.1.4.1.2636.3.1.2.0
192.168.88.2    : waiting...
172.16.100.14   : is a Juniper MX960 Internet Backbone Router
192.168.88.6    : connecting...
192.168.88.1    : is a Juniper MX480 Midplane Internet Backbone Router
172.16.100.1    : is a Juniper MX960 Internet Backbone Router
192.168.88.7    : connecting...
192.168.88.8    : connecting...
192.168.88.3    : waiting...
192.168.88.2    : is a Juniper MX480 Midplane Internet Backbone Router
192.168.88.9    : connecting...
192.168.88.3    : is a Juniper MX480 Midplane Internet Backbone Router
192.168.88.10   : connecting...
192.168.88.4    : executing snmpget -v 2c -c $nMpC0mmun1ty 192.168.88.4 1.3.6.1.4.1.2636.3.1.2.0
192.168.88.5    : executing snmpget -v 2c -c $nMpC0mmun1ty 192.168.88.5 1.3.6.1.4.1.2636.3.1.2.0
192.168.88.6    : executing snmpget -v 2c -c $nMpC0mmun1ty 192.168.88.6 1.3.6.1.4.1.2636.3.1.2.0
192.168.88.7    : executing snmpget -v 2c -c $nMpC0mmun1ty 192.168.88.7 1.3.6.1.4.1.2636.3.1.2.0
192.168.88.4    : waiting...
192.168.88.5    : waiting...
192.168.88.9    : executing snmpget -v 2c -c $nMpC0mmun1ty 192.168.88.9 1.3.6.1.4.1.2636.3.1.2.0
192.168.88.6    : waiting...
192.168.88.7    : waiting...
172.16.100.13   : is a Juniper MX960 Internet Backbone Router
192.168.88.11   : connecting...
192.168.88.9    : waiting...
192.168.88.10   : executing snmpget -v 2c -c $nMpC0mmun1ty 192.168.88.10 1.3.6.1.4.1.2636.3.1.2.0
192.168.88.6    : is a Juniper MX480 Midplane Internet Backbone Router
192.168.88.12   : connecting...
192.168.88.5    : is a Juniper MX480 Midplane Internet Backbone Router
192.168.88.9    : is a Juniper MX960 Internet Backbone Router
192.168.88.13   : connecting...
192.168.88.14   : connecting...
192.168.88.10   : waiting...
192.168.88.7    : is a Juniper MX480 Midplane Internet Backbone Router
192.168.88.11   : executing snmpget -v 2c -c $nMpC0mmun1ty 192.168.88.11 1.3.6.1.4.1.2636.3.1.2.0
192.168.88.12   : executing snmpget -v 2c -c $nMpC0mmun1ty 192.168.88.12 1.3.6.1.4.1.2636.3.1.2.0
192.168.88.11   : waiting...
192.168.88.14   : executing snmpget -v 2c -c $nMpC0mmun1ty 192.168.88.14 1.3.6.1.4.1.2636.3.1.2.0
192.168.88.13   : executing snmpget -v 2c -c $nMpC0mmun1ty 192.168.88.13 1.3.6.1.4.1.2636.3.1.2.0
172.16.100.12   : is a Juniper MX960 Internet Backbone Router
192.168.88.12   : waiting...
192.168.88.14   : waiting...
192.168.88.13   : waiting...
192.168.88.12   : is a Juniper MX480 Midplane Internet Backbone Router
192.168.88.10   : is a Juniper MX960 Internet Backbone Router
192.168.88.4    : is a Juniper MX480 Midplane Internet Backbone Router
192.168.88.14   : is a Juniper MX480 Midplane Internet Backbone Router
172.16.100.8    : is not a router
192.168.88.13   : is a Juniper MX960 Internet Backbone Router
192.168.88.8    : executing snmpget -v 2c -c $nMpC0mmun1ty 192.168.88.8 1.3.6.1.4.1.2636.3.1.2.0
192.168.88.8    : waiting...
192.168.88.11   : is not a router
192.168.88.8    : is not a router

Scanned      : 28 IPs
Found        : 25 routers
Wall time    : 18.874931097 seconds
hephaestus(scan)> exit

hephaestus> exit
niall@concrescence ~/Code/CSC3002/HEAD/code $