Standard Library

Batteries Included
Objective
Standard Modules:
socket
os
urllib
sys
telnetlib
subprocess
paramiko
time
json
logging
requests
re
os module
The OS module provides a portable way of using operating system
dependent functionality. For instance to get a directory list on a
windows machine you can issue :
os.system(ls l)
# will return a unix style list:
drwxr-xr-x 2 Dick Administ 0 Feb 27 22:54 aaaa
-rw-r--r-- 1 Dick Administ 645 Feb 28 22:41 ftp.py

os.system(dir *.*)
# will return windows style list
Volume in drive C has no label.
Volume Serial Number is F017-710B

Directory of C:\cs260\ftp

02/27/2012 10:54 PM <DIR> .

02/27/2012 10:54 PM <DIR> ..
02/27/2012 10:54 PM <DIR> aaaa
02/28/2012 10:41 PM 645 ftp.py
1 File(s) 645 bytes
3 Dir(s) 44,729,069,568 bytes free
os module
os.chdir(path) change the current working directory to path

os.getcwd() get current working directory

os.getcwdu() get the current working directory as a unicode object

os.listdir(path) returns a list of the names of the entries in the directory given by path; list
is in arbitrary order and doesnt contain , or .. .
>>> print os.listdir('.')
['aaaa', 'ftp.py']

os.mkdir(path[,mode]) create a directory named path with mode mode. default mode is
0777. Mode is ignored on Windows.

os.rename(old , new ) rename file or directory from old to new.

os.rmdir(path) remove (delete) the directory path. This only works if the directory is
empty.
sys module
This module provides a number of functions and variables that
can be used to manipulate different parts of the Python runtime
environment.
The argv list contains the arguments passed to the script, when
the interpreter was started. The first item contains the name of
the script itself.
import sys c:\DevNetCode\Python>python test.py
script name is test.py
print("script name is", sys.argv[0]) there are no arguments!

if len(sys.argv) > 1:
print("there are", len(sys.argv) - 1, c:\DevNetCode\Python>python test.py one two three
"arguments:") script name is test.py
for arg in sys.argv[1:]: there are 3 arguments:
print(arg) one
else: two
print("there are no arguments!") three
sys module
Working with modules
The path list contains a list of directory names where Python looks for
extension modules. When you start Python, this list is initialized from a
mixture of built-in rules, the contents of the PYTHONPATH environment
variable, and the registry contents (on Windows). But since its an ordinary
list, you can also manipulate it from within the program:

import sys

print("path has", len(sys.path), "members") path has 7 members

# add the sample directory to the path
sys.path.insert(0, "samples")
import sample
this is the sample module!
Traceback (innermost last):
File "sys-path-example-1.py", line 11, in ?
import random # oops!
ImportError: No module named random

# nuke the path

sys.path = []
import random # oops!
sys module
Checking the host platform
The platform variable contains the name of the host platform:
import sys

if sys.platform == "win32":
import ntpath
pathmodule = ntpath
elif sys.platform == "mac":
import macpath
pathmodule = macpath
else:
# assume it's a posix platform
import posixpath
pathmodule = posixpath

print(pathmodule)

Typical platform names are win32 for Windows 9X/NT and mac for Macintosh. For
Unix systems, the platform name is usually derived from the output of the uname -
r command, such as irix6, linux2, orsunos5 (Solaris).
sys module
Exiting the program
When you reach the end of the main program, the interpreter is automatically
terminated. If you need to exit in midflight, you can call the sys.exit function
instead. This function takes an optional integer value, which is returned to the
calling program.

import sys

print("hello")

sys.exit(1)

print("there")
subprocess module
The subprocess module allows us to:
spawn new processes
connect to their input/output/error pipes
obtain their return codes
It offers a higher-level interface than some of the other available
modules, and is intended to replace the following functions:
os.system()
os.spawn*()
os.popen*()
popen2.*()
commands.*()
subprocess module
subprocess.call()
We can run the command line with the arguments passed as a list of strings
or by setting the shell argument to a True value. Note, the default value of the
shell argument is False.
>>> import subprocess
>>> subprocess.call('ping 8.8.8.8, shell=True)

Pinging 8.8.8.8 with 32 bytes of data:

Reply from 8.8.8.8: bytes=32 time=3ms TTL=56
Reply from 8.8.8.8: bytes=32 time=3ms TTL=56
Reply from 8.8.8.8: bytes=32 time=3ms TTL=56
Reply from 8.8.8.8: bytes=32 time=3ms TTL=56

Ping statistics for 8.8.8.8:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 3ms, Maximum = 3ms, Average = 3ms
0
subprocess module
Return Codes
You can use subprocess.call return codes to determine the success of the
command.
Every process will return an exit code and you can do something with your script
based on that code.
If the return code is anything else than zero, it means that an error occurred.

stdin, stdout and stderr

One of the trickiest part with subprocess was how to work with pipes and to pipe
commands together.
PIPE indicates that a new pipe to the child should be created.
The default setting is "None", which means that no redirection will occur.
The standard error (or stderr) can be STDOUT, which indicates that the stderr data
from the child process should be captured into the same file handle as for stdout.
subprocess module
subprocess.Popen()
The underlying process creation and management in the subprocess module
is handled by the Popen class. subprocess.popen is replacing os.popen.
# Import the module
import subprocess

# Ask the user for input

host = input("Enter a host to ping: ")

# Set up the echo command and direct the output to a pipe

p1 = subprocess.Popen(['ping', host], stdout=subprocess.PIPE)

# Run the command

output = p1.communicate()[0]

print(output)
time module
There is a popular time module available in Python which
provides functions for working with times, and for converting
between representations. The functiontime.time() returns the
current system time in ticks since 12:00am, January 1,
1970(epoch).
import time # This is required to include time module.

ticks = time.time()
print("Number of ticks since 12:00am, January 1, 1970:", ticks)

Number of ticks since 12:00am, January 1, 1970: 1471928153.15123

time module
Getting current time
To translate a time instant from a seconds since the epoch floating-point value
into a time-tuple, pass the floating-point value to a function (e.g., localtime)
that returns a time-tuple with all nine items valid.
import time;
Local current time : time.struct_time(tm_year=2016, tm_mon=8,
localtime = time.localtime(time.time()) tm_mday=23, tm_hour=13, tm_min=0, tm_sec=38, tm_wday=1,
print ("Local current time :", localtime) tm_yday=236, tm_isdst=0)

Getting formatted time

You can format any time as per your requirement, but simple method to get
time in readable format is asctime().
import time

localtime = time.asctime( Local current time : Tue Aug 23 13:01:41 2016

time.localtime(time.time()) )
print("Local current time :", localtime)
time module
time.sleep(seconds)
The method sleep() suspends execution for the given number of seconds.
import time

time.sleep(1) # delays for 1 seconds

time.sleep(10) # delays for 10 seconds
time.sleep(60) # delays for 1 minute
time.sleep(3600) # delays for 1 hour

import time
import subprocess

def main():

while True:
subprocess.call("ping -n 1 8.8.8.8")
time.sleep(5) # delays for 1 seconds

if __name__ == '__main__':
main()
logging module
Logging is a means of tracking events that happen when some
software runs.
Logging provides a set of convenience functions for simple
logging usage. These are debug(), info(), warning(),
error() and critical().
The logging functions are named after the level or severity of
the events they are used to track.
logging module
When to use logging
Task you want to perform
Display console output for ordinary usage of a command
line script or program
Report events that occur during normal operation of a
program (e.g. for status monitoring or fault investigation)
Issue a warning regarding a particular runtime event
Report an error regarding a particular runtime event
Report suppression of an error without raising an exception
(e.g. error handler in a long-running server process)
The best tool for the task
print()
logging.info() (or logging.debug() for very detailed output
for diagnostic purposes)
warnings.warn() in library code if the issue is avoidable and
the client application should be modified to eliminate the
warning
logging.warning() if there is nothing the client application
can do about the situation, but the event should still be
noted
Raise an exception
logging.error(), logging.exception() or logging.critical() as
appropriate for the specific error and application domain
logging module
The standard levels and their applicability are described below
(in increasing order of severity):
Level When its used
DEBUG Detailed information, typically of interest only when diagnosing problems.
INFO Confirmation that things are working as expected.
An indication that something unexpected happened, or indicative of some
WARNING problem in the near future (e.g. disk space low). The software is still working as
expected.
Due to a more serious problem, the software has not been able to perform some
ERROR
function.
A serious error, indicating that the program itself may be unable to continue
CRITICAL
running.
logging module
Example
The INFO and DEBUG message doesnt appear because the default level is
WARNING. The printed message includes the indication of the level and the
description of the event provided in the logging call
import os
import re
import logging

received_packages = re.compile(r"Received = (\d)")

ip = '172.16.1.5'
ping_out = os.popen("ping -n 2 " + ip, "r")
logging.debug("... pinging " + ip)

line = ping_out.readlines()
n_received = re.findall(received_packages, str(line))
result = int(n_received[0])
if result == 0:
logging.error("no response")
elif result == 1:
logging.warning("alive, but 50 % packet loss")
elif result == 2:
logging.debug("alive")
else:
logging.critical("unknown issue")
re module
The re module provides regular expression tools for advanced
string processing.
r - Pythons raw string notation for regular expression patterns

re.findall(r'\bf[a-z]*', 'which foot or hand fell

fastest')

['foot', 'fell', 'fastest']

socket module
Sockets are the endpoints of a bidirectional communications
channel. Sockets may communicate within a process, between
processes on the same machine, or between processes on
different continents.
Sockets may be implemented over a number of different
channel types: Unix domain sockets, TCP, UDP, and so on.
The socket library provides specific classes for handling the
common transports as well as a generic interface for handling
the rest.
socket module
Sockets have their own vocabulary:
Term Description
domain The family of protocols that is used as the transport mechanism. These values are constants such
as AF_INET, PF_INET, PF_UNIX, PF_X25, and so on.
type The type of communications between the two endpoints, typically SOCK_STREAM for connection-
oriented protocols and SOCK_DGRAM for connectionless protocols.
protocol Typically zero, this may be used to identify a variant of a protocol within a domain and type.
hostname The identifier of a network interface:A string, which can be a host name, a dotted-quad address,
or an IPV6 address in colon (and possibly dot) notation
A string "<broadcast>", which specifies an INADDR_BROADCAST address.
A zero-length string, which specifies INADDR_ANY, or
An Integer, interpreted as a binary address in host byte order.
port Each server listens for clients calling on one or more ports. A port may be a Fixnum port number, a
string containing a port number, or the name of a service.
 socket module
Creating a Socket
s = socket.socket (socket_family, socket_type, protocol=0)

Here is the description of the parameters

socket_family: This is either AF_UNIX or AF_INET, as explained earlier.
socket_type: This is either SOCK_STREAM or SOCK_DGRAM.
protocol: This is usually left out, defaulting to 0.
#Client Socket #Server Socket

import socket import socket

# create an INET, STREAMing socket # create an INET, STREAMing socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# now connect to the web server on port 80 - the normal http port
s.connect(("www.python.org", 80))
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# bind the socket to a public host, and a well-known port
serversocket.bind((socket.gethostname(), 80))
# become a server socket
serversocket.listen(5)
socket module
Server Socket Methods
Method Description
s.bind() This method binds address (hostname, port number pair) to
socket.
s.listen() This method sets up and start TCP listener.
s.accept() This passively accept TCP client connection, waiting until
connection arrives (blocking).

Client Socket Methods

Method Description
s.connect() This method actively initiates TCP server connection.
socket module
General Socket Methods
Method Description
s.recv() This method receives TCP message
s.send() This method transmits TCP message
s.recvfrom() This method receives UDP message
s.sendto() This method transmits UDP message
s.close() This method closes socket
socket.gethostname() Returns the hostname.
socket module
A Simple Server
To write Internet servers, we use the socket function available in socket
module to create a socket object.
import socket # Import socket module

s = socket.socket() # Create a socket object

host = socket.gethostname() # Get local machine name
port = 12345 # Reserve a port for your service.
s.bind((host, port)) # Bind to the port

s.listen(5) # Now wait for client connection.

while True:
c, addr = s.accept() # Establish connection with client.
print('Got connection from', addr)
c.send('Thank you for connecting')
c.close() # Close the connection
socket module
A Simple Client
Let us write a very simple client program which opens a connection to a given
port 12345 and given host. This is very simple to create a socket client using
Python's socket module function.

import socket # Import socket module

s = socket.socket() # Create a socket object

host = socket.gethostname() # Get local machine name
port = 12345 # Reserve a port for your service.

s.connect((host, port))
print(s.recv(1024))
s.close() # Close the socket when done
urllib package
urllib.request is a Python module for fetching URLs (Uniform
Resource Locators). It offers a very simple interface, in the form
of the urlopen function. This is capable of fetching URLs using a
variety of different protocols. It also offers a slightly more
complex interface for handling common situations - like basic
authentication, cookies, proxies and so on.
Fetching URLs

import urllib.request

with urllib.request.urlopen('http://python.org/') as response:

html = response.read()
urllib.request module
HTTP is based on requests and responses - the client makes requests and
servers send responses. urllib.request mirrors this with a Request object
which represents the HTTP request you are making.
import urllib.request

req = urllib.request.Request(http://www.google.com)
response = urllib.request.urlopen(req)
the_page = response.read()

Note that urllib.request makes use of the same Request interface to handle all
URL schemes. For example, you can make an FTP request like so:

req = urllib.request.Request(ftp://ftp.google.com/)
urllib package
You can use a POST to transmit arbitrary data to your own application. In the
common case of HTML forms, the data needs to be encoded in a standard
way, and then passed to the Request object as the data argument. The
encoding is done using a function from the urllib.parse library.

import urllib.parse
import urllib.request

url = "http://www.someserver.com/cgi-bin/register.cgi"
values = {'name': Bill','location' : 'Northampton','language' : 'Python' }

data = urllib.parse.urlencode(values)
req = urllib.request.Request(url, data)
response = urllib.request.urlopen(req)
the_page = response.read()
urllib package
Headers
When you create a Request object you can pass a dictionary of headers in.
import urllib.parse
import urllib.request

url = http://www.someserver.com/cgi-bin/register.cgi
user_agent = Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)
values = {name : Michael Foord,location : Northampton,language : Python }

headers = { User-Agent : user_agent }

data = urllib.parse.urlencode(values)
req = urllib.request.Request(url, data, headers)
response = urllib.request.urlopen(req)
the_page = response.read()
urllib package
Handling Exceptions
URLError - URLError is raised because there is no network connection (no
route to the specified server), or the specified server doesnt exist.
HTTPError - The HTTPError instance raised will have an integer code
attribute, which corresponds to the error sent by the server.
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError

req = Request("http://www.google.com")

try:
response = urlopen(req)
except HTTPError as e:
print('The server couldn\'t fulfill the request.')
print('Error code: ', e.code)
except URLError as e:
print('We failed to reach a server.')
print('Reason: ', e.reason)
else:
print("everything is fine")
telnetlib module
The telnetlib module provides a Telnet class that
implements the Telnet protocol.
class telnetlib.Telnet(host=None, port=0[, timeout])

Telnet represents a connection to a Telnet server

telnetlib module
Telnet Objects
Telnet.read_until(expected, timeout=None)
Read until a given byte string, expected, is encountered or until timeout seconds have
passed.
Telnet.read_all()
Read all data until EOF as bytes; block until connection closed.
Telnet.write(buffer)
Write a byte string to the socket, doubling any IAC characters. This can block if the
connection is blocked. May raise OSError if the connection is closed.
Telnet.close()
Close the connection.
telnetlib module
import telnetlib
import sys
Example import time

HOST = "192.168.61.1"
user = "admin"
password = "topsecret"
reading_timeout = 5
try:
tn = telnetlib.Telnet(HOST)
tn.read_until(b"Username: ",reading_timeout)
tn.write(user.encode('ascii') + b"\n")
tn.read_until(b"Password: ",reading_timeout)
tn.write(password.encode('ascii') + b"\n")
tn.write(b"terminal length 0\n")
time.sleep(1)
tn.write(b"sh run\n")
tn.write(b"exit\n")
print(tn.read_all().decode('ascii'))

except EOFError:
print("\n\n* Program Exiting...\n")
sys.exit()

finally:
print("Closing connection....")
tn.close()
paramiko module
Paramiko is a Python (2.6+, 3.3+) implementation of the SSHv2
protocol, providing both client and server functionality. While it
leverages a Python C extension for low level cryptography.
Paramiko itself is a pure Python interface around SSH
networking concepts.
import paramiko

ssh = paramiko.SSHClient()
ssh.connect('127.0.0.1', username=admin', password=password')
paramiko module
Host Keys
One of the complicating aspects of SSH authentication is host keys.
The default behavior with an SSHClient object is to refuse to connect to a host
(''paramiko.RejectPolicy'') who does not have a key stored in
your local 'known_hosts' file.

import paramiko

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

ssh.connect('127.0.0.1', username='admin', password='password')

paramiko module
Private Key File
If you do intend to authenticate using a private key, you need to create keyfile
named private_key_file with the key in it and add it to your Python project.

1 -----BEGIN DSA PRIVATE KEY-----

2 [contents omitted]
3 -----END DSA PRIVATE KEY-----
paramiko module
Running simple commands
SSH uses the same type of input, output, and error handles
Stderr - Errors are sent to standard error,
Stdout - output goes to standard out
Stdin - if you want to send data back to the application, you write it to standard in.

# Send the command

stdin, stdout, stderr = ssh.exec_command(command)

# Wait for the command to terminate

while not stdout.channel.exit_status_ready() and not
stdout.channel.recv_ready():
time.sleep(1)

stdoutstring = stdout.readlines()
stderrstring = stderr.readlines()
json Module
JSON
The json library can parse JSON from strings or files. The library parses JSON
into a Python dictionary or list. It can also convert Python dictionaries or lists
into JSON strings. >>> import json
.
>>> json dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
'["foo", {"bar": ["baz", null, 1.0, 2]}]
>>> print json .dumps("\"foo\bar")
"\"foo\bar"
>>> print json.dumps(u'\u1234')
"\u1234"
>>> print json.dumps('\\')
"\\
.
>>> print json dumps({"c": 0, "b": 0, "a": 0}, sort_keys=True)
{"a": 0, "b": 0, "c": 0}
>>> from StringIO import StringIO
>>> io = StringIO()
>>> json.dump(['streaming API'], io)
>>> io.getvalue()
'["streaming API"]'
json Example
Parsing JSON
Take the following string containing JSON data:
json_string = '{"first_name": "Guido", "last_name":"Rossum"}' You can also convert the following to
JSON:
It can be parsed like this:
d = {
import json 'first_name': 'Guido',
parsed_json = json.loads(json_string) 'second_name': 'Rossum',
'titles': ['BDFL', 'Developer'],
and can now be used as a normal dictionary: }

print(json.dumps(d))
print(parsed_json['first_name']) '{"first_name": "Guido", "last_name":
"Guido" "Rossum", "titles": ["BDFL",
"Developer"]}'
requests Module
Requests is the only Non-GMO HTTP library for Python, safe for
human consumption.
Requests allows you to send organic, grass-fed HTTP/1.1 requests, without
the need for manual labor. There's no need to manually add query strings to
your URLs, or to form-encode your POST data. Keep-alive and HTTP
connection pooling are 100% automatic, powered by urllib3, which is
embedded within Requests.

Pip Install Requests

To install Requests, simply run this simple command in your terminal of
choice:
$ pip install requests
 requests Example
#Begin by importing the Requests module:
import requests

#Now, lets try to get a webpage. For this example, lets get GitHubs public timeline
r = requests.get('https://api.github.com/events')

#Now, we have a Response object called r. We can get all the information we need from this object.
#Requests simple API means that all forms of HTTP request are as obvious. For example, this is how you make
an HTTP POST request:

r = requests.post('http://httpbin.org/post', data = {'key':'value'})

#Nice, right? What about the other HTTP request types: PUT, DELETE, HEAD and OPTIONS? These are all just as
#simple:

r = requests.put('http://httpbin.org/put', data = {'key':'value'})

r = requests.delete('http://httpbin.org/delete')
r = requests.head('http://httpbin.org/get')
r = requests.options('http://httpbin.org/get')

#Thats all well and good, but its also only the start of what Requests can do.
Summary