Advanced Linux Programming M.

Barbeau

Writing Device Drivers Under Linux

Outline:

1. Introduction
2. User-mode drivers
o Access to I/O map devices
o Access to memory map devices
o Finding devices
o Data link access with Linux PF_PACKET
3. Kernel-mode drivers
o Block/char device
o Polling/interrupt mode
o Major/minor number
o Registration of a device driver
o Interrupt handling

1. Introduction

What is a device driver? A software abstraction of a device (peripheral or

chip)!

Types of drivers: user-mode / kernel-mode.

2. User-mode drivers

Direct access to devices within a user application

The x86 architecture has two memory maps: one for memory and one for
I/O.

Access to I/O Map devices

Opening the I/O map

Access to the I/O map under Linux: root user, system calls ioperm(2) and
iopl(2).

April 1, 2002 1
Advanced Linux Programming M. Barbeau

ioperm(): opens a block of consecutive addresses in the I/O map, parameters

are a base address and a length, in bytes, works for ports located from
0x0000 to 0x03ff.

iopl(): opens the entire I/O map (note: I/O mapped PCI devices are located
above 0x03ff.)

Reading/Writing the I/O map

System calls: inb(), inw(), and inl(), for byte, word and long reading; outb(),
outw(), and outl(), for byte, word, long writing. (see sys/io.h)

Access to memory-map devices

Abstraction is device /dev/mem and system call mmap().

Opening the memory map

Open /dev/mem, a file descriptor is returned.

Call mmap() with the file descriptor, offset and size (in the file). An address
mapped to the physical address space is returned. Address can be cast to a
data type and used like a normal pointer.

Where to find the address of a device? From manufacturer's documentation!

From directory /proc/bus!

Example:

more /proc/bus/pci/device

0070 10d90531 b 00002001 4c000000 00000000

00000000 00000000 00000000 00000000 00000100 00000100
00000000 00000000 00000000 00000000 00040000 tulip

Vendor ID: 10d9 (managed by PCI SIG), Device ID: 0531 (vender's
choice), also printed on the card.

Configured Base Address Register zero (BAR0): 00002001

April 1, 2002 2
Advanced Linux Programming M. Barbeau

Address of BAR0 is 0x2000 least significant bit 1 (0) indicates I/O

(memory) map.

Pitfalls: Multiple access must be thread safe. Disastrous effects of writing

wrong I/Os. Improper access may freeze the computer.

Data link access with Linux PF_PACKET

This section contains code extracted from a user level Wireless (Ethernet)
LAN driver. Only the data link access is addressed. The complete example
available at:

http://www.scs.carleton.ca/~barbeau/Courses/SETP/index.html

Data:
// wireless interface configuration
struct Ifconfig
{
int sockid; // socket descriptor
int ifindex; // index of the interface
WLANAddr hwaddr; // MAC address
int mtu; // maximum transmission unit
};

// label of device, e.g. "eth0"

char device[MAX_NAME_LEN];
// interface configuration
Ifconfig ifconfig;
// mutex to prevent concurrent transmission
pthread_mutex_t transmutex;
// ID of the receive thread
pthread_t threadID;

// constructor
WLANProtocol::WLANProtocol(char * device, Outcome & retval)
{
// undefined the socket descriptor
ifconfig.sockid = -1;

// copy the device name

strcpy(this->device, device);

// initialize the mutex

pthread_mutex_init(&transmutex, NULL);

April 1, 2002 3
Advanced Linux Programming M. Barbeau

retval = init();
}

// wrapper for an invocation of the Receive() operation

static void * startReceive(void * arg)
{
// make the thread asynchronously cancellable
pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);

((WLANProtocol *)arg)->Receive();
pthread_exit(NULL); // this statement should never be executed
}

Outcome WLANProtocol::init()
{
// (1) create device level socket
// - PF_PACKET : low level packet interface
// - SOCK_RAW : raw packets including link level header
// - ETH_P_ALL : all frames will be received
if ((ifconfig.sockid = socket(PF_PACKET, SOCK_RAW,
htons(ETH_P_ALL))) == -1)
{
aLog.log("WLANProtocol, cannot open socket: %s",
strerror(errno));
return NOK;
}

// (2) fetch the interface index

struct ifreq ifr;
strcpy(ifr.ifr_name, device);
if (ioctl(ifconfig.sockid, SIOGIFINDEX, &ifr) < 0)
{
aLog.log("WLANProtocol, failed to fetch ifindex: %s",
strerror(errno));
return NOK;
}
ifconfig.ifindex = ifr.ifr_ifindex;
aLog.log("WLANProtocol, ifindex: %d", ifconfig.ifindex);

// (3) fetch the hardware address

if (ioctl(ifconfig.sockid, SIOCGIFHWADDR, &ifr) == -1)
{
aLog.log("WLANProtocol, failed to fetch hardware address: %s",
strerror(errno));
return NOK;
}
memcpy(&ifconfig.hwaddr.data, &ifr.ifr_hwaddr.sa_data,
WLAN_ADDR_LEN);
aLog.log("WLANProtocol, hwaddr: %s", ifconfig.hwaddr.wlan2asc());

// (4) fetch the MTU

if (ioctl(ifconfig.sockid, SIOCGIFMTU, &ifr) == -1)
{
aLog.log("WLANProtocol, failed to the MTU: %s", strerror(errno));
return NOK;
}

April 1, 2002 4
Advanced Linux Programming M. Barbeau

ifconfig.mtu = ifr.ifr_mtu;
aLog.log("WLANProtocol, MTU: %d", ifconfig.mtu);

// (5) add the promiscuous mode

struct packet_mreq mr;
memset(&mr,0,sizeof(mr));
mr.mr_ifindex = ifconfig.ifindex;
mr.mr_type = PACKET_MR_PROMISC;
if (setsockopt(ifconfig.sockid, SOL_PACKET, PACKET_ADD_MEMBERSHIP,
(char *)&mr, sizeof(mr)) < 0)
{
aLog.log("WLANProtocol, failed to add the promiscuous mode: %d",
strerror(errno));
return NOK;
}

// (6) bind the socket to the interface (device)

struct sockaddr_ll sll;
memset(&sll, 0, sizeof(sll));
sll.sll_family = AF_PACKET;
sll.sll_ifindex = ifconfig.ifindex;
sll.sll_protocol = htons(ETH_P_ALL);
if (bind(ifconfig.sockid, (struct sockaddr*)&sll, sizeof(sll)) < 0)
{
aLog.log("WLANProtocol, failed to bind the socket: %d",
strerror(errno));
return NOK;
}

// (7) create attribute of a received thread (detached)

pthread_attr_t attr;
if (pthread_attr_init(&attr) != 0)
{
aLog.log("WLANProtocol, failed to create thread attribute!");
return NOK;
}
if (pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED)!=0)
{
aLog.log("WLANProtocol, failed to set thread attribute!");
return NOK;
}

// (8) create a receive thread

int error =
pthread_create(&threadID, &attr, startReceive, (void *) this);
if (error)
{
aLog.log("WLANProtocol, failed to create thread %s!",
strerror(error));
return NOK;
}

return OK;
}

// receive data over a socket

void WLANProtocol::Receive()

April 1, 2002 5
Advanced Linux Programming M. Barbeau

{
Message * msg; // received message
Outcome result;
unsigned char * buff; // pointer to received data
unsigned int i; // frame length
struct sockaddr_ll from; // source address of the message
socklen_t fromlen = sizeof(struct sockaddr_ll);
int error;

// (1) initialisation
#define THREADPERMSG
#ifdef THREADPERMSG
// (1.1) thread per message
aLog.log("WLANProtocol, thread per message model");

// actualize the DemuxArg class

typedef DemuxArg<ProtocolUI, Message> DemuxArgClass;

// create the attribute of a msg thread (detached)

pthread_attr_t attr;
if (pthread_attr_init(&attr) != 0)
{
aLog.log("WLANProtocol, failed to create thread attribute!");
assert(NULL); // fail
}
if (pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED)!=0)
{
aLog.log("WLANProtocol, failed to set thread attribute!");
assert(NULL); // fail
}
#else
// (1.2) single thread model
aLog.log("WLANProtocol, single thread model");

// only one message created and reused

msg = new(nothrow) Message(ifconfig.mtu, result);
if (msg == NULL || result == NOK)
{
aLog.log("WLANProtocol, failed to create msg!");
assert(NULL); // fail
}
#endif

// (2) infinite loop

while (true)
{
#ifdef THREADPERMSG
// (3.1) thread per message, one mssg is created per rcvd frame
msg = new(nothrow) Message(ifconfig.mtu, result);
if (msg == NULL || result == NOK)
{
aLog.log("WLANProtocol, failed to create msg!");
// sleep for 10 milliseconds before re-trying
usleep(10000);
continue;
}
#else

April 1, 2002 6
Advanced Linux Programming M. Barbeau

// (3.2) clear the contents of the message

msg->Reset();
#endif

// (4) get a pointer to the message buffer

buff = msg->Push(ifconfig.mtu);
assert(buff);

// (5) loop until a non-empty frame has been received on "device"

while (true)
{
// (6) wait and receive a frame
fromlen = sizeof(from);
i = recvfrom(ifconfig.sockid, buff, ifconfig.mtu, 0,
(struct sockaddr *) &from, &fromlen);
if (i == -1)
{
aLog.log("WLANProtocol, cannot receive data: %s",
strerror(errno));
// sleep for 10 milliseconds before re-trying
usleep(10000);
}
break; // exit the loop
}
aLog.log("WLANProtocol, frame received");

// (7) a non-empty frame received, truncate non-used tail

msg->Truncate(ifconfig.mtu-i);

// (8) pass the message to the high-level protocol

#ifdef THREADPERMSG
// (8.1) thread per message, a thread for each mssg is created

// build the arguments of a thread

DemuxArgClass *aDemuxArg = new(nothrow) DemuxArgClass(this, msg);
if (aDemuxArg == NULL)
{
aLog.log("WLANProtocol, failed to create argument");
delete msg;
}

// create a thread
// the thread is reponsible for deleting "aDemuxArg" and "msg"
pthread_t msg_tid; // identifier of the thread
error = pthread_create(
&msg_tid, &attr, demux<DemuxArgClass, ProtocolUI>,
(void *)aDemuxArg);
if (error)
{
aLog.log("WLANProtocol, failed to create thread: %s", error);
delete msg;
delete aDemuxArg;
}
#else
// (8.2) single thread model
xDemux(NULL, msg);
#endif

April 1, 2002 7
Advanced Linux Programming M. Barbeau

}
}

// destructor
WLANProtocol::~WLANProtocol()
{
aLog.log("WLANProtocol, shutdown");
// cancel the receive thread
int error = pthread_cancel(threadID);
if (error != 0)
{
aLog.log("WLANProtocol, failed to cancel thread");
}

// close the socket

if (ifconfig.sockid != -1) close(ifconfig.sockid);
}

Sending a frame:

// sends a frame
Outcome WLANSession::xPush(Message *msg)
{
assert(msg);

if (ifconfig->sockid == -1)
{
aLog.log("WLANSession::xPush(), sockid is NULL");
return NOK;
}

// (1) prepend the header onto the message

unsigned char *buff = msg->Push(WLAN_HEADER_LEN);
memmove(buff, &hdr, WLAN_HEADER_LEN);

// (2) set the from address

struct sockaddr_ll from;
int fromlen = sizeof(from);
from.sll_family = AF_PACKET;
from.sll_ifindex = ifconfig->ifindex;

// (3) lock the mutex on transmission

pthread_mutex_lock(transmutex);

// (4) send a frame

int sentlen = sendto(
ifconfig->sockid, buff, msg->Size(), 0,
(sockaddr *) &from, fromlen);

April 1, 2002 8
Advanced Linux Programming M. Barbeau

// (5) lock the mutex on transmission

pthread_mutex_unlock(transmutex);

if (sentlen == -1 )
{
aLog.log("WLANSession::xPush(), sendto failed");
return NOK;
}
return OK;
}

3. Kernel-mode drivers

Block device
A block device is something that can host a file system such as a disk. A
block device can only be accessed as multiples of a block, where a block is
usually one kilobyte of data.
Buffering is used.
Character device
A character device is one that can be accessed like a file, and a char driver is
in charge of implementing this behavior. This driver implements the open,
close, read and write system calls. The console and parallel ports are
examples of char devices.
No buffering.

April 1, 2002 9
Advanced Linux Programming M. Barbeau

Polling mode I/O

A process performs a system call (enters kernel-mode), device
driver starts the device, periodic inspection of the device by the
device driver, then the system call returns.

Polling mode I/O

Advantages: No other processes are running while a critical operation is

performed, no interrupt management overhead, and no interrupt support
required from the hardware.

Interrupt mode

A device driver, an IRQ, interrupt service routine (ISR), and a device bottom
half are involved.

A process performs a system call and blocks in state

TASK_INTERRUPTIBLE.

The device driver starts the device and performs the I/O, an IRQ is issued,
ISR is executed, bottom half is marked for execution or a function is put in
the task queue.

The ret_from_sys_call code block is executed (runs after any system call),
the process state is switched to TASK_RUNNING, and scheduler is
invoked.

Role of the ISR: completion of the I/O (i.e. check the device status, data
transfer from device to primary memory) and marks bottom half for later
execution (with the mark_bh() kernel function).

April 1, 2002 10
Advanced Linux Programming M. Barbeau

Role of the bottom half: postponed interrupt handling processing (initialized

with the init_bh() kernel function).

Interrupt mode

Advantage: While an I/O is being performed, the CPU can be allocated to

other processes.

Major/minor number

Major number: standard identity of a device driver (e.g. 2 for floppy, 3 for
IDE hard disk, 6 for a parallel interface, see file /usr/include/linux/major.h).

Minor number: identity of physical device, a driver can drive several

physical devices (e.g. 0 for 1st disk, 1 for 2nd disk).

Registration of device driver

Device drivers in Linux are known as modules and can be loaded
dynamically into the kernel using the insmod command.
A single module can be compiled and linked to the kernel.
When a module is loaded or when a machine is booted, a device driver is
registered with the OS (using a kernel function). A major number, a device
name, and a list of functions supported by the driver (an instance of struct
file_operations). Examples of supported functions: open(), read(), write(),
and ioctl().
Names and operations are stored in kernel tables (chrdevs[]/blkdevs[] for
character/block driver).

April 1, 2002 11
Advanced Linux Programming M. Barbeau

Interrupt handling
The request_irq() kernel function associates an ISR with an IRQ. Fails if an
ISR is already associated to the IRQ.

Demultiplexing of IRQs: several devices on the same IRQ, a master ISR de-
multiplexes the IRQ to several ISRs (hardware is polled to determine the
cause of the IRQ and the ISR is selected).

Organization of a device driver

APIs to character device drivers and block device drivers are slightly
different.

Each device is represented by a special file in /dev. Created by the kernel a

boot time or with:

mknod /dev/<dev_name> <type> <major_number> <minor_number>

<type> is c for character device and b for block device.

Interface to a device driver (file /usr/include/linux/fs.h):

struct file_operations {
struct module *owner;
loff_t (*llseek) (struct file *, loff_t, int);
ssize_t (*read) (struct file *, char *, size_t, loff_t *);
ssize_t (*write) (struct file *, const char *, size_t, loff_t *);
int (*readdir) (struct file *, void *, filldir_t);
unsigned int (*poll) (struct file *, struct poll_table_struct *);
int (*ioctl) (struct inode *, struct file *, unsigned int,
unsigned long);
int (*mmap) (struct file *, struct vm_area_struct *);
int (*open) (struct inode *, struct file *);
int (*flush) (struct file *);
int (*release) (struct inode *, struct file *);
int (*fsync) (struct file *, struct dentry *, int datasync);
int (*fasync) (int, struct file *, int);
int (*lock) (struct file *, int, struct file_lock *);
ssize_t (*readv) (struct file *, const struct iovec *,
unsigned long, loff_t *);
ssize_t (*writev) (struct file *, const struct iovec *,
unsigned long, loff_t *);
ssize_t (*writepage) (struct file *, struct page *, int,
size_t, loff_t*, int);
};

April 1, 2002 12
Advanced Linux Programming M. Barbeau

Look the same as an interface to the file system! A device driver may
support only a subset of the operations.

Example: Disk driver (pseudo code, full code in drivers/block/hd.c)

static struct file_operations hd_fops =

{
NULL, /* lseek - default */
block_read, /* read - general block-dev read */
block_write, /* write - general block-dev write */
NULL, /* readdir */
NULL, /* select */
hd_ioctl, /* ioctl */
NULL, /* mmap */
hd_open, /* open */
hd_release, /* release */
block_fsync /* fsync */
}

Initialization function: called in the kernel initialization code (e.g. tty_init())

or when the module is loaded (i.e. init_module()).

Kernel initialization code: Function chr_dev_init() in file

drivers/char/mem.c, for character devices, or function blk_dev_init() in file
drivers/block/ll_rw_bl.c, for block devices.

Example: Disk driver (pseudo code)

int hd_init(void)
{
/* registration of the interface with the kernel */
if (register_blkdev(MAJOR_NR, "hd", &hd_fops)
{
printk("hd: unable to get major %d for harddisk\n",
MAJOR_NR);
return -1;
}
blk_dev[MAJOR_NR].request_fn = DEVICE_REQUEST;
read_ahead[MAJOR_NR] = 8; /*8 sectors (4kB) read-ahead*/
hd_gendisk.next = gendisk_head;
gendisk_head = &hd_gendisk;
timer_table[HD_TIMER].fn = hd_times_out;
return 0;
}

April 1, 2002 13
Advanced Linux Programming M. Barbeau

Implementation of operations:

static int hd_ioctl(struct inode * inode,

struct file *file, unsigned int cmd, unsigned long arg)
{ ... }

The hd_open() function: called when a user-space process calls open() for
device /dev/hdi.

static int hd_open(struct inode * inode, struct file *filp)

{
int target;
/* get minor number of device */
target = DEVICE_NR(inode->i_rev);
/* check if number is acceptable */
if (target >= NR_HD)
{
return -ENODEV;
}
/* sleep while the target is busy */
while (busy[target])
{
sleep_on(&busy_wait);
}
/* increase the reference count */
access_count[target]++;

return 0;
}

static void hd_release(struct inode, struct file * file)

{ ... }

Further Reading

A good overview of user-mode device drivers can be found in:

• B. Nakatani, User-Mode Device Drivers, Embedded Linux Journal,
March/April 2002, pp. 47-48.
A good overview of kernel-mode device drivers can be found in:
• G. Nutt, Kernel Projects for Linux, Addison Wesley, 2001.
A detailed treatment of drivers for Linux can be found in:
• A. Rubini, Linux Device Drivers, O'Reilly, 2001.

April 1, 2002 14