Introduction

In this series of articles I describe how you can write a Linux loadable kernel module (LKM) for an embedded Linux device. This is the second article in the series — please read “Writing a Linux Kernel Module — Part 1: Introduction” before moving on to this article, as it explains how to build, load and unload loadable kernel modules (LKMs). Such description is not repeated in this article.

Character Device Drivers

A character device typically transfers data to and from a user application — they behave like pipes or serial ports, instantly reading or writing the byte data in a character-by-character stream. They provide the framework for many typical drivers, such as those that are required for interfacing to serial communications, video capture, and audio devices. The main alternative to a character device is a block device. Block devices behave in a similar fashion to regular files, allowing a buffered array of cached data to be viewed or manipulated with operations such as reads, writes, and seeks. Both device types can be accessed through device files that are attached to the file system tree. For example, the program code that is presented in this article builds to become a device /dev/ebbchar, which appears on your Linux system as follows:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ lsmod
Module Size Used by
ebbchar 2754 0
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ls -l /dev/ebb*
crw-rw-rwT 1 root root 240, 0 Apr 11 15:34 /dev/ebbchar

This article describes a straightforward character driver that can be used to pass information between a Linux user-space program and a loadable kernel module (LKM), which is running in Linux kernel space. In this example, a C user-space application sends a string to the LKM. The LKM then responds with the message that was sent along with the number of letters that the sent message contains. Later in the article I describe why we need to solve synchronization problems that arise with this approach, and I provide a version of the program that uses mutex locks to provide a solution.

Before describing the source code for the driver in this article, there are a few concepts that need to be discussed, such as device driver major and minor numbers, and the File Operations data structure.

Major and Minor Numbers

Device drivers have an associated major and minor number. For example, /dev/ram0 and /dev/null are associated with a driver with major number 1, and /dev/tty0 and /dev/ttyS0 are associated with a driver with major number 4. The major number is used by the kernel to identify the correct device driver when the device is accessed. The role of the minor number is device dependent, and is handled internally within the driver. You can see the major/minor number pair for each device if you perform a listing in the /dev directory. For example:
molloyd@beaglebone:/dev$ ls -l
crw-rw---T 1 root i2c 89, 0 Jan 1 2000 i2c-0
brw-rw---T 1 root disk 1, 0 Mar 1 20:46 ram0
brw-rw---T 1 root floppy 179, 0 Mar 1 20:46 mmcblk0
crw-rw-rw- 1 root root 1, 3 Mar 1 20:46 null
crw------- 1 root root 4, 0 Mar 1 20:46 tty0
crw-rw---T 1 root dialout 4, 64 Mar 1 20:46 ttyS0

Character devices are identified by a ‘c‘ in the first column of a listing, and block devices are identified by a ‘b‘. The access permissions, owner, and group of the device is provided for each device. Regular user accounts on the BeagleBone are members of some of these groups and therefore have permissions to use the i2c-0 and ttyS0 devices etc. See:
molloyd@beaglebone:/dev$ groups
molloyd dialout cdrom floppy audio video plugdev users i2c spi

The device that is developed in this article appears as a device (/dev/ebbchar) in the /dev directory.

It is possible to manually create a block or character device file entry and later associate it with your device (e.g., sudo mknod /dev/test c 92 1), but this approach is prone to problems. One such problem is that you have to ensure that the number you choose (e.g., 92 in this case) is not already in use. On the BeagleBone, you could examine the file /usr/src/linux-headers-3.8.13-bone70/include/uapi/linux/major.h for a list of all system device major numbers. However, a device that idenfies a “unique” major number using this approach would not be very portable, as the major number of the device could clash with that of another device on another Linux SBC or Linux distribution. The code that is provided in this article automatically identifies an appropriate major number to use.

The File Operations Data Structure

The file_operations data structure that is defined in /linux/fs.h holds pointers to functions (function pointers) within a driver that allows you to define the behavior of certain file operations. For example, Listing 1 is a segment of the data structure from /linux/fs.h. The driver in this article provides an implementation for the read, write, open, and release system call file operations. If you do not provide an implementation for one of the entries in this data structure then it will simply point to NULL, making it inaccessible. Listing 1 is somewhat intimidating, given the number of operations available. However, to build the ebbchar LKM we only need to provide an implementation for four of the entries. Therefore, Listing 1 is provided mainly as a reference that you can use if you need to provide additional functionality within the driver framework.

Listing 1: The file_operations Data Structure of the /linux/fs.h (Segment)

For further information, there is an excellent guide on the File Operations data structure at: Kernel.org Virtual File Systems

Get Source Code

Source Code for this Discussion

Get Source Code

All of the code for this discussion is available in the GitHub repository for the book Exploring BeagleBone. The code can be viewed publicly at: the ExploringBB GitHub Kernel Project directory, and/or you can clone the repository on your BeagleBone (or other Linux device) by typing:

The /extras/kernel/ebbchar directory is the most important resource for the next part of this article. The auto-generated Doxygen documentation for these code examples is available in HTML format and PDF format.

The Device Driver Source Code

The source code for the ebbchar device driver is provided in Listing 2. Similar to the code in the first article in this series, there is an init() function and an exit() function. However, there are additional file_operations functions that are required for the character device:

  • dev_open(): Called each time the device is opened from user space.
  • dev_read(): Called when data is sent from the device to user space.
  • dev_write(): Called when data is sent from user space to the device.
  • dev_release(): Called when the device is closed in user space.

Drivers have a class name and a device name. In Listing 2, ebb (Exploring BeagleBone) is used as the class name, and ebbchar as the device name. This results in the creation of a device that appears on the file system at /sys/class/ebb/ebbchar.

Listing 2: The ebbchar LKM (/extras/kernel/ebbchar/ebbchar.c)

In addition to the points described by the comments in Listing 2, there are some additional points:

  • This code has a fixed message size of 256 characters — this will be improved in later articles though the dynamic allocation of memory.
  • This code is not multi-process safe — that is addressed later in this article.
  • The ebbchar_init() function is much longer than the last article. That is because it is now automatically allocating a major number to the device, registering the device class, and registering the device driver. Importantly, you will notice that if anything goes wrong that the code carefully “backs out” of the successful operations. To achieve this I have repeated code (which I always dislike), but the alternative is to use goto statements, which is even less palatable (albeit slightly tidier).
  • The PTR_ERR() is a function that is defined in linux/err.h that retrieves the error number from the pointer.
  • The functions sprintf() and strlen() are available in the kernel through the inclusion of linux/kernel.h and indirectly through linux/string.h respectively. The functions in string.h are architecture dependent.

The next step is to build this code into a kernel module.

Building and Testing the LKM

A Makefile is required to build the LKM, as provided in Listing 3. This Makefile is very similar to the Makefile in the first article in the series, with the exception that it also builds a user-space C program that interacts with the LKM.

Listing 3: The Makefile for the LKM and the User-space Program (/extras/kernel/ebbchar/Makefile)

Listing 4 is a short program that requests a string from the user, and writes it to the /dev/ebbchar device. After a subsequent key press (ENTER) it then reads the response from the device and displays it in the terminal window.

Listing 4: The User-space Program for Testing the LKM (/extras/kernel/ebbchar/testebbchar.c)

In addition to the points described by the comments in Listing 4, there are some additional points:

  • The %[^\n]%*c uses the scanset specifiers, which are represented by %[] to use the ^ character to stop reading after the first occurrence of the \n character. In addition, the %*c ignores the trailing character, ensuring that the subsequent getchar() function works as required. Essentially, the scanf() code just reads in a sentence. If scanf() was used with a regular %s call then the string would terminated at the first occurrence of the space character.
  • The getchar() allows the program to pause at that point until the ENTER key is pressed. This is necessary to demonstrate a problem with the current code formulation.
  • The program then reads the response from the LKM and displays it in the terminal window.

All going well, the process to build the kernel module should be straightforward, provided that you have installed the Linux headers as described in the first article. The steps are as follows:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ make
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ls -l *.ko
-rw-r--r-- 1 molloyd molloyd 7075 Apr 8 19:04 ebbchar.ko
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ls -l test
-rwxr-xr-x 1 molloyd molloyd 6342 Apr 8 19:23 test
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo insmod ebbchar.ko
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ lsmod
Module Size Used by
ebbchar 2521 0

The device is now present in the /dev directory, with the following attributes:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ cd /dev
molloyd@beaglebone:/dev$ ls -l ebb*
crw------- 1 root root 240, 0 Apr 8 19:28 ebbchar

You can see that the major number is 240 — this is automatically assigned by the code in Listing 2.

We can then use the testebbchar program (Listing 4) to test that the LKM is working correctly. For the moment, the test program must be executed with root privileges — that issue is addressed shortly.
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo insmod ebbchar.ko
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo ./test
Starting device test code example...
Type in a short string to send to the kernel module:
This is a test of the ebbchar LKM
Writing message to the device [This is a test of the ebbchar LKM].
Press ENTER to read back from the device...
Reading from the device...
The received message is: [This is a test of the ebbchar LKM(33 letters)]
End of the program
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo rmmod ebbchar

The printk() output can be viewed by examining the kernel logs as follows:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo tail -f /var/log/kern.log
Apr 11 22:24:50 beaglebone kernel: [358664.365942] EBBChar: Initializing the EBBChar LKM
Apr 11 22:24:50 beaglebone kernel: [358664.365980] EBBChar: registered correctly with major number 240
Apr 11 22:24:50 beaglebone kernel: [358664.366061] EBBChar: device class registered correctly
Apr 11 22:24:50 beaglebone kernel: [358664.368383] EBBChar: device class created correctly
Apr 11 22:25:15 beaglebone kernel: [358689.812483] EBBChar: Device has been opened 1 time(s)
Apr 11 22:25:31 beaglebone kernel: [358705.451551] EBBChar: Received 33 characters from the user
Apr 11 22:25:32 beaglebone kernel: [358706.403818] EBBChar: Sent 45 characters to the user
Apr 11 22:25:32 beaglebone kernel: [358706.404207] EBBChar: Device successfully closed
Apr 11 22:25:44 beaglebone kernel: [358718.497000] EBBChar: Goodbye from the LKM!

You can see that 33 characters are sent to the LKM but 45 characters are returned — this is due to the addition of the 12 characters “(33 letters)” to the string data that specifies the length of the string that was originally sent. This addition is performed as a test in order to be certain that the code is sending and receiving unique data.

There are two significant problems with the current LKM. The first is that the LKM device can only be accessed with superuser permissions, and the second is that the current LKM is not multi-process safe.

User Access to the Device using Udev Rules

Throughout this article, the program that interfaces to the LKM device is executed using sudo. It would be very useful to set up our LKM device so that it can be accessed by a particular user or group, while still protecting the file system. To address this issue, you can use an advanced feature of Linux called udev rules that enables you to customize the behavior of the udevd service. This service gives you some user-space control over devices on your Linux system.

For example, to give user-level access to the ebbchar device, the first step is to identify the sysfs entry for the device. You can achieve this by using a simple find:
root@beaglebone:/sys# find . -name "ebbchar"
./devices/virtual/ebb/ebbchar
./class/ebb/ebbchar
./module/ebbchar

We then need to identify the KERNEL and SUBSYSTEM values about which to write the rule. You can use the udevadm command to perform this task:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ udevadm info -a -p /sys/class/ebb/ebbchar
Udevadm info starts with the device specified by the devpath and then walks up the chain of parent
devices. It prints for every device found, all possible attributes in the udev rules key format. A
rule to match, can be composed by the attributes of the device and the attributes from one single
parent device.
  looking at device '/devices/virtual/ebb/ebbchar':
  KERNEL=="ebbchar"
  SUBSYSTEM=="ebb"
  DRIVER==""

The rules are contained in the /etc/udev/rules.d directory. A new rule can be added as a file using these values, where the file begins with a priority number. Using a name such as 99-ebbchar.rules creates a new rule with the lowest priority, so that it does not interfere with other device rules. The rule can be written as in Listing 5 and placed in the /etc/udev/rules.d directory as follows:
molloyd@beaglebone:/etc/udev/rules.d$ ls
50-hidraw.rules 50-spi.rules 60-omap-tty.rules 70-persistent-net.rules 99-ebbchar.rules
molloyd@beaglebone:/etc/udev/rules.d$ more 99-ebbchar.rules
#Rules file for the ebbchar device driver
KERNEL=="ebbchar", SUBSYSTEM=="ebb", MODE="0666"

Listing 5: Udev rules file for the ebbchar device driver (/extras/kernel/ebbchar/99-ebbchar.rules)

Once the rules file is added to your system, you can test it using:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo insmod ebbchar.ko
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ls -l /dev/ebbchar
crw-rw-rwT 1 root root 240, 0 Apr 9 00:44 /dev/ebbchar

You can see that user and group now have the permissions required to read from and write to this device. Interestingly, the sticky bit is also set. The sticky bit usually means that write permission is not sufficient to delete files. Therefore, in the /tmp directory any user can create files, but no user can delete another user’s files. The sticky bit is represented by a capital T in the final character place. This usually appears as a lower-case t unless the executable (x) bit for others is set; however, when the x bit is not set it appears as a capital T. However, it is not entirely clear why the sticky bit is being set by udev — it appears to be unusual to udev rules under Debian. At this point the test application can be executed without requiring superuser permissions.

The strace Command

The strace command is a very useful debugging tool that can execute a program in order to intercept and record the system calls that it performs. The system call name, the arguments passed, and the resulting return value are all visible, which makes it a valuable tool for solving runtime issues. Importantly, you do not need the source code for the executable in order to view the output of strace. For example, you can utilize strace on your user-space application in order to view the communication between the user-space program and the kernel module, which results in the following for the test application:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo apt-get install strace
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ strace -v
usage: strace [-dffhiqrtttTvVxx] [-a column] [-e expr] … [-o file]
  [-p pid] … [-s strsize] [-u username] [-E var=val] …
  [command [arg …]]
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo strace ./test
execve("./test", ["./test"], [/* 15 vars */]) = 0

write(1, "Starting device test code exampl"..., 37Starting de…) = 37
open("/dev/ebbchar", O_RDWR) = 3
write(1, "Writing message to the device [T"..., 60Writing message …) = 60
write(3, "Testing the EBBChar device", 26) = 26
write(1, "Reading from the device...\n", 27Reading from the device…) = 27
read(3, "", 100) = 0
write(1, "The received message is: [Testin"..., 66The received …) = 66
write(1, "End of the program\n", 19End of the program) = 19
exit_group(0) = ?

The system call output gives us impressive insight into the communication that takes place between the user-space program test and the /dev/ebbchar device driver.

LKM Synchronization Problems

There is a serious problem with the LKM that is described in Listing 2. In the first article in this series I pointed out that LKMs do not execute sequentially and that they can be interrupted. Those facts have important consequences for the code that is written in this article, which can be demonstrated as follows:

Step 1: At the first terminal window shell you can execute the test application, but do not allow it to run to completion (by not pressing ENTER when prompted), as follows:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ./test
Starting device test code example...
Type in a short string to send to the kernel module:
This is the message from the first terminal window
Writing message to the device [This is the message from the first terminal window].
Press ENTER to read back from the device...

Step 2: Then at a second terminal window shell you can execute the same test application simultaneously as a second process on the Linux device. For example:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ./test
Starting device test code example...
Type in a short string to send to the kernel module:
This is the message from the second terminal window
Writing message to the device [This is the message from the second terminal window].
Press ENTER to read back from the device...

Step 3: You can then return to the first terminal window shell and press ENTER to run the program to completion, which results in the following output (shaded output is the repeated output from Step 1):
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ./test
Starting device test code example...
Type in a short string to send to the kernel module:
This is the message from the first terminal window
Writing message to the device [This is the message from the first terminal window].
Press ENTER to read back from the device...

Reading from the device...
The received message is: [This is the message from the second terminal window(51 letters)]
End of the program
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$

You can see that the received message is actually the message that was sent by the test application from Step 2, which is running in the second terminal window shell (not the first as might be expected). This is because the message that was sent in Step 2 overwrote the string message that was being stored by the LKM as a result of Step 1.

Step 4: You can return to the second terminal shell and run it to completion by pressing ENTER, which results in (the shaded output is the repeated output from Step 2):
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ./test
Starting device test code example...
Type in a short string to send to the kernel module:
This is the message from the second terminal window
Writing message to the device [This is the message from the second terminal window].
Press ENTER to read back from the device...

Reading from the device...
The received message is: []
End of the program
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$

No string is received. That is because the LKM is not storing any messages at that point in time. It has already delivered the stored message to the first terminal window test application and reset the buffer index to 0.

Adding Mutex Locks

The Linux kernel provides a full implementation of semaphores — a data type (struct semaphore) that is used for controlling access by multiple processes to a shared resource. The easiest way to use this semaphore code within the kernel is to use mutexes, as there is a full set of helper functions and macros available.

A simple way to prevent the problems described above is to prevent two processes from using the /dev/ebbchar device at the same time. A mutex is a lock that can set (put down) before a process begins using a shared resource. The lock can then be released (brought up) when the process is finished using the shared resource. When the lock has been set, no other process can access the locked code region. Once the mutex lock has been released by the process that locked it, the shared region of code is once again available to be accessed by the other process, which in turn locks the resource.

There are only very minor changes required to the code in order to implement mutex locks. An outline of these changes is provided in Listing 6 below:

Listing 6: Outline of the Changes to the ebbchar.c Program Code to Introduce mutex Locks

The full source code example is available in the /extras/kernel/ebbcharmutex/ directory. The final LKM is called ebbcharmutex.c. If you now perform the same test on the code that contains the mutex locks, you will observe a different behavior. For example:

Step 1: Using the first terminal window shell you can load the module and execute the test application, which results in the following output:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbcharmutex$ sudo insmod ebbcharmutex.ko
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbcharmutex$ lsmod
Module Size Used by
ebbcharmutex 2754 0
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbcharmutex$ ls -l /dev/ebb*
crw-rw-rwT 1 root root 240, 0 Apr 12 15:26 /dev/ebbchar
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbcharmutex$ ./test
Starting device test code example...
Type in a short string to send to the kernel module:
Testing the ebbchar LKM that has mutex code
Writing message to the device [Testing the ebbchar LKM that has mutex code].
Press ENTER to read back from the device...

Step 2: Then using the second terminal window shell you can attempt to execute the test application to create the second process:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbcharmutex$ ./test
Starting device test code example...
Failed to open the device...: Device or resource busy
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbcharmutex$

As expected and required, the second process fails to access the device.

Step 3: Returning to the first terminal window, the program can be allowed to run to completion by pressing ENTER:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbcharmutex$ ./test
Starting device test code example...
Type in a short string to send to the kernel module:
Testing the ebbchar LKM that has mutex code
Writing message to the device [Testing the ebbchar LKM that has mutex code].
Press ENTER to read back from the device...

Reading from the device...
The received message is: [Testing the ebbchar LKM that has mutex code(43 letters)]
End of the program

At this point the second terminal window shell can now execute the test program, whereupon it will acquire the mutex lock and run correctly.

Conclusion

DoxygenHTML DoxygenPDF
Click for the HTML and PDF version of the auto-generated Doxygen code documentation

There are several different issues described in this article. The important outcomes of this article are that:

  • You can now create your own device such as /dev/ebbchar, which you can write information to and read information from. This is important, as it provides a bridge between the Linux user space and the Linux kernel space. It enables you to develop advanced drivers, such as communication devices, which can be controlled by C code that is running in user space.
  • You can appreciate why it is important to be aware of the synchronization issues that can arise with kernel module programming.
  • You can use udev rules to alter the properties of a device as it is loaded.

The next article in this series examines how you can interface to GPIO devices, such as physical button and LED circuits, enabling you to write your own custom drivers for embedded Linux applications that interface to custom hardware. You can read that article here: Writing a Linux Kernel Module — Part 3: Interfacing to GPIOs (coming soon!). Finally, Listing 7 and Listing 8 are provided for your reference — they provide a list of the standard error states that are used in this series of articles, along with their associated error numbers and descriptions.

Listing 7: Error States for LKM Development (/usr/include/asm-generic/errno-base.h)

Listing 8: Error States for LKM Development (/usr/include/asm-generic/errno.h)