Grenoble INP ENSIMAG

cole Nationale Suprieure dInformatique et de Mathmatiques Appliques

Internship report
(Second Year Embedded Software Engineer Internship)
At Oxbern SAS

Development and implementation of the

CrystalCubes firmware, a 3D LED Display

El Khadiri Yassine
2A Option SLE
June 22 2015 September 18 2015

2
Pierre Schefler
Oxbern

Olivier Alphand
Ensimag

3 parvis Louis Nel

38016 Grenoble Cedex 1

681 Rue de la Passerelle

38400 Saint-Martin-dHres

CONTENTS

Contents
1 Introduction

2 General Overview
2.1 Oxbern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Crystal Cube . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5
5
6

3 Mission Objectives
3.1 Problems with the current prototype . . . . . . . . . . . . . . . . . . . . .
3.2 Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7
7
7
8

4 Firmware Architecture
4.1 High level overview . . . . . . . . . .
4.2 The STM32F4 Microprocessor . . . .
4.2.1 System Memory . . . . . . . .
4.2.2 System Clock . . . . . . . . .
4.3 Peripherals and Application Modules
4.3.1 LEDs/ADC/PWM . . . . . .
4.3.2 SDIO/FATFS/DataBase . . .
4.3.3 LTDC/I2C/STemWin/GUI .
4.3.4 USB . . . . . . . . . . . . . .
4.3.5 Lua . . . . . . . . . . . . . .
4.4 Crystallography Application . . . . .

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

9
9
10
11
11
12
12
13
14
14
15
15

5 Results

16

6 Hurdles and personal experience

16

7 Conclusion

17

A The STMs Peripheral Pinout Configuration

18

B Clock Configuration

19

C Simple Database file example

20

D GUI Examples

22

CONTENTS
Abstract
The projects objective is to develop a firmware for a 3D LED display.
The firmware has to be fully modular, drive the LEDs, allow for brightness control
through a potentiometer, control a touch screen, load files from an SD card and
communicate with a computer through a USB connection and run Lua scripts.
We chose to target STs STM32F429 micro-controller as it provides all of the
necessary peripherals. We first tested and validated the usage of each one of them on
STs STM32F429 Discovery Board, then we proceeded onto building the full firmware
that runs on our custom PCB.
We were able to lay down all of the necessary groundwork of the firmware but some
modules like the Lua interpreter and the USB connection are still work in progress.

Introduction

This internships objective is to develop and implement a firmware from scratch for the
CrystalCube, a 9x9x9 LED display. A firmware is a piece of software that provides low
level control over hardware peripherals on a micro-controller. This particular relationship
with the underlining hardware makes it so that, in general, firmwares are written from the
ground up. The embedded software engineer has to come up with a suitable architecture
for the system at hand and squeeze the most out of the given hardware by either using
existing drivers, modifying and adapting them for usage on the current system, or even
writing them from scratch if necessary.

2
2.1

General Overview
Oxbern

Oxbern is a startup created on June 2015 by Pierre Schefler as part of his entrepreneurial
project. The idea of making a 3D LED display was born during his first years group
project (see figure 1) where they decided to make it as a multidisciplinary exercise. It wasnt
supposed to go any further but then came a crystallographic researcher from the Louis Nel
institute showing interest in using the cube as a way the easily represent crystallographic
patterns.

Figure 1: First year (and the first) prototype

2.2

2 GENERAL OVERVIEW

Crystal Cube

The Crystal Cube (see figure 2) is Oxberns main product. Its a 3D 9x9x9 LED display
for visualizing crystallographic patterns. Each LED can be toggled separately. Users can
interact with the Cube through 10 buttons and a 20 by 2 Serial LCD screen. For advanced
usage, a simplistic API can be used to control the Cube through a USB connection with a
Personal Computer. To add new patterns, users must reload the firmware running on one
of the two arduinos necessary for the Cubes operation.

Figure 2: Second year prototype running on 2 Arduinos

Mission Objectives

3.1

Problems with the current prototype

The prototypes first iterations were based on Arduino boards (see figures 1 and 2). Two
Arduino boards were necessary : one to drive the LEDs and an other one to manage the
HMI 1 and handle the internally stored patterns.
From a production stand point this wasnt a viable solution as it not only was too cumbersome and clumsy but also lacked scalability because of the Arduinos lack of memory,
processing power and advanced peripherals. Hence, it was decided that the final version
of the product includes the possibility of controlling the LEDs brightness through a knob
at the back of the Cube, a touch Screen for a more user friendly interface instead of the
Serial LCD screen and the buttons, the possibility of launching Lua scripts directly on the
cube and a more complete USB API.

3.2

Solution

Thus came the prototypes next iteration running on one of STs prototyping boards : the
STM32F429 Discovery Board [6]. As mentioned above, this internships main objective is
to develop a firmware to drive the Cubes LEDs, control the touch display, interact with
the SD card and handle a USB connection with the provided software.
The firmware has to run on STs[8] STM32F429 Microcontroller [11], as it provides all
of the necessary hardware peripherals needed for this project. This means that a certain
familiarity with this chip and its internals, without being a thorough knowledge of all its
specifics, had to be acquired throughout the firmwares development.
More specifically, acquaintance with the following peripherals had to be gained :
SPI (Serial Peripheral Interface) for driving the LEDs
TIM (Timer) and ADC (Analog to Digital Converter) for the LEDs brightness level
handling
SDIO (Secure Digital Input Output) for SD card management
LTDC (LCD-TFT Driver Controller) and I2C (Inter-Integrated Circuit) for driving
the touch screen
SDRAM for managing system memory and screen buffers
USB device stack for handling the USB communication with a computer
For every peripheral, ST provides some general purpose drivers[5] and example code to
quickly get started.
1

Human-machine interface

3 MISSION OBJECTIVES

3.3

Guidelines

I was asked to follow two specific guidelines during this project :

Test and develop code for each peripheral in a separate project before integrating
them all into a single code base.
Make the firmware as modular as possible with the ability to easily replace and/or
remove modules with as little impact as possible on the rest of them and so that
hardware modifications wont have as much impact on the application layer.

Firmware Architecture

4.1

High level overview

The firmwares architecture can be represented by a top down diagram (see figure 3). Each
peripheral can work relatively independently from the others (ex. touch screen updating
and the LEDs brightness adjustment). Hence, it was decided to use FreeRTOS as a
minimalistic scheduler to easily implement the different modules that have to seem like
running in parallel, as required by the new specifications of the product.

Figure 3: A general overview of the firmwares architecture

This block diagram represents the natural cutout that stands out in the firmwares
architecture between the different peripherals and the overlaying middleware. All of the
firmwares modules are built on top of the HAL2 Libraries. Then come the middlewares
on top of that :
FreeRTOS : Handles scheduling between tasks
FATFS : Handles the file system on the sd card
STemWin : Handles widget and graphics and touch events on the touch screen
Lua Interpreter : Handles Lua Scripts and provides a Lua environment
USB Core and CDC stack : Handles the USB connection with the provided computer
software
2

Hardware Abstraction Layer

10

4 FIRMWARE ARCHITECTURE

As stated before, each stacked blocks with the same color on the figure 3 has been
validated in its own separate project3 then put back to back with the others in the form
of a FreeRTOS task.

Figure 4: Code base repository architecture

Indeed, this directory structure[12] follows the guidelines about having a flexible module
implementation as each directory can technically be its own project with minimal changes
to the overall firmware. Also, the use of HAL drivers makes it easy to change the microcontroller if needed (within the same family).

4.2

The STM32F4 Microprocessor

The STM32F429 is an upper mid-tier ARM processor that can run at 180 MHz with 192
KB of SRAM and 1024 KB of FLASH memory while providing various functionality thanks
to its peripherals (see Appendix A). In comparison, the Arduino Mega runs at 16 MHz with
8 KB of SRAM and 128 KB of FLASH memory. As we are targeting an ARM processor
3

Exception made for the Lua interpreter and the USB updating system which are still WIP at the time
of writing this paper

4.2 The STM32F4 Microprocessor

11

we used the ARM GCC Cross Compilation Toolchain on GNU/Linux and GNU/Linux
compatible flashing tools [14].
4.2.1

System Memory

The LCD-TFT screen has a resolution of 800 by 480 pixels. For an ARGB88884 representation, a buffer of 1500 KB is needed. Hence, it was mandatory to add an external SDRAM.
We chose an SDRAM of 8 MB to be able to also load and store heavier databases from
the SD Card. And to simplify memory management we decided to move the heap onto the
SDRAM, leaving the stack more space where to grow if needed as show in figure 5.

Figure 5: Chip memory layout

Heap allocation is handled by Newlib5 [4]s malloc through our implementation of the
sbrk function.6
4.2.2

System Clock

The system clock tree on the STM32F429 has to be fine-tuned according to the peripherals
that are to be activated. STs STM32CubeMx[10] comes with a clock configuration tool
4

8 bits per color and 8 bit for the alpha channel

Newlib is a C library intended for use on embedded systems.
6
More
info
about
using
newlib
at
https://balau82.wordpress.com/2010/12/16/
using-newlib-in-arm-bare-metal-programs/
5

12

4 FIRMWARE ARCHITECTURE

(See Appendix B) that helps with that. The biggest constraint in our system is the USB
Full Speed peripheral that has to run at 48MHz, bringing down the system clock from
180MHz to 168MHz.

4.3
4.3.1

Peripherals and Application Modules

LEDs/ADC/PWM

The Crystal Cubes LEDs are driven using 10 daisy chained shift registers[7]. One to select
a plane and the 9 others control the 81 leds on that plane. For example, to draw a square
in the second plane, we have to populate the sink drivers with these values :

1
2
3
4
5
6
7
8
9
10
11
12
13
14

uint16 buffer = {
0 b0000000000000000 ,
0 b0000000000000000 ,
0 b0000000000000000 ,
0 b0000000001111100 ,
0 b0000000001000100 ,
0 b0000000001111100 ,
0 b0000000000000000 ,
0 b0000000000000000 ,
0 b0000000000000000 ,
0 b0000000000000010
//
^^^^^^^
// unused b i t s
};

//
//
//
//
//
//
//
//
//
//

9 t h column
8 t h column
7 t h column
6 t h column
5 t h column
4 t h column
3 rd column
2nd column
1 s t column
plane s e l e c t i o n

This is done by sending them over the chips SPI7 peripheral, at a rate of 5.25 MBits/s.
The LEDs brightness is controlled with a potentiometer sampled by the ADC8 peripheral that outputs a 12 bit digit which is then turned into a percentage and used as the
PWMs duty cycle generated via the STMs Timer peripheral as shown in figure 6.

7
8

Serial Peripheral Interface : A simple synchronous bus interface used for short distance communication.
Analog to Digital Converter

4.3 Peripherals and Application Modules

13

Figure 6: PWM output for LED brightness control

4.3.2

SDIO/FATFS/DataBase

Data storage is based on FAT formated SD cards as the chip embeds an SDIO9 interface
and ST provides a modified FATFS[1] middleware that goes with it (see figure 7).
The database files that contain the crystallographic patterns are based on a JSON
format[3]. A lightweight C json parser[15] is used to parse through the database files and
build the necessary structures for the Crystallographic application. (see Appendix C for
an example of a simple database file)

Secure Digital Input Output

14

4 FIRMWARE ARCHITECTURE

Figure 7: FatFs Layered Architecture

4.3.3

LTDC/I2C/STemWin/GUI

The touch screen is a Newhaven Display 7" capacitive panel[2] with a resolution of 800 by
480 at a 24 bits color depth. It has a built-in driver for the touch panel and no controller.
This is where the on board LTDC module comes in handy as it only needs to be configured
with the correct parameters (buffer location, screen resolution, pixel format...) to start
displaying the buffers content onto the screen.
The touch panel has a built-in micro-controller that uses an I2C interface to send touch
related information (see page 6 of [2]).
For the graphical interface10 (see Appendix D for an example of the gui), ST provides
a modified version of emWin[13] called STemWin to use with their micro-controllers. This
middleware is LCD controller independent as it only works on the video memory buffer
the user provides. Pointer input devices and touch events are handled through callback
routines called by the user 11 .
4.3.4

USB

ST provides a USB core and USB CDC stack middleware[9] for handling serial communication with a PC. For correct operation, the USB peripheral clock needs to be at 48MHz
and enough heap space (at least 1024 bytes) with a working allocator should be available.
10
11

WIP at the time of writing this paper

https://www.segger.com/emwin-pointer-input-devices.html

4.4 Crystallography Application

15

Each packet received over the USB connection is sent to a state machine for decoding and
executing the encapsulated command according to the API.12
4.3.5

Lua

Lua scripts are handled by the Lua interpreter with specific native binded functions to
control the Cubes peripherals.

4.4

Crystallography Application

The Crystallography application follows a state machine (see figure 8). When turning the
cube on, if an SD card is present, the user is prompted to select a DataBase file or a Lua
script to load. If the SD Card is not detected an Error Message is displayed until the user
inserts and sd card in the slot.
If the user selects a Lua script, the Lua interpreter is launched with the given script.
If the user selects a DataBase file, the file is parsed and if it is well formatted, the user
is prompted to select a pattern to display. If there was an error parsing the database file
then an error message is displayed and the user is prompted to select another file.
User interface examples are provided at Appendix D.

Figure 8: Crystallography Applications state machine

12

The API is still work in progress

16

6 HURDLES AND PERSONAL EXPERIENCE

Results

The firmware is still work in progress as we are still tweaking PCB Layouts and solving
issues with some peripherals. At the time of editing this paper basic firmware functionality
is in place. Some issues with STs SDIO drivers reentrancy for use in a FreeRTOS aware
environment still need to be solved, the USB API still needs to be implemented on the PC
software side and the Lua interpreter isnt part of the firmware yet.

Hurdles and personal experience

This internship was quite heavy with new experiences as we practically built a hole computer system from the grounds up. It feels fulfilling despite having spent countless ours
debugging and fighting electronic failures. As I spent most of my time in the gnu debugger
I discovered the use of python scripts to custom format typedefs which Im sure will come
in handy in future projects. While were on the topic of python scripting I also had the
occasion to sharpen my python skills by writing scripts that helped my mentor handle his
electronic components database and automate to a certain degree the process of ordering
the necessary components in a given design.
One of the biggest hurdles I had - and Im still dealing with - is STs HAL drivers
implementation. One of the key elements thats supposed to make the firmware work the
way it was designed is having a real time kernel handle the different processes, in our case
FreeRTOS. The problem is that STs HAL drivers arent RTOS friendly as they are not
always reentrant. For simple peripherals like the SPI or the I2C buses, it doesnt seem to
be a problem, however, it is with time constrained peripherals such as the SDIO and the
USB. After spending quite some time searching the internet and reading forums frequented
by professional embedded systems engineers, it seems that the HAL drivers cause a lot of
problems to many people and the general wide spread solution is to extensively modify or
even rewrite them13 .
1 #i f (USE_RTOS == 1 )
2
/ Reserved f o r f u t u r e use /
3
#e r r o r USE_RTOS s h o u l d be 0 i n t h e c u r r e n t HAL r e l e a s e
4 #e l s e

Figure 9: Code found in the file stm32f4xx_hal_def.h of the current HAL drivers release

13

http://www.eevblog.com/forum/microcontrollers/sts-%28stm32cube%
29-software-ecosystem-is-terrible-how-can-we-fix-it/

17

Conclusion

Even though some modules are still not quite finished yet, we laid all the necessary groundwork for a fully modular and easily customizable firmware. This will insure the firmwares
compatibility with future updates of the underlining hardware to some extent and allow
for a more straightforward code maintenance.

18

A THE STMS PERIPHERAL PINOUT CONFIGURATION

The STMs Peripheral Pinout Configuration

19

Clock Configuration

20

C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

C SIMPLE DATABASE FILE EXAMPLE

Simple Database file example

{
" children " : [
{
" d e s c r i p t i o n " : "" ,
" image " : " " ,
"name" : " c u b i q u e c e n t r " ,
" options " : [
],
" points " : [
[8 ,8 ,0] ,
[8 ,6 ,0] ,
[6 ,6 ,0] ,
[6 ,8 ,0] ,
[7 ,7 ,1] ,
[6 ,8 ,2] ,
[6 ,6 ,2] ,
[8 ,8 ,2] ,
[8 ,6 ,2]
],
" type " : " p a t t e r n "
},
{
" children " : [
{
" d e s c r i p t i o n " : "" ,
" image " : " " ,
"name" : " c u b i q u e f a c e s c e n t r e s " ,
" options " : [
],
" points " : [
[8 ,8 ,0] ,
[8 ,6 ,0] ,
[6 ,6 ,0] ,
[6 ,8 ,0] ,
[8 ,7 ,1] ,
[6 ,7 ,1] ,
[7 ,6 ,1] ,
[7 ,8 ,1] ,
[7 ,7 ,0] ,
[8 ,8 ,2] ,
[8 ,6 ,2] ,
[6 ,6 ,2] ,
[6 ,8 ,2] ,
[7 ,7 ,2]
],
" type " : " p a t t e r n "
}
],
"name" : " gourpe 1 " ,

21
50
51
52
53
54
55
56

" type " : " group "

}
],
"name" : "New d a t a b a s e " ,
" type " : " group " ,
" v e r s i o n " : "1"
}

22

D GUI EXAMPLES

GUI Examples

SD card content
Click an item to load it.

Some folder

Some database

Some other folder

Some script

Path to group
1/12

Pattern name

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus et varius diam. Class aptent taciti
sociosqu ad litora torquent per conubia nostra, per
inceptos himenaeos. Aenean nulla nisi, auctor et
pharetra non, suscipit at erat. Suspendisse potenti. Vestibulum id orci quis mi pretium interdum. Sed
vehicula nec arcu non suscipit. Quisque at arcu ut nisl
molestie efficitur. Morbi pulvinar urna in semper feugiat. Etiam sit amet malesuada est. In hac habitasse
platea dictumst. Donec tempus vehicula accumsan. In
vitae leo tellus.

REFERENCES

23

References
[1] ChanN. Fatfs. http://elm-chan.org/fsw/ff/00index_e.html.
[2] Newhaven Display. Nhd-7.0-800480ef-atxl-ctp. http://www.newhavendisplay.com/
specs/NHD-7.0-800480EF-ATXL-CTP.pdf.
[3] ECMA. Json. http://www.json.org/.
[4] Jeff Johnston Maintainers : Corinna Vinschen. Newlib. https://sourceware.org/
newlib/.
[5] ST Microelectronics. Description of stm32f4xx hal drivers. http://www.st.com/
st-web-ui/static/active/jp/resource/technical/document/user_manual/
DM00105879.pdf.
[6] ST Microelectronics. Discovery kit for stm32f429/439 lines. http://www.st.com/
st-web-ui/static/active/en/resource/technical/document/user_manual/
DM00093903.pdf.
[7] ST Microelectronics. Low voltage 16-bit constant current led sink driver with
low power saving. http://www.st.com/web/en/resource/technical/document/
datasheet/CD00126635.pdf.
[8] ST Microelectronics. Semiconductor manufacturing company. http://www.st.com/
web/en/home.html.
[9] ST Microelectronics. Stm32cube usb device library. http://www.st.com/st-web-ui/
static/active/en/resource/technical/document/user_manual/DM00108129.
pdf.
[10] ST Microelectronics. Stm32cubemx. http://www.st.com/web/catalog/tools/
FM147/CL1794/SC961/SS1533/PF259242?sc=stm32cube.
[11] ST Microelectronics. Stm32f427xx stm32f429xx. http://www.st.com/web/en/
resource/technical/document/datasheet/DM00071990.pdf.
[12] Oxbern. Ccube firmware. https://github.com/ppff/CCube_Firmware.
[13] SEGGER. emwin. https://www.segger.com/emwin.html.
[14] texane. stlink. https://github.com/texane/stlink.
[15] udp. json-parser. https://github.com/udp/json-parser.