GENIVI Lifecycle Webinar

09th Dec 2014

David Yates
GENIVI System Architect and Lifecycle Topic Owner
Continental Automotive Gmbh
This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
)GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014
1
 Scope of Presentation

The aim of this presentation is to provide an overview of the Lifecycle architecture within GENIVI detailing
where we believe the Automotive world requires extensions to existing open source solutions.

The following topics will be covered:

Welcome & Introduction

Lifecycle Domain Overview
Component Overview
Startup/Shutdown Concept
Introduction to NSM (session management)
Introduction to Resource Management
Roadmap
Location of further information (AMM presentations)

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014 2
 Lifecycle Overview
1*: Get internal states Events:
State chart State change notification Phone session
Plug in for: Diag,SWL,Coding
State chart session
ADC,
PMIC
Supply Plug in for power State change protocol
Management handling (register for
HMI, Phone,
shutdown, get
Node State SWL/Update states,handshake for
Reaction on conditions Management Diagnostics state changes)
Events:
Turn off display, drives, mute audio, Good Ctrls
Poor
1* Bad
State chart
Plug in for: Events: Node observing for CPU load,
Sensors, Thermal Full operational
Devices memory, appl. crash
Error startup
Management Get states
Node
Resource
Reaction on conditions Set LUC config
Turn on fan, reduce audio volume, Last-User-Context

Boot config Resource

Plug in for:
Wakeup Management
reason, Power Boot
node /
Management Management
vehicle Config
network

19-Jun-17 3
 Lifecycle Manifest
Package
Product Component
Platform Component Node State Node State Node Startup
Machine Manager Controller
Supply
Manager
systemd
Supply Node State Boot
Management Management Management

Thermal Power Resource cgroup service

Management Management Management
Thermal
Manager
Node Health
Power Event Node Resource Monitor
Collector Mgr
This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014 4
 Startup/Shutdown Management

takes care about

Boot
Management Startup Management

takes care about

Node State
Management Shutdown Management

Why do we have this split?

systemd stops and unloads all components during its shutdown concept. This requires a lot of time to make them functional
again in the event of a cancel shutdown event being received

An IVI system must be able to resume operation without losing any context and without the need for a reboot. Therefore Node
State Management will only call registered consumers in the shutdown phase. This event notification will drive the components
into a stable state and persistent data written to disk. The processes are not killed or unloaded from memory.

Traditional shutdown will still be required for some legacy components and system services (filesystems, etc.) Therefore, once
registered components have been notified and have gone into their quiet state, systemd will be used to shut down selected
components, unmount filesystems and halt the system.
This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014
5
 Shutdown preparation in Startup Phase

Before systemd
Runlevel replacement
kernel GENIVI extensions

initrd
Start NSM via systemd
A
Mandatory targets B
C
(Base System & Early Features)
BASE_RUNNING

Node State
Manager
(during NSC init)
focussed.target
(last user context)
LUC_RUNNING

unfocussed.target(s)

FULLY_RUNNING
J

FULLY_ lazy.target
OPERATIONAL
This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014
6
 Shutdown Execution

Consumer I

Consumer H Writing LUC

Node Startup
Consumer G systemd app1.service
Node State

Controller
Manager

Consumer F

Consumer E
Node Startup
Consumer D Controller
systemd app2.service
Consumer C Writing LUC

Consumer B

Consumer A Enables:
1. Shutdown activities are trigger able without
Node State Unmount FFS
Machine unloading the components.
NodeState = 2. Legacy components can be shut down in their
NsmNodeState_Shutdown Turn power off
traditional way.
3. Full flexibility on where to integrate systemd
based shutdown units.

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014
7
 Use Cases
Phone
Vehicle Node State Node State Manager
Data Machine
set method
Thermal PhoneSession
Mgmt events/data SWLSession
. .. SWL
request
system
Supply restart
LucRunning
Mgmt Audio
. FullyOperational Signal
.
HMI
Vehicle events/data
Shutdown
Bus Phone
.
Audio
Navigation
HMI
lifecycle requests
Navigation

19-Jun-17 8
 Resource Management - Goals
Resource management contains the functionality to ensure that the node runs in a stable and defined
manner.

To do this, it will monitor and limit different aspects of SW component behavior including system resources
(i.e. CPU load and memory) and critical run-time observation.

Resource allocation will be configurable on a component basis through the use of cgroups.

Health management will provide a configurable escalation strategy defining actions to be taken in the case of
system failures.

The Resource Management component does not aim to implement access control. Access control strategy
must be implemented when setting the final system architecture.

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014
9
 Health Management
Health Management will ensure that the node runs in a stable and defined manner. To do this it is planned to have the
following multi layered observation system and escalation strategy:
register failure & monitoring of userland
Platformcomponents
Platform components attempt recovery
andintegrated
integrated request app/service restart Boot
and
applications RecoveryClients
Recovery Clients Management
applications

notify start/ execute request node restart

alive restart recovery NHM NSM
start/ notify alive
restart
systemd

notify alive
/dev/watchdog

forward NHM heartbeat externally or to internal HW Watchdog

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014
10
 Concepts for the System
Health Management - NHM

The Node Health Monitor will work in conjunction with systemd to monitor component failures in the system. It will
be responsible for :
monitoring systemd to automatically record and track failures per component (i.e. application, service)
providing an interface with which components can register failures when not using the systemd monitoring
maintaining failure statistics over multiple lifecycles for the system and components
the service name will be used to identify and track component failures
statistics on number of failures in number of lifecycles will be maintained (i.e. 3 failures in last 32 lifecycles)
monitoring the wakeup and shutdown events to catch unexpected system restarts
provide an interface for components to read system and component error counts
provide an interface for recovery clients to request a node restart

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014 11
 Concepts for the System
Health Management NHM cont..

Additionally the Node Health Monitor will test a number of product defined criteria with the aim to ensure
that userland is stable and functional. For instance it will be able to validate that :

there is enough free system memory

the CPU is not reporting an excessively high load for a sustained period
defined file accessibility is possible
defined processes are still running
communication is possible (D-Bus)
a user defined process can be executed with an expected result

If the NHM believes that there is an issue with user land then it will be capable to initiate a system restart

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014 12
 Concepts for the System
Health Management Recovery Client

A Recovery Client is a component that is executed when a failure has been detected in the system. There can
be a one to one relationship between apps and recovery clients or one client can handle multiple apps. It should
contain enough functionality to be able to :

request the error status count from the NHM (based on name of the service file failing)

based on the error count, escalate the recovery action, for instance:
file system mount failure, recovery action could be to format the file system and request a node restart
if it is an application that has failed multiple times then we may want to delete that applications persistency
data and restart the application
when possible, request that the SW is uninstalled or rolled back to a previous version

request systemd to restart the application

request a node restart via the NHM

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014 13
 Resource Management - Goals
Resource management contains the functionality to ensure that the node runs in a stable and
defined manner.

To do this, it will monitor and limit different aspects of SW component behavior including system
resources (i.e. CPU load and memory) and critical run-time observation.

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014
14
 Resource Management

Resource Management Node State Mgmt

<<refine>> <<refine>>
<<refine>>

cgroups
systemd
(access via sysfs or
systemd wrapper)
Starts services Control system resources
Configure cgroups Report/Handle resource
allocation errors
Node Resource Node State Manager
Manager
Monitor system resources Evaluate node Handle node
Kill resource abusers restart requests restart requests
(policy dependent)

Application Supply Control Logic

Component
P3

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014 15
 NRM Interfaces
The Node Resource Manager (NRM) provides the following D-Bus interface
org.genivi.NodeResourceManager.Info

which contains the following methods:

GetMemoryStatus -> provides the memory status of a particular cgroup or of the complete node
GetProcessorUsage -> read the current CPU usage for either a particular cgroup or for the complete node

and the following two signals

CgroupMemoryStatus -> triggers when the memory status of a particular cgroup is changed. The client can
register for notification about a specific cgroup through the use of the GroupName parameter
NodeMemoryStatus -> triggered when the memory status of the node is changed

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014
16
 Status and Roadmap
Horizon Intrepid Jupiter Kronos
Apr 2014 Oct 2014 Apr 2015 Oct 2015
Cgroup Adopted comp., provided by the
systemd (Kernel) OSS community
specific specific specific specific

Node Startup GENIVI funded OSS component (implemented by specific specific specific specific
Codethink)
Controller
Node OSS Component (implemented and maintained by specific specific specific specific
State Manager Continental)

Node Product specific library n/a n/a n/a n/a

State Machine

Node OSS Release upcoming (implemented by placeholder placeholder abstract specific

Resource Mgr Continental)

Node OSS Component (implemented and maintained by abstract abstract specific specific
Health Monitor Continental)

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014 17
 Links
Link to project pages :

http://wiki.projects.genivi.org/index.php/Lifecycle_cluster

http://projects.genivi.org/node-startup-controller/

http://projects.genivi.org/node-state-manager/

Links to git repositories :

http://git.projects.genivi.org/?p=lifecycle/node-health-monitor.git;a=summary

http://git.projects.genivi.org/?p=lifecycle/node-startup-controller.git;a=summary

http://git.projects.genivi.org/?p=lifecycle/node-state-manager.git;a=summary

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014
18
 Questions

Thanks for your time and attention.

Any questions??

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0)
GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries
19-Jun-17 Copyright GENIVI Alliance 2014 19