NCS Service Manager

NCS 3.0
March 2014

October 26, 2016

Confidential

Agenda
1 Overview and Background
How NCS fits in with the rest of the
world, what it does, and why
2 NETCONF and YANG
Why NETCONF was invented and how
it is different from previous
management protocols
Walk-through of the YANG modeling
language
3 System Setup
Installing NCS, packages, NETSIM
device simulator
4 Device Manager
Work with devices, synchronization,
templates, policies, etc
October 26, 2016

5 Service Manager
Work with high-level services
6 Alarm Manager
The alarm model and how to work
with alarms
7 System Administration
Deeper topics about installation
choices, logging, trouble shooting,
high availability, clustering, backups,
disaster recovery, etc
8 NED Development
NED types, Yang models and Java
code

Confidential

Model- and Transaction-Based Management

NETCONF, CLI, SNMP, Web UI, Java,
JavaScript, MTOSI, REST
Transform to Network
and Service Status

Transform Service Configuration

Changes
to Network Configuration
Changes

Service Manager
Mapping Logic

Collect Status and

Notifications

Deploy
Network Changes

CDB
Transaction
Manager

End-to-end
Transactional
Integrity

Device Manager

Cisco CLI
SNMP

NETCONF

Cisco, Juniper
October 26, 2016

Confidential

Traditional Service Activation Solution

WebUI
Business
Logic

Error
Handling

CLI + SNMP
Scripts
Typically WebUI only
Typically uses backend with CLI-scripting and SNMP PDUs
Half of the code (according to NSN) is error handling/recovery. Error handling
code is also very difficult + expensive to test

October 26, 2016

Confidential

Traditional Service Activation Solution Business Logic

Modify

Error
Handling

Delete

Business
Logic

Create

WebUI

CLI + SNMP
Scripts
Service creation is what people think of when asked to describe the service
Service deletion is also needed, but a bit simpler than creation
Service modification is the worst; usually requires more code than create +
delete combined. There are just so many ways things can be changed.

October 26, 2016

Confidential

The NCS Approach There has got to be a simpler way

Business
Logic
Model

REST

NC

Error
Handling

Modify

CLI

Delete

WebUI

Create

Model

CLI + SNMP
Scripts

Order of magnitude less work, or more if test is included

Clean model to model mapping
Transactions: No service code required for error handling (detection, recovery, testing)
UI Generation: No service code required for management interfaces (of many types)
NEDs: No service code required for device communication

October 26, 2016

Confidential

Spying on the Create Method

Service
Model

Service
Undo
Instance

Model-to-model
mapping

Create

NCS Database APIs

DB API to DB API
mapping

NCS stores the device level

undo information (reverse-diff)
for each service instance as a
binary object inside the service
instance

NCS

NCS Database APIs

Device
Models

October 26, 2016

Device
Changes

To
devices

Confidential

Delete is Easy

Service
Model

Service
Undo
Instance

Delete is easy.
Apply undo info.

Delete

NCS Database APIs

NCS
NCS Database APIs

Device
Models

October 26, 2016

Device
Changes

To
devices

Confidential

Modifications are Delete & Create In Memory

First delete, but

only in memory

Device
Models

Create

Service
Undo
Instance

Delete

Service
Model

Device
Changes
Diff before/after

October 26, 2016

Then run same

create method
as always

NCS

Update undo information

Actual changes to devices
Confidential

Service Management
Service modeling

Minimum of
Development

Mapping to device model

Service activation
Device effects
Service dry-run

Service Manager

Service check-sync

Mapping Logic

Service restoration

CDB
Transaction
Manager

Device Manager

Service testing
Mapping customer - service - device
Aggregated operational data
Notifications

October 26, 2016

Confidential

10

Service Model?
For example:
Video streaming service
Which channels?
Which quality level?
Which endpoint?

Configure video streaming

The Service Model is not about

collecting all low level
information in one place, it's
about abstraction

Turn knobs on devices to

make it happen

Eventually, the Service

Application will have to figure out
the low level details

October 26, 2016

Confidential

11

NCS and Service Models 3 steps

YANG: You write a service model
corresponding to the service you would
like to configure
SERVICE APPLICATION: You write
service logic using the Java APIs and the
FASTMAP algorithm to map the service
config to the actual device level config
changes
NCS: The Devices are handled by the NCS
Device Manager and provides their defined
data-models.

October 26, 2016

Confidential

Turn knobs on devices to

make it happen

12

The Service Model

View your system as a black box
Which (minimal set of) parameters does an operator need to enter when creating
the service?
Keep in mind:
Avoid exposing internal object models (common mistake)
Think usability for operators and API clients

October 26, 2016

Confidential

13

Service Model Structure

services

Properties (optional)
"Service catalogue"
If you want several service
instances to share settings

A-service
B-service
C-service

SLA-profile: gold, silver,

Channel package: L, M, S,

<service data>

properties

<property data>

devices
device

When defining your service

model, think about
Relationships to devices and
device configs?

C-Service
Application

<device data>
October 26, 2016

Confidential

14

Tasks of the Service Application

Validation
Must or code

Activate Video Streaming !

Mapping Logic
The heart of the
service manager

Turn knobs on devices to

make it happen

Test
Self-test method

October 26, 2016

Confidential

15

Service Model from Web-site Service Example

augment /ncs:services {
list web-site {
key name;
unique url;
unique ip;

leaf ip {
mandatory true;
type inet:ipv4-address;
}
leaf port {
description "UDP port to bind to.";
mandatory true;
type inet:port-number;
}
leaf lb-profile {
description "The load balancer to use.";
mandatory true;
type leafref {
path "/ncs:services/ncs:properties/
wsp:web-site/wsp:profile/wsp:name";
}
}

leaf name {
tailf:info "Unique service id";
type string;
}
// service-data holds NCS undo-info, etc
uses ncs:service-data;
ncs:servicepoint websiteservice;
// Above name is registered by service app
leaf url {
mandatory true;
type inet:uri;
}

October 26, 2016

}
}

Confidential

16

Service Properties from Web-site Service Example

augment /ncs:services/ncs:properties {
container web-site {
list profile {
key name;
leaf name {
type string;
}

list backend {
key name;
leaf name {
description "A backend to use. The backend
must support the webserver
module.";
type leafref {
path "/ncs:devices/ncs:device/ncs:name";
}
}
leaf drop-ratio {
description "Number of dropped packets per
1000 recieved";
config false;
tailf:callpoint ws-stats;
type uint32;
}
}

leaf lb {
description "The load balancer to use";
mandatory true;
type leafref {
path
"/ncs:devices/ncs:device/ncs:name";
}
}

October 26, 2016

Confidential

17

TM Forum: Customers and Customer-facing services

admin@ncs% show customers
customer exxon {
rank
12;
status inactive;
}
customer spotify {
rank
5;
status prospective;
}
customer volvo {
rank
10;
status active;
}

The customer and CFS

concepts are entirely
optional to use in NCS
October 26, 2016

Customer
Customer-facing
service, CFS

Each customer
facing service
points to one
or more
Resourcefacing services.

admin@ncs% show services customer-service

customer-service volvo-public-web {
customer volvo;
service /services/wse:web-site
[name='volvo-global'];
service /services/wse:web-site
[name='volvo-sweden'];
service /services/sec:security
[name='volvo-threat-level-2'];
}

New resource facing services can be

prepared in advance, tested, and then
deployed by just changing a reference in
the customer facing service.
admin@ncs% set services customer-service volvopublic-web service /services/sec:security
[name='volvo-threat-level-4']

Confidential

18

TM Forum: Service Catalogue and Resource-facing services

admin@ncs% show services properties
properties {
web-site {
profile bronze {
description "1 Web server";
lb
lb0;
backend www0;
}
profile gold {
description "3 Web servers";
lb
lb0;
backend www0;
backend www1;
backend www2;
}
profile silver {
description "2 Web servers";
lb
lb0;
backend www1;
backend www2;
}
}
}
October 26, 2016

admin@ncs% show services web-site

web-site volvo-global {
description "Volvo public web .com";
url
www.volvo.com;
ip
136.15.64.11;
port
80;
lb-profile gold;
}
web-site volvo-sweden {
description "Volvo public web .se";
url
www.volvo.se;
ip
136.15.64.16;
port
80;
lb-profile silver;
}

Service catalogue
Confidential

Resource-facing
services, RFS
19

Service Related Actions

Are all services in sync? Check Service Manager vs. Device Manager
admin@ncs% request services check-sync
sync-result {
service-id /ncs:services/wse:web-site[wse:name='volvo-global']
in-sync true
}

Has someone reconfigured

devices so that it breaks
the service?

Specific requests for service instances

admin@ncs% request services web-site volvo-global
Possible completions:
check-sync
- Check if device config is not according to the service
get-modifications - Get the configuration data this service created
re-deploy
- Run/Dryrun the service logic again
self-test
- Perform self-test of the service
admin@ncs% set services global-settings collect-forward-diff true

October 26, 2016

Confidential

Re-deploy will recalculate

the device configuration
and apply it (again)

20

Service instance request get-modifications

admin@ncs% request services web-site
volvo-global get-modifications
cli devices {
device lb0 {
config {
lbConfig {
+
listener 136.15.64.11 80 {
+
service volvo-global {
+
number 1;
+
URL-pattern www.volvo.com;
+
backend 192.168.0.9 8008 {
+
}
+
backend 192.168.0.10 8008 {
+
}
+
backend 192.168.0.11 8008 {
+
}
+
session {
+
type IP;
+
}
+
}
+
}
}
October 26, 2016

device www0 {
config {
wsConfig {
+
listener 192.168.0.9 8008 {
+
}
}
interface eth0 {
+
alias 0 {
+
ipv4-address 192.168.0.9;
+
}
}
}
}
device www1 {
config {
wsConfig {
+
listener 192.168.0.10 8008 {
+
}
}
interface eth0 {
+
alias 0 {
+
ipv4-address 192.168.0.10;
Confidential

21

Device Modifications in the WebUI

Which device config

did the service
write?
How do we
automatically delete
the service and clean
up the device
configs?

October 26, 2016

Confidential

22

Service Device Relationship Maintained, in both directions

admin@ncs> show devices device service-list
NAME SERVICE LIST
----------------------------------------------------------------------------------------lb0
[ "/ncs:services/web-site{volvo-global}" "/ncs:services/web-site{volvo-sweden}" ]
www0 [ "/ncs:services/web-site{volvo-global}" ]
www1 [ "/ncs:services/web-site{volvo-global}" "/ncs:services/web-site{volvo-sweden}" ]
www2 [ "/ncs:services/web-site{volvo-global}" "/ncs:services/web-site{volvo-sweden}" ]
admin@ncs> show services web-site device-list
NAME
DEVICE LIST
-------------------------------------volvo-global [ lb0 www0 www1 www2 ]
volvo-sweden [ lb0 www1 www2 ]
admin@ncs> show services web-site used-by-customer-service
USED BY CUSTOMER
NAME
SERVICE
-----------------------------------These are operational mode commands. Leave config mode:
volvo-global [ volvo-public-web ]
admin@ncs% exit
volvo-sweden [ volvo-public-web ]

or prepend run in front:

admin@ncs% run show devices device service-list

October 26, 2016

Confidential

23

Service Modifications, Re-deploy and Dry-run

Update service definition in the catalogue, apply upgrade to one service immediately
admin@ncs% set services properties web-site profile gold description "4 Web servers" backend www4
admin@ncs% request services web-site volvo-global re-deploy dry-run
cli {
device {
name lb0
data config {
lbConfig {
listener 136.15.64.11 80 {
service volvo-global {
+
backend 192.168.0.12 8008 {
+
}
}
}
}
}
}

October 26, 2016

device {
name www4
data config {
wsConfig {
+
listener 192.168.0.12 8008 {
+
}
}
interface eth0 {
+
alias 0 {
+
ipv4-address 192.168.0.12;
+
}
}
}
}

Confidential

24

Device Modifications, Re-deploy and Dry-run

Device changes may clobber services, e.g.:
admin@ncs% delete devices device www2 config interface eth0 alias 0
admin@ncs% commit
admin@ncs% request services web-site volvo-global check-sync
in-sync false
admin@ncs% request services web-site volvo-global re-deploy dry-run
cli {
device {
name www2
data config {
interface eth0 {
+
alias 0 {
+
ipv4-address 192.168.0.11;
+
}
}
}
}
}
admin@ncs% request services web-site volvo-global re-deploy
admin@ncs% request services web-site volvo-global check-sync
in-sync true
October 26, 2016

Confidential

25

LAB 5.1 - Playing with the service manager

$ cd examples.ncs/web-server-farm/web-site-service
$ make clean all
$ ncs-netsim start
$ ncs
$ ncs_cli u admin
http://127.0.0.1:8080

$ Your shell prompt

> NCS CLI, operational mode
% NCS CLI, configuration mode
# ConfD CLI
October 26, 2016

Confidential

26

A Web Site Service Example

Modelled in
Service
Manager

module web-site

Text
module lb

Managed by
Device Manager

module interfaces

module webserver
module interfaces
October 26, 2016

Confidential

27

Implementing a new Service

October 26, 2016

Confidential

28

LAB 5.2 Try (CLI or Web UI)

1. Sync from device
2. Create /services/properties/web-site/profile
sla-gold
lb: lb0
backend: www1 www2

3. Create /services/service/type/web-site
url, ip, port
lb-profile: sla-gold

October 26, 2016

Confidential

29

Have a look at the code behind the Service

examples.ncs/web-server-farm/web-site-service
services
web-site.yang
web-props.yang

website-service/src/java/src/com/example/website
WebSiteServiceRfs.java : create() callback (depending on the FASTMAP algorithm)

namespaces
Generated stuff

October 26, 2016

Confidential

30

Service Application create() method and YANG files

WebSiteServiceRFS.java

web-site.yang

web-site-props.yang

public class WebSiteServiceRFS {

@ServiceCallback(
servicePoint="websiteservice",
callType=ServiceCBType.CREATE)
public Properties create (
ServiceContext context,
NavuNode service,
NavuNode root,
Properties opaque)
throws DpCallbackException
{
try {
NavuNode myService = service;
NavuContainer myProfile = root.
container(Ncs._services_).
container(Ncs._properties_).
container(webSite.prefix,
webSite._web_site_).
list(webSiteProps._profile_).
elem(myService.leaf(
webSite._lb_profile_).toKey());

module web-site {
prefix wse;
augment /ncs:services {
list web-site {
key name;
leaf name { type string;
}
uses ncs:service-data;
ncs:servicepoint
websiteservice;

module web-site-props {
prefix wsp;
augment /ncs:services/
ncs:properties {
container web-site {
list profile {
key name;
leaf name {
type string;
}
leaf lb {
mandatory true;
type leafref {
path
"/ncs:devices/ncs:device/nc
s:name";
}
}
list backend {
key name;
leaf name {
type leafref {

October 26, 2016

leaf lb-profile {
mandatory true;
type leafref {
path "/ncs:services/
ncs:properties/wsp:website/wsp:profile/wsp:name";
}
}
leaf url {
mandatory true;
Confidential

31

Service Application create() method and YANG files

WebSiteServiceRFS.java

web-site.yang

web-site-props.yang

public class WebSiteServiceRFS {

@ServiceCallback(
servicePoint="websiteservice",
callType=ServiceCBType.CREATE)
public Properties create (
ServiceContext context,
NavuNode service,
NavuNode root,
Properties opaque)
throws DpCallbackException
{
try {
NavuNode myService = service;
NavuContainer myProfile = root.
container(Ncs._services_).
container(Ncs._properties_).
container(webSite.prefix,
webSite._web_site_).
list(webSiteProps._profile_).
elem(myService.leaf(
webSite._lb_profile_).toKey());

module web-site {
prefix wse;
augment /ncs:services {
list web-site {
key name;
leaf name { type string;
}
uses ncs:service-data;
ncs:servicepoint
websiteservice;

module web-site-props {
prefix wsp;
augment /ncs:services/
ncs:properties {
container web-site {
list profile {
key name;
leaf name {
type string;
}
leaf lb {
mandatory true;
type leafref {
path
"/ncs:devices/ncs:device/nc
s:name";
}
}
list backend {
key name;
leaf name {
type leafref {

In Navu, use
strings: "services",
or hashes: Ncs._services_

October 26, 2016

leaf lb-profile {
mandatory true;
type leafref {
path "/ncs:services/
ncs:properties/wsp:website/wsp:profile/wsp:name";
}
}
leaf url {
mandatory true;

Navu chaining
Confidential

32

Namespace .java files

namespace/webSite. java

/*
* BEWARE BEWARE BEWARE BEWARE BEWARE BEWARE BEWARE BEWARE BEWARE
* This file has been auto-generated by the confdc compiler.
* Source: ../load-dir/web-site.fxs
* BEWARE BEWARE BEWARE BEWARE BEWARE BEWARE BEWARE BEWARE BEWARE
*/
package com.example.websiteservice.websiteservice.namespaces;

public class webSite extends ConfNamespace {

public static final int

_native = 472211213;
public static final String _native_ = "native";
public static final int
_name = 1998270519;
public static final String _name_ = "name";
public static final int
_get_modifications = 1195376060;
public static final String _get_modifications_ = "get-modifications";
public static final int
_lb_profile = 1348396882;
public static final String _lb_profile_ = "lb-profile";
public static final int
_description = 202559156;

October 26, 2016

Confidential

33

Alternate Mapping Approaches

Implementation alternatives
Java only

Service
Model

Most expressive power, but also most work

Mapping

Make calls to external applications

Implemented in
Java
Template
Java + Template

Execute complex algorithms

Template only
Only simple mappings
Implemented in minutes (e.g. in CLI)

Java instantiating template

Do the complex computations in Java
Apply the bulk of the settings in template

Device
Models

October 26, 2016

Java exports variables to the template

Confidential

34

Service Mapping with Templates

examples.ncs/datacenter/qinq/packages/qinq-template/templates/qinq-template.xml
<vlans xmlns="http://pica8.org/yang"
<config-template
xmlns="http://tail-f.com/ns/config/1.0"
tags="merge" annotation="Pica">
servicepoint="sp-qinq-template">
<vlan-id>
<devices xmlns="http://tail-f.com/ns/ncs">
<id>{/s-vlan}</id>
<device tags="nocreate" >
</vlan-id>
<name>{/core-switch/switch}</name>
</vlans>
<config>
<interface xmlns="http://pica8.org/yang">
<vlan xmlns="urn:ios" tags="merge">
<aggregate-ethernet>
<vlan-list>
<name>{trunk-interface}</name>
<id>{/s-vlan}</id>
<family tags="merge">
</vlan-list>
<ethernet-switching>
</vlan>
<port-mode>trunk</port-mode>
<interface xmlns="urn:ios"
<vlan>
tags="merge" annotation="Cisco">
<members>
<FastEthernet>
<vlanid>{/s-vlan}</vlanid>
<name>{trunk-interface}</name>
</members>
<switchport tags="merge">
</vlan>
<mode>
</ethernet-switching>
<trunk/>
</family>
</mode>
</aggregate-ethernet>
<trunk>
<gigabit-ethernet>
<allowed>
<name>{edge-interface/interface}</name>
<vlan>
<family tags="merge">
<vlans>{/s-vlan}</vlans>
<ethernet-switching>
October 26, 2016

Confidential

35

Service Mapping with Java + Templates

pe-template.xml

<config-template
xmlns="http://tail-f.com/ns/config/1.0">
<devices xmlns="http://tail-f.com/ns/ncs">
<device>
<name>{$PE}</name>
<config tags="merge">
<vrf xmlns="urn:ios" tags="merge">
<definition>
<name>{/name}</name>
<rd>{/as-number}:1</rd>
<address-family>
<ipv4>

<interface xmlns="urn:ios" tags="merge">

<GigabitEthernet
when="{starts-with($PE_INT_NAME,
'GigabitEthernet')}">
<name>
{substring($PE_INT_NAME,16)}.{$VLAN_ID}
</name>
<description>
Link to CE / {$CE} {$CE_INT_NAME}
</description>
<encapsulation>
October 26, 2016

l3vpnRFS.java

@ServiceCallback(
servicePoint="l3vpn-servicepoint",
callType=ServiceCBType.CREATE)
public Properties create() {

Template peTemplate =
new Template(context, "pe-template");
Template ceTemplate =
new Template(context,"ce-template");
TemplateVariables vpnVar = new
TemplateVariables();

vpnVar.putQuoted("VLAN_ID", )
vpnVar.putQuoted("LINK_PE_ADR", )
peTemplate.apply(service, vpnVar);
ceTemplate.apply(service, vpnVar);
}

Confidential

36

LAB 5.3 Setup a new project

1.
2.

Create a new NCS runtime workspace

- create a directory and cd to it
Build the device
$ cd <ncs>/packages/neds/cisco-ios/src

make all
3.

Back to the new NCS runtime workspace, create simulated network

ncs-netsim create-network <ncs>/packages/neds/cisco-ios 4 c

4.

Create NCS environment

ncs-setup --dest .
5.

Start with:

ncs-netsim start
ncs
ncs_cli u admin

October 26, 2016

Confidential

37

LAB 5.4 Write a service model

Create a service package (in packages)

cd packages
ncs-make-package --service-template myService
Create eclipse environment

cd <project-top>
ncs-setup --eclipse-setup

(this can be done to refresh eclipse project when new packages are added)

Import project into eclipse

Now, write a service model
What should you abstract?
What input do you need from the user?

Think Meal, not Fries, Burgers and Coke!

October 26, 2016

Confidential

38

Services Sharing Resources

October 26, 2016

Confidential

39

FASTMAP Considerations
FASTMAP is the patented algorithm that records the modifications in the create()
callback, and is responsible for automatically handling the modify and delete case.
FASTMAP has a few assumptions:
Service application code only creates and sets data
Set operations must only be used on parts of the device models "owned" by that
service instance, so that there are no potential conflicts with other service
instances other than what can be achieved with sharedCreate()
One service instance must not depend on other service instances
If the creation of a service instance interacts with other service instances in a more
complex way than the above, you can implement that behavior in the
preModifications() callback. You would then have to deal with creation, modification
and deletion of your service without relying on the FASTMAP algorithm

October 26, 2016

Confidential

40

Services Sharing Resources

Service Instances

A1
Operator Creates A1

In create():

Device Changes

Recorded Service Data

a1

A1: a1 + dns

dns

The Service Application makes some device

changes (a1) and creates a shared
resource, e.g. some DNS settings common
to all service instances

NCS records all

device level changes
made by the service
application

if not exist(dns):
create(dns)
October 26, 2016

Confidential

41

Services Sharing Resources

Service Instances

A1

Device Changes

Recorded Service Data

a1

A1: a1 + dns

dns
A2

Operator Creates A2
In create():
if not exist(dns):
create(dns)
October 26, 2016

a2

A2: a2

The Service Application makes

some device changes (a2). The
shared resource already exists,
no action required
Confidential

NCS records all

device level changes
made by the service
application
42

Services Sharing Resources: The missing link

Service Instances

A1

a1

Operator Deletes A1
A2

In create():

Recorded Service Data

A1: a1 + dns

dns
a2

Service A2 breaks
when shared resource
dns is deleted

if not exist(dns):
create(dns)
October 26, 2016

Device Changes

A2: a2

NCS deletes all

recorded device
level changes made
by the service
application.
NCS has no way of
knowing A2
depends on dns.

Conditional creation of shared resources is

a bad idea in service code.
We need some other way to do this.
Confidential

43

Services Sharing Resources, using SharedCreate

Service Instances

A1
Operator Creates A1

In create():

Device Changes

Recorded Service Data

a1

A1: a1 + (dns)

dns

The Service Application makes some device

changes (a1) and creates a shared
resource, e.g. some DNS settings common
to all service instances

NCS records all

device level changes
made by the service
application

sharedCreate(dns)

October 26, 2016

Confidential

44

Services Sharing Resources, using SharedCreate

Service Instances

A1

Device Changes

Recorded Service Data

a1

A1: a1 + (dns)

dns
A2

Operator Creates A2
In create():
sharedCreate(dns)

October 26, 2016

a2

A2: a2 + (dns)

The Service Application makes

some device changes (a2). The
shared resource already exists,
no action required
Confidential

NCS records all

device level changes
made by the service
application
45

Services Sharing Resources, using SharedCreate

Service Instances

Device Changes

A1

a1

Operator Deletes A1
A2

A1: a1 + (dns)

dns
a2

A2: a2 + (dns)

Service A2 survives
In create():
sharedCreate(dns)

October 26, 2016

Recorded Service Data

NCS deletes the

recorded device
level changes made
by the service
application.
NCS leaves dns in
place, as it is still
referred.

Let all service instances create the shared

resource (using sharedCreate).
The dependency is recorded.
Confidential

46

Services Sharing Resources, using preModification Callback

Service Instances

A1
Operator Creates A1
A2

Device Changes

Recorded Service Data

a1

A1: a1

dns
a2

A2: a2

Operator Creates A2
In preModification():
if not exist(dns):
create(dns)
October 26, 2016

Before any
service instance
is created, dns is
created if it
doesn't already
exist.

As an alternative to sharedCreate() from

all service instances, requires shared
resources may be created as necessary, in
advance of the service creation
Confidential

47

Services Sharing Resources, using preModification Callback

Service Instances

A1
Operator Deletes A1
A2

Operator Deletes A2
In preModification():
if not exist(dns):
create(dns)
October 26, 2016

Device Changes

a1

Recorded Service Data

A1: a1

dns
a2

A2: a2

The pre- and postModification() callbacks

can be used to modify the transaction
outside the scope of the FASTMAP
algorithm.
See example 15-pre-modification.
Confidential

Since the
creation of the
shared dns
resource was
never recorded
as part of a
service, it will
linger even if all
service
instances are
deleted
48

Reactive FASTMAP
Reactive FASTMAP allows services
to adapt to changes in the
environment
to be activated in stages

October 26, 2016

Confidential

49

The Java APIs

October 26, 2016

Confidential

50

Mapping Logic approach

Mapping Service States to Device Configurations is a big challenge
Correctness
Changes
Maintenance
.
Traditional approach
Write code for all service life-cycle atoms
Create, Read, Update Delete: programmers need to implement all them
Error handling: what if any of the above fails midway through?

User interface
Adaptor programming, how to make the device changes happen
Persistence mapping
October 26, 2016

Confidential

51

The NCS Approach to Service Logic

Resource Manager
Registering Service Logic
NAVU API
Gives programmatic access to the service
and device models
Lazy DOM
Manipulate the DOM and you are done
No mapping code to devices
No mapping code to persistence
Programmers know how to manipulate datastructures
Transactions

FastMap

Device Model

CDB

Transaction

Service Model

NETCONF, CLI, SNMP,

FastMap
Reduce the transformation to one single
create method
October 26, 2016

Confidential

52

package-meta-data.xml in the Website service

The service component

Callback components
October 26, 2016

Confidential

53

NCS APIs

October 26, 2016

Confidential

54

NCS Block overview

CLI

Web
UI

NETCONF

SNMP

REST

MTOSI

Other

Radius
etc

AAA API

Service
Models

Service Manager

CDB
CDB API

Core Engine
Alarm
Manager
Alarm API

FastMap Application
Abstract

FastMap Application
Abstract

FastMap Application

Device Manager

Device
Models

Network Element Driver API (Java)

NETCONF
NED

October 26, 2016

NSMUX
API
NAVU API

Inventory

Data Provider (DP)

API

Management Agent API (MAAPI) (Java, JavaScript)

Cisco CLI
NED

Confidential

SNMP
NED

Generic
NED

55

NCS APIs
APSs are decoupled from NCS
Communication over Stream sockets
A NCS lib can have any number of subscribers
CLI

Tasks that can be done:

Configure NCS and the devices
Traverse the schema
Subscribe to changes
Publish notifications
Provide data
and more

socket

action()

OK

NCS

ncs.jar

Application

October 26, 2016

Confidential

56

NCS API:s

Threads from pool dispatched for each

distinct component
NEDs have separate pool, controlled by
NCS
Applications ARE threads, and are not
pooled

Package C

Services, and callbacks

Package B

Package A

Requests are multiplexed by respective

Multiplexer

Service Manager

NCS

Fast Map

Transaction Manager

Service Models

NCS Java VM
Service components

Application
components

CDB

Callback
components

NED components

Device Models

Device Manager

October 26, 2016

Confidential

57

Two ways of reading and manipulating data

Using CDB API
Read any type of data
Write OPERATIONAL data (tailf:cdb-oper)
Subscribe to changes in CDB
Cdb, CdbSubscription and CdbSession

Using MAAPI
The Agent API, full access to NCS data manipulation
Transaction control,
start new
attach to existing

Maapi

October 26, 2016

Confidential

58

NAVU API Navigation Utilities

DOM-based database traversal
LAZY implementation
Minimize round trips

Wrapper for CDB or MAAPI sessions

Use CDB for Reading data and Writing Operational data
Use Maapi when
Starting transactions and writing configuration data
Attaching to transactions to write configuration data

NAVU is not FastMap

Two different concepts

NAVU can be used anywhere with NCS

October 26, 2016

Confidential

59

NAVU API basics

NAVU needs to know two things
1. How to communicate with NCS
2. Where is the Root node

NavuContext
Defines the NCS connection (MAAPI / CDB / R/RW etc)
Already defined when using FastMap

NavuContainer
A representation of a Yang structure:
A container
A single list entry

October 26, 2016

Confidential

60

NAVU basics
Working from the root container
NavuContainer child1 = top.list(test._simple_list_)
.elem(new ConfKey(element1))
.list(test._child_list_)
.elem(new ConfKey(1));

NAVU will check the

existence of keys

NavuLeaf child1Attr = child1.container(test._child_container_).leaf(test._childAttr_);

module test {
...
container top{
list simple-list {
key name;
leaf name { type string; }
leaf data { type uint64; }

LAZY
We havent retrieved
the child leaf.
Traversing containers
and leafs do not trigger
the element to get
collected.

list child-list {
key index;
leaf index { type int64; }
container child-container {
precense Optional subsystem;
leaf childAttr { type string;}
...
October 26, 2016

Confidential

61

NAVU basics
Evaluating
if (child1.container(test._child_container_).exists()){
//Do something
}
else {
child1.container(test._child_container_).create();
child1Attr.set(new ConfBuf(VAL));
}
module test {
...
container top{
list simple-list {
key name;
leaf name { type string; }
leaf data { type uint64; }

Calling NCS
.exists()
.create()
list keys
etc

Causes NAVU to
evaluate

list child-list {
key index;
leaf index { type int64; }
container child-container {
leaf childAttr { type string;}
...

October 26, 2016

Confidential

62

NAVU Looping
Traversing all the elements in a list
E.g. all live devices
for (NavuContainer device :
root.container(Ncs._devices_).list(Ncs._device_).elements())
{
//Read the live-data from the device
ConfUInt64 packets = (ConfUInt64)device
.container(Ncs._live_status_)
.container(switch._data_)
.leaf(switch._packets_per_second_).value();
//Print it and forget about it
System.out.println(packets.toString() + packets/s in device
+ device.getKey().toString());
}

October 26, 2016

Confidential

63

LAB 5.5 Implementing Mapping logic

Setup Eclipse with the generated project
(remember: ncs-setup --eclipse-setup)

Create a service in the CLI

Implement the create method of the RfsService
Start with reading out the parameters of the service via the NavuContainer service
Set the parameters on the Devices in the devices/device tree

Debug using Eclipse

Create new: Run Configration -> Java Application
in VM Argumentss set:
-DTAILF_CLASSLOADER = FALSE
Main class: com.tailf.ncs.NcsJVMLauncher

Disable automatic startup of the java-jvm when debugging with Eclipse

October 26, 2016

Confidential

64

Setting up NAVU

Use CDB or MAAPI

Create NavuContext
Create Navu Root container
Create SCHEMA root
Create Top Container

Confusing?
Navu Root the entire system
Schema Root / of the module
Top container root node of the current
module

//Create root with Maapi session

NavuContainer superRoot =
new NavuContainer(
new NavuContext(maapi,th));
/root points to / i.e. a super root
//create a node that is the schema root
//instead
NavuContainer root =
superRoot.container(new Ncs().hash());
OR
//Create root with CDB
NavuContainer root =
new NavuContainer(cdb, (new Ncs().hash());

//Finally navigate in namespace

NavuContainer rootNode =
root.container(Ncs._devices);
October 26, 2016

Confidential

65

Setting up the context using Maapi

Maapi basics:
Create socket
Create Maapi object
Start Session
Load Schemas

MUST be loaded for NAVU

Done by Resource manager
When finished:
Keep session for reuse
Kill transaction
maapi.finishTrans(th);

// Setup socket to server

Socket sock = new Socket("localhost",
Conf.NCS_PORT);
// Start MAAPI session for admin user, originating from
localhost
Maapi maapi = new Maapi(sock);
maapi.startUserSession("admin",
InetAddress.getLocalHost(),
"maapi",
new String[] { "admin" },
MaapiUserSessionFlag.PROTO_TCP);
maapi.loadSchemas(); //only on standalone
// Start a read transaction towards the running
configuration.
int th = maapi.startTrans(Conf.DB_CANDIDATE,
Conf.MODE_READ_WRITE);
//Do something
maapi.applyTrans(th);
maapi.finishTrans(th);
maapi.candidateCommit();

October 26, 2016

Confidential

66

Setting up the context using Cdb

Cdb basics:
Create socket
Create Cdb object
Start Cdb Session
BUT: A Maapi load schema must be
performed.
When finished:
Kill Session
CDB sessions must be shortlived
Keeps READ locks towards CDB

October 26, 2016

// create new socket and Cdb instance.

Socket sock = new Socket(host, Conf.PORT);
Cdb cdb = new Cdb("test",sock);
// start session towards running data
CdbSession session =
cdb.startSession(Cdb.CDB_RUNNING);
//Read some value
ConfPath path =
new ConfPath(/test/servers/
server{%s}/port",
new Object[] { new String("www") });
ConfValue val = session.getElem(path);
// now do something with the read value
....
// end session when we are finished
session.endSession();

Confidential

67

Data types

ConfObject ConfObject [ ]

ConfPath

Hash representation of Yang node

Used internally

Representing a key path

ConfValue

ConfKey (ConfObject)

ConfInt32
ConfBoolean
ConfUInt32
ConfBuf
And plenty more

Representing a Yang List key

ConfTag (ConfObject)
Representing a Yang Element

Automatic namespaces

ConfObject [ ] example:
/servers/server{www1}/port
3
2
1
0

Generate shorthand code

from Yang
Useful for basic spell checks
Basically names for strings

[ConfTag, ConfTag, ConfKey, ConfTag]

October 26, 2016

Confidential

ncsc --emit-java
confdc --emit-java

68

NCS design patterns

October 26, 2016

Confidential

69

Subscribing to changes with NAVU

What happens if the Properties change

Model

Does not trigger the FastMap logic

How to update the devices

NCS subscription mechanism

Subscribe to changes in a specific subtree of the model
All changes below that point will trigger the subscription
Two types of subscription
NavuCdbConfigSubscriber - configuration
NavuCdbOperSubscriber - alarms, states etc.
Operational data in CDB (tailf:cdb-oper;)
Triggers only when the updater has used a lock while updating

Subscribe to
sub-tree

Both require the implementation of a diff interface:

NavuCdbConfigDiffIterate Interface
NavuCdbOperDiffIterateInterface

October 26, 2016

Confidential

70

Subscribing to changes with NAVU

Use Cdb to connect
Register Callback
Iterate over changers
Setup subscription
Cdb cdb = new Cdb("mycdb",socket);
//Create the subscriber
NavuCdbOperSubscriber sub =
new NavuCdbOperSubscriber(cdb,new
NcsAlarms(),"AlarmSubscriber");
//Register a subscription point
boolean regstatus =
sub.register(new MyIterate(),
"/alarms/alarm-list-container/alarms",
new NcsAlarms());
sub.start();
October 26, 2016

Iterate over the changes

public DiffIterateResultFlag iterate(
Integer subId,
NavuContainer root,
ConfObject[] kp,
DiffIterateOperFlag op,
ConfValue oldValue,
ConfValue newValue) {
//Navigate to our alarmlist
NavuList alarmList = root.
container(NcsAlarms._alarms_).
container(NcsAlarms._alarm_list_container_).
list(NcsAlarms._alarm_);
//Get our modified alarm-id i.e wich alarm was
// modifyed or added.
String kpString = Conf.kpToString(kp);
ConfKey key = (ConfKey)kp[0];

// check if the current alarm was added

// or changed state.
switch(op){
case MOP_CREATED:{ . . . }
case MOP_MODIFIED: { . . . }
Confidential

71

CDB Subscribers
NAVU is quick, but sometimes an application is what we need
Most generic functionality can be built around subscriptions
Define API write data model
Subscribe to changes Create subscriber
Callback based
NCS Mux (Resource manager) keeps thread pool

NCS Mux

Model

callback

October 26, 2016

Confidential

Java
application
Java
application
Java
application
72

Data providers
For operational data
Responsible for providing data to NCS on request
Data not stored in NCS
Fetched only when asked for

callback
get_elem

October 26, 2016

Confidential

NCS Mux

Model

Java
application
Java
application
Java
application

73

Data provider example

Define a part of the model to provide data for
@DataCallback(callPoint=work.callpoint_workPoint,
callType=DataCBType.GET_ELEM)
public ConfValue getElem(DpTrans trans, ConfObject[] keyPath)
throws DpCallbackException {

ConfInt32 kv = (ConfInt32) ((ConfKey)

keyPath[1]).elementAt(0);
Item i = MyDb.findItem( kv.intValue() );
if (i == null) return null; // not found
// switch on xml elem tag
ConfTag leaf = (ConfTag) keyPath[0];
switch (leaf.getTagHash()) {
case work._key:
return new ConfInt32(i.key);
...
case work._comment:
return new ConfBuf(i.comment);
default:
throw new DpCallbackException("xml tag not
handled");
}
October 26, 2016

Confidential

74

Full system
Using data providers and subscribers to build entire systems
Each component responsible for subset
All functions exposed through data model
Each sub-component can share its properties through the model

(CDB)
Common Model

Java
subscriber

Confidential

Java data
provider

Java data
provider

Java
subscriber

Java
subscriber

October 26, 2016

75

Augmenting the data model

The NCS build system augments the NCS models
Services under /services/service/type
Devices under /devices/device/config

Anything can be augmented anywhere

BUT: some places in the data-model has special meaning
The device manager
The service manager
The alarm manager

October 26, 2016

Confidential

76

LAB 5.6 Write a data provider

Create an external service inventory
Device meta-data:
Location
Serial number
Up-time

config false

cd packages
ncs-make-package --data-provider-template myInventory
Augment the service model with meta-data
Create a external data provider that holds this information
com.tailf.dp.annotations.DataCallback

Methods are described in:

com.tailf.dp.DataCallback

GET_ELEM, GET_NEXT,

October 26, 2016

Confidential

77

Callpoints and Alarms

October 26, 2016

Confidential

78

Callpoints
> request devices sync direction from-device
Trigger callbacks when parts of the Yang model is modified
Transactions, Validation, Actions, Notifications and more
Implemented via Annotations:
com.tailf.dp.proto: ActionCBType, DataCBType, TransCBType

The annotations correspond implementations of Interfaces

DpActionCallback, DpDataCallback, DpTransCallback

NCS will implement the interfaces

All you need to do is to annotate the code

October 26, 2016

Confidential

79

Callpoints Example

Action callpoint
Leaf name myaction
Many actions can share actionpoint
ConfTag name = myaction
Input and output are ConfXMLParam
Class MyActionCallbacks{
@ActionCallback(callPoint = myAp", callType = ActionCBType.INIT)
public void init(DpActionTrans actx) throws DpCallbackException {
//Init shared resources if needed
}
@ActionCallback(callPoint = myAp", callType = ActionCBType.ACTION)
public ConfXMLParam[] action(DpActionTrans actx, ConfTag name,
ConfObject[] kp, ConfXMLParam[] params)
throws DpCallbackException {
//Check which ConfTag was called (if registered with many)
//Read input parameters
//Perform action
October 26, 2016
Confidential

list simple-list {
...
tailf:action myaction {
tailf:actionpoint myAp;
input {
leaf mode {
type string;
mandatory true;
}
output {
leaf time {
type string;
mandatory true;
}

80

October 26, 2016

Confidential

81

NCS API Overview

CLI

Web
UI

NETCONF

SNMP

REST

MTOSI

Other

Radius
etc

AAA API

Service
Models

Service Manager
SYSTEM API

CDB

Core Engine
HA API

Alarm
Manager
Alarm API

CDB API

FastMap Application
Abstract

FastMap Application
Abstract

FastMap Application

Device Manager

Device
Models

Network Element Driver API (Java)

NETCONF
NED

October 26, 2016

NSMUX
API
NAVU API

Inventory

Data Provider (DP)

API

Management Agent API (MAAPI) (Java, JavaScript)

Cisco CLI
NED

Confidential

SNMP
NED

Generic
NED

82

Data Provider and Database Mapping

External
Database
October 26, 2016

Confidential

83

Common Applications Roles in NCS

Service Application: Translates
service level changes to device
configuration changes. MAAPI/NAVU
Network Element Driver:
Communicates with a device within a
product family. NEDAPI
Subscriber: Reacts to configuration
changes. CDBAPI.
Data Provider: Provides current value
of some status/information.
DPAPI (read-only parts)

October 26, 2016

Action Provider: Provides

implementation of operator actions,
e.g. software upgrade, clear alarm,
DPAPI (action parts)

External Database: Plugging in

external databases so it becomes
accessible to operators and applications
through MAAPI, i.e. an integral part of
the NCS system. DPAPI
Validator: Called during commit
sequence to check validity of (some
part/aspect of) the configuration.
MAAPI

Confidential

84

Common Applications Roles in NCS, page 2

Set hook: Application that triggers
when a particular leaves are modified,
may update the transaction. Use with
extreme caution, easily ruins the
transactional integrity. MAAPI
Transaction hook: Application that
modifies the contents of a transaction.
Use with extreme caution, easily ruins
the transactional integrity. MAAPI
Data feeder: An application that is
repeatedly storing operational data,
e.g. performance readings, into a
database table using the CDBAPI.

October 26, 2016

Event Receiver: Application that

receives events, e.g. SNMP Traps, and
creates alarms or takes other actions.
ALARM API, SYSTEM API

Transform: Application that translates

bi-directionally between two data
models, typically for different
management interface standards.
DPAPI + MAAPI
Policy, Command and Post-commit
script: Scripts to enforce policies or
perform additional actions at commit
time or when specific commands are
executed. MAAPI

Confidential

85

Common Applications Roles in NCS, page 3

Discovery application: Discovers
hosts, links, topology or something else
and populates result in NCS.
MAAPI and/or CDBAPI

Scheduler: Runs some application or

action at regular or configured times.
MAAPI
Management agent: E.g. prettier
WebUI, MTOSI or SOAP interface, self
service portal, integration with NB
workflow engine. MAAPI
Authentication plugin: Calls out to
an external Radius, Tacacs, LDAP, etc.
server to authenticate users. AAAAPI
October 26, 2016

Authorization callback: Application

that authorizes individual commands or
data access in the system. SYSTEM API
Completion plugin: Application that
alters the standard TAB-completion
behavior in the Command Line
Interfaces. SYSTEM API

User Type plugin: Application

implementing an additional base type
that can be used in the YANG models.
SYSTEM API

Confidential

86

Common Applications Roles in NCS, page 4

Upgrade client: An application that
participates in a software upgrade
campaign to update the data stored in
CDB beyond the automatic conversion.
CDBAPI
Message formatters: Application
reformatting or translating operator
messages, e.g. error messages.
SYSTEM API

October 26, 2016

Inter-process communications
plugin: An application to replace the
default IPC mechanism (TCP sockets)
used between NCS and applications.
Useful if clients need to use for
example UNIX-domain sockets or TIPC.
SYSTEM API

Wizard: Application that lets operator

fill in a few values, then computes
reasonable defaults for the rest. MAAPI

Confidential

87