10 Things That Will Kill Your IoT Application,

and How to Fix Them — Quickly

“Experience”
Defined
Broad
Industry
Range

© 2020 Aeris Communications, Inc. All Rights Reserved

Poll Question 1:

(1) Just learning about the industry

(2) In the early stages of a new connected product or
service program
(3) Just launched a new connected thing
(4) Experienced with connected things at large scale
 Lesson #1
A wide variety of issues can arise with products and
services built on connectivity, wreaking large-scale havoc
with performance, cost, and control.

Lesson #2
Prevention is ideal.
But when stuff happens, then
rapid detection, isolation, and resolution
are essential to minimize costs and regain control.

Lesson #3
Getting some brains into the situation is key to #2.
(Will explain in a bit)
© 2020 Aeris Communications, Inc. All Rights Reserved
Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su

Data
Sessions
per Hour

~0.7M
00:00 04:00 08:00 12:00 16:00 20:00

Total Data Transmitted

Not okay
Normal
Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su

Data per
Session
Cellular and apps just work
… if we have bars
Upstream from
the smartphone?
Little knowledge!

© 2020 Aeris Communications, Inc. All Rights Reserved

ATTACHMENT AND DATA RATES
Web App Mobile App
Subject to UE+eNB (or MS+BTS)
Link Budgets and Applicable Pathloss Models
App & Integration APIs

Application Logic
IOT SOLUTION
Device PROVIDER’S
CLOUD & APPS Data Lake

Required
Cloud Infrastructure

VPN Internet

Context

© 2020 Aeris Communications, Inc. All Rights Reserved

End to End,
in Simpler
Terms

Connected Radio Mobile IoT Solution

Things Access Core Provider’s
Networks Cloud
(RANs)

and data transit between them, which can also cause issues

© 2020 Aeris Communications, Inc. All Rights Reserved

Local Interfaces Software

IoT solution provider’s app(s)

Head Unit
Third-party apps

The Connected
(controlled or not)

Wi-Fi Cellular
Solution provider’s edge Thing’s Local
Telematics
Device
processing & control
Device firmware, OS,
Network
and configuration
Cellular radio module
firmware and configuration
Car Network SIM firmware, including
roaming preferences
and/or . . . Power
from vehicle
Analog & digital I/O and/or battery

(more software)

© 2020 Aeris Communications, Inc. All Rights Reserved

 Those “10” IoT Solution Killers
IoT Solution Call flow error  significant overages from device retries
Provider’s Cloud Dominant end-customer traffic spike  latency for other customers
Infrastructure upgrade error  nationwide outage for 80% of app features

Mobile Core Contract-usage mismatch  25% wasted cost

Surprise traffic surges  scrambles to prevent cross-customer impairment
Transit “Backhoe fading”  complete service outage

RANs Device ping-ponging  no reporting

MNO network changes  multiple (often silent) outages
Busy towers  reporting delays, failed OTAs
Poor indoor coverage  remote health monitoring app failure
Outdoor coverage gaps  mission critical automotive features impaired

Connected OTA bug  multi $M overage

Things Hacked devices used as DDOS attackers  out of business
Unauthorized third-party app use  multi $M overage
Automotive remote operation failure  4-hour support calls
Antenna failure  20% service outage
App / network timing mismatch  5x higher video upload latency
Misconfiguration  1/3 expected battery life, lost devices
Reporting restarts when interrupted  overages or outright failure
Stolen devices used for rogue SMS service  large overage charges

© 2020 Aeris Communications, Inc. All Rights Reserved

Poll Question 2:

(1) No, wow, I had no idea all this can go wrong.

(2) I’ve seen some of these, but I guess there may be
new surprises for us ahead.
(3) Totally, I’ve seen this all before.
(4) Yes, been there, seen that, but you missed a couple

[ if (4) — share them with us! ]

 Relative Frequency x Impact
IoT Solution
Provider’s Cloud

Mobile Core

Transit

RANs

Connected
Things

© 2020 Aeris Communications, Inc. All Rights Reserved

Our Repair Manual
1 Rapid detection, root-cause isolation,
remediation, and validation —
tapping core visibility and control
to see and act from end to end
2 Prevention through best practices in …
coverage-optimized roaming
network-aware app features
remote device configuration
life-cycle management
security
cloud infrastructure and app hardening
predictive monitoring

© 2020 Aeris Communications, Inc. All Rights Reserved

 Internet Fail
laptop Wi-Fi?

Wi-Fi router?

cable modem?

neighbor?

Cable ISP? (sigh)

personal hotspot

Result: Set for now!

© 2020 Aeris Communications, Inc. All Rights Reserved
 Internet Fail
laptop Wi-Fi?

Wi-Fi router?

cable modem?

neighbor?

Cable ISP? (sigh)

personal hotspot

Result: Set for now!

© 2020 Aeris Communications, Inc. All Rights Reserved
 IoT Fail

© 2020 Aeris Communications, Inc. All Rights Reserved

The Question for IoT Solution Management
 Mobile Core As Remote Brainpower
Control Authentication — who gets on the network
Over… Policy Definition and Enforcement — to do
what, and how much
Charging — at what rates
Billing — basis, pooling, timing
OSS

Remote All device interactions with the RAN and call flows upstream
Visibility Device location, by tower
Into…
IP address to device mapping
Traffic source and destination both directions, showing application
usage (customer’s or third parties’)
All activity sliced by customer,* device, and device pool subset*
“Crowdsourced” info from multiple customers re: network
performance, loading, and issues across multiple carriers*

© 2020 Aeris Communications, Inc. All Rights Reserved * Aeris only

 Customer Questions
When there’s a material deviation, we’re asked . . .
What % of my devices are affected?
Specific to one of my device types, or all?
In what geography?
A coverage or connectivity issue? Or in the data network?
On one MNO, or all?
A core purpose-built
for IoT solution An Aeris problem?
performance In my cloud?
assurance, expertly
Or in device’s edge network?
operated, can
answer these Which application(s)? Mine, or third parties’ ?
Upstream, downstream, or both?
What cloud endpoints (mine or others) are involved?
What’s this going to cost me? How can we control this better?

© 2020 Aeris Communications, Inc. All Rights Reserved

 Not All Mobile Cores Serve IoT Well
Visibility Available & Shared Full None
Frequency
Elements x Impact Aeris MNO MVNO

Notes: Purpose-built (not bought)

core assures and enhances
Focus is consumer mobile
subscriptions, not IoT
Advertising aside, limited
by dependency on MNOs,
IoT solution performance narrow window into core,
Core operations and little bench strength in
Run directly by our outsourced, and software network ops
responsive, expert in large vendors’ hands,
global team complicating support and
customization

© 2020 Aeris Communications, Inc. All Rights Reserved

For Example . . .
Tracking service with 1.7M devices.
We solved new-product performance issue
via cell-tower interaction analysis their MNO
declined to do, enabling timely launch.

0.7M runaway devices. Our forensics on

device-cloud interactions isolated root
cause quickly, saving $millions.

Off-grid solar provider in Africa.

Reduced TCO by 50% with coverage-
optimized multi-MNO roaming, plus
device configuration consulting.

© 2020 Aeris Communications, Inc. All Rights Reserved

Our Repair Manual
1 Rapid detection, root-cause isolation,
remediation, and validation —
tapping core visibility and control
to see and act from end to end
2 Prevention through best practices in…
coverage-optimized roaming
network-aware app features
remote device configuration
life-cycle management
security
cloud infrastructure and app hardening
predictive monitoring

© 2020 Aeris Communications, Inc. All Rights Reserved

 Benefits of Multiple MNO
Relationships and Roaming
Preference Control Primary MNO
poorer tower position

Preventing
Coverage Roaming-partner MNO

Issues
better tower position chosen
by Aeris-controlled roaming
priority list

8,000-mile US drive test on Aeris coverage-optimized multi-RAN

network showed >20% better performance than best single MNO
— our customers in Africa see even larger gains

© 2020 Aeris Communications, Inc. All Rights Reserved

 Network-Aware Apps
Device and network visibility critical in app/UX design
 Easy to create apps assuming the “happy path”
 The connected thing isn’t always so
 Example: remote operations best inactive
if vehicle has poor coverage
 Visibility into complete local “thing network” status
helps agents avoid long, complex support calls
 App instrumentation and logging is essential here

Large potential
support issues

© 2020 Aeris Communications, Inc. All Rights Reserved

 Device Configuration
To increase end-to-end reliability and reduce device-driven costs:
 Randomize network connection timing (initial and retries)
 For sessions conservation and network purge immunity, create
PDP context (GSM) or register (LTE) only when data ready to
send, then delete PGP context / deregister immediately after
 Set network mode for required use only (voice/SMS -> circuit
switched, or data -> packet switched; not both if unnecessary)
 Drive self-recovery reset with a watchdog timer when edge-
network/-device failure affects connectivity
 Support OTA roaming list updates; treat them as read only

© 2020 Aeris Communications, Inc. All Rights Reserved

Life Cycle Management
Ensure that OTA updates to SIM, radio module, device, and edge
network elements are flawless, especially for fixed applications
 Define small, representative pilot subset of devices
 Ensure detailed monitoring of end-to-end network behavior
 Test updated devices through reasonable range of conditions
 Evaluate carefully
 Update remaining devices in controlled stages, validating each one

© 2020 Aeris Communications, Inc. All Rights Reserved

 Security
We’ve seen no malicious security breaches in our customer base
 Current, common best practices are working well:

firmware lockdown no default data authentication

and secure boot passwords encryption protocols
tight role-based traffic lock-down to abnormality
access control only necessary routes alerts

Experience also suggests:

 Containment of self-DoS attacks via throttling and data caps
 Close monitoring for unsanctioned use of edge networks open to
Wi-Fi clients

© 2020 Aeris Communications, Inc. All Rights Reserved

Cloud Hardening
Manage end-to-end app performance proactively
 Continuous [ measure  analyze  improve  validate ] cycle
 Cloud, device, web, and mobile scope
 Architecture for horizontal and vertical scaling
Protect your cloud
 Throttling at load balancer, API gateway, and internal routers,
differentiated by device and application classes
 Timeouts in synch with devices
Instrument everything, and test, test, test
 Performance at full scale (in pre-production)
 Configuration changes that could yield service outages
(very, very carefully!)
© 2020 Aeris Communications, Inc. All Rights Reserved
00:00 04:00 08:00 12:00 16:00 20:00

On Prediction
 AI/ML-driven Smart Alarms

Confidence
Interval

© 2020 Aeris Communications, Inc. All Rights Reserved

IoT solutions have plenty of opportunities to go astray