SpringOne2015 StephaneMaldiniGlennRenfroDavidTuranski HighPerformanceStreamProcessing

SPRINGONE2GX
WASHINGTON, DC
High Performance Stream Processing

By Stephane Maldini, Glenn Renfro, David Turanski
@smaldini, @cppwfs, @dturanski
U n l e s s ot h erwi se i ndi cat ed , t hese sl id es ar e 201 3-2 01 5 P ivo tal So ftwa re, In c. and lic e nse d u nd e r a C re ativ e C ommons Attrib u tion-N onC omme rc ial li c e ns e : h tt p: / / c r ea t i vec o m m o n s .o r g/ l i c en s es / by- n c / 3 .0 /
Performance as it pertains to:

Message flow
Serialization
Processing
Un l ess otherwi s e indi cate d, t he se sl i des ar e 201 3 - 201 5 P ivo t al So ft war e, Inc . and lic e nse d u nd e r a
C reat iv e Co mmons At tr i butio n-No nComme r cial lic e ns e : ht t p:// cr e at iv ec o mmo n s.o rg /lic e nse s/b y -nc /3 .0/
Message Flow: The Myth?
1 million events a second?
It Depends
https://spring.io/blog/2015/06/17/spring-xd-benchmarks-part-1
Hardware
Check Network speed

iperf
Check Disk Read Write speed
dd
Processor Speed (specs)
Hardware: Network
1 Gb Ethernet
Msg Size
Msgs/Sec
10 Gb Ethernet
Msg Size
Msgs/Sec
100
1,250,000
1,000
125,000
1,000
1,250,000
10,000
12,500
10,000
125,000
100,000
1,250
100,000
12,500
100 12,500,000
Hardware: Disk
dd bs=1M count=256 if=/dev/zero of=test

will just commit your 128 MB of data into a RAM buffer
initially fast but server is still writing to disk after test
dd bs=1M count=256 if=/dev/zero of=/tmp/testfile

conv=fdatasync
This tells dd to require a complete sync once, right before it
exits.
Ensures all data is on the disk before calculating result
https://romanrm.net/dd-benchmark
Message Size
Message Size
10
Transports
Rabbit
perftest
https://www.rabbitmq.com/java-tools.html
Kafka
ProducerPerformance
kafka-consumer-perf-test.sh
https://engineering.linkedin.com/kafka/benchmarking-apachekafka-2-million-writes-second-three-cheap-machines
11
Configuration
Kafka
BatchSize
o Default is 16384
o .0.8.2.1 vs .0.8.1.1
Rabbit
prefetch
o Default for XD is 1
12
Testing Tools
Spring XD
load-generator-source
throughput
https://github.com/spring-projects/spring-xd-modules
13
Batching
Spring AMQP
Batch
Size
Msgs
Per Sec
1
https://github.com/cppwfs/rabbitmqsink
https://github.com/cppwfs/rabbitmqsource
18,465
10 158,564
100 453,926
14
Network Hops
Adds a cost per hop

Options:
Direct Binding
Composed Modules
Custom Module
15
One last little thing

JMX is disabled by default
When enabled it took a performance hit
because how SI was capturing stats via its
exporters
16
Object Serialization
POJO
byte[] required for transporting data between
remote processes
XD uses Kryo except when the payload type is
byte[] or String
XD supports optimizing Kryo for known payload
types
17
Serialization Benchmarks
An excellent comparative JVM serializers benchmark:
https://github.com/eishay/jvm-serializers/wiki
Best case: ~1500 ns

18
Serialization Benchmarks
Domain Object (as JSON):
{"media":
{"uri": "http://javaone.com/keynote.ogg", "title":"Javaone Keynote", "width":640,
"height":480,
"format":"video/mpg4","duration":18000000,"size":58982400,"bitrate":
262144,"persons":["Bill Gates", "Steven Jobs"],
"player":"JAVA","copyright":"" },
"images":[
{"uri": "http://javaone.com/keynote_large.jpg","title":"Javaone Keynote","width":
1024,"height":768,"size":"LARGE"},
{ "uri": "http://javaone.com/keynote_small.jpg", "title":"Javaone
Keynote","width": 320,"height":240, "size":"SMALL"}]
}
19
Object Serialization
Size matters: YMMV

Manually optimized Kryo ser/deser ~ 1500 ns = 1.5 s = .0015 ms.
Kafka XD;1000B messages ~ 500,000 msg/sec
2000 ns per message
Serialization overhead
~ 285174 msg/sec (source|sink)
At 50,000 msg/sec, the overhead may still be significant
(source|p1|p2|p3|sink)
20
Optimizing Kryo in XD
Disable references - If you know payload

types do not contain cyclic references.
xd.codec.kryo.references=false (in servers.yml)
This is a global setting for all streams

Register a custom serializer for a known
payload type
Install a jar with containing the required beans in

XD will auto-configure these
xd/lib.
21
Custom Serializers in XD
package spring.xd.bus.ext;
...
XD scans this
package for beans of
type KryoRegistrar
@Configuration
Each Registration associates
public class CustomKryoRegistrarConfig {
a type to a serializer and a
@Bean
unique ID
public KryoRegistrar myCustomRegistration() {
List<Registration> registrations = new ArrayList<>();
registrations.add(new Registration(MyObject.class, new MySerializer(),62));
return new KryoRegistrationRegistrar(registrations);
}
public PojoCodec(java.util.List<KryoRegistrar> kryoRegistrars, boolean useReferences)
22
Custom Serializers in XD
public class AddressSerializer extends Serializer<Address> {
@Override
public void write(Kryo kryo, Output output, Address address) {
output.writeString(address.getStreet());
output.writeString(address.getCity());
output.writeString(address.getCountry());
}
@Override
public Address read(Kryo kryo, Input input, Class<Address> type) {
return new Address(input.readString(),input.readString(),input.readString());
23
Serializable Domain Object

(+) Simple: This works
out of the box with no
additional
configuration
(-) Requires access to
source or wrapping
(-) Internal
benchmarks indicate
public class Address implements KryoSerializable {

...
@Override
public void write(Kryo kryo, Output output) {
output.writeString(this.street);
output.writeString(this.city);
output.writeString(this.country);
}
@Override
public void read(Kryo kryo, Input input) {
this.street = input.readString();
this.city = input.readString();
this.country = input.readString();
}
24
Benchmarking Your Custom Serializers

The spring-xd-samples repo includes a serializationbenchmarks project
https://github.com/spring-projects/spring-xd-samples/tree/master/
serialization-benchmarks
Lets look at some code
Sample Results
Ser (ns)
Desr (ns)
Baseline
2710.5
4590.9
Serializable Domain
Object
1663.3
2579.1
Custom Serializers
1126.1
2873.8
25
What about Processing ?
Source Msg/s > Sink Msg/s ?

Rate limited by Sink
Blocking transformation (http, file) ?
Rate limited by blocking Processor
Polling Sources Pausing ?
Rate limited by small Prefetch properties

26
Mitigating Cost of IO
Negative impact ?
Scale Out ?
o Works up to a point
o Network cost
Scale Up ?
o Message passing Overhead
o More In-Flight Data
27
Blocking IO
request A
P
r
o
c
e
s
s
o
r
request B
request C
R
e
m
o
t
e
Request Latency
Network Latency
Sink
Rate Degradation = -(A + B + C) ms
28
Asynchronous Boundary
R
e
m
o
t
e
P
r
o
c
e
s
s
o
r
Rate degradation = -(Async Hand O) ms
Request Latency
Network Latency
Sink
29
Asynchronous IO
Mitigate temporarily slow processors/sink
Back to degraded mode when queue full
Async Hand-Off generates Garbage
30
Reactor Core: Efficient Asynchronous
Trade-off Memory vs Garbage generation

Pre-Allocated Ring Buffer
Concurrent consuming without duplicating
buffer content
Ring Buffer Consumer Sequences
31
Ring What?
schedule Message<?> execution

get and publish
next available
slot
Event Loop
Thread
read published slot
execute Message<?>
32
Reactor Core: Efficient Asynchronous
The Spring XD Module

https://github.com/spring-projects/spring-xd/
blob/master/spring-xd-reactor
public interface Processor<I, O> {
Publisher<O> process(Stream<I> inputStream);
}
33
public class PongMessageProcessor implements Processor<Message, Message> {
@Override
public Stream<Message> process(Stream<Message> inputStream) {
return inputStream.map(message ->
new GenericMessage<String>(message.getPayload() + "-pojopong")
);
}
}
34
R
e
m
o
t
e
P
r
o
c
e
s
s
o
r
Parallel Scatter Gather !
Request Latency
Network Latency
Sink
35
Reactor Stream and RxJava
Compose asynchronous results

Without blocking (unlike future.get())
Reduce the processor/sink backlog !
36
Scatter Gather
public class AsyncNetworkProcessor implements Processor<Message, String> {
@Override
public Observable<String> process(Observable<Message> inputStream) {
return inputStream.flatMap(message ->
Observable.zip(
postHttp(/userProfile/ + message.getHeader(user_id)),
postHttp(/userLocation/ + message.getHeader(user_id)),
(respA, respB) -> respA + , + respB
)
);
}
public Observable<String> postHttp(String endpoint){
// An asynchronous HTTP call to forward response as CSV
}
}
37
Reactor Stream and RxJava
Some operators help tuning the right packet

size to send over the network
MicroBatching !
38
MicroBashing
public class AsyncNetworkProcessor implements Processor<String, String> {
@Override
public Stream<String> process(Stream<String> inputStream) {
return inputStream.window(1000, 1, TimeUnit.SECONDS)
.flatMap(messages ->
messages.reduce(, (prev, next) -> prev + , + next
)
);
}
}
39
Questions
40
Learn More. Stay Connected.

@springcentral
Spring.io/video
Microservices to Fast Data

John T. Davies
41

SpringOne2015 StephaneMaldiniGlennRenfroDavidTuranski HighPerformanceStreamProcessing

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

SpringOne2015 StephaneMaldiniGlennRenfroDavidTuranski HighPerformanceStreamProcessing

Hochgeladen von

Copyright:

Verfügbare Formate

SPRINGONE2GX

High Performance Stream Processing

Performance as it pertains to:

Message Flow: The Myth?

1 million events a second?

Message Flow: The Myth?

Message Flow: The Myth?

Check Network speed

dd bs=1M count=256 if=/dev/zero of=test

dd bs=1M count=256 if=/dev/zero of=/tmp/testfile

Adds a cost per hop

One last little thing

byte[] required for transporting data between

Best case: ~1500 ns

Size matters: YMMV

Disable references - If you know payload

This is a global setting for all streams

Install a jar with containing the required beans in

Serializable Domain Object

public class Address implements KryoSerializable {

Benchmarking Your Custom Serializers

What about Processing ?

Source Msg/s > Sink Msg/s ?

Rate limited by small Prefetch properties

Rate degradation = -(Async Hand O) ms

Reactor Core: Efficient Asynchronous

Trade-off Memory vs Garbage generation

schedule Message<?> execution

Reactor Core: Efficient Asynchronous

The Spring XD Module

public class PongMessageProcessor implements Processor<Message, Message> {

Parallel Scatter Gather !

Reactor Stream and RxJava

Compose asynchronous results

Reactor Stream and RxJava

Some operators help tuning the right packet

Learn More. Stay Connected.

Microservices to Fast Data

Das könnte Ihnen auch gefallen