_

v
a
o
y
@_
cee-tee-oh@
jf

rog

v
a
o
y
/
m
o
c
.
b
githu

REQUIREMENTS
parallel file Downloads
Parallel Download parts
interrupt/pause/resume
Progress events
Checksum caching

LETS LOOK IF WE CAN USE IT!

1. No traceable license
2. No website or docs
3. No traceable sources
4.Its an app, not a lib

JAVA.NET.URLCONNECTION
1. Memory wasteful
(buffering)
2. Minimal API
3. Blocking streams

WHAT WERE LOOKING FOR

1. Async/non-blocking
2. Event callbacks

WHAT IS IT GOING TO TAKE

1. Reactor
2. nio

pattern for lightweight

concurrency
Event driven
Threads reuse
Uses non-blocking Io

h"p://www.dre.vanderbilt.edu/~schmidt/PDF/reactor-siemens.pdf

h"p://gee.cs.oswego.edu/dl/cpjslides/nio.pdf

In Java, Reactor
means NIO

SocketChannel channel= SocketChannel.open();!

socketChannel.connect(new!
InetSocketAddress("http://remote.com", 80));!
...!
Selector selector = Selector.open();!
channel.configureBlocking(false);!
SelectionKey k = channel.register(selector,!
SelectionKey.OP_READ);!
k.attach(handler);!
!

while (!Thread.interrupted()) {!
selector.select();!
Set selected = selector.selectedKeys();!
Iterator it = selected.iterator();!
while (it.hasNext())!
SelectionKey k = (SelectionKey)(it.next();!
((Runnable)(k.attachment())).run();!
selected.clear();!
}!

NIO LIBRARIES
Most of them are servers
Netty, grizzly, etc.

Apache Mina
Apache HTTP components
asyncclient
Ning http client

Client and server nio

library
Evolved from netty
Latest release October 2012

NIO LIBRARIES
Most of them are servers
Netty, grizzly, etc

Apache Mina
Apache HTTP components
asyncclient
Ning http client

try (AsyncHttpClient asyncHttpClient = new AsyncHttpClient()) {

ListenableFuture<Response> future = asyncHttpClient
.prepareGet("http://oss.jfrog.org/api/system/ping")
.execute(new AsyncCompletionHandler<Response>() {

@Override
public Response onCompleted(Response response) {
System.out.println(response.getResponseBody());
return response;
}

@Override
public void onThrowable(Throwable t) {
t.printStackTrace();
}
});
Response response = future.get();
}

Request producer
Generates a request message and stream out its
content without buering it in memory

Response consumer
Processes a response message without buering its
content in memory

try$(CloseableHttpAsyncClient1asyncHttpClient1=1HttpAsyncClients.createDefault())1{1
1111asyncHttpClient.start();1
1111Future<HttpResponse>1future1=1asyncHttpClient.execute(1
111111111111HttpAsyncMethods.createGet("http://oss.jfrog.org/api/system/ping"),1
111111111111new$AsyncByteConsumer<HttpResponse>()1{1
1
1111111111111111@Override1
1111111111111111protected$void$onResponseReceived(final$HttpResponse1response)1{1
11111111111111111111System.out.println(response.getStatusLine().getReasonPhrase());1
1111111111111111}1
1
1111111111111111@Override1
1111111111111111protected$void$onByteReceived(final$CharBuffer1buf,1final$IOControl1ioctrl)1{1}1
1
1111111111111111@Override1
1111111111111111protected$void$releaseResources()1{1}1
1
1111111111111111@Override1
1111111111111111protected$HttpResponse1buildResult(final$HttpContext1context)1{1
11111111111111111111return$(HttpResponse)1context.getAttribute("http.response");1
1111111111111111}1
1
111111111111},1null);1
111HttpResponse1response1=1future.get();1
}1

"All problems in computer science can be

solved by another level of indirection"
David Wheeler

public interface HttpProviderDownloadHandler {

void onResponseReceived(int statusCode, Map<String, List<String>> headers);

boolean onBytesReceived(ByteBuffer buf);

void onFailed(Throwable error);

void onCanceled();

void onCompleted();
}

Feature/Library
Maturity
Download cancela=on
Progress hooks

Ning client
Good
Easy
Events not granular
enough
Documenta=on
A bit sparse
Server-side counterpart None, client only

H3p Async Client

Very new (early 2014)
With bugs
Just use onByteReceived()
Minimal
org.apache.h"pcomponents
h"pcore-nio

900
800
700
600

Ning

500

AHAC

400
300
200
100
0
Small le

Medium le

Large le

h"p://blogs.atlassian.com/2013/07/h"p-client-performance-io/

And that one for

discovering that
range header is
lost on redirect

h"ps://github.com/h"p2/h"p2-spec/issues/46

HttpAsyncClientBuilder builder = HttpAsyncClients.custom();

// add redirect strategy that copies "range" headers, if exist
builder.setRedirectStrategy(new DefaultRedirectStrategy() {
@Override
public HttpUriRequest getRedirect(HttpRequest request, HttpResponse response,


HttpContext context)
HttpUriRequest redirectRequest = super.getRedirect(request, response, context);
// copy "Range" headers, if exist
Header[] rangeHeaders = request.getHeaders(HttpHeaders.RANGE);
if (rangeHeaders != null) {

for (Header header : rangeHeaders) {


redirectRequest.addHeader(header);

}
}
return redirectRequest;
}});

public sta=c String encodeUrl(String urlStr) {

URLEncoder.encode(urlStr, "UTF-8");
...
}

Utility class for HTML form encoding.

This class contains static methods for
converting a String to the
application/x-www-formurlencoded MIME format.
For more information about HTML form
encoding, consult the HTML
specification.

h"p://www.safaribooksonline.com/library/view/h"p-the-deniave/1565925092/ch04s07.html

 Write to separate files,

combine on finish
Write to same file, seeking
to the right position

Implements SeekableByteChannel
java.nio.channels.FileChannel#write(
java.nio.ByteBuffer src, long position)

FileProgressInfo

FilePartProgressInfo

PersistentFileProgressInfo
Save the total size, sha1, number of parts
State of each part (oset, size, completed...)

VM level
O/S level

Prevent same VM threads writing to the file when

we started closing it
Closing sequence:
Release file locks
Close channels
Rename a file to it's final name (remove .part)
Erase progress info

ReentrantReadWriteLock.ReadLock writeToFileLock = rwl.readLock();

ReentrantReadWriteLock.WriteLock closeFileLock = rwl.writeLock();

public void close() throws IOException {
this.closeFileLock.lock();
}

public int write(int partIndex, ByteBuffer buf) {
if (!this.writeToFileLock.tryLock()) {
throw new IllegalStateException("File is being closed");
}
...
}

Multiple downloader instances writing to the same

file
Needed for writing:
Partial download file
Persistent download progress

FileLock lock = fileChannel.tryLock();

//Non-shared: (0L, Long.MAX_VALUE, false)
if (lock == null) {
throw new OverlappingFileLockException();
}
return lock;
}

private FileLock lock(FileChannel fileChannel) throws IOException {

FileLock lock = fileChannel.tryLock(Long.MAX_VALUE - 1, 1, false);
if (lock == null) {
throw new OverlappingFileLockException();
}
return lock;
}
WTF?!

HTTP/2
Mostly standardizing Google's
spdy
Header compression
multiplexing (framing)
prioritization

On the way clears some

ambiguities
E.g. compressed content length

h"p://chimera.labs.oreilly.com/books/1230000000545/ch12.html#REQUEST_RESPONSE_MULTIPLEXING

RTFM: RFC 2616

Ultimate book: HTTP: The Definitive Guide
Amazon
Safari

Reactor pattern
Doug Lea on NIO