Nosql: Basics: Alexey Zinovyev, Java/Bigdata Trainer in Epam

NoSQL: Basics
Alexey Zinovyev, Java/BigData Trainer in EPAM

With IT since 2007
With Java since 2009 About
With Hadoop since 2012
With EPAM since 2015
Contacts
E-mail : Alexey_Zinovyev@epam.com
Twitter : @zaleslaw @BigDataRussia
Facebook: https://www.facebook.com/zaleslaw
vk.com/big_data_russia Big Data Russia
vk.com/java_jvm Java & JVM langs

Training from Zinoviev Alexey 3
The Good Old Days

One of these fine days...

We have a NoSQL job for you, son!

But you like SQL and HATE nontraditional data

Lets talk about it, boy...

The Database Market

Lets use Cassandra to keep logs ONLY ..

For logs
ONLY?

Case #0 : PosgreSQL & Cassandra

Modern
Backend Java in 2016
Development in 1997
Backend Development in 2017
10^6
rows in
MySQL

GB->TB->PB->?

This chapter covers
Highload applications
Sharding & replication
NoSQL databases

Case #1 : Likes in Classmates

HIGHLOAD
APPLICATIONS

Case #2 : Pet E-Commerce

Simple web application #1








Big Table

Sharding

Classic Sharding

Sharding with Locator

Case #3 : Supercomputer for President

Motivation: Scale-up vs scale-out
16 CPUs Scale - Up 48 CPUs
16 CPUs Scale - Out 16 CPUs 16 CPUs 16 CPUs

Motivation: scalability
5TB
I
300GB
50GB

Motivation: Fault tolerance

Motivation: Fault tolerance

Case #4 : DDos attack against retail company

Replication

You'd measure performance!

NOSQL

Replication

CAP
Theorem

Databases
Cassandra
Hbase
Neo4j
Riak

Do you know data models in
these databases?

Databases with types
Cassandra (column family)

Hbase (column family)
Neo4j (graph)
Riak (key-value)

Database
party

Whats the problem with RBDMSs
Caching
Master/Slave
Cluster
Table Partitioning
Sharding

Evolution

Flowers in your garden
Data Model Performance Scalability Flexibility Complexity Functionality
Keyvalue high high high none variable (none)

Stores


Stores
Column high high moderate low minimal
Store


Stores
Store
Document high variable high low variable (low)
Store (high)


Stores
Store
Store (high)
Graph variable variable high high graph theory
Database


Stores
Store
Store (high)
Graph variable variable high high graph theory
Database
Relational variable variable low moderate relational
Database algebra

Gentle NoSQL
Scalability
Nodes and Data Centers
Easy to add new server
Specific data model
Eventual consistency

ACID in
SQL

Atomicity in NoSQL
read-write-modify (CAS)
key/row manipulation is atomic
API for atomic operations
bad support of transactions

BASE
basic availability all queries will be finished

soft state state can be changed without writing
eventual consistency

Database Data model Query API Data storage system
Cassandra Column Family Thrift Memtable/SSTable

CouchDB Documents Map/Reduce Append-only-B-tree

Hbase Column Family Thrift, REST Memtable/SSTable on

HDFS


HDFS
MongoDB Documents Cursor B-tree


HDFS
Neo4j Edges/Verticies Graph On-disk linked lists


HDFS
Neo4j Edges/Verticies Graph On-disk linked lists
Riak Key/Value Nested hashes, Hash

REST

Whats about Query Languages
in these databases?

Mongo
16 mb
JavaScript at the bottom
2d, 3d, B-tree indexes
3.0 version is very hot
integration with Kafka

Cassandra
CQL
2 billions columns in row
No ACID, of course
Can spend all your RAM
JVM - based

Riak
Links to another keys

REST
Consistency level in
each query
Ring of nodes

Neo4j
Vertices, edges
ACID
REST API + Cypher
2d index
Not so good for
distributed data
And what should we choose?

Network Rule
Can your data be presented as network or graph?
If yes -> Neo4j
If no -> continue

BigData Rule
Do you have TB or PB of data?
If yes -> NoSQL
If no -> Maybe you should stay with SQL?

Easy Rule
Do you need in more complex operation than read/write by

key?
If yes -> continue
If no -> Riak, Memcached

Hierarchy Rule
Do you have nested data? Is R >> W in your system?
If yes -> MongoDB, CouchDB
If no -> continue

Hadoop Rule
Is Hadoop integration required? Can your data be presented

as flat table?
If yes -> Hbase
If no -> continue

Availability Rule
Do you need in high availability? Are you agree with

eventual consistency?
If yes -> Cassandra
If no -> continue

Case #5 : Go back to PostgreSQL

CASSANDRA

Cassandra Features
LINEAR SCALABILITY MASTER-LESS ARCHITECTURE TUNABLE CONSISTENCY
LOREM IPSUM DOLOR AMET

Nulla nu nisi
Risus purus id fusce
MULTI
LobortisDATA
ipsum felis sed
CENTER SUPPORT MASSIVE WRITE AND FAST RESPONSE FLEXIBLE DATA MODEL

Cassandra Ring Architecture

Cassandra Replication
SimpleStrategy
NetworkTopologyStrategy
Cassandra Tunable Consistency
Write Consistency Read Consistency
ALL ALL
EACH_QUORUM EACH_QUORUM
QUORUM QUORUM
LOCAL_QUORUM
LOCAL_QUORUM
ONE
ONE
TWO
TWO
LOCAL_ONE
LOCAL_ONE
ANY

DATA MODEL

Data Model: Map<Map<Key, Value>>
Wide rows (2 billion cols)

Column names are data

KeySpace
CREATE KEYSPACE Excelsior WITH REPLICATION ={ 'class' :

'SimpleStrategy', 'replication_factor' : 3 };
CREATE KEYSPACE "Excalibur" WITH REPLICATION ={'class' :

'NetworkTopologyStrategy', 'dc1' : 3, 'dc2' : 2};

Tables
CREATE TABLE users

( id int PRIMARY KEY, first_name text,
last_name text, age int, hobbies list<test>);
CREATE TABLE users

( id int, first_name text, last_name text,
age int, hobbies list<test>, PRIMARY KEY(id));

Cassandra CQL (KEYS)
No Use
NO joins Data denormalization
NO subqueries Data duplication
NO Group By functionality PK, Indexes
NO aggregation functions Counters,
CAS Transactions, Batches
Keys Types Advanced Data Structures
partition key
Simple primary key PRIMARY KEY (empID)
partition key clustering key
Compound primary key PRIMARY KEY (empID, subdeptID, depID)
partition key clustering key
Composite primary key PRIMARY KEY ((block_id, breed), color)
User-defined types
CREATE TYPE address (
street text,
city text,
zip_code int,
phones set<text> );

User-defined types
CREATE INDEX state_key ON users (state);
SELECT * FROM users WHERE gender = 'f' AND state = 'TX'

ALLOW FILTERING;

Tables
CREATE TABLE users (
user_id text PRIMARY KEY,
first_name text, last_name text,
emails set<text> );
INSERT INTO users (user_id, first_name, last_name, emails)
VALUES ('frodo', 'Frodo', 'Baggins',
{'f@baggins.com', 'baggins@gmail.com'});
TTL
INSERT INTO clicks ( userid, url, date, name) VALUES (

3715e600-2eb0-11e2-81c1-0800200c9a66, 'http://apache.org',
'2013-10-09', 'Mary')
USING TTL 86400;

Ordering
CREATE TABLE timeseries (
event_type text,
insertion_time timestamp,
PRIMARY KEY (event_type, insertion_time))
WITH CLUSTERING ORDER BY (insertion_time DESC);
SELECT * FROM emp
WHERE empID IN (130,104) ORDER BY deptID DESC;

Slicing
//To retrieve events for the 12th of January 2014 between

3:50:00 and 4:37:30:
SELECT * FROM timeline WHERE day='12 Jan 2014' AND (hour,

min) >= (3, 50) AND (hour, min, sec) <= (4, 37, 30);

Batching
BEGIN BATCH
INSERT INTO purchases (user, balance) VALUES ('user1', -8)

IF NOT EXISTS;
INSERT INTO purchases (user, expense_id, amount,

description, paid) VALUES ('user1', 1, 8, 'bur', false);
APPLY BATCH;

Counters
CREATE TABLE counters.page_view_counts (counter_value

counter, url_name varchar, page_name varchar, PRIMARY KEY
(url_name, page_name) );
UPDATE counters.page_view_counts SET counter_value =

counter_value + 1 WHERE url_name='www.datastax.com' AND
page_name='home';

Lightweight Transactions
INSERT INTO users (login, email, name, login_count) VALUES

('jdoe', 'jdoe@abc.com', 'Jane Doe', 1) IF NOT EXISTS;
UPDATE users SET email = janedoe@abc.com WHERE login =

'jdoe' IF email = jdoe@abc.com;

JAVA DRIVER

DataStax Java Driver
DataStax developed a new protocol that doesn't have RPC

limitations (Asynchronous I/O)
Low-level API with simple mapping

Works with CQL3
QueryBuilder reminds CriteriaAPI
Accessor-annotated interfaces
Async
Query

Accessor

Do you want more adventures?

Other Cassandras OM
Achilles : well documented and provides transactions

Astyanax : connection pool, thread safety and pagination
Pelops : old project, good bycicle
PlayORM : strange but powerful thing
Easy-Cassandra : simple annotations + CRUD
Thrift as low level API
CASSANDRA INTERNALS

Cassandra write path (COMMITLOG)

Cassandra write path (MemTable)

Cassandra write path (SSTables)

Cassandra write path

Cassandra write path (FLUSH)

Cassandra write path (COMPACTION)

Cassandra READ path

Bloom
Filter

Case #6 : Strong Wall with Bloom Filters

Cassandra Virtual nodes

CASSANDRA TOOLBOX

# vim /etc/cassandra/conf/cassandra.yaml
cluster_name: 'cassandra_cluster
seeds: "192.168.10.1, 192.168.10.2, 192.168.10.3
listen_address: 192.168.10.1
# vim /etc/cassandra/cassandra-topology.properties
Config # Cassandra Node IP=Data Center:Rack
changes 192.168.10.1=dc1:rac1
192.168.10.2=dc1:rac1
192.168.10.3=dc1:rac1
# default for unknown nodes
default=DC1:r1

$ assandra
$ nodetool status
-- Address Load Tokens Owns (effective) Host ID Rack

UN 192.168.10.1 128.89 KB 256 62.4% 1a360230-95c7-44ec-a61f-f314e374da6e rack1
Check UN 192.168.10.2 89.97 KB 256 67.4% e92cc1f5-69e2-4fe0-bf4b-28c1bf5b0131 rack1
cluster UN 192.168.10.3 147.77 KB 256 70.2% 27f5040b-072f-4ec7-bb7c-62021a454e39 rack1
$ cqlsh 192.168.10.1
cqlsh> create keyspace data with
replication={'class':'SimpleStrategy','replication_factor':2};
cqlsh> use data;

DataStax Studio

NoSQL: Basics + Mongo
Alexey Zinovyev, Java/BigData Trainer in EPAM

MONGO WORLD

Case #7 : Customer wants MongoDB

Mongo
Driver

BSON (something like JSON)
Adds data types that JSON did not support (ISO Dates,
ObjectId, etc.)
Optimized for performance

Adds compression

MONGO TOOLBOX

Install

Install

Install

Robo 3T [Robomongo]

Demo with JavaDriver 3.3

Do you have an ORM framework?

ODM

Morphia
Object Document Mapper

Specified with annotations
Implemented with reflection
Runtime validation

Morphia advantages
Integrated with Spring, Guice and other DI frameworks

Lifecycle Method Annotations (@PrePersist, @PostLoad)
Built on top of Mongo Java Driver
More better than old-style queries by BSON-object
It has convenient Query API

Morphia
Entity

Morphia
Entity

Other Mongos OM
Jongo : mongo - shell queries in Java-code

EclipseLink : different support of different NoSQL
databases
MJORM : Google Code, XML mapping + MQL (SQL syntax

for Mongo data extracting)
DataNucleus : support many Js as JDO, JPA

Modern
Web App

Polyglot Persistance
Redis: Rapid access for reads and writes. No need to be

durable
RBDMS: Needs transactional updates and has tabular

structure.
Riak: Needs high availability across multiple locations.

Can merge inconsistent writes

Polyglot Persistance
Neo4j: Rapidly traverse links between friends and

ratings.
MongoDB: Lots of reads, infrequent writes. Powerful

aggregation mechanism.
Cassandra: Large-scale analytics on large cluster. High

volume of writes on multiple nodes

SPRING INTEGRATION

Spring
Data

Spring Data MongoDB
Templating : connection configs, collection lifecycle

(create, drop), Map/Reduce + Aggregation
Mapping: @Document, @Index, @Field

Repository support: geospatial queries, queries derived
from method signatures (at runtime)
Paging, sorting, CRUD operations

<mongo:mongo host="${mongo.host}" port="${mongo.port}">
<mongo:options
connections-per-host="${mongo.connectionsPerHost}
threads-allowed-to-block-for-connection
multiplier="${mongo.threadsAllowedToBlockForConnectionMultiplier}
connect-timeout="${mongo.connectTimeout}
max-wait-time="${mongo.maxWaitTime}
Mongo auto-connect-retry="${mongo.autoConnectRetry}
socket-keep-alive="${mongo.socketKeepAlive}
config socket-timeout="${mongo.socketTimeout}
slave-ok="${mongo.slaveOk}
write-number="1
write-timeout="0
write-fsync="true"/>
</mongo:mongo>
<mongo:db-factory dbname= "test" mongo-ref="mongo"/>

Spring &
MongoDB

Best Practice: SQL for NoSQL

Any questions?

Nosql: Basics: Alexey Zinovyev, Java/Bigdata Trainer in Epam

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Nosql: Basics: Alexey Zinovyev, Java/Bigdata Trainer in Epam

Hochgeladen von

Copyright:

Verfügbare Formate

NoSQL: Basics

Alexey Zinovyev, Java/BigData Trainer in EPAM

Twitter : @zaleslaw @BigDataRussia

vk.com/big_data_russia Big Data Russia

vk.com/java_jvm Java & JVM langs

Training from Zinoviev Alexey 4

Training from Zinoviev Alexey 5

Training from Zinoviev Alexey 6

Training from Zinoviev Alexey 7

Training from Zinoviev Alexey 8

Training from Zinoviev Alexey 9

Training from Zinoviev Alexey 10

Training from Zinoviev Alexey 11

Training from Zinoviev Alexey 12

Training from Zinoviev Alexey 15

Training from Zinoviev Alexey 16

Training from Zinoviev Alexey 17

Training from Zinoviev Alexey 18

Training from Zinoviev Alexey 19

Training from Zinoviev Alexey 20

Training from Zinoviev Alexey 21

Training from Zinoviev Alexey 22

Training from Zinoviev Alexey 23

Training from Zinoviev Alexey 24

Training from Zinoviev Alexey 25

Training from Zinoviev Alexey 26

Training from Zinoviev Alexey 27

Training from Zinoviev Alexey 28

Training from Zinoviev Alexey 29

Training from Zinoviev Alexey 30

Training from Zinoviev Alexey 31

Training from Zinoviev Alexey 32

Training from Zinoviev Alexey 33

16 CPUs Scale - Up 48 CPUs

16 CPUs Scale - Out 16 CPUs 16 CPUs 16 CPUs

Training from Zinoviev Alexey 34

Training from Zinoviev Alexey 35

Training from Zinoviev Alexey 36

Training from Zinoviev Alexey 37

Training from Zinoviev Alexey 38

Training from Zinoviev Alexey 39

Training from Zinoviev Alexey 40

Training from Zinoviev Alexey 41

Training from Zinoviev Alexey 42

Training from Zinoviev Alexey 43

Training from Zinoviev Alexey 44

Training from Zinoviev Alexey 45

Cassandra (column family)

Training from Zinoviev Alexey 46

Training from Zinoviev Alexey 47

Training from Zinoviev Alexey 48

Training from Zinoviev Alexey 49

Data Model Performance Scalability Flexibility Complexity Functionality

Keyvalue high high high none variable (none)

Training from Zinoviev Alexey 50

Data Model Performance Scalability Flexibility Complexity Functionality

Keyvalue high high high none variable (none)

Training from Zinoviev Alexey 51

Data Model Performance Scalability Flexibility Complexity Functionality

Keyvalue high high high none variable (none)

Training from Zinoviev Alexey 52

Data Model Performance Scalability Flexibility Complexity Functionality

Keyvalue high high high none variable (none)