You are on page 1of 53

BonFIRE Open Calls Information Telcon, 7 February 2012

Building service testbeds on FIRE

Technical Overview of BonFIRE Facility Architecture entry points, Available infrastructure capacity, Setting up experiments Florian Schreiner, Fraunhofer Institute FOKUS Kostas Kavvousanakis, EPCC W: www.bonfire-project.eu E: bonfire@bonfire-project.eu

Agenda BonFIREs Experimental Facility Architecture BonFIREs Infrastructure, available Resources BonFIREs Experiment Support Examples of current BonFIRE Experiments Demo of BonFIRE Experiment Setup and Monitoring Procedure

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 2

High Level View

Controlled deployment Monitoring Support Networks:


Std. Int. Emultd. Contrld. Permanent and on-request infrastructure facilities

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 3

Three Scenarios Service Experiments on top of three different Network Infrastructures


1. Extended multi-site clouds connected through standard internet 2. Cloud scenario with emulated network (IBBTs Virtual Wall) 3. Extended Cloud scenario with controlled network (implies federation)

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 4

Monitoring dashboard

Architecture
LDAP

Identity Server
(Used by Portal, Experiment Manager, Broker and Testbeds)

Monitoring

Read/ Write

Message Queue

SSH

SSH SSH Gateway Gateway

Monitoring GUI

Monitoring API

VM (Monitoring Aggregator)

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 5

BonFIRE Offering (1/2)


Support experiments over multiple heterogeneous cloud testbeds using a single declarative experiment descriptor. Support geographically distributed experiments. Support experiment monitoring at both resource level (e.g. CPU usage, temperature, packet delay etc.) and application level. Support the deployment of different software stacks over a variety of differently configured resources (compute, storage, network etc.) in multiple heterogeneous cloud testbeds. Support elasticity within an experiment, i.e. dynamically create, update and destroy resources from a running node of the experiment, including. cross-testbed elasticity.

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 6

BonFIRE Offering (2/2)


Support experiment management including experiment sharing, repeating and result collation and storage. Support the definition of an entire infrastructure in a single uniform experiment description. Study the possible federation of the BonFIRE testbeds with a variety of external cloud facilities, such as those provided by Federica or OpenCirrus. Support advanced network emulation via the Virtual Wall, including:
Dynamic modifications of running experiments (at the moment the network topology and node images have to be fully configured at the start of the experiment.) Additional generic network (e.g. overlay routing) and application layer functionality

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 7

Architecture Principles
It must always be possible to include testbeds over which BonFIRE has no control in a BonFIRE federation if the appropriate adapters are written and deployed into the BonFIRE system. It should not be necessary to force a testbed to change its functionality in order to be included in BonFIRE. Always provide APIs to BonFIRE functionality, in addition to any BonFIRE graphical user interfaces (GUIs). Allow experimenters full access to the specific functionality of particular testbeds; i.e. functionality that belongs only to one of the clouds and is not available through the BonFIRE API. Allow federated functionality to exclude specific functionality of a testbed if this makes common tasks easier to achieve. Support incremental adoption of the BonFIRE system by experimenters. Support declarative specification of experiments as far as possible.
Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 8

Agenda BonFIREs Experimental Facility Architecture BonFIREs Infrastructure, available Resources BonFIREs Experiment Support Examples of current BonFIRE Experiments Demo of BonFIRE Experiment Setup and Monitoring Procedure

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 9

BONFIRES INFRASTRUCTURE & RESOURCES


10
BonFIRE 10

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIREs Operations Support

To ensure the operation of the experimental facility Appropriate planning and control of the evolution of the permanently provided infrastructure Provision of on-request resources for large scale experiments in a well defined environment Support monitoring of experiments under well defined conditions Allowing repetition of experiments under similar conditions

11

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 11

The BonFIRE Service Oriented Infrastructure

Component 1 The Core Infrastructure Provided permanently by the BonFIRE consortium members Used as general purpose platform for medium scaled experiments Architecture and software provided by BonFIRE
12
Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 12

The BonFIRE Service Oriented Infrastructure

Component 2 Infrastructure/Resources on request On-request provisioning Not exclusively for BonFIRE Availability for limited times for large scale testing

13

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 13

The BonFIRE Service Oriented Infrastructure

Component 3 Experiment-owned infrastructure Experiments may have own small scale testbeds BonFIRE as extension to these testbeds or vice versa Effort needed to allow the connection of experiments infrastructure and the BonFIRE infrastructure
14
Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 14

BonFIREs Infrastructures and Resources


EPCC: 1 frontend + 2 worker nodes + 1 storage node, upto-date OpenNebula v3.0 w/ OCCI server HLRS: 1 frontend + 10 worker nodes, up-to-date OpenNebula v2.0.1 w/ OCCI server HP: Using the Cells infrastructure IBBT: Using the VirtualWall infrastructure INRIA: OpenNebula, using parts of Grid5k infrastructure PSNC: OpenNebula, exclusively for controlled network experimentation
15
Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 15

Agenda BonFIREs Experimental Facility Architecture BonFIREs Infrastructure, available Resources BonFIREs Experiment Support Examples of current BonFIRE Experiments Demo of BonFIRE Experiment Setup and Monitoring Procedure

16

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 16

BONFIRES EXPERIMENT SUPPORT


17
BonFIRE 17

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIREs Experiment Support

To Support the experiments carried out by BonFIRE customers


Support Tools Tutorials, Guidelines Individual support via email

Help experimenters to install software/applications and their own tools into the BonFIRE facility Support migration of an experiment from the experimenter's in-house testbed into BonFIREs facility Help experimenters to find appropriate third party tools and install these tools into the BonFIREs facility Clearly define the services a customer may ask for
Enable customers to understand the capabilities of BonFIRE

18

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 18

Agenda BonFIREs Experimental Facility Architecture BonFIREs Infrastructure, available Resources BonFIREs Experiment Support Examples of current BonFIRE Experiments Demo of BonFIRE Experiment Setup and Monitoring Procedure

19

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 19

Examples for current embedded and Open Call Experiments


BonFIRE is currently running 3 embedded experiments and 4 experiments from the 1st Open Call

Virtual Clusters on federated sites (1st


Open Call Experiment)

QoS-Oriented Service Engineering for Federated Clouds (Embedded


Experiment)

Elasticity in Cloud-based web applications (Embedded Experiment)

20

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 20

Example use case: VCOC Virtual Clusters on federated sites


To get a better understanding about the deployment and usage of a multi-site, Cloud virtual cluster
Three main topics: I. The time to deploy a virtual cluster on a federated infrastructure; II. The performance penalties associated to a distributed cluster; III. And the benefit from the distributed nature of the federation to tackle site failures

21

FOKUS, ATOS Origin and other members of the BonFIRE Fraunhofer Institute Centre of Supercomputing of Galicia (CESGA) consortium 2010

BonFIRE 21

Example use case: VCOC Virtual Clusters on federated sites


Experiment Hypothesis:
H1. For each application and dataset it is possible to deploy a distributed virtual cluster where the total time-to-solution (including time to deploy, execution of the application and to retire of the virtual cluster) is more favourable in a federated Cloud than in a single provider. H2. On a heterogeneous Cloud infrastructure, it is possible to adapt dynamically a virtual distributed cluster to guarantee the Quality of Service (i.e., the time-to-solution) using the performance of the application as information to trigger the change. H3. A federated Cloud infrastructure can be used to reduce the lost time due to failures for a virtual cluster without a significant overcost.

22

FOKUS, ATOS Origin and other members of the BonFIRE Fraunhofer Institute Centre of Supercomputing of Galicia (CESGA) consortium 2010

BonFIRE 22

VCOC requirements on the testbed

Controlled deployment Controlled network Very fine-grained monitoring Elasticity and cross-site elasticity Steering triggered by events

A Cloud facility for experimentation


23
FOKUS, ATOS Origin and other members of the BonFIRE Fraunhofer Institute Centre of Supercomputing of Galicia (CESGA) consortium 2010

BonFIRE 23

Examples for current embedded and Open Call Experiments


BonFIRE is currently running 3 embedded experiments and 4 experiments from the 1st Open Call

Virtual Clusters on federated sites (1st


Open Call Experiment)

QoS-Oriented Service Engineering for Federated Clouds (Embedded


Experiment)

Elasticity in Cloud-based web applications (Embedded Experiment)

24

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 24

Problem Statement
Make better QoS provisioning decisions

Why?

Portability and federation between providers

How to predict performance of applications deployed on different IaaS clouds?

Models Resource description


25

Hardware or benchmark scores BonFIRE 25

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

Experiment Hypothesis

Hypothesis: If IaaS resources are described in terms of application benchmark scores then PaaS performance models will be faster, cheaper and more accurate Approach: replace low-level resource metrics (e.g. CPU architecture, clockspeed, RAM, EC2 compute units) with benchmark scores based on defined patterns of communication and computation (e.g. dwarfs)

?
IaaS SLA IaaS Provisioning IaaS Virtualisation

26

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 26

Role in BonFIRE
BonFIRE Capability Heterogeneous hardware Detailed monitoring information Uniform request and access methods Repeatability of hardware allocation Cross-site federation Control of the network Access to long-term storage and experimental data Authentication and authorisation Why the experiment needs it The models must be predictive on a wide range or target platforms Monitoring info on several levels is required to create the models To make access to all the resources at different sites easy So that benchmark scores may be compared with test application data For complex test applications To simulate different network conditions Essential for off-line analysis of experiments Ensuring that experiments are not interfered with

The vast majority of these capabilities are of use to all experiments. The experiment will help BonFIRE by providing concrete requirements for these features and by testing and validating the infrastructure, the documentation and the support systems. BonFIRE 27 27 Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

Examples for current embedded and Open Call Experiments


BonFIRE is currently running 3 embedded experiments and 4 experiments from the 1st Open Call

Virtual Clusters on federated sites (1st


Open Call Experiment)

QoS-Oriented Service Engineering for Federated Clouds (Embedded


Experiment)

Elasticity in Cloud-based web applications (Embedded Experiment)

28

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 28

Technical overview
Develop a web-app for test purposes, and generate an image inside a VM to be instantiated at any time. Create a dummy load generator. Define load scenarios. An application must be developed in order to: Collect /store/process KPI (on demand or periodically). Check, in all scenarios, the SLAs accomplishment. Manage rules of SLAs. Generate changes over the testbed scheme. Define a testbed in the cloud with some minimum initial requirements eg: Number of VM Geographical location of resources. Network topology Identify the KPIs for each load scenario Set the upper and lower thresholds of the initial testbed scheme by load tests. Define the set of KPIs values that trigger the release or the enlargement of the resources deployed. Using a time diagram of a load model, resize the testbed resources in advance. With the information collected about KPI in a defined period of time generate a load model inferring the ideal deployment.

29

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 29

Example of Embedded Experiments to prepare the Infrastructure


Practical example Elasticity in Cloud-based web applications: The experiment tests in a comprehensive manner the whole functionality offered by BonFIRE:
CRUD of VMs Complex network topologies Geographical dispersed allocation of resources, DNS, storage, etc. Measurement of the latency of BonFIRE as infrastructure provider in a wide variety of load scenarios. Resize or modify resources dynamically. Dynamic configuration of load balancers. The use of resources through federation with other cloud providers in some scenarios. Monitor API
BonFIRE 30

Improvements to

30

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

Experiment Objectives
To optimize the distribution of web-apps containers. To investigate new scalability policies. To be able to predict the ideal deployment under the upcoming load. Learn how to resize dynamically the number of resources deployed, from a load model of a web application, inside a cloud federation infrastructure. Develop algorithms and procedures to avoid overprovisioning resources and to satisfy SLAs with web-app users. To study the concept of infinite elasticity in the cloud.

31

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 31

Experiment: in more detail


http/https KPI-collector Changes-generator TESTBED MANAGEMENT DLG
http://www.web-app.com

KPI/ SLAs POLICIES/ LOAD TESTS

EXPERIMENT -MANAGER

KPI

LB OPERATIONS WS AS WS AS DB WEB-APP WS AS

CLOUD-MANAGER

MONITORING
VM-REPOSITORY

32

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 32

What is needed from BonFIRE?

A cloud ready to run VMs offering the various web app containers and databases. Definition of different network schemes. Access to resources on-demand. Federation with other clouds providers. A service to finely monitor the VM containers. Access to geographically separated sites.

33

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 33

Benefits for BonFIRE

Comprehensive test of BonFIRE. Better understanding of BonFIRE elasticity capabilities. Monitoring process. Users interface (OVF, portal) Dynamic testbed configuration. Creation of scalability policies. Simulation of realistic scenarios with geogr. sep. sites. Improvement of some features of the BonFIRE facility:
Load balancing. Evolution of web traffic generators.

34

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 34

Agenda BonFIREs Experimental Facility Architecture BonFIREs Infrastructure, available Resources BonFIREs Experiment Support Examples of current BonFIRE Experiments Demo of BonFIRE Experiment Setup and Monitoring Procedure

35

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 35

Building service testbeds on FIRE

Demo

This Demo
Client-server application
Server VM on EPCC BonFIRE site

Monitoring
VM on EPCC BonFIRE site Display CPU load Configure and monitor application metric

Interactions with the BonFIRE Portal Cross-site elasticity Controlled network


37
BonFIRE 37

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

The user application


The experiment will use the iperf client-server application Scenario: assess the impact of multiple iperf clients accessing the same iperf server
Impact on iperf server VM performance
5 4 3 2 1

Iperf Server

Here is how BonFIRE supports this application

Monitoring

38

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 38

Building service testbeds on FIRE

Set up experiment

What is an experiment?
Collection of Compute, Storage and Network resources, each associated with a BonFIRE site Software implementing the experiment logic

40

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 40

Building service testbeds on FIRE

Create VMs

Clients and Servers


I already have a server Create client VM and point to server

42

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 42

Building service testbeds on FIRE

Monitoring

Set up Monitoring
BonFIRE base images include Zabbix monitoring client Zabbix monitoring server deployed as a separate VM

44

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 44

Building service testbeds on FIRE

Cross-site Elasticity

Monitoring for (Cross-Site) Elasticity


162 # Check metric values: 163 values = experiment.zabbix.metric('system.cpu.util[,system,avg1]', :type => :numeric, :hosts => server).values 164 avg3, avg5 = [values[0..3].avg, values[0..5].avg] 169 # Here if the CPU usage of the iperf server is too low, we'll spawn a 170 # new client. If it's too high, we'll shut a client down. 171 if clients_count <= 10 && values.length >= 3 && avg3 <= 20 172 session.logger.warn "Scaling UP (avg=#{avg3})!" 173 vm = create_client(session, experiment, locations, { 174 'aggregator_ip' => aggregator_ip, 175 'iperf_server' => server_ip 176 }) 177 sleep(10) until vm.ssh.accessible? 178 elsif clients_count > 1 && values.length >= 5 && avg5 >= 22 179 session.logger.warn "Scaling DOWN (avg=#{avg5})!" 180 # Delete the first client of the location which has the most clients: 181 locations.sort{|loc1,loc2| 182 loc2[1].size <=> loc1[1].size 183 }.first[1].shift.delete 186 end

46

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 46

Building service testbeds on FIRE

Controlled network

Emulated Control
Virtual Wall: Large-scale testbed in IBBT Based on Emulab software Add a private network with configured parameters:
> > > > > > > > > private_network = experiment.networks.submit( :location => ibbt, :name => "network-experiment#{experiment['id']}", :bandwidth => 1000, :latency => 0, :size => 24, :lossrate => 0, :address => "192.168.0.0" )

48

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 48

Controlled network
To interconnect with FEDERICA during the summer
Advanced network experimentation, supported by GEANT3 Ability to control routing on FEDERICA slices Possibility to route EPCC-PSNC traffic through FEDERICA

To provide access to AutoBAHN


GEANT3 experimental Bandwidth on Demand service Involving traffic between EPCC and PSNC

Option to route traffic through Virtual Wall


Emulated, highly configurable.

49

Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 49

Building service testbeds on FIRE

Summary

Facility for services experimentation


6 sites; 350 virtualised cores; 700GB of RAM; 30TB storage
On-request access to 2,000 additional, multi-core nodes

Real and emulated networks


Emulab-based Virtual Wall Controlled networks

Experiment Descriptors
Portal Restfully JSON DSL (OVF on the way)

Advanced monitoring
Zabbix on all VMs Infrastructure monitoring

What will you add to it?


51
Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 51

Building service testbeds on FIRE

Thank you for your attention

+ =

W: www.bonfire-project.eu E: bonfire@bonfire-project.eu
53
Fraunhofer Institute FOKUS, ATOS Origin and other members of the BonFIRE consortium 2010

BonFIRE 53