Sie sind auf Seite 1von 1


Cloud Computing to Support Water Resources Modeling

Scott D. Christensen,
Nathan R. Swain,
E. James Nelson,
Norman L. Jones,

A Background

C Applications

Advances in water resources modeling are providing us with better information,

however, they require more computational power to run. Cloud computing
enables universal access to cost-effective computing, yet there still remains a
significant technical barrier to accessing these resources. Here we present a set
of Python tools, TethysCluster and CondorPy, that have been developed to
lower the barrier to modeling in the cloud by providing :

Stochastic Analysis

Tethys Platform

and is often accounted for my performing a

stochastic analysis which requires running
hundreds or thousands of model simulations.
For a spatially-distributed, physics-based models
such as GSSHA running thousands of models

(2) a batch scheduling system to queue and dispatch the jobs to the
computing resources

may take months or even years. TethysCluster

(3) data management for job inputs and outputs

(4) the ability to dynamically create, submit, and monitor computing jobs

done much faster using cloud computing.

and CondorPy enable this type of analysis to be

HTCondor is a software system that that enables


computing resources and scheduling computing

and lots of time. One way to alleviate this


interface for HTCondor, and allows jobs to be

created, submitted and monitored from a Python
scripting environment. This interface facilitates the
use of HTCondor in a web environment like Tethys
Platform (see panel D).

Large modeling tasks often require a large amount of

computing resources. Commercial cloud providers





computation by decomposing the domain into

smaller models. This results in a series of
hierarchical models whose execution must be
coordinated. CondorPy facilitates running this
type of workflow with HTCondor in a parallel
computing environment.





E Summary
Two Python modules have been developed to lower the technical barrier to

Ensemble Forecast Processing

accessing cloud computing for performing large modeling tasks. TethysCluster

TethysCluster and CondorPy are used by the Streamflow Prediction Tool (a Tethys web app) to

automates the process of provisioning diverse cloud resources and configuring

them with HTCondor. CondorPy interfaces with HTCondor to enable computing

process each ensemble forecast every 12 hours when a new forecast is available. TethysCluster can

jobs to programmatically be created, submitted, and monitored.

be used to automatically provision and de-provision cloud computing resources.

CondorPy and TethysCluster have been integrated into Tethys Platform enabling

such as Amazon Web Services (AWS), and Microsoft

web apps to easily perform large computing tasks.


however configuring them HTCondor can prove

challenging. StarCluster is a Python module that



Medium-Range Weather Forecasts. A scheduled Python script creates 52 jobs using CondorPy to


CondorPy has been integrated into

the Tethys Platform Python SDK in the
form of a job manager that enables
developers to define computing jobs
and submit them to the HTCondor
pools to offload large computing

domains often requires powerful computers

automatically process a 52-member ensemble forecast produced by the European Center for

jobs. It enables diverse computing systems to be

CondorPy serves as a cross-platform, high-level

Job Manager

Top: large watershed shown divided into hierarchical sub-basis.

Bottom: Diagram showing the parallelization and hierarchy of the

High Throughput Computing (HTC) by managing


Probabilistic flood map resulting from 5000 model runs using the
spatially-distributed physics-based hydrologic model GSSHA.

Running high fidelity models over large

B CondorPy and TethysCluster

Cloud computing resources are easy to

provision through admin site of Tethys
Portal, the web interface of Tethys
Platform. TethysCluster works behind
the scenes to automatically configure
the cloud resources into an HTCondor
computing pool.


Hierarchical Modeling

While TethysCluster and CondorPy can be used independently to provision

computing resources and perform large modeling tasks, they have also been
integrated into Tethys Platform, a development platform for water resources
web apps, to enable computing support for modeling workflows and decision
support systems deployed as web apps.

Cluster Management

Tethys Platform is a water resources

web development platform that
lowers the barrier to creating web
apps. Tethys Platform provides open
source web GIS and visualization tools
all integrated into a unified Python

Uncertainty is inherent to hydrologic modeling,

(1) programmatic access to dynamically scalable computing resources

linked together into a unified computing pool.

D Tethys Platform Integration





computing resources with AWS. TethysCluster is an

adaptation of StarCluster and expands its functionality
to work with both Linux and Windows resources with
AWS as well as Azure.

Screenshot of a Tethys
web app, the
Streamflow Prediction
Tool, which uses
CondorPy and
TethysCluster to
process ensemble



This material is based upon work supported by the National Science Foundation under Grant No. 1135483.