Al.I.

Cuza University of Iai

Faculty of Economics and Business Administration
Department of Accounting, Information Systems and
Statistics

Data Analysis & Data

Science with R
Data Input and Output
(Import and Export)
By Marin Fotache

Scripts associated with this

presentation

Scripts

03a_basics_of_data_input_output.R:
http://1drv.ms/1DRMOTC
03b_intermediate_data_input_output:
http://1drv.ms/1JKeKwi

PostgreSQL

scripts (for creating the DB to be imported

in R see script 03a...)
00a_1_creating_tables__sales.sql:
http://1drv.ms/1JKeRbh
00a_2_populating_tables__sales.sql:
http://1drv.ms/1JKeYU0
01_creare_bd_vinzari_PostgreSQL.sql:
http://1drv.ms/1JKf522

02_populare_bd_vinzari_PostgreSQL .sql:
http://1drv.ms/1JKfem5

Scripts associated with this

presentation (cont.)
Oracle

scripts (for creating the DB to be imported in R

see script 03b...)
01-01a_creating_tables__sales.sql
http://1drv.ms/1LBNImg
01-01a_ro_creare_bd_vinzari.sql
http://1drv.ms/1AqQTvg
01-01b_populating_tables__sales.sql
http://1drv.ms/1LBNGLe
01-01b_ro_populare_bd_vinzari.sql
http://1drv.ms/1A5a60K

Web sites with R tutorials for data

input/output
R

Data Import/Export
http://cran.r-project.org/doc/manuals/r-re
lease/R-data.html
Beginner's guide to R: Get your data
into R
http://www.computerworld.com/article/2497
164/business-intelligence/beginner-s-guid
e-to-r-get-your-data-into-r.html
Reading/Writing Data: Part 1
https://www.youtube.com/watch?v=aBzA
els6jPk&index=9&list=PLjTlxb-wKvXNSD
fcKPFH2gzHGyjpeCZmJ

Web sites with R tutorials for data

input/output (cont.)
Importing

Data Into R from Different

Sources
http://www.r-bloggers.com/importing-data-i
nto-r-from-different-sources/
Data Import & Export in R
http://science.nature.nps.gov/im/datamgmt
/statistics/r/fundamentals/index.cfm
Reading data from the new version of
Google Spreadsheets
http://blog.revolutionanalytics.com/2014/
06/reading-data-from-the-new-version-of-g

Loading data into statistical

packages
Traditional solutions:

Direct import from external data files (Excel, CSV, text files etc.) using
their menus

Save intermediate results from the data sources into common format
files (XML, CSV, JSON ) and then import these intermediate files into
the package;

Create data sources using ODBC or JDBC

Some

more recent options:

Customized (for data source and the destination package) ETL

procedures

Connecting to special APIs or web/data services which provide data sets

in formats easy to import (e.g. Google Analytics)

Import data from web servers log into NoSQL data stores

Performing database query in a database server directly from the

statistical package.

Sources of Data in R (adaptated

from [Kabacoff, 2011])

Hadoop

NoSQL
Data
Stores

Loading data sets stored

within packages
See

previous presentation
Many packages include datasets, such as ggplot2; aftera package is
loaded, all of its datasets are available:
> library(ggplot2)
> str(diamonds)
'data.frame': 53940 obs. of 10 variables:
$ carat : num 0.23 0.21 0.23 0.29 0.31 0.24 0.24 0.26 0.22 0.23 ...
$ cut
: Ord.factor w/ 5 levels "Fair"<"Good"<..: 5 4 2 4 2 3 3 3 1 3 ...
$ color : Ord.factor w/ 7 levels "D"<"E"<"F"<"G"<..: 2 2 2 6 7 7 6 5 2 5 ...
$ clarity: Ord.factor w/ 8 levels "I1"<"SI2"<"SI1"<..: 2 3 5 4 2 6 7 3 4
5 ...
$ depth : num 61.5 59.8 56.9 62.4 63.3 62.8 62.3 61.9 65.1 59.4 ...
$ table : num 55 61 65 58 58 57 57 55 61 61 ...
$ price : int 326 326 327 334 335 336 336 337 337 338 ...
$ x
: num 3.95 3.89 4.05 4.2 4.34 3.94 3.95 4.07 3.87 4 ...
$ y
: num 3.98 3.84 4.07 4.23 4.35 3.96 3.98 4.11 3.78 4.05 ...
$ z
: num 2.43 2.31 2.31 2.63 2.75 2.48 2.47 2.53 2.49 2.39 ...

Loading data objects saved in the

workspace of a previous session
Save

the workspace associated with the current

session (the workspace contains all the of the
existing data objects at a point in time ):
> ws.name <- paste("work", Sys.Date(),
+ ".RData", sep="")
> save.image(file=ws.name)
Restore

(load) a previously saved workspace

(and all the data object in the workspace):
> load("work2014-09-12.RData")
When

more worspaced have been saved, one

can choos wich one to load
> load(file.choose())

Data entered from the keyboard (1)

The

simplest method of data entry

function edit() launches a text editor that
allows entering your data manually
If

the data frame exists:

> student_gi <- edit(student_gi)
or
> fix(student_gi)

Data entered from the keyboard

(2)
If

the data frame does not exist, follow two

steps:

1 Create an empty data frame (or matrix) with the

variable names and types you want to have in the final
dataset.

> mydata <- data.frame(age=numeric(0),

+ gender=character(0),
weight=numeric(0))

2 Invoke the text editor on this data object, enter data,

and save the results back to the data object.
> mydata <- edit(mydata)
Or
> fix(mydata)

Data entered from clipboard

One

can copy into clipboard small sections of data in a table

(e.g. a spreadsheet, a Web HTML table) using control-C
(copy command)

On

Windows, command read.table handles clipboard data

with a header row that is separated by tabs, and stores the
data in a data frame (x):

> x <- read.table(file = "clipboard", sep="\t",

+ header=TRUE)
On

Mac, the pipe ("pbpaste") function the equivalent:

copy without header

> x <- read.table(pipe("pbpaste"), sep="\t",

+ header=FALSE)

copy with header

> y <- read.table(pipe("pbpaste"), sep="\t",

+ header=TRUE)

Import from local CSV/delimited

text files
read.table()

reads a file in table format

and saves it as a data frame

> mydataframe = read.table(file,

+ header=logical_value,
+
sep="delimiter", row.names="name")
file is a delimited ASCII file
header is a logical value indicating whether the first
row contains variable names (TRUE or FALSE )
sep specifies the separating data values
row.names is an optional parameter specifying
one or more variables to represent row identifiers.

Data input from local delimited text

files
Data

frame births2006 is located in directory

DataSets/births2006
Delimitator in the source file is Tab (\t)
The name of the file to be imported is births2006.txt
Qualifying the subdirectory is a bit different from an operating
system to another
>
+
+
+
+
+
+
+
+
+

switch(Sys.info()[['sysname']],
Windows= {births2006 <- read.table(
"births2006\\births2006.txt",
fileEncoding = "UTF-8", header = TRUE, sep="\t")},
Linux = {births2006 <- read.table(
"births2006/births2006.txt",
fileEncoding = "UTF-8", header = TRUE, sep="\t")},
Darwin = {births2006 <- read.table(
"births2006/births2006.txt",
fileEncoding = "UTF-8", header = TRUE, sep="\t")} )

Data input from local delimited text

files (cont.)
When

we are not sure about the file name,

instead of the filename one can use the
function file.choose():

births2006.2 <- read.table(file.choose(),

+ fileEncoding = "UTF-8",
+ header = TRUE,
+ sep="\t")

Data input from local CSV files

Import

one dataset from Irina Dan's Ph.D. thesis concerning

a study of using e-documents in companies
Data source is a tab delimited file - companyinfo.csv
Current working directory is .../DataSets
File to be imported is located in directory
.../DataSets/IrinaDan
>
+
+
+
+
+
+
+
+
+

switch(Sys.info()[['sysname']],
Windows= {comp <- read.table(
"IrinaDan\\companyinfo.csv",
header=TRUE, sep=";", stringsAsFactors=FALSE)},
Linux = {comp <- read.table(
"IrinaDan/companyinfo.csv",
header=TRUE, sep=";", stringsAsFactors=FALSE)},
Darwin = {comp <- read.table(
"IrinaDan/companyinfo.csv",
header=TRUE, sep=";", + stringsAsFactors=FALSE)}
)

Data input from local CSV files

(cont.)
Import

a dataset from Dragos Cogean's Ph.D. thesis

which compares two cloud database services, Mongo
and MySQL
Data sources are tab delimited files
The .txt (tab-delimited) file resides in directory
\DataSets\DragosCogean
Notice the second version of function switch()
>
+
+
+
+
+
+
+
+

InsertMongoALL <- switch(Sys.info()[['sysname']],

Windows= { read.table(
"DragosCogean\\InsertMongo_ALL.txt",
fileEncoding = "UTF-8", header=TRUE,
sep="\t", stringsAsFactors=FALSE)},
Darwin = { read.table(
"DragosCogean/InsertMongo_ALL.txt",
fileEncoding = "UTF-8", header=TRUE,
sep="\t", stringsAsFactors=FALSE)} )

Data input from local CSV files

(cont.)
Import

Toyota Corolla second hand cars

data set (located in
...\DataSets\ToyotaCorolla directory);
Also notice the third version of function
switch()
> ToyotaCorolla <- read.table(
+
switch(Sys.info()[['sysname']],
+ Windows =
{"ToyotaCorolla\\ToyotaCorolla.csv"},
+ Darwin= {"ToyotaCorolla/ToyotaCorolla.csv"}),
+
fileEncoding = "UTF-8", header = TRUE,
+
sep=","
+ )

Data input from text file available on

web
Heart

attack data set

Description available at:

http://courses.statistics.com/software/R/tables4R.htm

The data set (as delimited text file) available at:

http://courses.statistics.com/Intro1/Lesson2/heartatk4R.txt
> heart.att = read.table(
http://courses.statistics.com/Intro1/Lesson2/heartatk
4R.txt
, header=TRUE)
> head(heart.att)
Patient DIAGNOSIS SEX DRG DIED CHARGES LOS AGE
1
1
41041
F 122
0 4752.00 10 79
2
2
41041
F 122
0 3941.00
6 34
3
3
41091
F 122
0 3657.00
5 76
4
4
41081
F 122
0 1481.00
2 80
5
5
41091
M 122
0 1681.00
1 55

Data input from CSV file available on

web
data set: 356 people polled on their
smoking status (Smoke) and their socioeconomic
status (SES).
The data file contains only two columns, and
when read R interprets them both as factors:
Smoking

> smoker <read.csv("http://www.cyclismo.org/tutorial/R/_static/

smoker.csv")
> head(smoker)
Smoke SES
1 former High
2 former High
3 former High
4 former High
5 former High
6 former High

Data input from CSV file available on

web
data set: 356 people polled on their
smoking status (Smoke) and their socioeconomic
status (SES).
The data file contains only two columns, and
when read R interprets them both as factors:
Smoking

> smoker <read.csv("http://www.cyclismo.org/tutorial/R/_static/

smoker.csv")
> head(smoker)
Smoke SES
1 former High
2 former High
3 former High
4 former High
5 former High
6 former High

Download and read

When

a data set is large, instead of the direct import...

> dat.csv <read.csv("http://www.ats.ucla.edu/stat/data/hsb2.csv")

...

one can proceed in two steps:

1. download the file

>
download.file("http://archive.ics.uci.edu/ml/machinelearning-databases/arrhythmia//arrhythmia.data",
destfile="data.csv")
trying URL 'http://archive.ics.uci.edu/ml/machinelearning-databases/arrhythmia//arrhythmia.data'
Content type 'text/plain; charset=UTF-8' length 402355
bytes (392 Kb)
opened URL
==================================================
downloaded 392 Kb
2. import the downloaded file

> df.2 <- read.csv("data.csv")

Importing data from Excel files

The

"simplest" way to read an Excel

(.xls/.xlsx) file is to save it in Excel as a text
(tab delimited) or csv file and then to read
it as in previous slides
Loading directly into R .xls/.xlsx files is
possible through various packages:

RO D BC
gdata
xlsReadW rite
XLConnect
xlsx

Problems (on Windows systems)

when loading some packages
> install.packages("xlsx")
> library(xlsx)

Loading required package: rJava

Loading required package: xlsxjars
xlsx

requires package rJava; on Windows systems that

sometimes creates problems (e.g. R 64 bits on Windows
7)
On my computer (Windows 7 64bit), a downloaded java
runtime in directory C:\\Program Files\\Java\\jre7\\ so
before package rJava or any other package which uses
rJava, I need to run:
> options(java.home="C:\\Program
Files\\Java\\jre7\\")

Reading data from Excel files

We'll

prefer package xlsx:

> library(xlsx)
ADL

master students (2013-2014) file is located in

directory .../DataSets/ADL)
Path is specified differently for Windows and Mac OS
systems:
> workbook <-

switch(Sys.info()[['sysname']],

+
Windows=
{ "ADL\\ADL2013_Studenti.xlsx"},
+

Darwin = { "ADL/ADL2013_Studenti.xlsx"})

> adl2013_stud <- read.xlsx(workbook, 1)

> str(adl2013_stud)

Import data from local PostgreSQL

databases
PostgreSQL

database serves is installed locally, on my laptop

(on Windows systems check if the PostgreSQL service is
started)
Package needed: RPostgreSQL
> install.packages("RPostgreSQL")
> library(RPostgreSQL)
Load

the PostgreSQL driver

> drv <- dbDriver("PostgreSQL")

Open

a connection

On Windows systems:

> con <- dbConnect(drv, dbname="bd2014", user="bd2014",

+

password="bd2014")
On Mac OS

> con <- dbConnect(drv, port=5433, dbname="sales2014",

+

user="sales2014", password="sales2014")

Import data from local PostgreSQL

databases (cont.)
Launch

the PostgreSQL query; the result of the query will be saved into data
frame invoice_detailed:

> invoice_detailed <+

dbGetQuery(con,

"SELECT i.invoiceNo, invoiceDate, i.customerId,

customerName, place, countyName, region,

comments, invoiceRowNumber, i_d.productId,

productName, unitOfMeasurement, category,

quantity, unitPrice, quantity * unitPrice AS amountWithoutVAT,

quantity * unitPrice * (1 + VATPercent) AS amount

FROM invoices i

INNER JOIN invoice_details i_d ON i.invoiceNo = i_d.invoiceNo

INNER JOIN products p ON i_d.productId = p.productId

INNER JOIN customers c ON i.customerid = c.customerid

INNER JOIN postcodes pc ON c.postCode = pc.postCode

INNER JOIN counties ON pc.countyCode =counties.countyCode

ORDER BY i.invoiceNo, invoiceRowNumber")

Import data from local PostgreSQL

databases (cont.)
Launch

the PostgreSQL query; the result of the query will be saved

into data frame invoice_detailed:

> head(invoice_detailed,3)
invoiceno invoicedate customerid customername place
1

1111

2012-08-01

1001 Client 1 SRL

Iasi

1111

2012-08-01

1001 Client 1 SRL

Iasi

1111

2012-08-01

1001 Client 1 SRL

Iasi

countyname

region comments invoicerownumber productid

Iasi Moldova

<NA>

Iasi Moldova

<NA>

Iasi Moldova

<NA>

productname unitofmeasurement

category quantity

Product 1

b500ml Category A

50

Product 2

kg Category B

75

Product 5

unit Category A

50

unitprice amountwithoutvat amount

1

1000

50000

62000

1050

78750

88200

7060

353000 437720

Saving the data frame(s)

Data

frame(s) will be saved (for further use) in

directory .../DataSets/sales

Path

qualification is different between Windows

and Mac systems:

> file.name <-

switch(Sys.info()[['sysname']],

Windows=
{ "sales\\invoice_detailed.RData"},
Darwin = { "sales/invoice_detailed.RData"})
> save(invoice_detailed, file = file.name)
After

saving, whenever needed, the data frame can

be loaded into Rstudio session with load function

Close connections/drivers
After
Close

the import, the resources must be freed

all PostgreSQL connections

for (connection in dbListConnections(drv) )

{
dbDisconnect(connection)
}
Frees

all the resources on the driver

> dbUnloadDriver(drv)

Import data from a remote PostgreSQL

databases (!!!!!!!!!!!!!!!!!!!)
...for

the moment it is impossible to externally

(outside FEAA) access the database servers

Access Oracle databases through

JDBC
Package

ROracle was intended to provide access to Oracle databases

Unfortunately, now package ROracle is not available
Next example was inspired by
http://www.r-bloggers.com/connecting-r-to-an-oracle-database-with-rjdbc
/
As the name suggests, the solution needs dealing with some Java
"things"

Requirements: JDK/JRE previously installed

Download ojdbc jar from www.oracle.com (in my case, ojdbc6.jar)
Set JAVA_HOME, set max. memory, and load rJava library
Sys.setenv(JAVA_H O M E= '/path/to/java_hom e')
on my Mac OS:

>
Sys.setenv(JAVA_HOME='/Library/Java/JavaVirtualMachines/jdk1.7
.0_45.jdk/Contents/Home')
> options(java.parameters="-Xmx2g")
> install.packages("rJava")

Access Oracle databases through

JDBC (cont.)

Getting some information

Java version

> .jinit()
> print(.jcall("java/lang/System", "S", "getProperty",
"java.version"))

classPath (just for the record)

> .jclassPath()
Load RJDBC package
> Install.packages(RJDBC)
> library(RJDBC)
Create connection driver and open connection
> jdbcDriver <JDBC(driverClass="oracle.jdbc.OracleDriver",
classPath="/Users/admin/Downloads/ojdbc6.jar")

Access Oracle databases through

JDBC (cont.)

Open connection

> jdbcConnection <- dbConnect(jdbcDriver,

+

"jdbc:oracle:thin:@//10.10.0.7:1521/orcl",

"bd2",

"bd2")
Launch the Oracle query and store the result into the data
frame st
> st <- dbGetQuery(jdbcConnection,
+ "SELECT * FROM studenti")

Close connection

> dbDisconnect(jdbcConnection)

Import data from MongoDB

Import data from Cassandra

Import data from Hadoop

Read HTML tables from the web

Package needed: XML

> install.packages("XML")
> library(XML)
> myURL <"http://www.jaredlander.com/2012/02/another-kindof-super-bowl-pool/"
> dfHTML <- readHTMLTable(myURL, which=1,
header=FALSE,
+

stringsAsFactors = FALSE)

> head(dfHTML,3)
V1

V2

V3

1 Participant 1 Giant A Patriot Q

2 Participant 2 Giant B Patriot R

Import XML files

Package needed: XML

> library(XML)

Web address of the xml file

> url <"http://www.statistics.life.ku.dk/primer/mydata.xml

"

Import

> indata <- xmlToDataFrame(url)

> head(indata, 5)
Girth Height Volume
1

8.3

70

10.3

8.6

65

10.3

8.8

63

10.2

10.5

72

16.4

Reading HTML pages with multiple

tables
Package needed: XML
> library(XML)
The web page contains text and a number of tables

> url.1 <'http://en.wikipedia.org/wiki/World_population'

> tbls.1 <- readHTMLTable(url.1)
> class(tbls.1)
[1] "list"
Display how many tables are on the page
> length(tbls.1)
[1] 28
Read only the 1st table on this page (which=1)
> tbl.1.1 <- readHTMLTable(url.1, which=1,
header=F, stringsAsFactors = FALSE)

Importing data from other

statistical packages
Package

needed: foreign
> install.packages("foreign")
> library(foreign)
Read a Stata data file (.dta)
> states <- read.dta("states.dta")
Read a local SPSS file
> spss1 <- read.spss("p004.sav",
+ use.value.labels = TRUE, to.data.frame =
TRUE)
Import the SPSS file directly from web address
> spss2 <read.spss("http://www.ats.ucla.edu/stat/spss/ex
amples/chp/p004.sav",
+ use.value.labels = TRUE,

Save/export R data objects

Save

a data frame as a .csv file

> write.csv(spss2, file = "spss2.csv")
Save a data frame as a tab delimited text file
> write.table(spss2, file = "spss2.txt",
+ sep = "\t", fileEncoding = "UTF-8")
Save a data frame as an Excel (xlsx) file (requires
package xlsx)
> write.xlsx(spss2, file = "spss2.xlsx",
sheetName="spss2")
> write.xlsx(echipe.4,
+ file = "Centralizator BD2_2013_SIA1.xlsx",
+ sheetName="t4.echipe",
+ row.names=FALSE, append=TRUE, showNA=FALSE)

Save/export R data objects (cont.)

Save

a dataframe as a .dta file (requires package

foreign)

> write.dta(spss2, file = "spss2.dta")

Save

to binary R format (can save multiple

datasets and R objects)

> save(invoice.details.ro,
+ file = "invoice.details.ro.RData")
> save(states, spss2, dat.xls,
+ file = "temp.RData")