Part V Working Remote And More

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 61

Working remote - ssh, scp and rsync

telnet - plain communication
telnet hostname port Useful to nd out whether some service works as it should Do NOT use it to work remotely, ssh is much more secure! Try telnet towel.blinkenlights.nl :-)

ssh and scp - security by encryption Developed 1995 to allow secure connections between computers ssh username@computername log in onto computername as username
scp -r * username@computername:<path_rel_to_home> scp *.txt username@computername:/<absolute_path> scp "username@computername:path/*.*"./ copy all remote les in folder path to current directory

Quotes necessary for remote wildcards

62

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013

Working remote - ssh, scp and rsync

Tunneling connections
ssh XX local 22 gname @gate XY ssh 22 rname @remote XX local ssh XX local
8888

ssh

You can ssh from local machine to gate You can (e.g.) ssh from gate to remote You cannot directly access remote Then use ssh for tunneling the connection
8888: < remote >:22 < gname >@ < gate > cat in other terminal 8888 file . txt < rname > @localhost : some / dir 8888 < rname > @localhost
63

ssh -L # then scp -P ssh -p

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013

Working remote - ssh, scp and rsync (2)

rsync - backup and syncronisation Data transfer with scp for few les and unique transfer rsync transmits only differences and compresses General syntax rsync [OPTION...] SRC... [DEST] Both remote and local Potentially useful examples: rsync sourcedir/ targetdir/ sync sourcedir to targetdir rsync sources targetdir/ le/dir sources into targetdir rsync sources user@host:targetdir/ for remote access same syntax as scp
rsync -avuCn <src> <dest>

-a archive: recursive; preserve permissions, times, . . . -v verbose -u update: skip les that are newer on the receiver -C exclude certain les (e.g. *~ #* .#* ,* _$* *$ *.old *.bak *.o *.a *.exe core .svn...) -n perform a trial run with no changes made
64

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013

Synchronising data - Unison

very nice to sync 2 locations GUI and backup support 2-way sync (including merge etc.) supports win and linux (MAC?) needs gtk and ssh download and infos:

http://www.cis.upenn.edu/~bcpierce/unison/

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 65

Working remote - screen

during a typical session, many terminals are used if working remote, this means opening many ssh connections for each login/logout, everything has to be closed/opened again screen solves all these problems by creating virtual terminals C-a c creates a new terminal C-a " shows a selection-list of available terminals C-a d or screen -d detach the screen C-a D D detach the screen and log out screen -r reattach the screen C-a A give a name to a terminal C-a w show the currently opened terminals C-a k kill a terminal
Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 66

Process management
schedule jobs
at 4:01 pm < job . txt # use atq for list , atrm to delete crontab -e # -l to list # m h dom mon dow command 12 14 * * * touch / home / neckel / cron . txt

priority
nice -n <value> <command> starts command with lower priority top shows processes with top computing time within top, r can renice a process nohup run a command immune to hangups ps -ef list all processes pidof nd out the process id for a program name kill <ID> kills process with the given Id (-9 to force) killall <name> kills processes with given name bg and fg send programms to background/foreground
67

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013

Some more commands you should know

basename " this / is / the / path / to / a . file " dirname " this / is / the / path / to / a . file " time sleep 3 time for i in ` seq 100000 ` ; do a = $ [3* $ i ]; done users groups du - sh df -h ln -s < file > < link > echo " 3.6+5*4 " | bc

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 68

Part VI Visualizing with Gnuplot

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 69

Now for Something Completely Different?!

Why gnuplot?
Fast way to visualize scientic data Vast functionality Its free and available on Windows, Linux, and MAC Often bash-tools helpful to preprocess/use data

Next steps
Shortly introduce basic functionality Visualization of 2d and 3d scientic data

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 70

Interactive
$ gnuplot G N U P L O T Version 4.2 p a t c h l e v e l 3 l a s t m o d i f i e d Mar 2008 System : L i n u x 2.6.27 11 g e n e r i c C o p y r i g h t (C) 1986 1993 , 1998 , 2004 , 2007 , 2008 Thomas W i l l i a m s , C o l i n K e l l e y and many o t h e r s Type help t o access t h e on l i n e r e f e r e n c e manual . The g n u p l o t FAQ i s a v a i l a b l e from h t t p : / / www. g n u p l o t . i n f o / f a q / Send bug r e p o r t s and s u g g e s t i o n s t o < h t t p : / / s o u r c e f o r g e . n e t / . . . > T e r m i n a l t y p e s e t t o wxt gnuplot> help g n u p l o t i s a commandd r i v e n i n t e r a c t i v e f u n c t i o n and data p l o t t i n g program . [...]
Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 71

Interactive Use

Command interpreter Help functionality

help for general help help commands for overview help plot for help on command

plot

quit or <Ctrl+D> to exit

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 72

A rst example
Plotting sin(x )
gnuplot> p l o t sin ( x ) gnuplot>

. . . leads to

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 73

Useful Commands
Comments
Comments start with #

Plotting
plot <function> for 2d data splot <function> for 3d data plot [xmin:xmax] sin(x) specify range for x -axis plot sin(x ), cos(x) multiple plots replot repeat last plot command help plot help on further options Escape linebreaks with \

Functions
Variables x, y, z Usual operators; 2*x**2+sin(x)/3.24
Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 74

Useful Commands Plotting

Some plot modiers
plot sin(x) title A nice plot specify title plot sin(x) with <style> specify style Syntax : w i t h < s t y l e > { { l i n e s t y l e | l s < l i n e s t y l e >} | {{ l i n e t y p e | l t < l i n e t y p e >} { l i n e w i d t h | lw < l i n e w i d t h >} { l i n e c o l o r | l c < c o l o r s p e c >} { p o i n t t y p e | p t < p o i n t t y p e >} { p o i n t s i z e | ps < p o i n t s i z e >} { f i l l | f s < f i l l s t y l e >} { nohidden3d } { p a l e t t e }} } where < s t y l e > i s e i t h e r l i n e s , p o i n t s , l i n e s p o i n t s , impulses , d o t s , s t e p s , . . . use test to see available style of your terminal
Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 75

Setting Variables and Parameters

Use set, e.g. gnuplot> gnuplot> gnuplot> gnuplot> gnuplot> s e t xrange [ 0 : 3 * p i ] s e t yrange [ 2 : 2 ] set x l a b e l X axis set y l a b e l Y axis plot sin ( x )

Use unset to reset settings, see help unset For xrange its restored with set xrange restore

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 76

Setting Variables and Parameters

Choose terminal (default: wxt) gnuplot> set t e r m i n a l g n u p l o t > s e t t e r m i n a l png t r u e c o l o r f o n t a r i a l 20 \ enhanced Save to le gnuplot> set out [ put ] filename gnuplot> p l o t sin ( x )

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 77

Save and Load

Save and load settings
Save settings: save lename. plt Load settings: load lename. plt load works also for plot commands

use also for scripts or collection of commands

Batch mode
Similar for batch mode: $ gnuplot filename.gplt run script filename.gplt Helpful hint: use parameter -persist $ gnuplot -persist filename.gplt

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 78

More Useful Commands

Functions, variables Use variables, calculations, functions
g n u p l o t > a = s q r t (16) * 10+2 gnuplot> p r i n t a 42.0 gnuplot> p r i n t p i 3.14159265358979 g n u p l o t > f ( x ) = a * s i n ( x+ p i ) gnuplot> p l o t f ( x )

Shell access Leaving Gnuplot to list directory discards settings Therefore shell-ecape: Single commands with ! command Spawn shell with shell , return with exit pwd and cd "folder" directly accessible (tab works)
Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 79

Plot Scientic Data

Assume data le gaussian1.dat # x f(x) error -2.000000 0.053991 0.015473 -1.800000 0.078950 0.025096 -1.600000 0.110921 0.037264 ... 2.000000 0.053991 0.004492 Plot with plot plot plot plot gaussian1 . d a t gaussian1 . d a t gaussian1 . d a t gaussian1 . d a t w lines l t 1 gaussian2 . d a t w lines l t 2 w l i n e s p o i n t s lw w e r r o r b a r s lw 2 u 1 : 3 w impulses w errorbars l t 1 lw 2 , \ w errorbars l t 2 lw 2 2 ps 2 p t 5 pt 3 lw 2 lw 2 , \ lw 2 p t 5 , \

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 80

Data within the Gnuplot le

Apart from data in les ( lename and ), data can be provided inline ( )

Easy way to pipe data to gnuplot without having to create

intermediate les

s e t term png s e t o u t t e s t . png p l o t w l i n e s 0 0 1 1 2 4 3 9 end replot then doesnt work

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 81

3d Data

Unordered data sets

s p l o t d i s t i l l e d . dat u 1:2:8

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 82

3d Data
3d data on a grid
# 0 0 0 x 0 1 2 y f (x ,y) 4 4 5

1 0 5 1 1 5 1 2 7 3 0 6 3 1 7 3 2 10 If grid has no missing entries, gnuplot creates surface s p l o t 3 d f i l e . dat w l i n e s

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 83

3d Grid Data
There are much more options for plotting We dont cover them all, just few examples s p l o t s i n ( x ) * y+11 w l i n e s s e t hidden3d replot s e t isosample 40 show view s e t view 60 ,15 s e t c o n t o u r base s e t pm3d a t b unset c o n t o u r s e t pm3d

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 84

Modify values
File one2ten.dat: 1 1 ... 10 10 Use of built-in modiers plot one2ten . d a t u 1 : ( $2 * $2 ) w l i n e s , \ u 1 : ( s q r t ( $2 ) ) w l i n e s , \ u 1 : ( $1+$2 ) w l i n e s

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 85

Using Bash Commands

Bash commands such as sed, awk, sort, tail, cat, paste, . . . can

be used as well

Allows powerful data processing on the y Examples: plot < c a t one2ten . d a t e l e v e n 2 t w e l v e . d a t \ u 1 : ( $2 * $2 ) w l i n e s p l o t one2ten . d a t u 1 : ( $2 * $2 ) w l i n e s , \ < sed s / 4 / 5 / g one2ten . d a t u 1 : ( $2 * $2 ) \ w lines p l o t < awk { p r i n t $1 , ( $2 <4) ? 0 : $2 } one2ten . d a t \ w lines p l o t < awk { x=x+$2 ; p r i n t $1 , x /NR } one2ten . d a t \ w steps

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 86

Finally

More information. . .
gnuplot website:

http://www.gnuplot.info/
Demo scripts for gnuplot:

http://gnuplot.sourceforge.net/demo/
not so Frequently Asked Questions:

http://t16web.lanl.gov/Kawano/gnuplot/index-e.html

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 87