sqoop architecture export

sqoop basic export

transactionality
staging table export

export
------------
1)generated record container class
(java code will be geneareted ,
data types of sql and java combination)
2)number of blocks
3)sqoop export

-->hadoop and hdfs is not good for small data .

below things are require to export

-------------------------------------
hostname
port
usernmae
pwd
db
schema
table
export dir

sqoop export --connect jdbc:mysql://localhost/zeyobron_analytics

--username root --password cloudera --table controlling_incremental_date_import
--m 1 --export-dir /user/cloudera/export_first_table/

1)Have already imported data directory consist of data

2)create mysql table (export_first_table)
3)execute sqoop export command

1 mapper -- 1 block (128 mb)

4
512 mb
128* 4

128 -- 1--120
128 --129 --256
128 --257--384
128 --

Export transactionality
-----------------------------
--->>in order to understand Export transactionality
export
export import of incremental last madified

modifications in sqoop;
---------------------------
have table in RDBMS
do sqoop import to hdfs
change a row in RDBMS
do sqoop last madified to attain changes in RDBMS

1) created table in rdbms

2) normal sqoop import to hdfs
3) update table in rdbms
4

sqoop import --connect jdbc:mysql://localhost/zeyobron_analytics

--username root --password cloudera --table customer
--m 1 --incremental lastmodified --check-column created --merge-key custid
--target-dir /user/cloudera/import_first_table/

>>data got changed here also

>>but in part file reducer file created (/user/cloudera/import_first_table/part-r-
00000)

--incremental lastmodified (already existing data if modified )

--check-column createdate --should be always date or timestarmp in table
--last-vlaue
if we write like below command it takes from 2019-10-23 modified data will be
effect update in the target dir
sqoop import --connect jdbc:mysql://localhost/zeyobron_analytics
--username root --password cloudera --table customer
--m 1 --incremental lastmodified --check-column created --last-vlaue 2019 -10-23
--merge-key custid --target-dir /user/cloudera/import_first_table/

--Merge-key(key to compare )
it is used to do do comparision between source and destination

source

2019-10-21 sai 1
2019-10-22 aditya 2
2019-10-23 bala 3 (bala updated with ram )
2019-10-24 zeyo 4
2019-10-25 menal 5

imported

2019-10-21 sai 1
2019-10-22 aditya 2
2019-10-23 bala 3 (became ram)
2019-10-24 zeyo 4
2019-10-25 menal 5

ckeck colum tag is mandatory

check column should be date or timestramp
last value is optional and depending on requirement
merge key is mandatory
mereg key should be primary key
if not first occurance will take place as below

source
2019-10-21 sai 1
2019-10-22 aditya 2
2019-10-23 bala 3 (bala updated with ram )
2019-10-24 zeyo 3
2019-10-25 menal 5

imported

2019-10-21 sai 1
2019-10-22 aditya 2
2019-10-23 ram 3 (became ram)
2019-10-24 zeyo 3
2019-10-25 menal 5

--> above case it will be ambiguity ,so fisrt occurance of 3 will be updated in the
iported area

incremental appned can also be achieved by last modified

Task 1 --

How to import date or timestamp column as string Type in HDFS for avro

Task 2 --

Try importing data using sqoop without target directory check what happens

Task 3 --

Do a sample Sqoop Export

Task 4 --

Create a table in RDBMS with 5 rows

Do sqoop import
Change value in RDBMS and try to achieve modification in hdfs

Task 5 --

Have 5 dates

2019-10-21 sai 1
2019-10-22 aditya 2
2019-10-23 bala 3
2019-10-24 zeyo 4
2019-10-25 meenal 5

2019-10-21 sai 1
2019-10-22 aditya 2
2019-10-23 anand 3
2019-10-24 zeyo 4
2019-10-25 meenal 5
do the import

change data at 23 as anand

and Do last modified as with 24th and check whether changes happened and after that
run with 23

Task 6 --

Do the sqoop import with 2 mappers (part-m-0000, part-m-00001)

And do changes for the records present in both the part files AT rdbms and do last
modified and tell me how many part files came at the after last modified