SPARK LOG ANALYSIS

log4j.rootLogger=${root.
logger}
5
5
log.dir=/var/log/spark
6
6
log.file=spark-worker-gmo-cl-data-01.log
7
7
max.log.file.size=200MB
8
-max.log.file.backup.index=10
8
+max.log.file.backup.index=5
9
9
log4j.appender.RFA=org.apache.log4j.RollingFileAppender
10
10
log4j.appender.RFA.File=${log.dir}/${log.file}
11
11
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
12
12
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
Find the log4j for spark and change the log directory manually
find / -ctime -0.1 2>/dev/null | grep log4j.properties <- find files that were m
odified in the last 24*60 * 0.1 minutes
/etc/hadoop/conf.cloudera.yarn/log4j.properties
/etc/hadoop/conf.cloudera.hdfs/log4j.properties
/etc/spark/conf.cloudera.spark/log4j.properties
/var/run/cloudera-scm-agent/process/2340-deploy-client-config/yarn-conf/log4j.pr
operties
/var/run/cloudera-scm-agent/process/2336-deploy-client-config/aux/client/log4j.p
roperties
/var/run/cloudera-scm-agent/process/2336-deploy-client-config/spark-conf/log4j.p
roperties
/var/run/cloudera-scm-agent/process/2330-deploy-client-config/hadoop-conf/log4j.
properties
/var/run/cloudera-scm-agent/process/2324-spark-SPARK_WORKER/config/log4j.propert
ies <- this one had old /var/log/spark and new index
/var/run/cloudera-scm-agent/process/2324-spark-SPARK_WORKER/aux/client/log4j.pro
perties
/var/run/cloudera-scm-agent/process/2324-spark-SPARK_WORKER/log4j.properties <this one had old /var/log/spark
/var/run/cloudera-scm-agent/process/2324-spark-SPARK_WORKER/hadoop-conf/log4j.pr
operties
log.dir=/var/log/spark
log.dir=/mnt/vol1/var/log/spark (worker default group advanced configuration saf
ety valve)
log.threshold=INFO
main.logger=RFA
root.logger=${log.threshold},${main.logger}
log4j.rootLogger=${root.logger}
-log.dir=/var/log/spark
+log.dir=/mnt/vol1/var/log/spark
log.file=spark-worker-gmo-cl-data-01.log
max.log.file.size=200MB
max.log.file.backup.index=5
log4j.appender.RFA=org.apache.log4j.RollingFileAppender
[dchtchou@gmo-cl-edge-02 SimpleApp]$ ls -a
. .. project run run~ simple.sbt simple.sbt~ src target
[dchtchou@gmo-cl-edge-02 SimpleApp]$ mkdir lib
[dchtchou@gmo-cl-edge-02 SimpleApp]$ cp /home/dchtchou/src/ion-pcm/SPARK/jars/sp
ark-assembly-1.1.0-SNAPSHOT-hadoop2.3.0-cdh5.1.0.jar lib
[dchtchou@gmo-cl-edge-02 SimpleApp]$ ./run
[info] Set current project to Simple Project (in build file:/home/dchtchou/src/i
on-pcm/SPARK/SimpleApp/)
[info] Compiling 1 Scala source to /home/dchtchou/src/ion-pcm/SPARK/SimpleApp/ta
rget/scala-2.10/classes...
[info] Packaging /home/dchtchou/src/ion-pcm/SPARK/SimpleApp/target/scala-2.10/si
mple-project_2.10-1.0.jar ...
[info] Done packaging.
[success] Total time: 6 s, completed Jul 30, 2014 5:50:35 PM
14/07/30 17:50:36 INFO SecurityManager: Changing view acls to: dchtchou
Compiled ok, but getting:
14/07/30 17:50:40 INFO SparkContext: Job finished: count at SimpleApp.scala:15,
took 0.025717261 s
Lines with a: 1, Lines with b: 0
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/
hive/HiveContext
at SimpleApp$.main(SimpleApp.scala:21)
at SimpleApp.main(SimpleApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.hive.HiveConte
xt
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 9 more
will re-sart of spark role help?
File /mnt/vol1/var/log/spark/spark-master-gmo-cl-edge-02.log
nope
put /etc/hive/conf.cloudera.hive/hive-site.xml on /etc/alternatives/spark-conf/
on every node
since this is spark home, we need to replace the spark assembly as well
export SPARK_HOME=/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark
mv /opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/assembly/lib/spark
-assembly-1.0.0-cdh5.1.0-hadoop2.3.0-cdh5.1.0.jar /opt/cloudera/parcels/CDH-5.1.
0-1.cdh5.1.0.p0.53/lib/spark/assembly/lib/spark-assembly-1.0.0-cdh5.1.0-hadoop2.
3.0-cdh5.1.0.jar.Original
cp spark-assembly-1.1.0-SNAPSHOT-hadoop2.3.0-cdh5.1.0.jar /opt/cloudera/parcels/
CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/assembly/lib/
do this by ftp
spark-assembly-1.1.0-SNAPSHOT-hadoop2.3.0-cdh5.1.0.jar
did this -no reboot required
scala>
hiveContext.hql("SELECT COUNT(*) FROM bom_table_top").collect().forea
ch(println)
ch(println)
14/07/30 19:15:45 INFO ParseDriver: Parsing command: SELECT COUNT(*) FROM bom_ta
ble_top
14/07/30 19:15:45 INFO ParseDriver: Parse Completed
14/07/30 19:15:45 INFO Analyzer: Max iterations (2) reached for batch MultiInsta
nceRelations
14/07/30 19:15:45 INFO Analyzer: Max iterations (2) reached for batch CaseInsens
itiveAttributeReferences
14/07/30 19:15:45 INFO metastore: Trying to connect to metastore with URI thrift
://gmo-cl-edge-02:9083
14/07/30 19:15:45 INFO metastore: Waiting 1 seconds before next connection attem
pt.
14/07/30 19:15:46 INFO metastore: Connected to metastore.
14/07/30 19:15:47 INFO Analyzer: Max iterations (2) reached for batch Check Anal
ysis
14/07/30 19:15:47 INFO deprecation: mapred.map.tasks is deprecated. Instead, use
mapreduce.job.maps
14/07/30 19:15:47 INFO MemoryStore: ensureFreeSpace(404207) called with curMem=0
, maxMem=278302556
14/07/30 19:15:47 INFO MemoryStore: Block broadcast_0 stored as values in memory
(estimated size 394.7 KB, free 265.0 MB)
14/07/30 19:15:47 INFO SQLContext$$anon$1: Max iterations (2) reached for batch
Add exchange
Prepare Expressions
14/07/30 19:15:47 INFO SparkContext: Starting job: collect at SparkPlan.scala:52
14/07/30 19:15:48 INFO FileInputFormat: Total input paths to process : 1
14/07/30 19:15:48 INFO DAGScheduler: Registering RDD 5 (mapPartitions at Exchang
e.scala:69)
14/07/30 19:15:48 INFO DAGScheduler: Got job 0 (collect at SparkPlan.scala:52) w
ith 1 output partitions (allowLocal=false)
14/07/30 19:15:48 INFO DAGScheduler: Final stage: Stage 0(collect at SparkPlan.s
cala:52)
14/07/30 19:15:48 INFO DAGScheduler: Parents of final stage: List(Stage 1)
14/07/30 19:15:48 INFO DAGScheduler: Missing parents: List(Stage 1)
14/07/30 19:15:48 INFO DAGScheduler: Submitting Stage 1 (MapPartitionsRDD[5] at
mapPartitions at Exchange.scala:69), which has no missing parents
14/07/30 19:15:48 INFO DAGScheduler: Submitting 2 missing tasks from Stage 1 (Ma
pPartitionsRDD[5] at mapPartitions at Exchange.scala:69)
14/07/30 19:15:48 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
14/07/30 19:15:48 INFO TaskSetManager: Re-computing pending task lists.
14/07/30 19:15:48 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 0, lo
calhost, PROCESS_LOCAL, 5668 bytes)
14/07/30 19:15:48 INFO Executor: Running task 1.0 in stage 1.0 (TID 1)
14/07/30 19:15:48 INFO BlockManager: Found block broadcast_0 locally
14/07/30 19:15:48 INFO HadoopRDD: Input split: hdfs://gmo-cl-name-01:8020/user/h
ive/warehouse/bom_table_top/part-m-00000:39554203+39554204
14/07/30 19:15:48 INFO HadoopRDD: Input split: hdfs://gmo-cl-name-01:8020/user/h
ive/warehouse/bom_table_top/part-m-00000:0+39554203
14/07/30 19:15:48 INFO deprecation: mapred.tip.id is deprecated. Instead, use ma
preduce.task.id
14/07/30 19:15:48 INFO deprecation: mapred.task.id is deprecated. Instead, use m
apreduce.task.attempt.id
14/07/30 19:15:48 INFO deprecation: mapred.task.is.map is deprecated. Instead, u
se mapreduce.task.ismap
14/07/30 19:15:48 INFO deprecation: mapred.task.partition is deprecated. Instead
, use mapreduce.task.partition
14/07/30 19:15:48 INFO deprecation: mapred.job.id is deprecated. Instead, use ma
preduce.job.id
14/07/30 19:15:49 INFO Executor: Finished task 0.0 in stage 1.0 (TID 0). 1868 by
tes result sent to driver
14/07/30 19:15:49 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 0) in
1338 ms on localhost (1/2)
14/07/30 19:15:49 INFO DAGScheduler: Stage 1 (mapPartitions at Exchange.scala:69
) finished in 1.356 s
14/07/30 19:15:49 INFO DAGScheduler: looking for newly runnable stages
14/07/30 19:15:49 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have
all completed, from pool
14/07/30 19:15:49 INFO DAGScheduler: running: Set()
14/07/30 19:15:49 INFO DAGScheduler: waiting: Set(Stage 0)
14/07/30 19:15:49 INFO DAGScheduler: failed: Set()
14/07/30 19:15:49 INFO DAGScheduler: Missing parents for Stage 0: List()
14/07/30 19:15:49 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[9] at map at
SparkPlan.scala:52), which is now runnable
14/07/30 19:15:49 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (Ma
ppedRDD[9] at map at SparkPlan.scala:52)
14/07/30 19:15:49 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesI
nFlight: 50331648, targetRequestSize: 10066329
14/07/30 19:15:49 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 2
non-empty blocks out of 2 blocks
14/07/30 19:15:49 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 0
remote fetches in 6 ms
14/07/30 19:15:49 INFO DAGScheduler: Stage 0 (collect at SparkPlan.scala:52) fin

ished in 0.094 s
14/07/30 19:15:49 INFO SparkContext: Job finished: collect at SparkPlan.scala:52
, took 2.344278732 s
[249340]
scala>
sweet!
spark://HOST:PORT
Connect to the given Spark standalone cluster master. Th
e port must be whichever one your master is configured to use, which is 7077 by
default.
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
hiveContext.hql("SELECT LEVEL, COUNT(*) AS level_counts FROM dchtchou_bom_ta
ble GROUP BY LEVEL").collect().foreach(println)
spark-shell --master spark://gmo-cl-edge-02:7077
14/07/30 19:27:34 INFO SparkILoop: Created spark context..
Spark context available as sc.
scala> 14/07/30 19:27:34 INFO SparkDeploySchedulerBackend: Connected to Spark cl
uster with app ID app-20140730192734-0000
change both master and worker memeory to 4g (CM)
scala>
hiveContext.hql("SELECT LEVEL, COUNT(*) AS level_counts FROM dchtchou
_bom_table GROUP BY LEVEL").collect().foreach(println)
_bom_table GROUP BY LEVEL").collect().foreach(println)
14/07/30 19:40:35 INFO ParseDriver: Parsing command: SELECT LEVEL, COUNT(*) AS l
evel_counts FROM dchtchou_bom_table GROUP BY LEVEL
nceRelations
pt.
14/07/30 19:40:38 INFO Analyzer: Max iterations (2) reached for batch Check Anal
ysis
14/07/30 19:40:38 INFO deprecation: mapred.map.tasks is deprecated. Instead, use
mapreduce.job.maps
, maxMem=278302556
14/07/30 19:40:38 INFO MemoryStore: Block broadcast_0 stored as values in memory
(estimated size 394.7 KB, free 265.0 MB)
7, maxMem=278302556
14/07/30 19:40:38 INFO MemoryStore: Block broadcast_0_meta stored as values in m
emory (estimated size 56.0 B, free 265.0 MB)
14/07/30 19:40:38 INFO BlockManagerInfo: Added broadcast_0_meta in memory on gmo
-cl-edge-02:51538 (size: 56.0 B, free: 265.4 MB)

14/07/30 19:40:38 INFO BlockManagerMaster: Updated info of block broadcast_0_met
a
4263, maxMem=278302556
14/07/30 19:40:38 INFO MemoryStore: Block broadcast_0_piece0 stored as values in
memory (estimated size 88.4 KB, free 264.9 MB)
14/07/30 19:40:38 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on g
mo-cl-edge-02:51538 (size: 88.4 KB, free: 265.3 MB)
14/07/30 19:40:38 INFO BlockManagerMaster: Updated info of block broadcast_0_pie
ce0
Add exchange
Prepare Expressions
14/07/30 19:40:39 INFO SparkContext: Starting job: collect at SparkPlan.scala:52
14/07/30 19:40:40 INFO NetworkTopology: Adding a new node: /default/192.168.1.17
:50010
:50010
:50010
:50010
:50010
:50010
:50010
:50010
:50010
:50010
:50010
:50010
:50010
:50010

14/07/30 19:40:41 INFO DAGScheduler: Registering RDD 63 (mapPartitions at Exchan
ge.scala:44)
cala:52)
14/07/30 19:40:41 INFO DAGScheduler: Submitting 1095 missing tasks from Stage 1
(MapPartitionsRDD[63] at mapPartitions at Exchange.scala:44)
14/07/30 19:40:41 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 0, gm
o-cl-data-04, NODE_LOCAL, 18008 bytes)
14/07/30 19:40:41 INFO TaskSetManager: Starting task 10.0 in stage 1.0 (TID 9, g
mo-cl-data-03, NODE_LOCAL, 18007 bytes)
14/07/30 19:40:41 INFO TaskSetManager: Starting task 9.0 in stage 1.0 (TID 10, g
mo-cl-data-02, NODE_LOCAL, 18008 bytes)
14/07/30 19:40:41 INFO TaskSetManager: Starting task 11.0 in stage 1.0 (TID 11,
gmo-cl-data-01, NODE_LOCAL, 18007 bytes)
14/07/30 19:40:42 INFO ConnectionManager: Accepted connection from [gmo-cl-data03/192.168.1.16]
14/07/30 19:40:42 INFO SendingConnection: Initiating connection to [gmo-cl-data04/192.168.1.17:37802]
14/07/30 19:40:42 INFO SendingConnection: Initiating connection to [gmo-cl-data-
02/192.168.1.15:48984]
14/07/30 19:40:42 INFO SendingConnection: Connected to [gmo-cl-data-04/192.168.1
.17:37802], 1 messages pending
mo-cl-data-04:37802 (size: 88.4 KB, free: 265.3 MB)
4833 ms on gmo-cl-data-04 (1/1095)
5593 ms on gmo-cl-data-03 (2/1095)
5607 ms on gmo-cl-data-02 (3/1095)
5597 ms on gmo-cl-data-02 (4/1095)
14/07/30 19:40:47 INFO TaskSetManager: Finished task 12.0 in stage 1.0 (TID 12)
in 5842 ms on gmo-cl-data-04 (5/1095)
5977 ms on gmo-cl-data-04 (6/1095)
14/07/30 19:40:48 INFO TaskSetManager: Finished task 9.0 in stage 1.0 (TID 10) i
n 6129 ms on gmo-cl-data-02 (7/1095)
6372 ms on gmo-cl-data-04 (9/1095)
14/07/30 19:40:48 INFO TaskSetManager: Finished task 10.0 in stage 1.0 (TID 9) i
n 6508 ms on gmo-cl-data-03 (10/1095)

14/07/30 19:40:48 INFO TaskSetManager: Starting
14/07/30 19:40:48 INFO TaskSetManager: Finished
6524 ms on gmo-cl-data-03 (11/1095)
task 26.0 in stage 1.0 (TID 26,

task 4.0 in stage 1.0 (TID 5) in
task 15.0 in stage 1.0 (TID 16)


14/07/30 19:43:54 INFO TaskSetManager: Finished task 1041.0 in stage 1.0 (TID 10
93) in 2456 ms on gmo-cl-data-02 (1091/1095)

14/07/30 19:43:56 INFO DAGScheduler: Stage 1 (mapPartitions at Exchange.scala:44
) finished in 194.792 s
14/07/30 19:43:56 INFO DAGScheduler: looking for newly runnable stages
14/07/30 19:43:56 INFO DAGScheduler: running: Set()
14/07/30 19:43:56 INFO DAGScheduler: waiting: Set(Stage 0)
14/07/30 19:43:56 INFO DAGScheduler: failed: Set()
14/07/30 19:43:56 INFO DAGScheduler: Missing parents for Stage 0: List()
14/07/30 19:43:56 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[67] at map at
SparkPlan.scala:52), which is now runnable
14/07/30 19:43:56 INFO DAGScheduler: Submitting 200 missing tasks from Stage 0 (
MappedRDD[67] at map at SparkPlan.scala:52)
gmo-cl-data-02, PROCESS_LOCAL, 8734 bytes)
14/07/30 19:43:56 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 1105
, gmo-cl-data-01, PROCESS_LOCAL, 8734 bytes)
14/07/30 19:43:56 INFO MapOutputTrackerMasterActor: Asked to send map output loc
ations for shuffle 0 to spark@gmo-cl-data-02:39656
14/07/30 19:43:56 INFO MapOutputTrackerMaster: Size of output statuses for shuff
le 0 is 2766 bytes

) in 228 ms on gmo-cl-data-03 (3/200)

task 24.0 in stage 0.0 (TID 1119


14/07/30 19:43:57 INFO TaskSetManager: Finished task 37.0
14/07/30 19:43:57 INFO TaskSetManager: Starting task 46.0
14/07/30 19:43:57 INFO MapOutputTrackerMasterActor: Asked
in stage 0.0 (TID 1132

to send map output loc


4, gmo-cl-data-02, PROCESS_LOCAL, 8734 bytes)

14/07/30 19:43:59 INFO DAGScheduler: Stage 0 (collect at SparkPlan.scala:52) fin
ished in 2.771 s
14/07/30 19:43:59 INFO SparkContext: Job finished: collect at SparkPlan.scala:52
, took 200.469593935 s
[16,476377]
[3,2220985]
[10,11041237]
[14,1852499]
[4,4947681]
[13,1353556]
[12,1686848]
[17,15441]
[15,1771336]
[18,3537]
[6,16810306]
[2,764589]
[7,49529326]
[19,5143]
[11,4848319]
[5,7586226]
[8,29843447]
[20,1743]
[9,123893626]
[1,249340]
scala>
SparkSQL can't do functions yet like concat...have to program it in Spark :)
14/08/01 03:49:46 INFO ParseDriver: Parsing command: SELECT
ASSEMBLY_NAME
,ASSEMBLY_DESC
,ASSEMBLY_PF
,ASSEMBLY_BU
,ASSEMBLY_TG
,ASSEMBLY_ITEM_ID
,BILL_SEQUENCE_ID
,COMPONENT_NAME
,COMPONENT_DESC
,COMPONENT_PF
,COMPONENT_BU
,COMPONENT_TG
,COMPONENT_ITEM_ID
,COMPONENT_QUANTITY
,EFFECTIVITY_DATE
,DISABLE_DATE
,ALTERNATE_BOM_DESIGNATOR
,ORGANIZATION_ID
,ORGANIZATION_CODE
,TEST_COMMENT
,concat('',table_top.COMPONENT_NAME,'') AS PATH
,1 AS LEVEL
,table_top.COMPONENT_NAME AS LEVEL_1
,LEVEL_2
,LEVEL_3
,LEVEL_4
,LEVEL_5
,LEVEL_6
,LEVEL_7
,LEVEL_8
,LEVEL_9
,LEVEL_10
,LEVEL_11
,LEVEL_12
,LEVEL_13
,LEVEL_14
,LEVEL_15
,LEVEL_16
,LEVEL_17
,LEVEL_18
,LEVEL_19
,LEVEL_20
,LEVEL_21
,LEVEL_22
,LEVEL_23
,LEVEL_24
,LEVEL_25
,LEVEL_26
,LEVEL_27
,LEVEL_28
,LEVEL_29
,LEVEL_30
,1 as level_partition
FROM bom_table_top
14/08/01 03:49:47 INFO SparkDeploySchedulerBackend: Registered executor: Actor[a
kka.tcp://sparkExecutor@gmo-cl-data-03:41181/user/Executor#1938503466] with ID 0
14/08/01 03:49:47 INFO BlockManagerMasterActor: Registering block manager gmo-cl
-data-03:59963 with 2.1 GB RAM
nceRelations
pt.
Exception in thread "main" org.apache.spark.sql.catalyst.errors.package$TreeNode
Exception: Unresolved attributes: 'concat(,'table_top.component_name,) AS
path#4,'table_top.component_name AS level_1#6, tree:
Project [assembly_name#8,assembly_desc#9,assembly_pf#10,assembly_bu#11,assembly_
tg#12,assembly_item_id#13,bill_sequence_id#14,component_name#15,component_desc#1
6,component_pf#17,component_bu#18,component_tg#19,component_item_id#20,component
_quantity#21,effectivity_date#22,disable_date#23,alternate_bom_designator#24,org
anization_id#25,organization_code#26,test_comment#27,'concat(,'table_top.comp
onent_name,) AS path#4,1 AS level#5,'table_top.component_name AS level_1#6,l
evel_2#31,level_3#32,level_4#33,level_5#34,level_6#35,level_7#36,level_8#37,leve
l_9#38,level_10#39,level_11#40,level_12#41,level_13#42,level_14#43,level_15#44,l
evel_16#45,level_17#46,level_18#47,level_19#48,level_20#49,level_21#50,level_22#
51,level_23#52,level_24#53,level_25#54,level_26#55,level_27#56,level_28#57,level
_29#58,level_30#59,1 AS level_partition#3]
LowerCaseSchema
MetastoreRelation default, bom_table_top, None
at org.apache.spark.sql.catalyst.analysis.Analyzer$CheckResolution$$anon
fun$apply$1.applyOrElse(Analyzer.scala:71)
at org.apache.spark.sql.catalyst.analysis.Analyzer$CheckResolution$$anon
fun$apply$1.applyOrElse(Analyzer.scala:69)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.s
cala:165)
at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala
:156)
at org.apache.spark.sql.catalyst.analysis.Analyzer$CheckResolution$.appl
y(Analyzer.scala:69)
at org.apache.spark.sql.catalyst.analysis.Analyzer$CheckResolution$.appl
y(Analyzer.scala:67)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$an
onfun$apply$2.apply(RuleExecutor.scala:62)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$an
onfun$apply$2.apply(RuleExecutor.scala:60)
at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.
scala:51)
at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimiz
ed.scala:60)
at scala.collection.mutable.WrappedArray.foldLeft(WrappedArray.scala:34)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.app
ly(RuleExecutor.scala:60)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.app
ly(RuleExecutor.scala:52)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.apply(RuleExecutor.s
cala:52)
at org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQ
LContext.scala:317)
at org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.sc
ala:317)
at org.apache.spark.sql.hive.HiveContext$QueryExecution.optimizedPlan$lz
ycompute(HiveContext.scala:250)
at org.apache.spark.sql.hive.HiveContext$QueryExecution.optimizedPlan(Hi
veContext.scala:249)
at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(S
QLContext.scala:320)
at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.s
cala:320)
at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycomput
e(SQLContext.scala:323)
at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContex
t.scala:323)
at org.apache.spark.sql.SchemaRDD.collect(SchemaRDD.scala:428)
at sparksql_shell$.main(sparksql_shell.scala:49)
at sparksql_shell.main(sparksql_shell.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:313)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:73)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Created a table dchthou_bom_table_n77_c7706_fab_02 through CREATE TBALE....AS SE

LECT
set the partition on it now
This works in spark!
SELECT ASSEMBLY_NAME
,concat('',table_top.COMPONENT_NAME,'')
,instr(concat('',table_top.COMPONENT_NAME,'')
,table_top.PATH) AS PATH
FROM bom_table_top as table_top
WHERE COMPONENT_NAME = "N77-C7706-FAB-2"
INSERT wtih PARTITION does not work
sparksql-shell: reading file:BOM_TABLE_LEVEL_JobCheck_LoadScript.hql
sparksql-shell: running query:
SELECT
LEVEL
,COUNT(*) level_counts
FROM dchtchou_bom_table_sparksql
GROUP BY LEVEL
ORDER BY LEVEL
[1,249340]
[2,764588]
[3,2220984]
[4,4947681]
[5,7586226]
[6,16810306]
[7,49529326]
[8,29843447]
[9,123893626]
sparksql-shell: finished in 1.98 min

-- =============================================================================
==================
-- = End level JobCheck duration: 0:02:04.017641, total duration: 0:57:27.331271
-- =============================================================================
==================
-- =============================================================================
==================
-- = Total levels JobCheck duration: 0:57:27.331271
-- =============================================================================
==================
[dchtchou@gmo-cl-edge-02 TestCasePartitionOnLevelTextFile]$ bash: o0: command no
t found
[dchtchou@gmo-cl-edge-02 TestCasePartitionOnLevelTextFile]$ [dchtchou@gmo-cl-edg
e-02 TestCasePartitionOnLevelTextFile]$
1: End level 1 duration: 0:00:19.851785
SELECT
LEVEL
,COUNT(*) level_counts
FROM dchtchou_bom_table_sparksql
GROUP BY LEVEL
ORDER BY LEVEL
[1,249340]
[2,764588]
[3,2220984]
[4,4947681]
[5,7586226]
[6,16810306]
[7,49529326]
[8,29843447]
[9,123893626]
[10,11041237]
[11,4848319]
[12,1686848]
[13,1353556]
[14,1852499]
[15,1771336]
[16,476377]
[17,15441]
[18,3537]
[19,5143]
[20,1743]
sparksql-shell: finished in 2.13 min
INSERT INTO dchtchou_bom_pairs_parquet
SELECT * FROM bom_pairs_tsv
See if we can catch bom_pairs_tsv
It's doing this weird thing:
:50010
14/08/07 03:58:58 INFO DAGScheduler: Registering RDD 5 (mapPartitions at Exchang
e.scala:69)
cala:52)
14/08/07 03:58:59 INFO DAGScheduler: Submitting 4394 missing tasks from Stage 1
(MapPartitionsRDD[5] at mapPartitions at Exchange.scala:69)
14/08/07 03:59:14 WARN TaskSchedulerImpl: Initial job has not accepted any resou
rces; check your cluster UI to ensure that workers are registered and have suffi
cient memory
[dchtchou@gmo-cl-edge-02 sparksql-shell]$ /home/dchtchou/src/ion-pcm/src/spark/s
parksql-shell/sparksql-shell.sh -q "SELECT COUNT(*) FROM dchtchou_bom_table_spar
ksql" 2>log.txt
sparksql-shell: using context:org.apache.spark.SparkContext@1629aeb2
SELECT COUNT(*) FROM dchtchou_bom_table_sparksql
C-c C-c[dchtchou@gmo-cl-edge-02 sparksql-shell]$ /home/dchtchou/src/ion-pcm/sr
c/spark/sparksql-shell/sparksql-shell.sh -q "SELECT COUNT(*) FROM dchtchou_bom_t
able_sparksql" 2>log.txt
sparksql-shell: using context:org.apache.spark.SparkContext@7a86b09d
SELECT COUNT(*) FROM dchtchou_bom_table_sparksql
C-c C-c[dchtchou@gmo-cl-edge-02 sparksql-shell]$

SPARK LOG ANALYSIS

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

SPARK LOG ANALYSIS

Hochgeladen von

Copyright:

Verfügbare Formate

log4j.rootLogger=${root.

14/07/30 19:15:49 INFO DAGScheduler: Stage 0 (collect at SparkPlan.scala:52) fin

-cl-edge-02:51538 (size: 56.0 B, free: 265.4 MB)

14/07/30 19:40:41 INFO FileInputFormat: Total input paths to process : 3

n 6508 ms on gmo-cl-data-03 (10/1095)

task 26.0 in stage 1.0 (TID 26,

in 1701 ms on gmo-cl-data-02 (25/1095)

task 41.0 in stage 1.0 (TID 41,

93) in 2456 ms on gmo-cl-data-02 (1091/1095)

ations for shuffle 0 to spark@gmo-cl-data-03:37009

, gmo-cl-data-04, PROCESS_LOCAL, 8734 bytes)

task 24.0 in stage 0.0 (TID 1119

, gmo-cl-data-03, PROCESS_LOCAL, 8734 bytes)

in stage 0.0 (TID 1132

task 60.0 in stage 0.0 (TID 1155

task 70.0 in stage 0.0 (TID 1165

task 90.0 in stage 0.0 (TID 1185

Created a table dchthou_bom_table_n77_c7706_fab_02 through CREATE TBALE....AS SE

sparksql-shell: finished in 1.98 min

Das könnte Ihnen auch gefallen