Sporule
SPORULE
Spark On Yarn Error - Failed to send RPC
2022-01-15
Problem I am new to the big data world, and I am trying to build a Hadoop cluster by using docker. The spark shell did not work with the error message below: Diagnose The problem looks like it can't connect to the ip address, so I start with testing the connection between spark to the ip address. The ping went through without problems. Then I look at yarn to see if I can find any logs there, I discovered the error messages below: It looks obvious that the job containers were killed because the …
Problem I use docker-compose to build my development Hadoop cluster, and Hive is one of the components. I received an error message: Diagnose I started hive with a debug mode by using: It returns error message: It looks like the system complained about the hostname thrift://hive-metastore.docker-hadoop_default:9083. I did some Google and the hive system does not like the character "_" Solution The hostname was auto-generated by Docker Compose, so I need to change the default name explicitly. D…
The Problem I built a Linux dev box docker image with some necessary run time environments such as go and python, but it didn't run and raised the error message exec user process caused "exec format error". The Cause I spent more than an hour finding the solution as nothing looked wrong for me. I have tried different docker image build options but no luck. After multiple attempts, I discovered the error message disappeared after I passed commands directly rather than using the run.sh in the Doc…