September 3, 2021
LATERAL VIEW is a very powerful concept in Hive. It is used when we have to work with data with complex types. Let’s see this with […]
September 1, 2021
You are getting the below error during DataNode startup. This post talks about how to fix the issue. 2013-04-11 16:25:50,515 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting […]
August 30, 2021
Changing the number of replicas on an existing topic is a 3 step process. Get the current information about the topic Configure new replica assignment in […]
August 27, 2021
You are trying to calculate median and you couldn’t find a function to calculate median in Hive. No worries you have come to the right place. […]
August 25, 2021
Pretty simple. Use the except() to subtract or find the difference between two dataframes. Do you like us to send you a 47 page Definitive guide […]
August 23, 2021
So you got this below execution and wondering how to fix it. In this post we are going to explain the possible causes for this exception which […]
August 20, 2021
java.net.BindException is a common exception when Spark is trying to initialize SparkContext. This is especially a common error when you try to run Spark locally. 16/01/04 […]
August 18, 2021
In Kafka, consumers connected to a topic are grouped into consumer groups. So a consumer group will have a list of consumers. So to get the […]
August 16, 2021
RDD, Dataframe and Dataset are all Spark APIs introduced in Spark at different points in time. The goal of these API is to help us work […]
August 13, 2021
When namenode is started or restarted, namenode will be in safemode for a period of time. At this time you will not be able to use […]
August 11, 2021
Downloading an entire bucket or folder inside a bucket is quite straightforward with AWS CLI. Install AWS CLI from here if you don’t have it already. […]
August 9, 2021
Sometimes you might see a stage being skipped in the DAG visualization in Spark web UI. In this post we are going to discuss couple of […]
August 6, 2021
It is a pretty common use case to export the contents of a Hive table into a CSV file. It’s pretty simple if you are using […]
August 4, 2021
In this post we will see how Spark decides the number of tasks and number of tasks to execute in parallel in a job. Let’s see […]
May 31, 2021
It is a very common use case to process the data in Spark and save the processed data or Spark dataframe directly into a Hive table. […]
May 28, 2021
NameNode NameNode is the heart of HDFS. NameNode maintains the metadata of HDFS – files, list of blocks, directories, permissions etc. The metadata is persisted on […]
May 26, 2021
Let’s redefine the question a little bit. Can multiple Kafka consumers from a consumer group read the same message from a partition? The short answer to […]
May 24, 2021
Broadcast variables are variables which are available in all executors executing the Spark application. These variables are already cached and ready to be used by tasks […]
May 21, 2021
Apache Pig was created by Yahoo. Apache Hive was created by Facebook. Both tools aimed at hiding the complexities of writing MapReduce jobs. Pig is similar […]
May 19, 2021
It is a pretty common use case to find the list of duplicate elements or rows in a Spark DataFrame and it is very easy to […]
