ABOUT – Hadoop In Real World


We are a group of Hadoop engineers who are passionate about Hadoop and related Big Data technologies. We have worked with as small as 20 nodes production Hadoop clusters to as big as 2,000 nodes production Hadoop cluster. Collectively we have seen a wide range of problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions. In this page, we will briefly talk about how we got here.


Year 2008, an engineering director at a client place asked us to look at this new tool called "Hadoop" and see whether it will solve the performance issues with ETL jobs we had at that time. Only a handful of companies were trying out Hadoop at that time and the Hadoop community was super small compared to its today's size. So we started reading Hadoop documentation on Apache's website and started playing around with it. Immediately we were fascinated by the concept and the technology and we were hooked to it. 4 months and 150 gallons of coffee later we had a 6 node cluster up and running executing our nightly ETL. We were able to solve the performance issues with our ETL jobs with Hadoop. Our Hadoop implementation and the success that came along with it spread through out the engineering department and the product teams. We started receiving lot of "exotic" requests to implement in Hadoop which was previously thought impossible or time consuming to implement. We started seeing our team grow.


With these new ideas/initiatives coming in and the product teams' willingness to pump money in to these initiatives, we looked to hire Hadoop engineers. Few weeks in to our hunt for Hadoop engineers we quickly realized there are not that many Hadoop engineers in the market because Hadoop was very new to the market. So we decided to hire good engineers and programmers and trained them in Hadoop. The training we had for the new engineers was quite powerful and very successful because we exactly know the challenges with Hadoop and our training program was designed to teach exactly those challenges and get them to "production ready" development ASAP.


Year 2011, we started seeing a decent number of Hadoop developers available in the market so we decided there is no need for our training program and we were dead wrong. Most of the candidates we interviewed for Hadoop developer positions were not good enough so we rejected a lot of candidates in the interviews. Even the ones we hired were strong developers but we realized there were not "production ready" for Hadoop. Because they lacked the understanding of the complexities and challenges involve in developing and maintaining production Hadoop environments. So we had no choice to continue our training program.


Year 2012, we packaged our Hadoop course online internally and made available for the Hadoop engineers that we hired and asked them to go over the course. The online course was just as effective as our class room training and saved us a lot of time and money. Year 2015, we saw a lot of main stream companies adopting Hadoop and also we saw lot of online courses and class room training being offered by other companies and just for the fun of it we looked at those course and they were garbage (to put it lightly). Not only they were totally not worth the money, they did not cover any real world concepts and the most shocking thing was, most of what was being thought was incorrect.

So we decided to make our course online for everyone to consume. Aug 2015, we went live with our free Hadoop Starter Kit course on udemy.com. The response was ecstatic and students were very happy with the content and were asking us for a comprehensive developer course. Student really liked that our teaching incorporated real world practices in Hadoop and the course was completely practical and actionable. Dec 2015, we went live with Hadoop Developer In Real World course.

Our student count continues to grow and the time of of writing we have about 25,000 students and they are very Happy and Successful. From day 1, our students satisfaction became our top priority so we made customer support top of our list. So it is safe to say we LOVE our students and our students LOVE us back. Check out some of awesome reviews from our awesome students.

So, that is our journey in a short page so far. We are looking forward to serve you !