A Deep-Dive Introduction to Spark for RDBMS Users

Earlier in the summer, I start a series of articles for IBM developerWorks. Those articles focus on Apache Spark from a RDBMS user perspective, of course, the database of choice was Informix.

The series starts by the ingestion of the data from Informix to Spark using JDBC. It comes with full examples in Java and link to the code in GitHub. Your favorite relational database is not Informix? Well, first, you’re probably wrong and, second, it should be fairly easy to adapt to MySQL, Dd2, Oracle, or PostgreSQL. I recently reused some of the code to connect to Microsoft SQL Server. I also cover how to write your own dialect, in Java. Find more about it on IBM’s developerWorks website.

The second article drills in basic data manipulation. I also explain the notion of lazy evaluation (and I am not thinking about students in college here) and, one of my favorite: the dataframe. Of course, there is plenty of code, all in Java.

More are to come. Do not hesitate to ask questions…