jar containing org.apache.hadoop.hive.dynamodb

2019-08-11 08:32发布


I was trying to programmatically Load a dynamodb table into HDFS (via java, and not hive), I couldnt find examples online on how to do it, so thought I'd download the jar containing org.apache.hadoop.hive.dynamodb and reverse engineer the process.

Unfortunately, I couldn't find the file as well :(.

Could someone answer the following questions for me (listed in order of priority).

  1. Java example that loads a dynamodb table into HDFS (that can be passed to a mapper as a table input format).
  2. the jar containing org.apache.hadoop.hive.dynamodb.



It's in hive-bigbird-handler.jar. Unfortunately AWS doesn't provide any source or at least Java Doc about it. But you can find the jar on any node of an EMR Cluster:


You might want to checkout this Article:

  • Amazon DynamoDB Part III: MapReducin’ Logs

Unfortunately, Amazon haven’t released the sources for hive-bigbird-handler.jar, which is a shame considering its usefulness. Of particular note, it seems it also includes built-in support for Hadoop’s Input and Output formats, so one can write straight on MapReduce Jobs, writing directly into DynamoDB.

Tip: search for hive-bigbird-handler.jar to get to the interesting parts... ;-)


1- I am not aware of any such example, but you might find this library useful. It provides InputFormats, OutputFormats, and Writable classes for reading and writing data to Amazon DynamoDB tables.

2- I don't think they have made it available publically.