STORM-1901: Avro Integration for Storm-Kafka

[vesense]

init version for code review, changes are following:

• separate Avro from storm-hdfs as a new subproject. (can used by storm-hdfs & storm-kafka, or others modules)
• add new Config.TOPOLOGY_AVRO_SERIALIZER in Config
• add new Config.TOPOLOGY_AVRO_CONFLUENT_SCHEMA_REGISTRY_URL in Config
• add DirectAvroSerializer and DefaultDirectAvroSerializer in storm-avro
• add new AvroSchema in storm-kafka
• add new DirectAvroTupleToKafkaMapper in storm-kafka
• new unit tests
• document improvements

[vesense]

@dossett I add you as the Committer Sponsors because you have done a lot in storm avro integration . Hope you have chance to take a look.

[vesense]

@ptgoetz @HeartSaVioR Could you also take a look?

[dossett]

Thanks @vesense I will review this week.

[harshach]

@vesense is this intended to be a schema registry, i.e users publishing their schemas IMO having schema registry again in storm is not a good idea.
By looking at the code seems to be a module containing avro related modules and mostly related to storm-kafka module. From what I see this is a AvroSerializer and AvroDeserializer if so shouldn't these part of storm-kafka module instead of being its own external module. Trying to understand where its applicable except the storm-kafka module at this point.

[dossett]

Should be AvroScheme and Scheme

[vesense]

Thx. Will fix.

[dossett]

@harshach My interpretation of the PR is that it moves some Avro speficif code I added to storm-hdfs into a more sharable module.

@vesense I like the idea of making it easier to read/write avro data in a Storm topology, thanks for looking at this. I don't know that it needs to be its own module though, maybe it could be somewhere in org.apache.storm.utils ?

[vesense]

The reason why I define the new interface DirectAvroSerializer and move avro to a sharable module is explained as above.

@harshach This PR is based on STORM-1504, you can take a look at it.

@dossett Your reviews have been addressed. Please have a look.

[vesense]

@dossett BTW, I have a question about ConfluentAvroSerializer:
I can't see any unit tests for ConfluentAvroSerializer. I wonder how kryo create the object by using construction ConfluentAvroSerializer(Kryo k, Map stormConf) with parameters (ConfluentAvroSerializer is specified by avro.serializer).