Producer and Consumer Contracts

Producer¶ ↑

Messages can be logged to databus using Log4j APIs. com.inmobi.messaging.ScribeMessagePublisher has to be configured as the Log 4J Appender.
Message format. Object of any one of these types can be logged via Log4j API. Any other kind is ignored.
    Type: byte[]
    Type: String
    Type: org.apache.thrift.TBase 
Message size: The serialized message size must be between 0 and 3 KB.
Example:
    //in log4j properties:
    log4j.logger.teststream=DEBUG,SA2
    log4j.appender.SA2=com.inmobi.messaging.ScribeMessagePublisher
    log4j.appender.SA2.hostname=localhost
    log4j.appender.SA2.Port=7912
    log4j.appender.SA2.ScribeCategory=teststream 

    Logger logger = Logger.getLogger("teststream");
    String msg = "i am flying on the other side"; logger.info(msg); //passing string object

In this example, the message will go to the stream name “teststream”. Producer could leverage - github.com/InMobi/messaging-client-scribe for sending messages

Consumer¶ ↑

Data gets collected in HDFS cluster in minute level directories at
    /databus/streams_local/[streamName]/YYYY/MM/DD/HR/MN
        contains data produced in the local DC of a particular stream
        YYYY/MM/DD/HR/MN represent Year/Month/Date/Hour/Minute in UTC 
    /databus/streams/[streamName]/YYYY/MM/DD/HR/MN
        contains the data produced in ALL the DCs of a particular stream
        YYYY/MM/DD/HR/MN represent Year/Month/Date/Hour/Minute in UTC 
Decoding messages from the file
    All data files in the minute level directores are gzip compressed
    Data is Base-64 encoded. Each message is separated by newline character 
Data immutability guarantee
    Timestamp directories YYYY/MM/DD/HR/MN are always created in increasing timeline.
    Minute directories are created at minute intervals in regular case. However it is not guaranteed that timestamp directories are created for all minutes. For example after creating 2012/01/03/08/05, the next min directory could be 2012/01/03/08/07.
    When the directory for a particular minute is created, the directory for previous minute becomes immutable