File Connector

Contents

• Overview
• Summary
• Source Files
• Configuration
• Diagnostics
• Metrics

Overview

• FileConnector is intended for consuming text files from the file system

• A demo that uses FileConnector is available in Streaming Text Files to Kafka.

Summary

Supports Datastream Updates	N
Checkpoint Type	Line Number
Diagnostics Aware	Y
Metrics Aware	N

Source Files

1. com.linkedin.datastream.connectors.file.FileConnector

2. com.linkedin.datastream.connectors.file.FileConnectorFactory

3. com.linkedin.datastream.connectors.file.FileProcessor

Configuration

• All Connector config properties must be prefixed with brooklin.server.connector.<connectorName>

• connectorName is an arbitrary user-supplied name specified in Brooklin configuration (brooklin.server.connectorNames)

Property	Description	Default
maxExecProcessors	Number of threads in the thread pool used for reading input files	5
numPartitions	Number of input file partitions	1

Diagnostics

URL	Single Brooklin instance GET /diag?q=status&type=connector&scope=:componentScope&content=:componentInput/ All Brooklin instances GET /diag?q=allStatus&type=connector&scope=:componentScope&content=:componentInput/
URL Params	componentScope Connector name in brooklin.server.connectorNames componentInput Only supported value is "position" Reports the number of lines read so far

Metrics

None