Question Regarding PutInfluxDatabaseRecord_2 1.10.0

[philrogers]

Hi Guys,
I have been using PutInfluxDatabaseRecord_Version 1.10 for quite some time using the batch feature.

I notice this now appears to be gone in version 2.x. Just wondering what is the best way to batch up a lot of data from nifi.

I have an avro schema, I tried using MergeRecord and merged 10 records together in an array then sent the merged flow file array to the PutInfluxDatabaseRecord when I look in the nifi logs it looks like it still only writes one record at a time.

2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient --> POST http://testserver:8086/api/v2/write?org=mycomp&bucket=nifibkt&precision=ns
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient Content-Type: text/plain; charset=utf-8
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient Content-Length: 1104
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient Content-Encoding: identity
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient Accept: application/json
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient User-Agent: influxdb-client-java/1.11.0
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient net-linkerrors,device=JPE19350550,hostname=xxxx-fab01,location=xxxx,metric_type=linkerrors,port=Ethernet22/4,state=SA alignmentErrors_avg=0.0,alignmentErrors_max=0.0,alignmentErrors_min=0.0,alignmentErrors_stddev=0.0,alignmentErrors_weight=1.0,fcsErrors_avg=0.0,fcsErrors_max=0.0,fcsErrors_min=0.0,fcsErrors_stddev=0.0,fcsErrors_weight=1.0,frameTooLongs_avg=0.0,frameTooLongs_max=0.0,frameTooLongs_min=0.0,frameTooLongs_stddev=0.0,frameTooLongs_weight=1.0,frameTooShorts_avg=0.0,frameTooShorts_max=0.0,frameTooShorts_min=0.0,frameTooShorts_stddev=0.0,frameTooShorts_weight=1.0,inDiscards_avg=0.0,inDiscards_max=0.0,inDiscards_min=0.0,inDiscards_stddev=0.0,inDiscards_weight=1.0,inErrors_avg=0.0,inErrors_max=0.0,inErrors_min=0.0,inErrors_stddev=0.0,inErrors_weight=1.0,outDiscards_avg=0.0,outDiscards_max=0.0,outDiscards_min=0.0,outDiscards_stddev=0.0,outDiscards_weight=1.0,outErrors_avg=0.0,outErrors_max=0.0,outErrors_min=0.0,outErrors_stddev=0.0,outErrors_weight=1.0,symbolErrors_avg=0.0,symbolErrors_max=0.0,symbolErrors_min=0.0,symbolErrors_stddev=0.0,symbolErrors_weight=1.0 1619158166290763005
2021-05-10 11:40:36,196 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient --> END POST (1104-byte body)

I also sometimes see this in the Nifi logs which I am not sure why this is happening as all flow files have content No Content Packets

2021-05-10 11:40:36,181 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient <-- 204 No Content http://testserver:8086/api/v2/write?org=vocus&bucket=nifibkt&precision=ns (2ms)
2021-05-10 11:40:36,181 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient Date: Mon, 10 May 2021 01:40:36 GMT
2021-05-10 11:40:36,181 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient <-- END HTTP

[bednar]

Hi @philrogers,

thanks for using our nifi-influxdb-bundle.

Hi Guys,
I have been using PutInfluxDatabaseRecord_Version 1.10 for quite some time using the batch feature.

I notice this now appears to be gone in version 2.x. Just wondering what is the best way to batch up a lot of data from nifi.

The MergeRecord is correct way how to prepare batches for PutInfluxDatabaseRecord_2.

I have an avro schema, I tried using MergeRecord and merged 10 records together in an array then sent the merged flow file array to the PutInfluxDatabaseRecord when I look in the nifi logs it looks like it still only writes one record at a time.
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient --> POST http://testserver:8086/api/v2/write?org=mycomp&bucket=nifibkt&precision=ns
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient Content-Type: text/plain; charset=utf-8
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient Content-Length: 1104
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient Content-Encoding: identity
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient Accept: application/json
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient User-Agent: influxdb-client-java/1.11.0
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient
2021-05-10 11:40:36,195 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient net-linkerrors,device=JPE19350550,hostname=xxxx-fab01,location=xxxx,metric_type=linkerrors,port=Ethernet22/4,state=SA alignmentErrors_avg=0.0,alignmentErrors_max=0.0,alignmentErrors_min=0.0,alignmentErrors_stddev=0.0,alignmentErrors_weight=1.0,fcsErrors_avg=0.0,fcsErrors_max=0.0,fcsErrors_min=0.0,fcsErrors_stddev=0.0,fcsErrors_weight=1.0,frameTooLongs_avg=0.0,frameTooLongs_max=0.0,frameTooLongs_min=0.0,frameTooLongs_stddev=0.0,frameTooLongs_weight=1.0,frameTooShorts_avg=0.0,frameTooShorts_max=0.0,frameTooShorts_min=0.0,frameTooShorts_stddev=0.0,frameTooShorts_weight=1.0,inDiscards_avg=0.0,inDiscards_max=0.0,inDiscards_min=0.0,inDiscards_stddev=0.0,inDiscards_weight=1.0,inErrors_avg=0.0,inErrors_max=0.0,inErrors_min=0.0,inErrors_stddev=0.0,inErrors_weight=1.0,outDiscards_avg=0.0,outDiscards_max=0.0,outDiscards_min=0.0,outDiscards_stddev=0.0,outDiscards_weight=1.0,outErrors_avg=0.0,outErrors_max=0.0,outErrors_min=0.0,outErrors_stddev=0.0,outErrors_weight=1.0,symbolErrors_avg=0.0,symbolErrors_max=0.0,symbolErrors_min=0.0,symbolErrors_stddev=0.0,symbolErrors_weight=1.0 1619158166290763005
2021-05-10 11:40:36,196 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient --> END POST (1104-byte body)

The problem is caused by outdated version of influxdb-client-java. This issue was already fixed by influxdata/influxdb-client-java#176

I also sometimes see this in the Nifi logs which I am not sure why this is happening as all flow files have content No Content Packets

2021-05-10 11:40:36,181 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient <-- 204 No Content http://testserver:8086/api/v2/write?org=vocus&bucket=nifibkt&precision=ns (2ms)
2021-05-10 11:40:36,181 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient Date: Mon, 10 May 2021 01:40:36 GMT
2021-05-10 11:40:36,181 INFO [Timer-Driven Process Thread-6] okhttp3.OkHttpClient <-- END HTTP

The 204 No Content is success response from InfluxDB v2.

I will prepare PR with updated version of influxdb-client-java... stay tuned

Regards

[bednar]

Hi @philrogers,

the issue is fixed in the 1.11.0 milestone. If you would like to use a dev version, please install bundle from: https://github.com/influxdata/nifi-influxdb-bundle/releases/tag/v1.11.0-SNAPSHOT.

Regards