Racing condition with parquet writer

[richiesgr]

Hi
After debugging a lot secor when writing Parquet from Avro message. I come to the conclusion that a possible racing condition can occurs.

• So Iv'e a topic with 300 Partitions
• Start 10 Pods - 7 Core per pod
• 7 Thread per pod

Result not enough parquet writer to handle every partition. So start a IndexOutOfBoundsException
This occurs into the parquet writer no code related to secor. As you know the parquet writer is not thread safe. Is there a possibility one Parquet writer is used by more that 1 thread ?

My investigations show me that assumption is correct because the only workaround I found at this moment is to have a much higher number of secor.consumer.threads to be sure that Parquet writer is not reused by mistake

Can you confirm ?
Thanks

[HenryCaiHaiying]

You can turn on secor debug and check the following log line to see how many writers are created: LOG.debug("created writer for path {}", path.getLogFilePath()); From the code, we are going to create a new writer for each path. You can also use jmap to count how many Parquet Writer objects in the heap

…

On Sun, Jan 3, 2021 at 6:27 AM Richard Grossman ***@***.***> wrote: Hi After debugging a lot secor when writing Parquet from Avro message. I come to the conclusion that a possible racing condition can occurs. - So Iv'e a topic with 300 Partitions - Start 10 Pods - 7 Core per pod - 7 Thread per pod Result not enough parquet writer to handle every partition. So start a *IndexOutOfBoundsException* This occurs into the parquet writer *no code related to secor*. As you know the parquet writer is not thread safe. Is there a possibility one Parquet writer is used by more that 1 thread ? My investigations show me that assumption is correct because the only workaround I found at this moment is to have a much higher number of *secor.consumer.threads* to be sure that Parquet writer is not reused by mistake Can you confirm ? Thanks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1792>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYJP7YVNGI3MOLCADWURJDSYB5FXANCNFSM4VR5DMXA> .

[richiesgr]

Hi
My question was if my assumption are valid ?
yes we can debug with many tools but As I read the code I see there is 1 parquet writer by file because it's stored in a hashset<path,writer> but is there by any chance a problem a race condition trying to use the same parquet writer by 2 threads ?
Because in this case it's more a design problem than a code bug

[HenryCaiHaiying]

From the code, it doesn't look like writers are going to be shared between threads. The edge case I can think of is when the consumer group rebalances, some of the work and related file paths being shifted from thread 1 to thread 2, but in that case the thread 1 should already relinquish the work on the given path. If you do see the problem happens after consumer rebalance (I think there are some logging categories indicating consumer rebalances), you can add some code to clear the hashset<path,writer> and writers on consumer rebalance (there is a callback when the consumer rebalance happens).

…

On Mon, Jan 4, 2021 at 4:16 AM Richard Grossman ***@***.***> wrote: Hi My question was if my assumption are valid ? yes we can debug with many tools but As I read the code I see there is 1 parquet writer by file because it's stored in a hashset<path,writer> but is there by any chance a problem a race condition trying to use the same parquet writer by 2 threads ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1792 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYJP74DHYAOM62ADA3DBD3SYGWQJANCNFSM4VR5DMXA> .