Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Change serializer for DiskBasedMap to handle non serializable java objects #580

Closed
ovj opened this issue Feb 20, 2019 · 0 comments
Closed

Comments

@ovj
Copy link
Contributor

ovj commented Feb 20, 2019

Current "com.uber.hoodie.common.util.SerializationUtils" requires us to have all objects to be java serializable.
Details of the discussion are here -> #495. We need to switch that to Kryo serializer so that we can handle non java serializable objects.

In this case "org.apache.avro.util.Utf8" is not java serializable but it can be present in GenericRecord (Hoodie Payload) we receive from input stream. Like this there can be other objects. We need something generic which can handle all such objects.

`Caused by: com.uber.hoodie.exception.HoodieSerializationException: unable to serialize object
at com.uber.hoodie.common.util.SerializationUtils.serialize(SerializationUtils.java:63)
at com.uber.hoodie.common.util.SerializationUtils.serialize(SerializationUtils.java:85)
at com.uber.hoodie.common.util.collection.DiskBasedMap.put(DiskBasedMap.java:169)
at com.uber.hoodie.common.util.collection.ExternalSpillableMap.put(ExternalSpillableMap.java:169)
at com.uber.hoodie.common.util.collection.ExternalSpillableMap.put(ExternalSpillableMap.java:42)
at com.uber.hoodie.io.HoodieMergeHandle.init(HoodieMergeHandle.java:154)
at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:72)
at com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:231)
at com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:185)
at com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:268)
... 31 more
Caused by: java.io.NotSerializableException: org.apache.avro.util.Utf8
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at java.util.HashMap.internalWriteEntries(HashMap.java:1790)
at java.util.HashMap.writeObject(HashMap.java:1363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at com.uber.hoodie.common.util.SerializationUtils.serialize(SerializationUtils.java:60)
... 40 more

`

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants