CompatibleFieldSerializerFactory sometimes doesn't work while add new object as field

[liusf1993]

I have a class in server side and client side. And the class is like this:

  static public class SomeClass {
    String value;
  }

In the newer version, there are some changes, and the new structure is like this.

static public class SomeClass {

   String value;
   AnotherClass anotherClass;
 }

 static public class AnotherClass {

   String k;
   String v;
 }

While a value appears twice or above, an IndexOutOfBoundsException is thrown.
Here is my test test code, both the code are tested in a same class, here I separate encode from decode.
Encode:

public class CompatibleTest {

  public static final Kryo kryo = new Kryo();

  static {
    CompatibleFieldSerializerFactory compatibleFieldSerializerFactory = new CompatibleFieldSerializerFactory();
    CompatibleFieldSerializerConfig config = compatibleFieldSerializerFactory.getConfig();
    config.setChunkedEncoding(true);
    kryo.setDefaultSerializer(compatibleFieldSerializerFactory);
    kryo.setRegistrationRequired(false);
  }


  @Test
  public void encode() throws IOException {
    String s = "Hello Kryo!";
    AnotherClass anotherClass = new AnotherClass();
    anotherClass.k = s;
    anotherClass.v = s;

    SomeClass someClass = new SomeClass();
    someClass.anotherClass = anotherClass;
    someClass.value = s;

    Output output = new Output(new FileOutputStream("file.bin"));
    kryo.writeObject(output, someClass);
    output.close();

    System.out.println(someClass);
  }

  static public class SomeClass {

    String value;
    AnotherClass anotherClass;
  }


  static public class AnotherClass {

    String k;
    String v;
  }
}

Decode:

public class CompatibleTest {

  public static final Kryo kryo = new Kryo();

  static {
    CompatibleFieldSerializerFactory compatibleFieldSerializerFactory = new CompatibleFieldSerializerFactory();
    CompatibleFieldSerializerConfig config = compatibleFieldSerializerFactory.getConfig();
    config.setChunkedEncoding(true);
    kryo.setDefaultSerializer(compatibleFieldSerializerFactory);
    kryo.setRegistrationRequired(false);
  }


  @Test
  public void decode() throws IOException {
    Input input = new Input(new FileInputStream("file.bin"));
    SomeClass object2 = kryo.readObject(input, SomeClass.class);
    System.out.println(object2);
    input.close();
  }

  static public class SomeClass {

    String value;

  }


}

I view the source code and learn that it is caused by references, Stirng`` Hello Kryo! appears 3 times, in the first time it is stored into MapReferenceResolver.writeObjects, and the next two store the index of the first Hello Kryo!. While read the Input, SomeClass doesn't have field anotherClass and the Class AnotherClass doesn't exist in the older version, the first Hello Kryo! can not be read into MapReferenceResolver.readObjects, so the next two Hello Kryo! will fail while read by references.

I also learned that the problem can be solved with kryo.setReferences(false); , but it may affect performance and circular references may exist in my project.

Is there any other thing I can do to solve the problem? Thank you in advance!

[NathanSweet]

What version of Kryo are you using? With v5 (master branch) it works for me. Eg, run the below, then comment run and the 2 classes, and uncomment the other code:

public class CompatibleTest {
	public static final Kryo kryo = new Kryo();

	static {
		CompatibleFieldSerializerFactory compatibleFieldSerializerFactory = new CompatibleFieldSerializerFactory();
		CompatibleFieldSerializerConfig config = compatibleFieldSerializerFactory.getConfig();
		config.setChunkedEncoding(true);
		kryo.setDefaultSerializer(compatibleFieldSerializerFactory);
		kryo.setRegistrationRequired(false);
		kryo.setReferences(true);
	}

	public void run () throws IOException {
		String s = "Hello Kryo!";
		AnotherClass anotherClass = new AnotherClass();
		anotherClass.k = s;
		anotherClass.v = s;

		SomeClass someClass = new SomeClass();
		someClass.anotherClass = anotherClass;
		someClass.value = s;

		Output output = new Output(new FileOutputStream("file.bin"));
		kryo.writeObject(output, someClass);
		output.close();

		System.out.println(someClass.value);
	}

	static public class SomeClass {
		String value;
		AnotherClass anotherClass;
	}

	static public class AnotherClass {
		String k;
		String v;
	}

// public void run () throws IOException {
// Input input = new Input(new FileInputStream("file.bin"));
// SomeClass object2 = kryo.readObject(input, SomeClass.class);
// System.out.println(object2.value);
// input.close();
// }
//
// static public class SomeClass {
// String value;
// }

	static public void main (String[] args) throws Exception {
		new CompatibleTest().run();
	}
}

[NathanSweet]

I take it back, I do see an issue when kryo.setReferences(true); is added (false is the default now). I'll modify my code above.

[NathanSweet]

Trace logs for writing:

00:00 TRACE: [kryo] Write: <not null> [0]
00:00 TRACE: [kryo] Write initial reference 0: com.esotericsoftware.kryo.CompatibleTest$SomeClass [1]
00:00 DEBUG: [kryo] Write: com.esotericsoftware.kryo.CompatibleTest$SomeClass [1]
00:00 TRACE: [kryo] Cached String field: value (com.esotericsoftware.kryo.CompatibleTest$SomeClass)
00:00 TRACE: [kryo] Cached AnotherClass field: anotherClass (com.esotericsoftware.kryo.CompatibleTest$SomeClass)
00:00 TRACE: [kryo] Register class name: com.esotericsoftware.kryo.CompatibleTest$SomeClass (com.esotericsoftware.kryo.serializers.CompatibleFieldSerializer)
00:00 TRACE: [kryo] Write fields for class: com.esotericsoftware.kryo.CompatibleTest$SomeClass
00:00 TRACE: [kryo] Write field name: anotherClass [2]
00:00 TRACE: [kryo] Write field name: value [14]
00:00 TRACE: [kryo] Write field AnotherClass: anotherClass (com.esotericsoftware.kryo.CompatibleTest$SomeClass) [19]
00:00 TRACE: [kryo] Cached String field: k (com.esotericsoftware.kryo.CompatibleTest$AnotherClass)
00:00 TRACE: [kryo] Cached String field: v (com.esotericsoftware.kryo.CompatibleTest$AnotherClass)
00:00 TRACE: [kryo] Register class name: com.esotericsoftware.kryo.CompatibleTest$AnotherClass (com.esotericsoftware.kryo.serializers.CompatibleFieldSerializer)
00:00 TRACE: [kryo] Write class name: com.esotericsoftware.kryo.CompatibleTest$AnotherClass [1]
00:00 TRACE: [kryo] Write: <not null> [55]
00:00 TRACE: [kryo] Write initial reference 1: com.esotericsoftware.kryo.CompatibleTest$AnotherClass [56]
00:00 DEBUG: [kryo] Write: com.esotericsoftware.kryo.CompatibleTest$AnotherClass [56]
00:00 TRACE: [kryo] Write fields for class: com.esotericsoftware.kryo.CompatibleTest$AnotherClass
00:00 TRACE: [kryo] Write field name: k [57]
00:00 TRACE: [kryo] Write field name: v [59]
00:00 TRACE: [kryo] Write field String: k (com.esotericsoftware.kryo.CompatibleTest$AnotherClass) [61]
00:00 TRACE: [kryo] Write class 1: String [0]
00:00 TRACE: [kryo] Write: <not null> [1]
00:00 TRACE: [kryo] Write initial reference 2: Hello Kryo! (String) [2]
00:00 TRACE: [kryo] Write: Hello Kryo! (String) [2]
00:00 TRACE: [kryo] Write chunk: 13 [13]
00:00 TRACE: [kryo] Write chunk: 75 [75]
00:00 TRACE: [kryo] End chunk.
00:00 TRACE: [kryo] Write field String: v (com.esotericsoftware.kryo.CompatibleTest$AnotherClass) [1]
00:00 TRACE: [kryo] Write class 1: String [0]
00:00 DEBUG: [kryo] Write reference 2: Hello Kryo! (String) [1]
00:00 TRACE: [kryo] Write chunk: 2 [2]
00:00 TRACE: [kryo] Write chunk: 4 [4]
00:00 TRACE: [kryo] End chunk.
00:00 TRACE: [kryo] Write chunk: 1 [1]
00:00 TRACE: [kryo] End chunk.
00:00 TRACE: [kryo] Write field String: value (com.esotericsoftware.kryo.CompatibleTest$SomeClass) [1]
00:00 TRACE: [kryo] Write class 1: String [0]
00:00 DEBUG: [kryo] Write reference 2: Hello Kryo! (String) [1]
00:00 TRACE: [kryo] Write chunk: 2 [2]
00:00 TRACE: [kryo] End chunk.
00:00 TRACE: [kryo] Object graph complete.

For reading:

00:00 TRACE: [kryo] Read: <not null> [1]
00:00 TRACE: [kryo] Read initial reference 0: com.esotericsoftware.kryo.CompatibleTest$SomeClass [1]
00:00 TRACE: [kryo] Cached String field: value (com.esotericsoftware.kryo.CompatibleTest$SomeClass)
00:00 TRACE: [kryo] Register class name: com.esotericsoftware.kryo.CompatibleTest$SomeClass (com.esotericsoftware.kryo.serializers.CompatibleFieldSerializer)
00:00 TRACE: [kryo] Read fields for class: com.esotericsoftware.kryo.CompatibleTest$SomeClass
00:00 TRACE: [kryo] Read field name: anotherClass
00:00 TRACE: [kryo] Read field name: value
00:00 TRACE: [kryo] Unknown field will be skipped: anotherClass
00:00 TRACE: [kryo] Read chunk: 75
00:00 TRACE: [kryo] Read chunk: 4
00:00 DEBUG: [kryo] Unable to read unknown data (unknown type).
com.esotericsoftware.kryo.KryoException: Unable to find class: com.esotericsoftware.kryo.CompatibleTest$AnotherClass
	at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:180)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:149)
	at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:677)
	at com.esotericsoftware.kryo.serializers.CompatibleFieldSerializer.read(CompatibleFieldSerializer.java:134)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:693)
	at com.esotericsoftware.kryo.CompatibleTest.run(CompatibleTest.java:55)
	at com.esotericsoftware.kryo.CompatibleTest.main(CompatibleTest.java:65)
Caused by: java.lang.ClassNotFoundException: com.esotericsoftware.kryo.CompatibleTest$AnotherClass
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:174)
	... 6 more
00:00 TRACE: [kryo] Read chunk: 1
00:00 TRACE: [kryo] Next chunk.
00:00 TRACE: [kryo] Read chunk: 2
00:00 TRACE: [kryo] Read class 1: String [1]
00:00 TRACE: [kryo] Read field String: value (com.esotericsoftware.kryo.CompatibleTest$SomeClass) [107]
00:00 TRACE: [kryo] Object graph complete.
Exception in thread "main" com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 2, Size: 1
Serialization trace:
value (com.esotericsoftware.kryo.CompatibleTest$SomeClass)
	at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:133)
	at com.esotericsoftware.kryo.serializers.CompatibleFieldSerializer.read(CompatibleFieldSerializer.java:170)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:693)
	at com.esotericsoftware.kryo.CompatibleTest.run(CompatibleTest.java:55)
	at com.esotericsoftware.kryo.CompatibleTest.main(CompatibleTest.java:65)
Caused by: java.lang.IndexOutOfBoundsException: Index: 2, Size: 1
	at java.util.ArrayList.rangeCheck(ArrayList.java:653)
	at java.util.ArrayList.get(ArrayList.java:429)
	at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:62)
	at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:841)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:713)
	at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:122)
	... 4 more

The problem is when writing, the AnotherClass fields are written first and it contains the "Hello Kryo!" string. A reference ID is written, then the characters. When writing that string later, for SomeClass, only the reference ID is written. When reading, the AnotherClass class doesn't exist, so its bytes are skipped -- including the bytes for the "Hello Kryo!" reference ID and characters. When the reference ID is encountered for SomeClass#value, Kryo doesn't know what to do because it never read the reference. This is described here, under readUnknownFieldData:
https://github.com/EsotericSoftware/kryo/#compatiblefieldserializer-settings

I'm afraid there's not much that can be done. The only way to make it safe us if you don't use references. You could disable references for String.class (override ReferenceResolver#useReferences) which might make it less likely. You can try to avoid adding/removing classes, but likely you are using CompatibleFieldSerializer because that is exactly what you want to do.

As a last resort, you could likely write your own serializer which stores ID -> reference mappings separate from the serialized data. Kryo writes the reference IDs, so you may want to modify Kryo to not do that. The IDs themselves are available from the ReferenceResolver.

FWIW, I prefer using TaggedFieldSerializer. It is efficient and backward compatible, but not forward compatible.