Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Base64 validation issue when handling binary payload sent with SENML_JSON format #1548

Closed
vshorin opened this issue Nov 27, 2023 · 3 comments
Closed

Base64 validation issue when handling binary payload sent with SENML_JSON format #1548

vshorin opened this issue Nov 27, 2023 · 3 comments
Labels
bug Dysfunctionnal behavior core Impact core of Leshan

Comments

@vshorin
Copy link

vshorin commented Nov 27, 2023

Version(s)

v2.0.0-M11 and up

Which components

leshan-core

Tested With

leshan-client-demo

What happened

When sending object 19 with a "send" operation from the leshan client, the server returns [BAD_REQUEST(400)] : Invalid Payload. if SENML_JSON is used and the Base64 version of the payload contains "w" in the end. If SENML_CBOR is used on the same operation, the server returns [CHANGED(204)]..
The server is using org.eclipse.leshan.core.util.base64.DefaultBase64Decoder method validateIsCanonical. There seems to be a typo here:

leshan/leshan-core/src/main/java/org/eclipse/leshan/core/util/base64/DefaultBase64Decoder.java

Line 185 in fa6f7d2

validChar = new char[] { 'A', 'Q', 'g', 'W' };

and

leshan/leshan-core/src/main/java/org/eclipse/leshan/core/util/base64/DefaultBase64Decoder.java

Line 197 in fa6f7d2

validChar = new char[] { 'A', 'Q', 'g', 'W', 'E', 'U', 'k', '0', 'I', 'Y', 'o', '4', 'M', 'c', 's', '8' };

where validChar array contains "W", but it seems that it should have contained "w" instead.

How to reproduce

  1. Have a leshan client that has object 19 defined
  2. Resource 19/0/0 should have value that ends with padded "w" when base64 encoded, like "ABCDEFG" => "QUJDREVGRw==" or "ABCDL" => "QUJDREw=":
public class FakeBinaryObject extends BaseInstanceEnabler {
...
    private byte[] value;

    public FakeBinaryObject() {
        this.value = "ABCDEFG".getBytes();
        // this.value = "ABCDL".getBytes();
    }

    @Override
    public synchronized ReadResponse read(LwM2mServer server, int resourceId) {
        LOG.info("Read on Fake Binary Object resource /{}/{}/{}", getModel().id, getId(), resourceId);
        switch (resourceId) {
        case RESOURCE_DATA:
            return ReadResponse.success(resourceId, this.value);
...
        default:
            return super.read(server, resourceId);
        }
    }
...
  1. Connect to the server
  2. Use send commands with object 19:
send -c SENML_CBOR current-value 19
send -c SENML_JSON current-value 19

Relevant Output

client logs:

send -c SENML_CBOR current-value 19
           InteractiveCommands 2023-11-27 12:47:45,850 [INFO] Sending Data to coaps://127.0.0.1:5684[LWM2M_SERVER 1111] using SENML_CBOR(112).
              FakeBinaryObject 2023-11-27 12:47:45,853 [INFO] Read on Fake Binary Object resource /19/0/0
              FakeBinaryObject 2023-11-27 12:47:45,853 [INFO] Read on Fake Binary Object resource /19/0/1
              FakeBinaryObject 2023-11-27 12:47:45,853 [INFO] Read on Fake Binary Object resource /19/0/2
              FakeBinaryObject 2023-11-27 12:47:45,853 [INFO] Read on Fake Binary Object resource /19/0/3
              FakeBinaryObject 2023-11-27 12:47:45,853 [INFO] Read on Fake Binary Object resource /19/0/4
              FakeBinaryObject 2023-11-27 12:47:45,853 [INFO] Read on Fake Binary Object resource /19/0/5
           InteractiveCommands 2023-11-27 12:47:45,888 [INFO] Data sent successfully to coaps://127.0.0.1:5684[LWM2M_SERVER 1111] [CHANGED(204)].
send -c SENML_JSON current-value 19
           InteractiveCommands 2023-11-27 12:47:51,702 [INFO] Sending Data to coaps://127.0.0.1:5684[LWM2M_SERVER 1111] using SENML_JSON(110).
              FakeBinaryObject 2023-11-27 12:47:51,702 [INFO] Read on Fake Binary Object resource /19/0/0
              FakeBinaryObject 2023-11-27 12:47:51,702 [INFO] Read on Fake Binary Object resource /19/0/1
              FakeBinaryObject 2023-11-27 12:47:51,702 [INFO] Read on Fake Binary Object resource /19/0/2
              FakeBinaryObject 2023-11-27 12:47:51,703 [INFO] Read on Fake Binary Object resource /19/0/3
              FakeBinaryObject 2023-11-27 12:47:51,703 [INFO] Read on Fake Binary Object resource /19/0/4
              FakeBinaryObject 2023-11-27 12:47:51,703 [INFO] Read on Fake Binary Object resource /19/0/5
           InteractiveCommands 2023-11-27 12:47:51,734 [INFO] Send data to coaps://127.0.0.1:5684[LWM2M_SERVER 1111] failed [BAD_REQUEST(400)] : Invalid Payload.

server logs:

org.eclipse.leshan.core.request.exception.InvalidRequestException: Invalid payload in [CON-POST   MID=47999, Token=4C65C148D04AAFBD, OptionSet={"Uri-Path":"dp", "Content-Format":"application/senml+json"}, acked "[{"bn":"/19/","n":"0/0","vd":"QU".. 197 bytes] from [Identity /127.0.0.1:60725[psk=test]]
        at org.eclipse.leshan.server.californium.send.SendResource.handlePOST(SendResource.java:102) [leshan-server-cf-2.0.0-M11.jar:?]
        at org.eclipse.californium.core.CoapResource.handleRequest(CoapResource.java:261) [californium-core-3.8.0.jar:?]
        at org.eclipse.leshan.core.californium.LwM2mCoapResource.handleRequest(LwM2mCoapResource.java:57) [leshan-core-cf-2.0.0-M11.jar:?]
        at org.eclipse.californium.core.server.ServerMessageDeliverer.deliverRequest(ServerMessageDeliverer.java:133) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.stack.BaseCoapStack$StackTopAdapter.receiveRequest(BaseCoapStack.java:204) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.stack.AbstractLayer.receiveRequest(AbstractLayer.java:84) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.stack.AbstractLayer.receiveRequest(AbstractLayer.java:84) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.stack.BlockwiseLayer.receiveRequest(BlockwiseLayer.java:568) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.stack.ReliabilityLayer.receiveRequest(ReliabilityLayer.java:296) [californium-core-3.8.0.jar:?]     
        at org.eclipse.californium.core.network.stack.AbstractLayer.receiveRequest(AbstractLayer.java:84) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.stack.BaseCoapStack.receiveRequest(BaseCoapStack.java:126) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.CoapEndpoint$1.receiveRequest(CoapEndpoint.java:309) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.UdpMatcher$2.run(UdpMatcher.java:289) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.elements.util.SerialExecutor$1.run(SerialExecutor.java:293) [element-connector-3.8.0.jar:?]
        at org.eclipse.californium.core.network.CoapEndpoint$6.run(CoapEndpoint.java:1366) [californium-core-3.8.0.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.eclipse.leshan.core.node.codec.CodecException: Unable to decode nodes : ...
        at org.eclipse.leshan.core.node.codec.senml.LwM2mNodeSenMLDecoder.decodeTimestampedNodes(LwM2mNodeSenMLDecoder.java:214) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.node.codec.DefaultLwM2mDecoder.decodeTimestampedNodes(DefaultLwM2mDecoder.java:223) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.server.californium.send.SendResource.handlePOST(SendResource.java:81) ~[leshan-server-cf-2.0.0-M11.jar:?]
        ... 17 more
Caused by: org.eclipse.leshan.senml.SenMLException: Unable to parse SenML JSON.
        at org.eclipse.leshan.senml.json.jackson.SenMLJsonJacksonEncoderDecoder.fromSenML(SenMLJsonJacksonEncoderDecoder.java:99) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.node.codec.senml.LwM2mNodeSenMLDecoder.decodeTimestampedNodes(LwM2mNodeSenMLDecoder.java:198) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.node.codec.DefaultLwM2mDecoder.decodeTimestampedNodes(DefaultLwM2mDecoder.java:223) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.server.californium.send.SendResource.handlePOST(SendResource.java:81) ~[leshan-server-cf-2.0.0-M11.jar:?]
        ... 17 more
Caused by: org.eclipse.leshan.core.util.json.JsonException: Node vd with value 'QUJDREVGRw' is not in valid Base64 format.
        at org.eclipse.leshan.senml.json.jackson.SenMLJsonRecordSerDes.deserialize(SenMLJsonRecordSerDes.java:181) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.senml.json.jackson.SenMLJsonRecordSerDes.deserialize(SenMLJsonRecordSerDes.java:33) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.util.json.JacksonJsonSerDes.deserialize(JacksonJsonSerDes.java:65) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]        
        at org.eclipse.leshan.senml.json.jackson.SenMLJsonJacksonEncoderDecoder.fromSenML(SenMLJsonJacksonEncoderDecoder.java:96) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.node.codec.senml.LwM2mNodeSenMLDecoder.decodeTimestampedNodes(LwM2mNodeSenMLDecoder.java:198) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.node.codec.DefaultLwM2mDecoder.decodeTimestampedNodes(DefaultLwM2mDecoder.java:223) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.server.californium.send.SendResource.handlePOST(SendResource.java:81) ~[leshan-server-cf-2.0.0-M11.jar:?]
        ... 17 more
Caused by: org.eclipse.leshan.core.util.base64.InvalidBase64Exception: Base64 string QUJDREVGRw is not in canonical form.
        at org.eclipse.leshan.core.util.base64.DefaultBase64Decoder.validateIsCanonical(DefaultBase64Decoder.java:209) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.util.base64.DefaultBase64Decoder.validateEncodedData(DefaultBase64Decoder.java:102) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.util.base64.DefaultBase64Decoder.decode(DefaultBase64Decoder.java:68) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]     
        at org.eclipse.leshan.core.util.base64.DefaultBase64Decoder.decode(DefaultBase64Decoder.java:63) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]     
        at org.eclipse.leshan.senml.json.jackson.SenMLJsonRecordSerDes.deserialize(SenMLJsonRecordSerDes.java:178) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.senml.json.jackson.SenMLJsonRecordSerDes.deserialize(SenMLJsonRecordSerDes.java:33) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.util.json.JacksonJsonSerDes.deserialize(JacksonJsonSerDes.java:65) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]        
        at org.eclipse.leshan.senml.json.jackson.SenMLJsonJacksonEncoderDecoder.fromSenML(SenMLJsonJacksonEncoderDecoder.java:96) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.node.codec.senml.LwM2mNodeSenMLDecoder.decodeTimestampedNodes(LwM2mNodeSenMLDecoder.java:198) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.node.codec.DefaultLwM2mDecoder.decodeTimestampedNodes(DefaultLwM2mDecoder.java:223) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.server.californium.send.SendResource.handlePOST(SendResource.java:81) ~[leshan-server-cf-2.0.0-M11.jar:?]
        ... 17 more
@vshorin vshorin added the bug Dysfunctionnal behavior label Nov 27, 2023
@sbernard31 sbernard31 added the core Impact core of Leshan label Nov 27, 2023
sbernard31 added a commit that referenced this issue Nov 27, 2023
@sbernard31
GH-1548: Fix DefaultBase64Decoder rejecting payload finishing by 'W'
aab18a4
@sbernard31
Copy link
Contributor

sbernard31 commented Nov 27, 2023 •
edited
Loading

@vshorin very good catch, very well described !

Thx a lot for taking time to report this 🙏

This is now fixed in master (commit aab18a4) and this will be available in next release 2.0.0-M15. (not yet planned, we just release the 2.0.0-M14)

Waiting the workaround is to copy/paster the fixed version and configure your client/server/bsserver like :

Map<ContentFormat, NodeDecoder> decoders = new HashMap<>(DefaultLwM2mDecoder.getDefaultNodeDecoders(false));
decoders.put(ContentFormat.TEXT, new LwM2mNodeTextDecoder(new DefaultLwM2mLinkParser(),
        new FixedDefaultBase64Decoder(DecoderAlphabet.BASE64, DecoderPadding.REQUIRED)));
decoders.put(ContentFormat.SENML_JSON,
        new LwM2mNodeSenMLDecoder(new SenMLJsonJacksonEncoderDecoder(false,
                new FixedDefaultBase64Decoder(DecoderAlphabet.BASE64URL, DecoderPadding.FORBIDEN),
                new DefaultBase64Encoder(EncoderAlphabet.BASE64URL, EncoderPadding.WITHOUT)), true));
decoders.put(ContentFormat.JSON,
        new LwM2mNodeJsonDecoder(new LwM2mJsonJacksonEncoderDecoder(), new DefaultLwM2mLinkParser(),
                new FixedDefaultBase64Decoder(DecoderAlphabet.BASE64, DecoderPadding.REQUIRED)));
builder.setDecoder(new DefaultLwM2mDecoder(decoders, DefaultLwM2mDecoder.getDefaultPathDecoder()));

The fix should also be deployed on the sandbox : https://leshan.eclipseprojects.io/
Let me know if it works for you now and so we can close this issue.

(I think this bug exists since 2.0.0-M10 : ce98924#diff-d6e9393e636f661cde99688c1bbfedbd664fb43c7aefddd6a7a06eb32ff78826R158-R185)

@sbernard31
Copy link
Contributor

@cyril2maq, @JaroslawLegierski you could be interested by that ☝️

@vshorin
Copy link
Author

vshorin commented Nov 28, 2023

@sbernard31 wow, that was fast! Thanks a lot for your work! The workaround is fine for now, so I'll close.

@vshorin vshorin closed this as completed Nov 28, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
bug Dysfunctionnal behavior core Impact core of Leshan
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sbernard31 @vshorin