[FR] Interest in an on-the-fly serial/g-code compression technique from Prusa's FW? Existing PR...

[scottmudge]

Description

I designed an implemented a simple but effective way to compress g-code over the serial connection by a factor of ~0.61, with no computational cost to the firmware. I've also written an associated OctoPrint plugin that compresses the data from the PC-side automatically, and manages the configuration state between firmware and PC.

It was noted by a few people this could benefit many other printers with slower MCUs, which may struggle to keep up with fast printing on low baud-rate serial connections.

Feature Workflow

Please see the Prusa Firmware PR here: prusa3d/Prusa-Firmware#2955

If there is interest, I will make the associated modifications to the Marlin FW. I will need someone to test with OctoPrint (with the associated plugin), as I do not have another printer with which I can flash Marlin.

[qwewer0]

Is it only beneficial for printing through serial or also from onboard SD card?

[scottmudge]

I did implement a version and a tool which compressed gcode files before they are stored on the SD card, but I didn't see any sort of performance benefit. The only benefit was compressing the gcode files to about ~60% of their original size. The downside is the gcode is no longer human readable.

SD card printing uses the SPI bus as I understand it, which is generally quite a bit faster than a standard 115,200 baud serial connection.

The compression method does require an interface between the gcode and the firmware. For OctoPrint, the OctoPrint-MeatPack plugin fills this role, where it automatically compresses outgoing data on-the-fly. As noted I wrote a packing/compression application for standalone gcode files to serve this role for SD printing, but the benefit was marginal at best, and it would be an extra step to compress the gcode files before uploading them to the SD card.

The primary purpose of this is to alleviate throughput bottlenecks on the serial port. An issue I haven't really observed with SD card printing.

[qwewer0]

Got it, thanks.

[sjasonsmith]

I was skeptical at first, but after reading your documentation in your Octoprint plugin repo it seems like a promising idea.

A little care might be needed to allow it to co-exist with the binary transfer protocol, or to prevent using them together if they are fundamentally incompatible.

[ellensp]

It is also incompatible with GCODE_CASE_INSENSITIVE, as it looks for specific UPPERCASE characters and numbers to pack.

[scottmudge]

The case insensitivity would be on the OctoPrint_MeatPack plugin side.

It would be trivial to add a catch to change all lowercase command letters -- 'x', 'y', 'e', and so on -- to their uppercase counterparts.

In fact, there would be no reason to send the lowercase variant, with the exception of lowercase 'e', used in exponent notation for floating points. Which I don't think RS-274 requires support for anyway.

I will add the case-conversion catch in the next version.

I've also just finished testing the whitespace removal feature, and after patching an issue with the official Prusa firmware (possibly Marlin as well? it looks like they share the cmdqueue code), where strtod() was parsing upper-case "E"s as exponent markers when they lack a whitespace before them, it works great.

The compression ratio is now close to 0.5, nearly doubling throughput over the standard serial connection.

[scottmudge]

Alright I modified the OctoPrint plugin to replace the necessary characters to their upper case. It only does this to "G" commands, as "M" commands can sometimes take string parameters.

I should also note that lower case letters won't harm anything. They'll just take 1.5 bytes to send versus 1.

But given that G0/G1 commands are 95% of a g-code file, and given that the plugin now ensures such outgoing commands have proper, packable letter cases, it shouldn't be a problem.

[ellensp]

@scottmudge agreed, with the sender capitalising GCODE_CASE_INSENSITIVE is no longer an issue.

[scottmudge]

So far it seems to be working well.

I've tested the following features, printing ~3 12-hour prints for a total of ~85MB raw gcode, ~46.75 MB effectively sent MB over serial:

• Normal packing method described by original MeatPack project.
• Removal of whitespace and automatic correction of lower-case to upper-case letters (in "G" commands only, as "M" commands sometimes receive arbitrary strings as parameters).
• Replacement of space ' ' character with 'E' in the packing table when white-space removal is active.
• Implementation of protocol versions to aid backward compatibility between firmware and plugin, in the event future features are added. By default if the MeatPack firmware module emits no protocol version, this indicates protocol version 0, where only the original packing method is supported.

I've encountered no issues, but I did need to correct one aspect of the firmware outside of the MeatPack code. As described previously, the strtod() method used to convert the ASCII strings to floats was parsing upper-case "E" characters as exponent markers. So when whitespace was removed between the Y parameter and the E parameter for G0/G1 commands, both the parsed Y-component was incorrect, and the E-component was lost.

To fix this, I implemented a modified version of strtod() which ignores uppercase 'E' characters. This allows for exponent notation to be used, but only with lower-case 'e' characters.

There was a hacky workaround already in the Prusa firmware code (possibly from its Marlin roots), but it was performing a secondary substring scan for the 'E' character, replacing it with \0 if it existed, extracting the float with standard strtod(), and then replacing the \0 with E before returning. It was horribly inefficient, so this has been corrected.

After all these fixes, I am getting an improved compression ratio of ~ 0.55:

Meaning the effective rate of the 115,200 baud serial connection is now ~210,000 baud. A significant improvement.

I plan to update the OctoPrint plugin and modified Prusa firmware release with these changes soon (from which I can work with your existing Marlin PR, but you will need to test it, as I do not have a Marlin-compatible MCU). Unfortunately I am quite busy with my other work, so this will likely happen in the next few days.

Thanks for the great ideas about culling the whitespace, it worked quite well after some debugging.

[ellensp]

Merged, closing

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.