What we pass to ZPAQ Backend

A big change to ZPAQ

(Note, there was a bug in libzpaq.cpp which limited decompression to levels 3,4, and 5. This has been corrected.)

The lz4_compresses function will now return a value that indicates to what percent a block potentially can be compressed or zero if the compression Threshold is not met. It's just a snapshot of the entire buffer, but the result is used to determine how the ZPAQ backend will handle it. To this end, we compute zpaq_ease to indicate how easy or hard it might be for ZPAQ to compress. 0 would be very hard, i.e randomized data. 255 would be easy, i.e. more text.

Ex. If lz4_compresses estimates a block is 75% compressible, that means, the estimate is the resulting block would be 75% it's original size, based on lz4. zpaq_ease therefore is the inverse of that, scaled to 255.

zpaq_ease = 255 - (75 * 2.55) = 191.25 = 191

This becomes part of the method string passed to ZPAQ.
zpaq_type is always 0 to let the backend determine how to handle the block.
zpaq_level is computed in util.c as 1 to 5, depending on the compression level.
control->zpaq_bs = block size computed in util.c
see Computations for Memory Overhead Required for LZMA and ZPAQ

The method string conforms to the ZPAQ standard: LB,E,T where

• L = ZPAQ Compression Level (1,2,3,4,5)
• B = Block Size 6-11
• E = ease of compression (textiness)
• T = Type of file. This is always 0 now to let backend decide.

 171         int zpaq_ease, zpaq_type, compressibility;
 172         char method[10]; /* level, block size, ease of compression, type */
 173
 174         if (!(compressibility=lz4_compresses(control, cthread->s_buf, cthread->s_len)))
 175                 return 0;
 176
...
 185         /* Compression level can be 1 to 5, zpaq version 7.15 */
 186         
 187         zpaq_ease = 255-(compressibility * 2.55);       /* 0, hard, 255, easy. Inverse of lz4_compresses */
 188         if (zpaq_ease < 25) zpaq_ease = 25;             /* too low a value fails */
 189         zpaq_type = 0;                                  /* default, binary data */
 190
 191         sprintf(method,"%d%d,%d,%d",control->zpaq_level,control->zpaq_bs,zpaq_ease,zpaq_type);
 192
 193         print_verbose("ZPAQ: Method selected: %s: level=%d, bs=%d, easy=%d, type=%d\n",
 194                        method, control->zpaq_level, control->zpaq_bs, zpaq_ease, zpaq_type);
 195
 196         zpaq_compress(c_buf, &c_len, cthread->s_buf, cthread->s_len, &method[0],
 197                         control->msgout, SHOW_PROGRESS ? true: false, thread);
 198