get back to kernel 4.14.67

[chavafg]

There is a performance degradation when using 4.19.10 kernel, so lets get back to 4.14.67

14:32:43 Report Summary:
14:32:43 +-----+----------------------+---------------------+---------------+--------+-------+--------+--------+------+------+-----+
14:32:43 | P/F |         NAME         |         FLR         |     MEAN      |  CEIL  |  GAP  |  MIN   |  MAX   | RNG  | COV  | ITS |
14:32:43 +-----+----------------------+---------------------+---------------+--------+-------+--------+--------+------+------+-----+
14:32:43 | *F* | boot-times           | Failed to load JSON | exit status 2 |        |       |        |        |      |      |     |
14:32:43 | *F* | memory-footprint     | 85.0%               | 128.1%        | 115.0% | 30.0% | 128.1% | 128.1% | 0.0% | 0.0% |   1 |
14:32:43 | P   | memory-footprint-ksm | 95.0%               | 101.6%        | 105.0% | 10.0% | 101.6% | 101.6% | 0.0% | 0.0% |   1 |
14:32:43 +-----+----------------------+---------------------+---------------+--------+-------+--------+--------+------+------+-----+
14:32:43 Fails: 2, Passes 1
14:32:43 Failed

http://jenkins.katacontainers.io/job/kata-metrics-tests-ubuntu-16-04-PR/311/console

[grahamwhaley]

@chavafg - that 'failed to load json' is more a sign that something went wrong with the test run (rather than failing a boundary value check iyswim).
We can either check the logs in more detail, but probably more useful is for somebody to use the report generator to do a hand run/compare locally between the kernel versions to get a better view of any impacts and regressions.

[chavafg]

@grahamwhaley yeah, I thought that about the boot time:
On the log I see:

14:19:07 ERROR: No kernel last line

But in addition, the memory foot print measurement failed.

[grahamwhaley]

For the 'no last line', the test looks for the phrase Freeing unused kernel memory in the kernel dmesg logs to locate the end of the kernel initialisation timestamp. Usual reasons for not finding that phrase would be:

• it changed in the kernel code (possible, but I would say unlikely, but pretty easy to check)
• some bad interaction with stdout, and we might be dropping or truncating the output - we have seen that before.

The memory footprint is more likely 'real' - so we should check that locally (it can have random fails on the metrics CI servers occasionally if we are unlucky with our timing :-( ).

[chavafg]

Maybe now these are the lines?:

14:19:07 >: [[    0.886457] Freeing unused kernel image memory: 836K
14:19:07 ]
14:19:07 >: [[    0.887657] Write protecting the kernel read-only data: 16384k
14:19:07 ]
14:19:07 >: [[    0.888820] Freeing unused kernel image memory: 2016K
14:19:07 ]
14:19:07 >: [[    0.889388] Freeing unused kernel image memory: 1208K

[grahamwhaley]

yep, looks like there was a kernel commit around August 2nd 2018 that changed the strings used in the free calls :-)
https://lkml.org/lkml/2018/8/5/172

Guess we'd better open an Issue - let me go do that.

[grahamwhaley]

Added Issue at kata-containers/tests#1036

[sboeuf]

@grahamwhaley @chavafg shouldn't we keep this issue opened, or open another one to switch back to 4.19.10 asap when we get the last line patch merged?

[grahamwhaley]

Last line patch submitted at kata-containers/tests#1044
@chavafg @sboeuf