[Syncd failed during boot] syncd#supervisord: syncd /usr/bin/syncd: error while loading shared libraries: libprotobuf.so.10: cannot open shared object file: No such file or directory#015

[gechiang]

Description
Starting with BRCM cached build 144 syncd fails to boot up. Syslog shows the following error:

Dec 19 00:10:15.674639 str-s6000-acs-8 NOTICE swss#orchagent: :- main: Enabling synchronous mode
Dec 19 00:10:15.675629 str-s6000-acs-8 NOTICE swss#orchagent: :- main: --- Starting Orchestration Agent ---
Dec 19 00:10:15.677173 str-s6000-acs-8 NOTICE swss#orchagent: :- loadFromFile: no context config specified, will load default context config
Dec 19 00:10:15.701299 str-s6000-acs-8 NOTICE swss#orchagent: :- insert: added switch: 0:
Dec 19 00:10:15.731777 str-s6000-acs-8 NOTICE swss#orchagent: :- RedisChannel: creating notification thread
Dec 19 00:10:15.738778 str-s6000-acs-8 NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec 19 00:10:15.755610 str-s6000-acs-8 NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec 19 00:10:15.759712 str-s6000-acs-8 NOTICE swss#orchagent: :- startRecording: started recording: sairedis.rec
Dec 19 00:10:15.760714 str-s6000-acs-8 NOTICE swss#orchagent: :- initSaiRedis: Enable redis pipeline
Dec 19 00:10:15.770226 str-s6000-acs-8 NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec 19 00:10:16.046110 str-s6000-acs-8 INFO pmon#/supervisord: start Installing collected packages: sonic-platform
Dec 19 00:10:16.229909 str-s6000-acs-8 WARNING systemd[1]: Configuration file /lib/systemd/system/copp-config.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Dec 19 00:10:16.358641 str-s6000-acs-8 INFO syncd#supervisord: syncd /usr/bin/syncd: error while loading shared libraries: libprotobuf.so.10: cannot open shared object file: No such file or directory#015
Dec 19 00:10:16.359327 str-s6000-acs-8 NOTICE syncd#dsserve: child /usr/bin/syncd exited status: 32512
Dec 19 00:10:16.369980 str-s6000-acs-8 INFO syncd#supervisord: syncd [5] child /usr/bin/syncd exited status: 32512
Dec 19 00:10:16.551518 str-s6000-acs-8 INFO syncd#supervisord: ledinit connecting stream socket: Connection refused
Dec 19 00:10:16.734945 str-s6000-acs-8 INFO pmon#/supervisord: start Successfully installed sonic-platform-1.0
Dec 19 00:10:16.737014 str-s6000-acs-8 INFO syncd#supervisor-proc-exit-listener: Process syncd exited unxepectedly. Terminating supervisor...
Dec 19 00:10:16.891564 str-s6000-acs-8 INFO /hostcfgd: Running cmd: 'sudo systemctl start pmon.service'
Dec 19 00:10:16.929671 str-s6000-acs-8 INFO syncd#supervisord 2020-12-19 00:10:09,367 INFO exited: dependent-startup (exit status 0; expected)
Dec 19 00:10:16.929671 str-s6000-acs-8 INFO syncd#supervisord 2020-12-19 00:10:16,373 INFO exited: syncd (exit status 2; not expected)
Dec 19 00:10:16.929671 str-s6000-acs-8 INFO syncd#supervisord 2020-12-19 00:10:16,735 WARN received SIGTERM indicating exit request
Dec 19 00:10:16.929671 str-s6000-acs-8 INFO syncd#supervisord 2020-12-19 00:10:16,754 INFO waiting for ledinit, supervisor-proc-exit-listener, rsyslogd to die
Dec 19 00:10:16.986644 str-s6000-acs-8 INFO /hostcfgd: Feature 'pmon.service' is enabled and started

Steps to reproduce the issue:

1. Install any image after 143 (144 and above) will show this issue

admin@str-s6000-acs-8:~$ show vers

SONiC Software Version: SONiC.HEAD.144-dirty-20201217.112925
Distribution: Debian 10.7
Kernel: 4.19.0-9-2-amd64
Build commit: 55a70758
Build date: Thu Dec 17 11:38:40 UTC 2020
Built by: johnar@jenkins-worker-22

Platform: x86_64-dell_s6000_s1220-r0
HwSKU: Force10-S6000
ASIC: broadcom
ASIC Count: 1
Serial Number: 1QBRX42
Uptime: 02:53:28 up  2:45,  1 user,  load average: 4.79, 5.94, 5.94

Docker images:
REPOSITORY                    TAG                              IMAGE ID            SIZE
docker-snmp                   HEAD.144-dirty-20201217.112925   5d427c4d2a03        526MB
docker-snmp                   latest                           5d427c4d2a03        526MB
docker-teamd                  HEAD.144-dirty-20201217.112925   e605de3b8bc3        524MB
docker-teamd                  latest                           e605de3b8bc3        524MB
docker-nat                    HEAD.144-dirty-20201217.112925   def65a05e5b8        527MB
docker-nat                    latest                           def65a05e5b8        527MB
docker-router-advertiser      HEAD.144-dirty-20201217.112925   74d7a302cac1        481MB
docker-router-advertiser      latest                           74d7a302cac1        481MB
docker-platform-monitor       HEAD.144-dirty-20201217.112925   e908239cecba        607MB
docker-platform-monitor       latest                           e908239cecba        607MB
docker-dhcp-relay             HEAD.144-dirty-20201217.112925   dc4b84838d18        487MB
docker-dhcp-relay             latest                           dc4b84838d18        487MB
docker-database               HEAD.144-dirty-20201217.112925   956d6138cd6d        480MB
docker-database               latest                           956d6138cd6d        480MB
docker-lldp                   HEAD.144-dirty-20201217.112925   0c387e118e68        521MB
docker-lldp                   latest                           0c387e118e68        521MB
docker-orchagent              HEAD.144-dirty-20201217.112925   df5d2eebe519        555MB
docker-orchagent              latest                           df5d2eebe519        555MB
docker-sonic-telemetry        HEAD.144-dirty-20201217.112925   ba87a6f4fa5a        555MB
docker-sonic-telemetry        latest                           ba87a6f4fa5a        555MB
docker-sonic-mgmt-framework   HEAD.144-dirty-20201217.112925   e5f3ba2b01ba        641MB
docker-sonic-mgmt-framework   latest                           e5f3ba2b01ba        641MB
docker-fpm-frr                HEAD.144-dirty-20201217.112925   19b061f7ae8e        540MB
docker-fpm-frr                latest                           19b061f7ae8e        540MB
docker-sflow                  HEAD.144-dirty-20201217.112925   fecb4e4770db        524MB
docker-sflow                  latest                           fecb4e4770db        524MB
docker-syncd-brcm             HEAD.144-dirty-20201217.112925   63d60d5e6939        575MB
docker-syncd-brcm             latest                           63d60d5e6939        575MB

admin@str-s6000-acs-8:~$

Describe the results you expected:
Boot up with all the services functioning correctly!

The change that went into this build (144) has the following changes:
Support SONiC Reproduceable Build-debian/pip/web packages (#5718) (detail / githubweb)

[gechiang]

syslog.txt

[lguohan]

https://sonic-jenkins.westus2.cloudapp.azure.com/job/broadcom/job/buildimage-brcm-cache/144/artifact/target/docker-syncd-brcm.gz.log

can you check dpkg --list in the sync docker? I can see the libprotobuf is installed, not sure why it is missing here.

[lguohan]

not sure why we installed the stretch backport here.

root@str-s6000-acs-8:/# dpkg --list | grep libpro
ii  libprocps6:amd64           2:3.3.12-3+deb9u1                 amd64        library for accessing process information from /proc
ii  libprotobuf-dev:amd64      3.6.1-4~bpo9+1                    amd64        protocol buffers C++ library (development files) and proto files
ii  libprotobuf-lite17:amd64   3.6.1-4~bpo9+1                    amd64        protocol buffers C++ library (lite version)
ii  libprotobuf17:amd64        3.6.1-4~bpo9+1                    amd64        protocol buffers C++ library

root@str-s6000-acs-8:/# ldd ./usr/lib/libsai.so.1.0
        linux-vdso.so.1 (0x00007ffc90ec5000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1b3a8db000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1b3a6be000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1b3a4b6000)
        libprotobuf.so.10 => not found
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1b3a134000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1b39e30000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1b39a91000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f1b3987a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1b44a9f000)

[lguohan]

https://sonic-jenkins.westus2.cloudapp.azure.com/job/broadcom/job/buildimage-brcm-cache/141/artifact/target/docker-syncd-brcm.gz.log

Get:1 http://debian-archive.trafficmanager.net/debian stretch/main amd64 libc-dev-bin amd64 2.24-11+deb9u4 [259 kB]
Get:2 http://debian-archive.trafficmanager.net/debian-security stretch/updates/main amd64 linux-libc-dev amd64 4.9.240-2 [1556 kB]
Get:3 http://debian-archive.trafficmanager.net/debian stretch/main amd64 libc6-dev amd64 2.24-11+deb9u4 [2364 kB]
Get:4 http://debian-archive.trafficmanager.net/debian stretch/main amd64 zlib1g-dev amd64 1:1.2.8.dfsg-5 [205 kB]
Get:5 http://debian-archive.trafficmanager.net/debian stretch/main amd64 libprotobuf10 amd64 3.0.0-9 [681 kB]
Get:6 http://debian-archive.trafficmanager.net/debian stretch/main amd64 libprotobuf-lite10 amd64 3.0.0-9 [125 kB]
Get:7 http://debian-archive.trafficmanager.net/debian stretch/main amd64 libprotobuf-dev amd64 3.0.0-9 [954 kB]

[lguohan]

the issue is caused by this pr #5718

usually, the packages from backport is not preferred, is there a particular reason to prefer backport packages?

@gechiang , there is something you can improve the saibcm packaging as well, you can specify a particular packet version in the control file. you can specify

Package libprotobuf-dev, libprotobuf10, libprotobuf-lite10 explicitly to avoid such issue.