Multiple segmentation faults

Question

I am trying to upgrade my Raspberry Pi Zero and have been fighting a multitude of segmentation faults, and I am at a loss as to how to fix this. When I enter sudo apt-get upgrade, this is the output:

osmc@OSMCPIZERO32:~$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
rbp1-kernel-osmc
The following packages will be upgraded:
armv6l-eventlircd-osmc armv6l-remote-osmc base-files-osmc diskmount-osmc libxslt1.1 mediacenter-addon-osmc
perftune-osmc rbp-bootloader-osmc rbp-userland-dev-osmc rbp-userland-osmc rbp1-mediacenter-osmc
wireless-firmware-osmc
12 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Need to get 58.6 MB of archives.
After this operation, 279 kB disk space will be freed.
Do you want to continue? [Y/n] yes
Get:1 http://archive.raspbian.org/raspbian/ jessie/main libxslt1.1 armhf 1.1.28-2+deb8u1 [213 kB]
Get:2 http://apt.osmc.tv/ jessie/main base-files-osmc all 1.9.5 [15.4 kB]
Get:3 http://apt.osmc.tv/ jessie/main diskmount-osmc all 1.4.4 [3,684 B]
Get:4 http://apt.osmc.tv/ jessie/main perftune-osmc all 1.2.0 [2,042 B]
Get:5 http://apt.osmc.tv/ jessie/main rbp-bootloader-osmc armhf 1.2.9-1 [2,165 kB]
Get:6 http://apt.osmc.tv/ jessie/main rbp-userland-dev-osmc armhf 1.2.0-5 [417 kB]
Get:7 http://apt.osmc.tv/ jessie/main rbp-userland-osmc armhf 1.2.0-5 [1,874 kB]
Get:8 http://apt.osmc.tv/ jessie/main mediacenter-addon-osmc all 3.0.618 [18.0 MB]
Get:9 http://apt.osmc.tv/ jessie/main rbp1-mediacenter-osmc armhf 16.1.0-12 [30.3 MB]
Get:10 http://apt.osmc.tv/ jessie/main armv6l-eventlircd-osmc armhf 1.2.0 [34.8 kB]
Get:11 http://apt.osmc.tv/ jessie/main armv6l-remote-osmc armhf 1.2.1 [1,343 kB]
Get:12 http://apt.osmc.tv/ jessie/main wireless-firmware-osmc all 1.1.2 [4,266 kB]
Fetched 58.6 MB in 3min 37s (269 kB/s)
dpkg-deb: error: subprocess tar was killed by signal (Segmentation fault)
dpkg: error processing archive /var/cache/apt/archives/libxslt1.1_1.1.28-2+deb8u1_armhf.deb (--unpack):
subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: subprocess tar was killed by signal (Segmentation fault)
dpkg: error processing archive /var/cache/apt/archives/base-files-osmc_1.9.5_all.deb (--unpack):
subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: subprocess tar was killed by signal (Segmentation fault)
dpkg: error processing archive /var/cache/apt/archives/diskmount-osmc_1.4.4_all.deb (--unpack):
subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: subprocess tar was killed by signal (Segmentation fault)
dpkg: error processing archive /var/cache/apt/archives/perftune-osmc_1.2.0_all.deb (--unpack):
subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: subprocess tar was killed by signal (Segmentation fault)
dpkg: error processing archive /var/cache/apt/archives/rbp-bootloader-osmc_1.2.9-1_armhf.deb (--unpack):
subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: subprocess tar was killed by signal (Segmentation fault)
dpkg: error processing archive /var/cache/apt/archives/rbp-userland-dev-osmc_1.2.0-5_armhf.deb (--unpack):
subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: subprocess tar was killed by signal (Segmentation fault)
dpkg: error processing archive /var/cache/apt/archives/rbp-userland-osmc_1.2.0-5_armhf.deb (--unpack):
subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: subprocess tar was killed by signal (Segmentation fault)
dpkg: error processing archive /var/cache/apt/archives/mediacenter-addon-osmc_3.0.618_all.deb (--unpack):
subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: subprocess tar was killed by signal (Segmentation fault)
dpkg: error processing archive /var/cache/apt/archives/rbp1-mediacenter-osmc_16.1.0-12_armhf.deb (--unpack):
subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: subprocess tar was killed by signal (Segmentation fault)
dpkg: error processing archive /var/cache/apt/archives/armv6l-eventlircd-osmc_1.2.0_armhf.deb (--unpack):
subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: subprocess tar was killed by signal (Segmentation fault)
dpkg: error processing archive /var/cache/apt/archives/armv6l-remote-osmc_1.2.1_armhf.deb (--unpack):
subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: subprocess tar was killed by signal (Segmentation fault)
dpkg: error processing archive /var/cache/apt/archives/wireless-firmware-osmc_1.1.2_all.deb (--unpack):
subprocess dpkg-deb --control returned error exit status 2
 Errors were encountered while processing:
/var/cache/apt/archives/libxslt1.1_1.1.28-2+deb8u1_armhf.deb
/var/cache/apt/archives/base-files-osmc_1.9.5_all.deb
/var/cache/apt/archives/diskmount-osmc_1.4.4_all.deb
/var/cache/apt/archives/perftune-osmc_1.2.0_all.deb
/var/cache/apt/archives/rbp-bootloader-osmc_1.2.9-1_armhf.deb
/var/cache/apt/archives/rbp-userland-dev-osmc_1.2.0-5_armhf.deb
/var/cache/apt/archives/rbp-userland-osmc_1.2.0-5_armhf.deb
/var/cache/apt/archives/mediacenter-addon-osmc_3.0.618_all.deb
/var/cache/apt/archives/rbp1-mediacenter-osmc_16.1.0-12_armhf.deb
/var/cache/apt/archives/armv6l-eventlircd-osmc_1.2.0_armhf.deb
/var/cache/apt/archives/armv6l-remote-osmc_1.2.1_armhf.deb
/var/cache/apt/archives/wireless-firmware-osmc_1.1.2_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Any thoughts or advice would be most appreciated before I tear out the rest of my hair. :) Thanks!

Answer 1

Thanks for all those who replied. I decided to just scrap the install and use a backup image I had made a few days prior. I'm pretty sure that the reason for all the seg faults is due to me (without realizing the future problems that would ensue at the time) editing my sources list in order to get the Weaved package installed on my Pi Zero. I didn't realize that only Raspbian repos were to be on that list. I was installing Weaved in order to figure out a way to remotely SSH my Pi when it was connected to a VPN tunnel, because I am having trouble forwarding packets via iptables to make it work that way. In the end, I didn't even use the Weaved service, and ended up corrupting my card as well. A lose-lose for me as it turns out. But thank god for making regular backups, because I've put literally hundreds of hours trying to configure this card exactly the way I want it to be (while learning a fair amount of network administration and Linux along the way..)

Answer 2

Cause

I had the similar problem, and also suspect that it is due to wrong apt source, especially the one in /etc/apt/sources.list.d/raspi.list.

It seems that /etc/apt/sources.list.d/raspi.list should always use debian source from raspberrypi (other than the official or mirror of official debian repo):

deb http://archive.raspberrypi.org/debian/ jessie main ui

If the official debian source is used, it probably introduces wrong binary format in upgrade, and they result in segment fault when execute. (Also, I didn't find the mirror list of archive.raspberrypi.org/debian.)

I found that at least the following binaries exit with segfault immediately after start at the moment:

1. /bin/tar
2. /bin/sed
3. /usr/bin/reset

The first two of above are used in unpacking or some configuration scripts during apt-get upgrade, and cause the failure. Even after the source files are restored to the original ones, the corrupted binaries still exist and interrupt all future upgrades.

Solution

Other than reinstall the whole system, the corrupted binaries can be recovered from the official image:

1. Download the raspbian image from raspberrypi.org.
2. Mount the second partition of the image (which is the root file system).
3. Copy the corresponding binaries from the official image to your pi.
4. Make sure the source files are correct (both /etc/apt/sources.list and /etc/apt/sources.list.d/raspi.list.
5. sudo apt-get clean && sudo apt-get update && sudo apt-get upgrade to see if it works (if not, see details below).

Find corrupted binaries

I didn't find a way to make apt-get be verbose, neither found its log with more details. But the cause of segment fault in apt-get can still be indicated via strace.

The following command exports the execve system call (it is used for executing another binary) trace of apt-get upgrade to file named execlist:

sudo strace -f -e trace=execve apt-get -y upgrade > execlist 2>&1

Wait until it finishes. if it doesn't end after a long time, hit ctrl+c to interrupt. Open the execlist and find the line containing killed by SIGSEGV. Look for execve just before these lines with the same pid, and the first argument of execve reveals the corrupted binary.

For example:

...
[pid 30462] execve("/bin/sed", ["sed", "-n", "-e\n\t\ts/^[[:space:]]*\\[global\\]/\\"..., "/etc/samba/smb.conf"], [/* 25 vars */]) = 0
Process 30463 attached
[pid 30463] execve("/usr/bin/tail", ["tail", "-n", "1"], [/* 25 vars */]) = 0
[pid 30462] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR,  si_addr=0x61096} ---
[pid 30463] +++ exited with 0 +++
[pid 30462] +++ killed by SIGSEGV +++
[pid 30461] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=30463, si_uid=0, si_status=0, si_utime=0, si_stime=3} ---
Segmentation fault
...

Shows pid 30462 is killed by SIGSEGV, which was started with execve("/bin/sed", ...). After that, you may run /bin/sed to check if it really is corrupted.