I'm currently trying to upgrade a ubuntu 14.04 box to xenial. I'm trying to do do release update, and its failing with errors like UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 382: invalid start byte

It looks like a known bug - I've tried that and had no luck finding the offending package, and disabled/removed my 2 non standard package.lst files for nodesource and veeam repositories.

The traceback reads something like this

Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-woadaq_z/xenial", line 8, in <module>
    sys.exit(main())
  File "/tmp/ubuntu-release-upgrader-woadaq_z/DistUpgrade/DistUpgradeMain.py", line 242, in main
    if app.run():
  File "/tmp/ubuntu-release-upgrader-woadaq_z/DistUpgrade/DistUpgradeController.py", line 1876, in run
    return self.fullUpgrade()
  File "/tmp/ubuntu-release-upgrader-woadaq_z/DistUpgrade/DistUpgradeController.py", line 1757, in fullUpgrade
    if not self.doPostInitialUpdate():
  File "/tmp/ubuntu-release-upgrader-woadaq_z/DistUpgrade/DistUpgradeController.py", line 943, in doPostInitialUpdate
    self.tasks = self.cache.installedTasks
  File "/tmp/ubuntu-release-upgrader-woadaq_z/DistUpgrade/DistUpgradeCache.py", line 806, in installedTasks
    for line in pkg._pcache._records.record.split("\n"):
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 382: invalid start byte
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/problem_report.py", line 416, in add_to_existing
    self.write(f)
  File "/usr/lib/python3/dist-packages/problem_report.py", line 369, in write
    block = f.read(1048576)
  File "/usr/lib/python3.4/codecs.py", line 319, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Original exception was:
Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-woadaq_z/xenial", line 8, in <module>
    sys.exit(main())
  File "/tmp/ubuntu-release-upgrader-woadaq_z/DistUpgrade/DistUpgradeMain.py", line 242, in main
    if app.run():
  File "/tmp/ubuntu-release-upgrader-woadaq_z/DistUpgrade/DistUpgradeController.py", line 1876, in run
    return self.fullUpgrade()
  File "/tmp/ubuntu-release-upgrader-woadaq_z/DistUpgrade/DistUpgradeController.py", line 1757, in fullUpgrade
    if not self.doPostInitialUpdate():
  File "/tmp/ubuntu-release-upgrader-woadaq_z/DistUpgrade/DistUpgradeController.py", line 943, in doPostInitialUpdate
    self.tasks = self.cache.installedTasks
  File "/tmp/ubuntu-release-upgrader-woadaq_z/DistUpgrade/DistUpgradeCache.py", line 806, in installedTasks
    for line in pkg._pcache._records.record.split("\n"):
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 382: invalid start byte
=== Command terminated with exit status 1 (Mon Apr  3 09:31:21 2017) ===

And there's nothing really helpful in the logs. How would I get do-release update to work?

up vote 44 down vote accepted

What you have there is the upgrade script itself tripping over invalid data somewhere. You need to find and remove the invalid data.

In this case, it was the package veeamsnap. Removing that package should fix it. But because this is different for each case, I'll describe the steps taken to reach that conclusion. It is a fairly complicated process.

This is a fun one, because python3 strings should all be in UTF-8. What you have here (discovered after the fact) is a C module (apt_pkg) somehow inserting non-UTF-8 data into a python3 string, therefore breaking every attempt at reading the string - notice how the error handler itself threw an exception too?

Into the unknown debugger we go!

The best way to diagnose issues like this is to cause the debugger to pause before the line that fails. With Python, when you have a series of nested calls like this the easiest way to add a debugger pause is to edit the file itself.

  1. Using your example, we can see that the failure in question is in the file /tmp/ubuntu-release-upgrader-woadaq_z/DistUpgrade/DistUpgradeCache.py line 806, so let's fire up a text editor and go to that line. The temp path will be different for each run, so make sure you use the one from your error output!

    screenshot of editor

  2. From here, we can first add a simple pause in the debugger, by inserting import pdb; pdb.set_trace(); on line 806 just before the error. Because this is Python, the indentation is important!

    screenshot of debugging statement

  3. Now we need to run the modified program. Don't run do-release-upgrade again; that'll probably download a new one. See in the error logs, the first line after "Original exception was"? The one with /tmp/ubuntu-release-upgrader-woadaq_z/xenial? That's the one you want to run. So run that file, as root (or sudo).

    Running that should get you into the debugger (pdb):

    screenshot of debugger

  4. From here, we figure out how many packages there are in total. The easy way to do that is to run sum(1 for _ in self). Wait a bit (this can take a while) and it will print a number. In this case, it was 76028.

    Now, since the error probably doesn't happen in the first few, and we don't want to manually step through >75000 packages, and we can't add an exception handler (because the error is so bad it breaks Python itself), we need an alternative.

  5. Remove the line added in step 4. Edit the code to print an incrementing number out for every package. For example, add foo = 0 above the loop on line 802 and foo += 1; print(foo) on line 807 (just before the erroring line).

    screenshot of number printing code

  6. Run the code again, using the same command as in step 3. It will print a large list of numbers. Let it keep running until it prints the error again. You might need to enlarge your window:

    screenshot of number output

    That last number should be the package it crashed on. Keep note of that number.

  7. Now that you know which package/number causes the crash, it's time to add the debugger pause with a condition to only execute on that package. For example, if you crash on package 72285, add if foo == 72285: import pdb; pdb.set_trace() just after the line that prints foo:

    screenshot of new pdb pause

  8. Run the code again. Now when you get into pdb it should be on the package that causes the crash. You can type in the name of the variable pkg to print its value, which will tell you the current package's name:

    screenshot of package name

    More generally, typing in the name of any variable will print its output.

  9. Remove the offending package and try the upgrade again (from a clean do-release-upgrade).

  • 7
    This is a very nice, very gentle intro to gdb, which can be used with different levels of proficiency by just about any user. +1 from me, and kudos. And, BTW, you may just add that typing pkg in the debugger will print the value of the variable of the same name, as defined on line 803. In other words, pkg is not a debugger instruction. Cheers. – MariusMatutiae Apr 3 '17 at 8:14
  • @MariusMatutiae Edited. And it's pdb ;) (This was actually intended more to be specific to solving this class of problems, but it's nice that you find it easy to follow as a general intro.) – Bob Apr 3 '17 at 8:59
  • To resolve this problem in particular, wouldn't it be easier to just add a line to the script that prints whatever that debug message wants to print for a package record that doesnt exist? (Theres that logging.debug message right above) Or does this assume the pkg variable might not be able to be printed at all due to the bug, and python debugger can print anything at all? – CausingUnderflowsEverywhere Apr 3 '17 at 16:37
  • If we still have the Super User blog, this would be an excellent addition to it! – Canadian Luke Apr 3 '17 at 17:00
  • @CausingUnderflowsEverywhere In theory, yes. In practice, a similar suggestion from the linked bug report apparently didn't work (I'm not sure why, just from what OP told me) and I ended up doing it interactively in case something else triggered the crash - e.g. didn't know that in this case it was the record property itself that could not be read. – Bob Apr 4 '17 at 7:34

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.