How urgent is a *** System restart required *** for security?

Question

To learn a bit of server administration I've set up a simple Ubuntu 14.04 server on which I run a personal website. I've set it to automatically install security updates, but leave out the other updates. This seems to work pretty fine. Occasionally I get a message when logging into the server (with ssh) saying:

*** System restart required ***

The times this happened I simple rebooted Ubuntu and all was fine. This is ok because it's a simple personal website. What I wonder about though, is how this works for webservers which should be up 99.9999etc% of the time? Do they simply not restart and risk the security being breached because security updates are not installed (which I cannot imagine)? Or do they take the downtime for granted (which I cannot imagine either)?

How should I handle this if this were a very important production server which I want to keep up and running? All tips are welcome!

[EDIT] I know I can do cat /var/run/reboot-required.pkgs to list the packages which cause the reboot. The command currently yields the following:

linux-image-3.13.0-36-generic
linux-base
dbus
linux-image-extra-3.13.0-36-generic
linux-base

but how do I know if the updates are little things of whether I have a serious security vulnerability if I don't do the restart?

[EDIT2] Okay, I now combined the commands I've found to be useful into one:

xargs aptitude changelog < /var/run/reboot-required.pkgs | grep urgency=high

If this doesn't output anything, there don't seem to be security issues with a high urgency.

One last question though: are low, medium, and high the only urgency possibilities, or are there any more like for example critical or extremelyimportant?

Answer 1

The is no simple answer as it depends on the updates made. If the kernel had a serious security problem then it is good to restart as soon as possible. If the kernel had only minor fixes then the restart could be postponed.

If you guarantee an availability > 99.9% then you will almost always have a clustered system where you can reboot the nodes one by one without interrupting the service.

So you reboot the first system and reatach it to the cluster. Then the second and so on. Then the service will never become unavailable.

Answer 2

addon for the topic solution

I perform similar check for 'reboot requirement' for zabbix monitoring system

I see 2 issue in 'Topic' solution:

1. aptitude usually works badly in scripts. I kill a few hours but still didn't make it work with zabbix
2. if only 1 changelog includes urgent update - your check will always show positive results

My logic is:

1. Check last change only in changelog for every package which requires system reboot
2. As an output show only highest priority update

Using Debian documentation I found 5 possible values for 'urgency' and also fact that it can followed by equal("=") or semicolon(":") characters. Also there're can be upper and lower case characters

So I ended up with following:

#!/bin/bash
##################################
# Zabbix monitoring script
#
# Checking urgency in changelog 
# for updates which require system restart
#
##################################
# Contact:
#  anton.lugovoi@yandex.ru
##################################
# ChangeLog:
#  20151205    initial creation
#  20151208    check uniq packages only 
##################################

case "$1" in

status)
    if [ -f /var/run/reboot-required ]; then
      echo 1
    else
      echo 0
    fi 
    ;;

urgency)
    if [ -f /var/run/reboot-required.pkgs ]; then
      while read pkg; do
        tmp=`/usr/bin/apt-get changelog $pkg | \
             /bin/grep -m1 -ioP '(?<=[Uu]rgency[=:])(low|medium|high|emergency|critical)' | \
             tr '[:upper:]' '[:lower:]'`
        if [ -n $tmp ]; then
          if   [ "$tmp" == "low" ] && \
               [ "$urgency" != "medium" ] && \
               [ "$urgency" != "high" ] && \
               [ "$urgency" != "emergency" ] && \
               [ "$urgency" != "critical" ]; then 
            urgency=low
          elif [ "$tmp" == "medium" ] && \
               [ "$urgency" != "high" ] && \
               [ "$urgency" != "emergency" ] && \
               [ "$urgency" != "critical" ]; then 
            urgency=medium
          elif [ "$tmp" == "high" ] && \
               [ "$urgency" != "emergency" ] && \
               [ "$urgency" != "critical" ]; then 
            urgency=high
          elif [ "$tmp" == "emergency" ] && \
               [ "$urgency" != "critical" ]; then 
            urgency=emergency
          elif [ "$tmp" == "critical" ]; then 
            urgency=critical
            break
          fi
        fi 
      done < <(sort -u /run/reboot-required.pkgs)
    else
      urgency=none
    fi

    case "$urgency" in
        none)      urgency=0 ;;
        low)       urgency=1 ;;
        medium)    urgency=2 ;;
        high)      urgency=3 ;;
        emergency) urgency=4 ;;
        critical)  urgency=5 ;;
        *)         urgency=42 ;;
    esac

    echo $urgency
    ;;
esac
exit 0

As a result:

• reboot_required_check.sh status returns 1 if reboot is required, 0 if isn't
• reboot_required_check.sh urgency returns highest 'urgency' level or '0' if reboot is not required

Hope it helps someone to save a time ;)

Answer 3

What I wonder about though, is how this works for webservers which should be up 99.9999etc% of the time? Do they simply not restart and risk the security being breached because security updates are not installed (which I cannot imagine)? Or do they take the downtime for granted (which I cannot imagine either)?

Big web servers are restarted when * System restart required * appears for security reasons.

But this is transparent to the user and the site is never down because big servers often run two or three servers that store exactly the same files and display the same site. The first one is the main server while the two others are secondary and are used only when the main server is down.