57

To learn a bit of server administration I've set up a simple Ubuntu 14.04 server on which I run a personal website. I've set it to automatically install security updates, but leave out the other updates. This seems to work pretty fine. Occasionally I get a message when logging into the server (with ssh) saying:

*** System restart required ***

The times this happened I simple rebooted Ubuntu and all was fine. This is ok because it's a simple personal website. What I wonder about though, is how this works for webservers which should be up 99.9999etc% of the time? Do they simply not restart and risk the security being breached because security updates are not installed (which I cannot imagine)? Or do they take the downtime for granted (which I cannot imagine either)?

How should I handle this if this were a very important production server which I want to keep up and running? All tips are welcome!

[EDIT] I know I can do cat /var/run/reboot-required.pkgs to list the packages which cause the reboot. The command currently yields the following:

linux-image-3.13.0-36-generic
linux-base
dbus
linux-image-extra-3.13.0-36-generic
linux-base

but how do I know if the updates are little things of whether I have a serious security vulnerability if I don't do the restart?

[EDIT2] Okay, I now combined the commands I've found to be useful into one:

xargs aptitude changelog < /var/run/reboot-required.pkgs | grep urgency=high

If this doesn't output anything, there don't seem to be security issues with a high urgency.

One last question though: are low, medium, and high the only urgency possibilities, or are there any more like for example critical or extremelyimportant?

  • I don't understand the question. Websites with larger traffic simply schedule this downtime during a period of time with less traffic. How urgent it is depends on what's being updated exactly. – Ramhound Sep 23 '14 at 10:48
  • 14
    I wonder how many people came here because they saw the question in the "Hot Network Questions" list and wondered what the expletives were... *raises hand* – David Richerby Sep 23 '14 at 17:35
  • 6
    @Ramhound: Ehm, no, they transparently switch over to a secondary server for the duration of the maintenance. – Lightness Races in Orbit Sep 24 '14 at 8:27
  • 1
    Re the last question: I'm having in mind to filter out low and medium and consider all other / unknown levels urgent: | grep 'urgency=' | egrep -v '=(low|medium)' – KajMagnus May 15 '16 at 6:45
46

The is no simple answer as it depends on the updates made. If the kernel had a serious security problem then it is good to restart as soon as possible. If the kernel had only minor fixes then the restart could be postponed.

If you guarantee an availability > 99.9% then you will almost always have a clustered system where you can reboot the nodes one by one without interrupting the service.

So you reboot the first system and reatach it to the cluster. Then the second and so on. Then the service will never become unavailable.

| improve this answer | |
  • 2
    Thanks for your answer. I added a little piece to my initial question; I know I can do cat /var/run/reboot-required.pkgs to get the packages which require the reboot. But how do I know if these are only minor fixes, or whether it is a serious security vulnerability? – kramer65 Sep 23 '14 at 9:58
  • 2
    @kramer65 each package has a changelog. E.g. the changlog for the kernel can be found here. – Uwe Plonus Sep 23 '14 at 10:59
  • 2
    Alright, so then it is up to the sysadmin (i.e.: in this case myself) to determine whether those changes are important? I have far too little knowledge to determine this for the Linux kernel, let alone for all the zillion other packages. Is there no central place where I can find a determination whether the update is absolutely needed for security? – kramer65 Sep 23 '14 at 11:23
  • 8
    @kramer65 Run aptitude changelog <package>, here is an example output: paste.ubuntu.com/8410798 (This is on a Debian system, not Ubuntu, but the same will work on Ubuntu too.) – nyuszika7h Sep 23 '14 at 13:28
  • 6
    Thanks for all the help here. I finally combined all the things I've learned here into one command: xargs aptitude changelog < /var/run/reboot-required.pkgs | grep urgency=high (added it to the initial question as well) which gives some output as to which packages have highly urgent patches. After that, individual packages can of course be inspected. Thanks a million for all the answers and ideas! – kramer65 Sep 24 '14 at 9:16
3

addon for the topic solution

I perform similar check for 'reboot requirement' for zabbix monitoring system

I see 2 issue in 'Topic' solution:

  1. aptitude usually works badly in scripts. I kill a few hours but still didn't make it work with zabbix
  2. if only 1 changelog includes urgent update - your check will always show positive results

My logic is:

  1. Check last change only in changelog for every package which requires system reboot
  2. As an output show only highest priority update

Using Debian documentation I found 5 possible values for 'urgency' and also fact that it can followed by equal("=") or semicolon(":") characters. Also there're can be upper and lower case characters

So I ended up with following:

#!/bin/bash
##################################
# Zabbix monitoring script
#
# Checking urgency in changelog 
# for updates which require system restart
#
##################################
# Contact:
#  anton.lugovoi@yandex.ru
##################################
# ChangeLog:
#  20151205    initial creation
#  20151208    check uniq packages only 
##################################

case "$1" in

status)
    if [ -f /var/run/reboot-required ]; then
      echo 1
    else
      echo 0
    fi 
    ;;

urgency)
    if [ -f /var/run/reboot-required.pkgs ]; then
      while read pkg; do
        tmp=`/usr/bin/apt-get changelog $pkg | \
             /bin/grep -m1 -ioP '(?<=[Uu]rgency[=:])(low|medium|high|emergency|critical)' | \
             tr '[:upper:]' '[:lower:]'`
        if [ -n $tmp ]; then
          if   [ "$tmp" == "low" ] && \
               [ "$urgency" != "medium" ] && \
               [ "$urgency" != "high" ] && \
               [ "$urgency" != "emergency" ] && \
               [ "$urgency" != "critical" ]; then 
            urgency=low
          elif [ "$tmp" == "medium" ] && \
               [ "$urgency" != "high" ] && \
               [ "$urgency" != "emergency" ] && \
               [ "$urgency" != "critical" ]; then 
            urgency=medium
          elif [ "$tmp" == "high" ] && \
               [ "$urgency" != "emergency" ] && \
               [ "$urgency" != "critical" ]; then 
            urgency=high
          elif [ "$tmp" == "emergency" ] && \
               [ "$urgency" != "critical" ]; then 
            urgency=emergency
          elif [ "$tmp" == "critical" ]; then 
            urgency=critical
            break
          fi
        fi 
      done < <(sort -u /run/reboot-required.pkgs)
    else
      urgency=none
    fi

    case "$urgency" in
        none)      urgency=0 ;;
        low)       urgency=1 ;;
        medium)    urgency=2 ;;
        high)      urgency=3 ;;
        emergency) urgency=4 ;;
        critical)  urgency=5 ;;
        *)         urgency=42 ;;
    esac

    echo $urgency
    ;;
esac
exit 0

As a result:

  • reboot_required_check.sh status returns 1 if reboot is required, 0 if isn't
  • reboot_required_check.sh urgency returns highest 'urgency' level or '0' if reboot is not required

Hope it helps someone to save a time ;)

| improve this answer | |
0

What I wonder about though, is how this works for webservers which should be up 99.9999etc% of the time? Do they simply not restart and risk the security being breached because security updates are not installed (which I cannot imagine)? Or do they take the downtime for granted (which I cannot imagine either)?

Big web servers are restarted when * System restart required * appears for security reasons.

But this is transparent to the user and the site is never down because big servers often run two or three servers that store exactly the same files and display the same site. The first one is the main server while the two others are secondary and are used only when the main server is down.

| improve this answer | |
  • 1
    While this is theoretically correct, Big web servers run custom versions of Linux. They won't see a System restart required dialogue, they update what they need to stay secure. In most cases, many if not all of the updates can be done while the system is running (I believe it is even possible to update a Linux kernel on a running system without a reboot). – joeeey May 2 '15 at 21:36
  • Interesting. I have a server on Amazon and I often restart it because of this message... I am running Ubuntu on my server. How to customize it so I don't have to reboot it every now and then? – rom May 2 '15 at 22:04
  • I don't have any experience with Amazon servers. Big web servers are run on dedicated servers and VPS's. Because of this, the system administrator has more control over the software. Does Amazon give you root shell access to your server? – joeeey May 3 '15 at 4:05
  • Yes it's possible to have root access. – rom May 3 '15 at 8:11
  • Then updating packages manually, and then restarting the affected services, and using something like Ksplice for the kernel updates would be one way. It is worth noting that Ksplice freezes execution of a computer so it is the only program running when applying a patch, so there may still be a little bit of downtime (due to the web server process being 'frozen'). This is where the answer by @Uwe Plonus comes in. – joeeey May 3 '15 at 9:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.