35

I noticed that my Python application is much slower when running it on python:2-alpine3.6 than running it without Docker on Ubuntu. I came up with two small benchmark commands and there's a huge difference visible between the two operating systems, both when I'm running them on an Ubuntu server, and when I'm using Docker for Mac.

$ BENCHMARK="import timeit; print(timeit.timeit('import json; json.dumps(list(range(10000)))', number=5000))"
$ docker run python:2-alpine3.6 python -c $BENCHMARK
7.6094589233
$ docker run python:2-slim python -c $BENCHMARK
4.3410820961
$ docker run python:3-alpine3.6 python -c $BENCHMARK
7.0276606959
$ docker run python:3-slim python -c $BENCHMARK
5.6621271420

I also tried the following 'benchmark', which doesn't use Python:

$ docker run -ti ubuntu bash
root@6b633e9197cc:/# time $(i=0; while (( i < 9999999 )); do (( i ++
)); done)

real    0m39.053s
user    0m39.050s
sys     0m0.000s
$ docker run -ti alpine sh
/ # apk add --no-cache bash > /dev/null
/ # bash
bash-4.3# time $(i=0; while (( i < 9999999 )); do (( i ++ )); done)

real    1m4.277s
user    1m4.290s
sys     0m0.000s

What could be causing this difference?

  • 1
    @Seth look again: timing starts after bash is installed, inside the launched bash shell – Underyx Jul 20 '17 at 11:55
45
+50

I've run the same benchmark as you did, using just Python 3:

$ docker run python:3-alpine3.6 python --version
Python 3.6.2
$ docker run python:3-slim python --version
Python 3.6.2

resulting in more than 2 seconds difference:

$ docker run python:3-slim python -c "$BENCHMARK"
3.6475560404360294
$ docker run python:3-alpine3.6 python -c "$BENCHMARK"
5.834922112524509

Alpine is using a different implementation of libc (base system library) from the musl project(mirror URL). There are many differences between those libraries. As a result, each library might perform better in certain use cases.

Here's an strace diff between those commands above. The output starts to differ from line 269. Of course there are different addresses in memory, but otherwise it's very similar. Most of the time is obviously spent waiting for the python command to finish.

After installing strace into both containers, we can obtain a more interesting trace (I've reduced the number of iterations in the benchmark to 10).

For example, glibc is loading libraries in the following manner (line 182):

openat(AT_FDCWD, "/usr/local/lib/python3.6", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
getdents(3, /* 205 entries */, 32768)   = 6824
getdents(3, /* 0 entries */, 32768)     = 0

The same code in musl:

open("/usr/local/lib/python3.6", O_RDONLY|O_DIRECTORY|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
getdents64(3, /* 62 entries */, 2048)   = 2040
getdents64(3, /* 61 entries */, 2048)   = 2024
getdents64(3, /* 60 entries */, 2048)   = 2032
getdents64(3, /* 22 entries */, 2048)   = 728
getdents64(3, /* 0 entries */, 2048)    = 0

I'm not saying this is the key difference, but reducing the number of I/O operations in core libraries might contribute to better performance. From the diff you can see that executing the very same Python code might lead to slightly different system calls. Probably the most important could be done in optimizing loop performance. I'm not qualified enough to judge whether the performance issue is caused by memory allocation or some other instruction.

  • glibc with 10 iterations:

    write(1, "0.032388824969530106\n", 210.032388824969530106)
    
  • musl with 10 iterations:

    write(1, "0.035214247182011604\n", 210.035214247182011604)
    

musl is slower by 0.0028254222124814987 seconds. As the difference grows with number of iterations, I'd assume the difference is in memory allocation of JSON objects.

If we reduce the benchmark to solely importing json we notice the difference is not that huge:

$ BENCHMARK="import timeit; print(timeit.timeit('import json;', number=5000))"
$ docker run python:3-slim python -c "$BENCHMARK"
0.03683806210756302
$ docker run python:3-alpine3.6 python -c "$BENCHMARK"
0.038280246779322624

Loading Python libraries looks comparable. Generating list() produces bigger difference:

$ BENCHMARK="import timeit; print(timeit.timeit('list(range(10000))', number=5000))"
$ docker run python:3-slim python -c "$BENCHMARK"
0.5666235145181417
$ docker run python:3-alpine3.6 python -c "$BENCHMARK"
0.6885563563555479

Obviously the most expensive operation is json.dumps(), which might point to differences in memory allocation between those libraries.

Looking again at the benchmark, musl is really slightly slower in memory allocation:

                          musl  | glibc
-----------------------+--------+--------+
Tiny allocation & free |  0.005 | 0.002  |
-----------------------+--------+--------+
Big allocation & free  |  0.027 | 0.016  |
-----------------------+--------+--------+

I'm not sure what is meant by "big allocation", but musl is almost 2× slower, which might become significant when you repeat such operations thousands or millions of times.

  • 12
    Just few corrections. musl is not Alpine’s own implementation of glibc. 1st musl is not a (re)implementation of glibc, but different implementation of libc per POSIX standard. 2nd musl is not Alpine’s own thing, it’s a standalone, unrelated project and musl is not used just in Alpine. – Jakub Jirutka Oct 25 '17 at 19:56
  • given that musl libc seems like a better more standards based*, not to mention newer implementation why does it seem to underperform glibc in these cases? *cf. wiki.musl-libc.org/functional-differences-from-glibc.html – Forest Oct 19 '18 at 13:41
  • Is the difference of 0.0028 seconds statistically significant? The relative deviation is only 0.0013% and you are taking 10 samples. What was the (estimated) standard deviation for those 10 runs (or even the max-min difference)? – Peter Mortensen Mar 28 at 23:30
  • @PeterMortensen For questions regarding benchmark results you should refer to the Eta Labs code: etalabs.net/libc-bench.html E.g. malloc stress test is repeated 100k times. The results could be heavily dependent on library version, GCC version and CPU used, just to name a few aspects. – Tombart Mar 29 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.