0

I am looking for some advice. We have a Self managed Gitlab CE with around 20 projects. I am trying to configure the CI for one Scala Sbt multi module project but the jobs are too slow; the first one is sbt -mem 4096 clean test and finishes in less than 3 minutes on my local machine but take more than 50 minutes on our runner.

My .gitlab-ci.yml file is quite simple, I have tried to improve the perfs with Coursier for the artifacts resolutions and caching the dependencies but no real gain.

Here is my config :

# https://www.scala-sbt.org/1.0/docs/Travis-CI-with-sbt.html

# Use a tag matching the Jdk_Scala_Sbt versions that we need so that we avoid 
# downloading Sbt and Scala on each build
image: hseeberger/scala-sbt:8u181_2.12.8_1.2.8

cache:
  # Cache all dependencies for all builds on this project
  key: "dependencies_in_${CI_PROJECT_NAME}"
  paths:
    - sbt_dependencies

before_script:
  - export JVM_OPTS="-Dfile.encoding=UTF8 -Xms2048M -Xmx2048M -Xss6M -XX:ReservedCodeCacheSize=512M"
  - mkdir -p sbt_dependencies/ivy

after_script:
  # Cleanup the cached directories to avoid unnecessary cache updates
  - rm -fv sbt_dependencies/ivy/.ivy2/.sbt.ivy.lock
  - find sbt_dependencies/ivy/.ivy2/cache -name "ivydata-*.properties" -print -delete
  - find sbt_dependencies/ivy/.sbt        -name "*.lock"               -print -delete

stages:
  - test

test:
  stage: test
  tags:
    # Tag the job to ensure it use the correct runner
    - sbt-1-2-8
    - scala-2-12-8
  script:
    - sbt -v --info -ivy sbt_dependencies/ivy -batch -mem 4096 clean test

This is executed by a docker runner, cache is enabled and working.

My last attempt is to add a stage to resolve the dependencies with sbt update and set them in the cache. This stage ran in less than 10 minutes when the cache exists.

Here is the status (top) of the host when the job is running :

top - 10:40:40 up 6 days, 19:08,  3 users,  load average: 2.61, 2.43, 2.39
Tasks: 146 total,   1 running, 145 sleeping,   0 stopped,   0 zombie
%Cpu(s): 22.6 us,  4.0 sy,  0.0 ni,  8.1 id, 62.3 wa,  0.0 hi,  3.0 si,  0.0 st
KiB Mem :  2036976 total,    68676 free,  1856276 used,   112024 buff/cache
KiB Swap:  2097148 total,   658060 free,  1439088 used.    29896 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                           
36510 root      20   0 7650632 1.637g   7996 S  51.5 84.2  13:50.11 java                                              
   34 root      20   0       0      0      0 S   3.0  0.0   6:47.53 kswapd0                                           
29731 root      20   0   41320   5300   1112 S   0.7  0.3  23:45.95 gitlab-runner                                     
35651 root      20   0   38152   4520    316 S   0.7  0.2   0:11.23 gitlab-runner                                     
    3 root      20   0       0      0      0 S   0.3  0.0   0:25.99 ksoftirqd/0        

Any advice on how to improve the speed ?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.