java,docker

Java 8/11 and Docker (Part 3)

Written by: Marcel Koert B.S.E.E. | Posted on: | Category:

The cores

This case is more difficult to diagnose, or better: it is more difficult to understand whether the JVM is considering CPU limits or not, since when using the " share and sharing cpu " (as Swarm probably does ) we cannot rely on the classic Runtime.getRuntime().availableProcessors()because there is a bug solved only starting from JVM 9 . In reality, this is not a trivial matter because many APIs rely on this value to size the thread pools, such as for the garbage collector or the fork-join.

This actually sent me into confusion: the response of http: // localhost: 8080 / jvm related to the cores will always be 4 (those of the host) starting the stack in Swarm mode , despite all the changes we can make, but actually we notice differences in performance by varying the limits of the cpu . The only case where it seems to have a correct value is when starting a single container with the parameter --cpuset-cpus:

docker run --name openjdk -p 8080: 8080 -d --cpuset-cpus = 0-1 cnj / openjdk-jvm-info

Unfortunately, starting a container in this way is not applicable to a production context: it is not managed by an orchestrator and does not scale . If you start a container in this way it is very likely that we don't need to set limits. In this mode, however (where the cores are explicitly assigned ) they availableProcessors are actually 2 and the parameters of the GC and JIT pools are correct: it is therefore probable that the Swarm mode refers to the concept of cpu quota and sharing , so the JVM does not is able to determine the exact cores.

The problem

As in the case of memory, the CPU limits also seem to be ignored, at least in part. What do you deduce from? Let's start from the standard situation and limit the use of a core:

1. version: '3.7'
2. services:
3.  jvm-info:
4.  image: cnj / openjdk-jvm-info
5.  ports:
6.  - "8080: 8080"
7.  environment:
8.  JAVA_OPTS:>
9.  -XX: + PrintFlagsFinal
10. -XX: + PrintGCDetails
11. deploy:
12. resources:
13. limits:
14. cpus: '1'

The contaniner takes really long to start: almost 12 seconds ! So something is happening, but from http: // localhost: 8080 / jvm there are always 4 cores. Let's see how the garbage collector thread pools were sized:

docker service logs -f jvm_jvm-info | grep ParallelGCThreads

According to the official documentation , there should have been 2 , instead they are 4 (like the host cores). Going to 1.5 cores, the boot takes 9 seconds, while at 2 cores 5 seconds: it seems that the JVM actually responds to the limits in terms of performance , but the "counted" cores are always 4 and this makes the sizing of some parameters bust . We just have to reset them manually !

The solution

This is where the second image generated by the project comes in handy! Looking for a solution on the net, in fact, you will find so much material that it is necessary to test and synthesize at some point. So starting from the post " OpenJDK and Containers " and from the slides of " Why you're going to fail run ning Java on Docker " (to name two) you can come up with a series of parameters for the JVM to be put in accordance with the cores set as limits, since in Swarm mode it does not happen by itself.

The base image fabric8 / java-centos-openjdk8-jdk seems to implement these considerations and automatically creates a set of parameters for the JVM.

By starting the stack defined by src / docker / fabric8 / stack-cpu.yml

1.  version: '3.7'
2.  
3.  services:
4.  jvm-info:
5.  image: cnj / fabric8-jvm-info
6.  ports:
7.  - "8081: 8080"
8.  environment:
9.  JAVA_OPTIONS:>
10. -XX: + PrintFlagsFinal
11. -XX: + PrintGCDetails
12. JAVA_MAX_CORE: 3
13. deploy:
14. resources:
15. limits:
16. cpus: '3'

where it is specified that the variable JAVA_MAX_CORE == resources.limits.cpus , a series of flags for JVM are generated for us starting from this value. Going to http: // localhost: 8081 / jvm , I am struck by the following " input arguments ":

1.  -XX: ParallelGCThreads = 3
2.  -XX: ConcGCThreads = 3
3.  -Djava.util.concurrent.ForkJoinPool.common.parallelism = 3
4.  -XX: CICompilerCount = 2
5.  -XX: + UseParallelGC
6.  -XX: + ExitOnOutOfMemoryError

Where -XX: ParallelGCThreads , -XX: ConcGCThreads and -Djava.util.concurrent.ForkJoinPool.common.parallelism coincide with the cores. Using -XX: + ExitOnOutOfMemoryError can be interesting, so, in the case of out-of-memory, the container stops and Docker starts a new one .

Conclusions

We have seen how JVM 8 (> = 131) can somehow adapt to the limits imposed by Docker on the Java process. The "problem" is more tied to the core, which are not calculated correctly when you limit the processor share , in percentage . Assigning specific cores instead seems to work (remember --cpuset-cpus?), But it is not supported by Docker's Swarm mode, nor by Kubernetes, so the cases of real applicability are drastically reduced. Limiting resources is an important operation to take into consideration when deploying an image in a highly competitive environment where many containers run. From experimental tests found on the net, it seems that up to 3/4 JVM per node we do not have great impacts, but for greater numbers the competition becomes a lot and above all at the CPU level we start paying for the context switch. Keeping in mind therefore that they are experimental tests that are difficult to verify, it is better to take advantage of the limits offered by Docker, remembering to set a set of parameters for the JVM congruous with the limits :

src / docker / openjdk / stack.yml

1.  version: '3.7'
2.  
3.  services:
4.  jvm-info:
5.  image: cnj / openjdk-jvm-info
6.  ports:
7.  - "8080: 8080"
8.  environment:
9.  JAVA_OPTS:>
10. -XX: + UnlockExperimentalVMOptions
11. -XX: + UseCGroupMemoryLimitForHeap
12. -XX: InitialRAMFraction = 2
13. -XX: MaxRAMFraction = 2
14. -XX: ParallelGCThreads = 3
15. -XX: ConcGCThreads = 3
16. -XX: CICompilerCount = 2
17. -Djava.util.concurrent.ForkJoinPool.common.parallelism = 3
18. -XX: + ExitOnOutOfMemoryError
19. deploy:
20. resources:
21. limits:
22. memory: 1024M
23. cpus: '3'

In the case of cores> = 2, -Djava.util.concurrent.ForkJoinPool.common.parallelism , -XX: ConcGCThreads and -XX: ParallelGCThreads will be set equal to the number of cores. If instead the cores <2 , it is better to replace the parallel GC with the serial one , removing -XX: ConcGCThreads and -XX: ParallelGCThreads in favor of -XX: + UseSerialGC .

If you are lucky enough to be able to jump to the latest version of Java (to date 11), you will not need any of these parameters because the JVM is now " cgroups aware ".

© 2019 Marcel Koert for MeloMar IT BV