ComputeOps : calcul intensif en conteneurs
The "DevOps" trend is leading more developers to deliver their code accompanied by a container image, or a file allowing such a container to be rebuilt (lightweight virtual machines). Thus, the application can be deployed much more easily, and in an execution context similar to that of the developer, without requiring the intervention of an administrator to install this or that application library. Docker is the emblematic tool of this movement.
This concept is now spreading at high speed in the world of the cloud and supercomputers; in the latter case, rather around the Singularity tool. In order to keep the possibility of using these computing resources, and to benefit from the flexibility provided by containers, it is becoming vital to prepare the use of these containers in our disciplines and in our computing grids.
Objectives
- Compare the different container technologies (docker, rocket, lxd, udocker, singularity, shifter): effect on performance, vectoring, access to compute accelerators, security, ease of administration and use. Is there a decisive advantage to disdaining the dominant tool (Docker) and turning to the "HPC" alternatives (Singularity, Shifter,...)?
- Study the interoperability of technologies. In particular, images and image reconstruction files. For example, could a user develop on his workstation in Docker, then deploy on the computing centre in Singularity?
- Validate the compatibility of containers with the grid.
- Prototype a job submission tool that can deploy a container application on a personal machine, the grid, clouds, supercomputers, etc., i.e. allow a physicist, via a single interface, to run his application on his computer or on any computing resource at his disposal.
- When running containers in production on a cluster, it is necessary to use a container orchestrator that distributes tasks over the available resources and on demand. Several solutions exist (Docker Swarm, Kubernetes, Mesos,...), so it is necessary to follow and compare the technologies.
Project leads
- Since 2019 : Martin Souchal
- 2017-2019 : Cécile Cavet
