Linux in the nuclear industry

Linux in the nuclear industry

19 Mars 2018

Mehdi DOGGUY

EDF

Who am I?

I 2006 : M1 Informatique, Paris 11

I 2007 : M2 MPRI, Paris 7

I 2008 - 2011 : PhD thesis in Computer Science, Paris 7

“Determinacy and confluence in synchronous process calculi”

I 2011 - 2012 : Post-Doc, ENS Ulm Affine typing in ReactiveML

I 2008 - present : Debian developer

Started with small packages in the OCaml team. . .

I 2012 - present : HPC Expert, EDF

1 / 24

Who am I?

I 2006 : M1 Informatique, Paris 11

I 2007 : M2 MPRI, Paris 7

I 2008 - 2011 : PhD thesis in Computer Science, Paris 7

“Determinacy and confluence in synchronous process calculi”

I 2011 - 2012 : Post-Doc, ENS Ulm Affine typing in ReactiveML

I 2008 - present : Debian developer

Started with small packages in the OCaml team. . .

I 2012 - present : HPC Expert, EDF

1 / 24

Who am I?

I 2006 : M1 Informatique, Paris 11

I 2007 : M2 MPRI, Paris 7

I 2008 - 2011 : PhD thesis in Computer Science, Paris 7

“Determinacy and confluence in synchronous process calculi”

I 2011 - 2012 : Post-Doc, ENS Ulm Affine typing in ReactiveML

I 2008 - present : Debian developer

Started with small packages in the OCaml team. . .

I 2012 - present : HPC Expert, EDF

1 / 24

Who am I?

I 2006 : M1 Informatique, Paris 11

I 2007 : M2 MPRI, Paris 7

I 2008 - 2011 : PhD thesis in Computer Science, Paris 7

“Determinacy and confluence in synchronous process calculi”

I 2011 - 2012 : Post-Doc, ENS Ulm Affine typing in ReactiveML

I 2008 - present : Debian developer

Started with small packages in the OCaml team. . .

I 2012 - present : HPC Expert, EDF

1 / 24

Who am I?

I 2006 : M1 Informatique, Paris 11

I 2007 : M2 MPRI, Paris 7

I 2008 - 2011 : PhD thesis in Computer Science, Paris 7

“Determinacy and confluence in synchronous process calculi”

I 2011 - 2012 : Post-Doc, ENS Ulm Affine typing in ReactiveML

I 2008 - present : Debian developer

Started with small packages in the OCaml team. . .

I 2012 - present : HPC Expert, EDF

1 / 24

Who am I?

I 2006 : M1 Informatique, Paris 11

I 2007 : M2 MPRI, Paris 7

I 2008 - 2011 : PhD thesis in Computer Science, Paris 7

“Determinacy and confluence in synchronous process calculi”

I 2011 - 2012 : Post-Doc, ENS Ulm Affine typing in ReactiveML

I 2008 - present : Debian developer

Started with small packages in the OCaml team. . .

I 2012 - present : HPC Expert, EDF

1 / 24

Who am I?

I 2006 : M1 Informatique, Paris 11

I 2007 : M2 MPRI, Paris 7

I 2008 - 2011 : PhD thesis in Computer Science, Paris 7

“Determinacy and confluence in synchronous process calculi”

I 2011 - 2012 : Post-Doc, ENS Ulm Affine typing in ReactiveML

I 2008 - present : Debian developer

Started with small packages in the OCaml team. . .

I 2012 - present : HPC Expert, EDF

1 / 24

About EDF

I World nuclear energy leader

I Europe leader in hydro power

I Industrial operator in Asia and United States

I Some key figures

I

71.2 billion euros annual revenue

I

37.6 M clients worldwide

I

584.7 TWh produced annually

I

136 GW production capacity

I

73 nuclear reactors, 78% of production

I

154 845 employees worldwide

I Large R&D and engineering divisions

https://www.edf.fr/en/the-edf-group/who-we-are/edf-at-a-glance

2 / 24

Intro

Since 2003, EDF has been using a custom distro on scientific workstations, HPC clusters and servers for industrial R&D and engineering.

EDF decided to make its distro publicly available and turn it into an Open Source community-driven project.

Outline of this presentation:

I business context

I scientific and engineering IT needs

I our solution

2 / 24

1 Scientific computing

needs

Scientific computing needs at EDF

I R&D

I

Conception

I

Information technology

I

Renewable energies

I

Electrical networks

I

. . .

I Engineering

I

Safety studies

I

Simulations

I Energy management

I

Reduce downtime on existing reactors

I

Planning consumption and production weeks in advance

3 / 24

Scientific computing at a glance

I Modeling

I

Approximate reality with a model

I

Often need for a modeler

I Simulation

I

Execution of a numerical code computing the behavior of the model system

I

A whole area of software development

I

Need for the fastest hardware to work on large arrays of floats

I Visualization

I

Results exploration and analysis

I

Need for the best graphics hardware and displays

4 / 24

Open Source scientific tools, by EDF

I Code Saturne

I

general-purpose computational fluid dynamics software

I

https://www.code-saturne.org/

I Code Aster

I

Structures and Thermomechanics Analysis for Studies and Research

I

https://www.code-aster.org/

I Salom´ e

I

Integration platform for numerical simulation

I

http://www.salome-platform.org/

I . . .

5 / 24

Typical IT user needs

I Scientific workstation / laptop

I

Modeling and visualizing

I

Developing scientific simulation codes

I

Accessing the HPC clusters

I High performance clusters

I

Thousands of compute nodes connected using fast network

I

More or less specialized depending on applications

I Computing chains

I

Servers or small clusters

I

Regular execution of the same code

I

Coupling with other components

6 / 24

An HPC ¹ cluster

1

High Performance Computing

2 Scibian

Our solution: Scibian

HPC cluster HPC cluster Server

Workstation Workstation

Server Application

Application Application Business needs User Developper

+customizations

8 / 24

What’s Debian?

I A technical project

I

Building a successful distribution

I

With a real impact on the world

I A philosophical & political project

I

Promoting and defending Free Software

I

With an impact on many users

I A social experiment

I

Thousands of volunteer contributors all over the world

9 / 24

What’s Debian?

I A technical project

I

Building a successful distribution

I

With a real impact on the world

I A philosophical & political project

I

Promoting and defending Free Software

I

With an impact on many users

I A social experiment

I

Thousands of volunteer contributors all over the world

9 / 24

What’s Debian?

I A technical project

I

Building a successful distribution

I

With a real impact on the world

I A philosophical & political project

I

Promoting and defending Free Software

I

With an impact on many users

I A social experiment

I

Thousands of volunteer contributors all over the world

9 / 24

What’s Debian?

I A technical project

I

Building a successful distribution

I

With a real impact on the world

I A philosophical & political project

I

Promoting and defending Free Software

I

With an impact on many users

I A social experiment

I

Thousands of volunteer contributors all over the world

9 / 24

What’s Debian?

I A technical project

I

Building a successful distribution

I

With a real impact on the world

I A philosophical & political project

I

Promoting and defending Free Software

I

With an impact on many users

I A social experiment

I

Thousands of volunteer contributors all over the world

Part of the landscape of Free Software:

I 1983 – GNU

I 1991 – Linux

I 1993 – Slackware, Debian, Red Hat

I 1998 – OSI created by former DPL; definition based on DFSG

I 2004 – Ubuntu

9 / 24

Free Software without distributions

Many different:

I project hosting solutions (sourceforge, github, author’s homepage, etc.)

I build systems (autotools, CMake, Rubygems, custom-made)

I causes of problems (dependencies, conflicting libraries, etc.) 1. download foo-1.0.tar.gz

checksum mismatch, missing public key, etc.

2. ./configure

error: missing bar, baz,. . .

3. foreach (bar, baz, . . . ) go to 1 until success 4. make

error: symbol not found 5. make install

error: cp: cannot create regular file /some/weird/path

10 / 24

A Free Software distribution

Alice Bob Carol Dave

soft ware pack ages feedback/bugs

feedback/bugs

1. Integration of upstream projects (avoid conflicting versions) 2. Efficient infrastructure to distribute software to users (mirrors) 3. Unified interface for software installation, upgrade and removal 4. Intermediate, unified support layer

11 / 24

A Free Software distribution

Alice Bob Carol Dave

soft ware

pack ages feedback/bugs

feedback/bugs

1. Integration of upstream projects (avoid conflicting versions) 2. Efficient infrastructure to distribute software to users (mirrors) 3. Unified interface for software installation, upgrade and removal 4. Intermediate, unified support layer

11 / 24

A Free Software distribution

Alice Bob Carol Dave

soft ware pack ages

feedback/bugs feedback/bugs

1. Integration of upstream projects (avoid conflicting versions) 2. Efficient infrastructure to distribute software to users (mirrors) 3. Unified interface for software installation, upgrade and removal 4. Intermediate, unified support layer

11 / 24

A Free Software distribution

Alice Bob Carol Dave

soft ware pack ages feedback/bugs

feedback/bugs

1. Integration of upstream projects (avoid conflicting versions) 2. Efficient infrastructure to distribute software to users (mirrors) 3. Unified interface for software installation, upgrade and removal 4. Intermediate, unified support layer

11 / 24

A Free Software distribution

Alice Bob Carol Dave

soft ware pack ages feedback/bugs

feedback/bugs

1. Integration of upstream projects (avoid conflicting versions) 2. Efficient infrastructure to distribute software to users (mirrors) 3. Unified interface for software installation, upgrade and removal 4. Intermediate, unified support layer

11 / 24

A Free Software distribution

Alice Bob Carol Dave

soft ware pack ages feedback/bugs

feedback/bugs

1. Integration of upstream projects (avoid conflicting versions) 2. Efficient infrastructure to distribute software to users (mirrors) 3. Unified interface for software installation, upgrade and removal 4. Intermediate, unified support layer

11 / 24

A very successful project

I +50 000 packages, The largest Free Software archive

I a dozen of ports (with 3 different kernels – Linux, Hurd, FreeBSD)

I 100% Free Software (including infrastructure)

I Known for stability, robustness, expertise of package maintainers

I

Huge focus on QA tests, including for packages in the long tail

I Stable releases every 2 years (+/- few months)

12 / 24

Who is doing Debian?

I Thousands of volunteers (including approx. 1 000 Debian Developers)

I Organized in teams, focusing on:

I

Packaging tasks: Ruby, KDE, scientific, etc.

and also: porters, security, release management, blends, translation, etc.

I

Distribution infrastructure: system administration, packages archive, build daemons, mirrors, quality assurance, etc.

I

Project support & communication: press, documentation, trademark, auditors & accounting, events, videos, etc.

I Real experts of their packages, and of their tasks in general

13 / 24

A distributed and independent organization

I Developers coming from 63 countries (us: 18%, de: 17%, fr:

10%)

I Distributed infrastructure

I Myriad of loosely-connected services ; easy to add your own

I Assets held by several Trusted Organizations

I No company with a major influence on Debian

14 / 24

Why choosing Debian for Scibian?

I One OS for workstations, servers and clusters

I

Full binary compatibility

I Appropriate release cycle

I

One major version approx. every 3 years

I

Frequent updates (for critical bugs and security fixes)

I Largest scientific software offering

I

Only Ubuntu matches, by following Debian repositories

I Designed for customization

I

Custom repositories, easy deployment

I

Modular and hookable installer

I Community openness

I

Easy to get interesting changes into the distribution

I Easy to integrate applications

I

Cool packaging helpers

I

Abundant documentation

15 / 24

Why choosing Debian for Scibian?

I One OS for workstations, servers and clusters

I

Full binary compatibility

I Appropriate release cycle

I

One major version approx. every 3 years

I

Frequent updates (for critical bugs and security fixes)

I Largest scientific software offering

I

Only Ubuntu matches, by following Debian repositories

I Designed for customization

I

Custom repositories, easy deployment

I

Modular and hookable installer

I Community openness

I

Easy to get interesting changes into the distribution

I Easy to integrate applications

I

Cool packaging helpers

I

Abundant documentation

15 / 24

Why choosing Debian for Scibian?

I One OS for workstations, servers and clusters

I

Full binary compatibility

I Appropriate release cycle

I

One major version approx. every 3 years

I

Frequent updates (for critical bugs and security fixes)

I Largest scientific software offering

I

Only Ubuntu matches, by following Debian repositories

I Designed for customization

I

Custom repositories, easy deployment

I

Modular and hookable installer

I Community openness

I

Easy to get interesting changes into the distribution

I Easy to integrate applications

I

Cool packaging helpers

I

Abundant documentation

15 / 24

Why choosing Debian for Scibian?

I One OS for workstations, servers and clusters

I

Full binary compatibility

I Appropriate release cycle

I

One major version approx. every 3 years

I

Frequent updates (for critical bugs and security fixes)

I Largest scientific software offering

I

Only Ubuntu matches, by following Debian repositories

I Designed for customization

I

Custom repositories, easy deployment

I

Modular and hookable installer

I Community openness

I

Easy to get interesting changes into the distribution

I Easy to integrate applications

I

Cool packaging helpers

I

Abundant documentation

15 / 24

Why choosing Debian for Scibian?

I One OS for workstations, servers and clusters

I

Full binary compatibility

I Appropriate release cycle

I

One major version approx. every 3 years

I

Frequent updates (for critical bugs and security fixes)

I Largest scientific software offering

I

Only Ubuntu matches, by following Debian repositories

I Designed for customization

I

Custom repositories, easy deployment

I

Modular and hookable installer

I Community openness

I

Easy to get interesting changes into the distribution

I Easy to integrate applications

I

Cool packaging helpers

I

Abundant documentation

15 / 24

Why choosing Debian for Scibian?

I One OS for workstations, servers and clusters

I

Full binary compatibility

I Appropriate release cycle

I

One major version approx. every 3 years

I

Frequent updates (for critical bugs and security fixes)

I Largest scientific software offering

I

Only Ubuntu matches, by following Debian repositories

I Designed for customization

I

Custom repositories, easy deployment

I

Modular and hookable installer

I Community openness

I

Easy to get interesting changes into the distribution

I Easy to integrate applications

I

Cool packaging helpers

I

Abundant documentation

15 / 24

Our solution: Scibian

HPC cluster HPC cluster Server

Workstation Workstation

Server Application

Application Application Business needs User Developper

+customizations

16 / 24

Life cycle

Scibian 6

Scibian 7

Scibian 8

Scibian 9

2016 2014

2012 2018 2020

hardware hardware

hardware hardware

hardware business app

business app

business app

17 / 24

Why a derivative?

I 6+ year support

I Custom security support

I Workstation and HPC hardware support backports (mainly for Infiniband, OmniPath, nVidia GPUs and newer Intel

micro-architectures)

I Support of some libs removed from Debian

I Upgrade to newer major versions of some software

18 / 24

Business Applications

Packaging of business applications follows a few rules:

I No maintainer scripts and no services

I No files outside /opt/$name-$version, except:

I

a script in /usr/bin

I

manpages

I

copyright file

I

icons and a desktop file

I Pre-Depends must be empty

I No alternative dependencies and no Provides

I All files are owned by root and writable only by root

I No setuid/setgid binaries (or other means to escalate privileges)

I . . .

Each application is made available on each published version of Scibian.

19 / 24

3 Tools integrated into

Scibian

Puppet HPC

The main goal of Puppet-HPC is to provide a common generic configuration management system that can be used effortlessly across multiple HPC clusters and organizations.

The Puppet-HPC software stack notably provides:

I Many generic Puppet modules (>80) for all technical components required on a HPC cluster

I Defined data model for representing the description of an HPC cluster based on Hiera

I Tools to easily deploy and manage the configuration with high-scalability requirements

It is heavily tested on Debian and used in production on thousands of Debian machines.

20 / 24

Goals behind Puppet HPC

I The code base can be re-used and the development effort is shared.

I The same code is run on many different environments, it is therefore more tested and more reliable.

I The code can be easily tested on a small testing environment even if the data is different from the production environment.

More details available at:

I https://edf-hpc.github.io/puppet-hpc/puppet_hpc_

reference-0.1.html

I https://github.com/edf-hpc/puppet-hpc

21 / 24

Scibian HPC Installation Guide

Standardize Scibian cluster deployments by:

I Defining a general architecture for an HPC Cluster

I Describing how to install it using tools packaged in Scibian

I Showing how to configure it using our Puppet modules

Document available at:

https://scibian.github.io/scibian-hpc-install-guide/

22 / 24

4 Why should I use

Scibian?

Why should I use Scibian?

I Still use Debian (We don’t even change the kernel!)

I More relaxed update rules

I A set of integrated tools for deployment and scientific computing infrastructures

I Meet and work with other industrial users

I Well tested solution in a high demanding production environment

I Integrate specific proprietary software (that is not possible to integrate in Debian)

I . . .

23 / 24

Merci!

Useful links:

I https://twitter.com/ScibianLinux

I https://github.com/scibian/

I https://github.com/edf-hpc

I https://lists.debian.org/debian-hpc/