Deployment of the Python package

80 6 Simulating Multi Body dynamics in Python: The PyChrono project

// NumPy typemaps that convert a function taking pointers to an array an its dimensions as arguments into a Python function returning a 3 dimensional NumPy array.

%{

%apply ( uint8_t** ARGOUTVIEW_ARRAY3, int* DIM1, int* DIM2, int* DIM3) {(uint8_t** vec, int* h, int* w, int* c)};

////

//// PixelRGBA8 Extension ////

// Function returning False if the array of data (Buf of the BufferT class) is NULL

%extend

chrono::sensor::SensorBufferT<std::shared_ptr<chrono::sensor::PixelRGBA8[]>>

{ public:

bool HasData() {

return !($self->Buffer==NULL);

} };

%extend

chrono::sensor::SensorBufferT<std::shared_ptr<chrono::sensor::PixelRGBA8[]>>

{ public:

void GetRGBA8Data(uint8_t** vec, int* h, int* w, int* c) {

*h = $self->Height;

*w = $self->Width;

*c = sizeof(PixelRGBA8)/sizeof(unsigned char);

*vec = reinterpret_cast<uint8_t*>($self->Buffer.get());

} };

6.2 The PyChrono project 81 manager (pip) can not do. This being said, pip packages can still be installed in Anaconda environments

• The creation of Anaconda Python packages, thanks to the conda-build tool.

• The distribution of Anaconda packages, through the Anaconda Cloud platform.

Anaconda also provides a GUI as alternative to the command line and a bootstrap version called Miniconda. In addition, Anaconda can be interfaced with PyCharm and Spyder, two of the most widespread Python IDE.

For these reasons we identified Anaconda packages as the right tool to deploy and distribute PyChrono.

6.2.3.1 Building the Conda Package

The building of a conda package required to call conda-build in a directory called Recipe, that contains:

• A metadata file meta.yaml where the information of the packages are written, such as package name, version, description, license, requirements for the builder and the user.

• Shell scripts (build.sh for Linux/MacOS and build.bat for Windows) that build and install the package

• An optional test file.

The Conda package has to be built for every Python version and Operative Sys-tem, meaning to distribute the package for Python 3.6, 3.7 and 3.8 and for Windows Linux and MacOS the package has to be built 9 times. In addition, we wanted the package to be built on the newest code and distributed every time a new commit is pushed, or at least any time the Python API changes. For these reasons the only vi-able way to accomplish this is through a CI/CD (Continuous Integration Continuous Deployment) tool. CI/CD services allow to build, test and deploy the latest version of the code on several different OS on a single machine or on a web service running on a server.

At the beginning the CI of Project Chrono relied on Travis CI while the CD of Conda packages was on Appveyor CI. Later, due to the desire to have a leaner CI/CD pipeline, to the increasing importance of PyChrono in the Chrono Project and to the size of PyChrono, whose building process could not fit anymore in the time limit of Appveyor, the whole CI/CD has been moved to GitLab CI.

After the building and testing phase the Conda packages are built whenever a manual action is triggered (making this a Continuous Delivery rather than a contin-uous Deployment)

In order accomplish this the builder must have:

• Anaconda (Miniconda is sufficient) installed and activate for the shell in use.

• All the Conda packages needed by the building process (conda-build, SWIG, CMake, Jinja etc)

• Conda packages dependencies of the specific package

82 6 Simulating Multi Body dynamics in Python: The PyChrono project

Fig. 6.2: The Chrono CI/CD pipeline. Pressing the Play button triggers the deploy-ment of PyChrono.

• Not-Conda packages dependencies.

Some libraries needed by the builder can not be installed through Conda (such as Irrlicht in our case) but whenever it is possible to satisfy a dependency through Ana-conda then using Conda is the best option since if specified in the metadata, Ana-conda packages needed by the users are installed automatically during the installation of the package.

Once the builder has everything it needs, we can launch the conda-build com-mand on the recipe folder. Conda-build will use the source location from the meta-data file, and then build according to the instructions in build.bat/.sh. We use CMake to generate, build and install the package. The installation is necessary for the pack-age relocation, which is the ability of Conda to install the built packpack-age in any prefix (the prefix depends on Conda install location and the virtual environment name).

6.2.3.2 Distributing and Installing a Conda Package

Anaconda Cloud is a server that hosts Conda packages and from which users can download and install packages. Developers can create an account and upload their packages to their Channel. Channels upload is protected by login but it is possible to create a token (tokens are strings that can be used as an alternative to login) to upload conda packages from the command line. CI tools often allow to encrypt these

6.2 The PyChrono project 83 tokens. Using the token the Conda package can be uploaded to a specific Anaconda Cloud channel, projectchrono in this case, thus being available for the users. The users who have Anaconda installed can install PyChrono just by activating Conda and typing

conda install -c projectchrono pychrono

Obviously this CD tool transfer a lot of work from the users to the developers, since the whole pipeline has to be set up and after that it still require some main-tenance, since new version of Python, of the building tools (such conda-build) or changes in the dependencies often lead to changes in the pre-build or build process.

This being said, distributing Conda packages makes the installation of PyChrono extremely easy and relieves the users of installing the whole versioning/genera-tion/SWIG/building pipeline, since only Anaconda is needed. PyChrono users pool had great benefits from this, having reached more than 3000 downloads at the time of writing this document.