Repository

The git repos we are maintaining use multiple branches with a specific scope:

  • master: for release code usable by the public

  • dev: for the code being developed (between releases)

  • n-xxxx: all temporary branches called focusing on resolution of an issue number #n with title xxxx (be it a feature ar bug)

The common master and dev branches are protected and developers must not work directly on them.

When starting a new development, you should always span a new temporary branch from the common dev branch. We strongly recommend that each new feature development or bug fix gets its own branch.

When you have checked extensively the modifications made on a feature branch, you can then merge them on the common dev branch. As a matter of fact, always check extensively your changes before merging them on any common branch. This merge should be made using a gitlab merge request.

After each block of code implementing a feature that has been checked, you must, without exception, commit and push.

Commits

Your commits should be frequent and atomic (never bundle lots of changes to multiple features inside a single commit). If you have made a lot of changes, you can add file changes to the commit interactively using git add -p.

Do not underestimate the value of a clear-cut, well formatted commit message. We recommend the following guidelines to be used when writing a commit message: https://chris.beams.io/posts/git-commit/

For the format of the commit message, we demand the following:

  • First line (header): commit short message (again see https://chris.beams.io/posts/git-commit/)

  • Body: details about the commit (if needed), written as complete sentences

  • Last line (if applicable): references to the issue(s) related to this commit (ex: See #1 #7 or Fix #5)

Tags

Tags are used (among other things) to indicates the commit number of a specific version of the repo. This is important to put in place once the repo is made available to the public so users and developers alike can track versions.

We use three numbers identifying a version: a.b.c

  • a: indicates major revisions (lot of changes perhaps with no backward compatibility)

  • b: indicates minor revisions (new feature introduced or a bunch of smaller ones)

  • c: indicates patch or bug-fix

Those version numbers should be at least used for the master branch, and can indicate releases. The tags indicating versions should be formatted: v{a}.{b}.{c} (for example: v1.0.1)

You can get the commits associated with each tag, which gives you the version number of these commits. But you can also get the version number associated with any commit in the history, by calling the following command (having checkout the commit to version):

$ git describe --tags --match "v[0-9]*"

This command will generate a version number with the following format: LAST_TAG_NAME-NB_COMMITS-COMMIT-CHECKSUM with LAST_TAG_NAME the closest tag in the commit history, COMMIT_CHECKSUM the checksum of the current commit and NB_COMMIT the number of commits separating the commit from the tag. For example in our case, with a commit checksum “gfface8d”, 3 commits after commit tagged with version “v1.2.0”, we get: v1.2.0-3-gfface8d

Changelog

A changelog file CHANGES must be maintained, detailing the changes brought about by a new revision. It is recommended you modify it along your other commits to not miss anything. Each set of changes is associated to a version number, which is associated to a tag.

We are using the following format:

...

v{a}.{b}.{c+1}
--------------

- change1
  - subchange1
  ...
- change2
...


v{a}.{b}.{c}
------------
(start of the changelog)

So each new version changes are appended above the changelog.

Between two releases, new changes are added to the top without a version number as section header (as version number still need to be determined). The addition as a version number as the top section header must be done upon releasing a new version.

Package requirements

It is mandatory that you keep an up-to-date file requirements.txt at the root of your project directory containing the package dependencies. You can update it through:

$ pip freeze > requirements.txt

In this purpose, it is best to have set a python virtual environment to not gather requirements of other projects. If you need help setting that up, see section Python environment.

Makefile

For easier usability, we provide a Makefile for developers. It defines many commands and their options, the more useful ones being:

  • make lint: check all linters + mypy

  • make lint-apply: apply isort and black formatting

  • make test: perform unit testing

  • make coverage: perform coverage tests on unit testing

  • make coverage-report: print the coverage of each source file

  • make coverage-report-html: show the coverage result in details in folder htmlcov

  • make doc: generate the documentation with sphinx

  • make notebook-test: execute all notebooks in and from doc/notebooks/

Gitlab CI/CD pipeline

This repository has an associated pipeline for continuous integration. It consists of a succession of operations that are applied on each commit and merge-request pushed on the online repository.

This pipeline runs the following jobs on every change pushed:

  • Lint: test code readability and coherence (see linters)

    • black

    • flake8

    • isort

    • mypy

  • Test

    • coverage tests (run unit tests and compute coverage)

    • coverage html report (same as abobe but outputs the report in html detailed form)

  • Build

    • doc (check errors when generating the doc)

  • Integration tests

    • py{3.x} (one job per version of python >=3.8 to run unit tests in)

  • Backward (test retrocompatibility)

    • backward-major (test latest major release unit tests on new code)

    • backward-minor (test latest minor release unit tests on new code)

    • backward-patch (test latest patch release unit tests on new code)

Note

while it is a good practice to only push commits that are guaranteed to pass those tests for every branch (by running those tests locally beforehand), they are only required to pass for the common dev and master branch.

In the case of a release (i.e a push on the master branch), the following jobs are appended:

  • Integration tests

    • install (test if the package can be installed from its source code)

    • notebooks (run all notebooks doc/notebooks/ and check they are working)

  • Build (build the distribution of the package)

  • Deploy (manually triggered: deploy new version of the package on PyPI (mcda)

Package structure

This package uses the following structure:

.
├── README.md                          # package readme in .md format
├── README.PYPI.md                     # PyPI package readme in .md format (deployed on PyPI)
├── CHANGES.txt                        # package changelog txt format
├── Dockerfile                         # development container dockerfile
├── Makefile                           # contains all helpers/utilities for developers
├── requirements.txt                   # up-to-date list of python packages and versions required (obtained with `pip freeze > requirements.txt`)
├── MANIFEST.in                        # indicates all non-source files to include when building package
├── pyproject.toml                     # Minimal configuration to build package distributed binaries
├── .gitlab-ci.yml                     # Continuous Integration script for gitlab pipeline
├── .readthedocs.yaml                  # Configuration for readthedocs integration
├── tox.ini                            # tox configuration file (used to test package vs other python versions)
├── setup.py                           # package installation setup
├── src                                # folder for python sources structured for packaging
│   .
│   └── mcda                           # package source code
│       ├── .                          # User API (externally usable features) package structure
│       └── internal                   # folder for package internal features
├── tests                              # folder containing python tests (unit or coverage)
│   .
├── doc                                # documentation folder
│   .
│   ├── notebooks                      # folder containing examples and tutorials notebooks
│   └── html                           # html documentation folder
├── scripts                            # folder with development useful scripts
│   .
├── .vscode                            # VS Code configuration (that we use as maintainers)
│   .
├── .devcontainer                      # VS Code development container configuration files
│   .
├── LICENSE                            # present package intent, license choice and authors (english)
├── LICENCE
│   ├── {complete_license_file}_EN.txt # complete license file in english
│   ├── {complete_license_file}_FR.txt # complete license file in french
│   ├── LICENCE.en                     # present package intent, license choice and authors (english)
│   └── LICENCE.fr                     # present package intent, license choice and authors (french)
│   .