Tutorial

Important note: Package Helper is no longer maintained. We recommend you to migrate to Package Helper 2.

Package Helper helps you create, develop and maintain a package. If you use all the tools presented in this tutorial, things will work this way:

• You create the file structure of your package in less than a minute with Cookiecutter.

• You use the IDE PyCharm. It is configured to:

  • Generate the documentation of your package locally,
  • Run the unit tests.

• Your project is on GitHub. When you push modifications:

  • ReadTheDocs automatically generates the documentation and publishes it online.
  • Travis-CI automatically runs the unit tests on several versions of Python,
  • Codecov displays what parts of the package are covered by the unit tests.

• When you make a release on GitHub, Travis-CI automatically publishes your package on PyPI. This way, any Python user can install it with pip install.

• Pyup helps you keep your development requirements up to date, such as sphinx or pytest.

Tick the tools that you want to use:

• Virtualenv: virtual environment. Recommended especially if your use a third-party package that is still in a 0.x.x release (which means that its API is not considered stable yet).
• Github: collaborative version control.
  • ReadTheDocs: put a beautiful documentation of the package on the web.
  • Travis-CI: run the unit tests with different versions of Python.
    • PyPI: make the package installable with pip install.
    • Codecov: check if the unit tests cover the whole package.
  • Pyup: maintain the development requirements up to date. Recommended for large projects.

Preliminaries: Once and for All

Create Accounts on the Websites

Ensure that you have accounts (preferably with the same login) on:

• GitHub,

• ReadTheDocs,

• Travis-CI,

• PyPI,

• Codecov,

• Pyup.

Install Cookiecutter

In a terminal (e.g. Anaconda Prompt):

pip install cookiecutter

Install Git

Git will be used to download the package template that will be used by Cookiecutter.

Install git: https://git-scm.com/downloads. You may need to restart your computer.

Change the Documentation Style in PyCharm

Perform this step only if you want to use the Numpy documentation style.

In the “Welcome to PyCharm” window (before you open a project): Configure → Settings → Tools → Python Integrated Tools → Docstrings → Docstring format → NumPy.

Register Your GitHub Account in PyCharm

In PyCharm:

1. Menu File → Settings → Version Control → GitHub.

2. Click on the “+” icon.

3. Fill in the form and validate.

Register Your GitHub Account in ReadTheDocs

On ReadTheDocs' website: Paramètres → Comptes liés. Check that your GitHub account is listed here.

Install Travis Client on Your Computer

• Under Windows:

  1. Install Ruby (https://rubyinstaller.org/ ).

  2. Run PyCharm as Administrator.

  3. In PyCharm's terminal, do: gem install -V travis. If it does not work, restart your computer and try again.

• Under Debian, run as root:

  apt-get update
  apt-get install cookie-cutter ruby ruby-dev gcc
  gem install -V travis
  

• Under Ubuntu 16, run:

  sudo apt-get install ruby-dev
  sudo gem install -V travis
  

If you experience troubles installing travis, cf. https://github.com/travis-ci/travis.rb#installation.

Create Your package

This section is adapted from: https://cookiecutter-pypackage.readthedocs.io/en/latest/tutorial.html. Creating your package should take less than 5 minutes with the minimal options (cookiecutter + PyCharm), and less than 30 minutes if you use all the goodies (GitHub, ReadTheDocs, Travis, etc).

Generate Your Package with Cookiecutter

1. Your project will need a project name (e.g. My Toy Package) and a project slug (typically my_toy_package). Before starting, check that your project slug is not used in PyPI, even if you do not plan to use PyPI for the moment.

2. In a terminal (e.g. Anaconda Prompt):

   1. Go to the parent directory of where you want to put the directory of your package, e.g. D:\GitHub\.

   2. cookiecutter https://github.com/francois-durand/package_helper.git

   3. Answer the questions. Here is an example (some explanations follow):

      full_name [F. Durand]: François Durand
      email [fradurand@gmail.com]: fradurand@gmail.com
      github_username [francois-durand]: francois-durand
      project_name [Python Boilerplate]: My Toy Package
      project_slug [my_toy_package]:
      project_short_description [Python Boilerplate contains all the boilerplate
      you need to create a Python package.]: My Toy Package is a beautiful
      package.
      pypi_username [francois-durand]:
      version [0.1.0]:
      use_pytest [y]:
      use_codecov [y]:
      use_pypi_deployment_with_travis [y]:
      add_pyup_badge [n]:
      Select command_line_interface:
      1 - No command-line interface
      2 - Click
      3 - Argparse
      Choose from 1, 2, 3 (1, 2, 3) [1]:
      create_author_file [y]:
      Select open_source_license:
      1 - GNU General Public License v3
      2 - MIT license
      3 - BSD license
      4 - ISC license
      5 - Apache Software License 2.0
      6 - Not open source
      Choose from 1, 2, 3, 4, 5, 6 (1, 2, 3, 4, 5, 6) [1]:
      

Some explanations now:

• use_pytest: there are essentially three ways to do unit tests in Python: unittest (the standard solution), pytest (the recommended test package) and doctest (where tests are integrated in the docstrings). If you are new to testing, I suggest using doctest. But even so, pytest is useful to configure your tests (as we will do a bit later). For this reason, in all cases, I suggest to answer yes.

• use_codecov: you use Codecov to assess the coverage of your code by your tests.

• use_pypi_deployment_with_travis: when you make a release on GitHub, Travis CI automatically publishes your package on PyPI.

• add_pyup_badge: a Pyup badge appears in the readme of your package. I suggest to answer no, even if you use Pyup.

• Click and Argparse allow you to easily call your program with unix-style command, e.g. python my_program.py --help. You can choose either of them without harm, even if you do not use it for the moment. But personally, I answer no.

• create_author_file: I suggest to answer yes.

In PyCharm, choose "Open a project" and select the directory of your package: the whole file structure is in there!

Create a Virtual Environment

A virtual environment is essentially a Python installation dedicated to your project, with its own versions of the third-party packages. It ensures that if you reuse this project several months later, it will still work…

In PyCharm:

1. Menu File → Settings → Project → Project Interpreter. (For Apple users: PyCharm → Preferences → Project → Project Interpreter.)

2. Click on the gear-shaped icon → Add.

3. Fill in the form: New environment using Virtualenv. This directory proposed is just fine. Validate.

Install Dev Requirements

The file requirements_dev.txt contains the list of the third-party packages that you need for development, such as sphinx or pytest, but that the end-user does not need when using your package.

In PyCharm's terminal:

1. Ensure you are in the directory of your package (e.g. D:\GitHub\my_toy_package).

2. Ensure that your virtual environment is activated: there should be (venv) at the beginning of the line. If not:

   Windows: venv\Scripts\activate
   Linux:   source venv/bin/activate
   

3. pip install -r requirements_dev.txt

Linux users: if cryptography is already installed with apt-get install, you may need to uninstall it before pip-installing the dev requirements.

Install Your Package in “Development Mode”

This way, your package behaves as if it were installed, but any change you make will have effect immediately.

In PyCharm's terminal, you should still be in the directory of your package. Do:

python setup.py develop

Add a Run Configuration for the Documentation

In PyCharm:

1. In the root of your project, add a directory named build.

2. Menu Run → Edit Configurations.

3. Plus icon (top left) → Python docs → Sphinx task.

4. Give a name to the configuration, e.g. Generate docs.

5. Input: the “docs” directory of your project.

6. Output: the “build” directory of your project.

7. OK.

Run this configuration: you should have a warning “Title underline too short”.

1. Go to the mentioned file and correct the problem.

2. Run the configuration again: normally, it generates the documentation.

3. To check the result, you can open the file build/index.html.

The template contains examples of classes, such as the file MyClass1.py. You can use them as models if you are new to Sphinx documentation.

Add a Run Configuration for the Unit Tests

In PyCharm:

1. Menu Run → Edit Configurations.

2. Add a new configuration by clicking the + button → Python tests → pytest.

3. Give a name to the configuration, e.g. All tests.

4. Ignore the warning and validate.

Run this configuration: normally, it runs all the tests of the project.

If you are new to testing, you can take example from the classes provided by the template, such as MyClass1.py.

Create the GitHub Repo

In PyCharm:

1. Menu VCS → Import into version control → Share project on GitHub.

2. Fill in the form and validate, e.g.:

   New repository name: my_toy_package
   Remote name: origin
   Description: My Toy Package is a beautiful package.
   

In a browser, you can go to your GitHub account to check that everything is there.

Set up ReadTheDocs

On ReadTheDocs' website:

1. Go to “My Projects”.

2. Import a Project → Importer manuellement.

3. Fill in the form and validate, e.g.:

   my_toy_package
   https://github.com/francois-durand/my_toy_package
   Git
   

4. Admin → Advanced settings.

5. Check “Installer votre projet dans un virtualenv via setup.py install”.

6. Validate.

Set up Travis CI

1. On Travis' website:

   1. Login using your Github credentials.

   2. It may take a few minutes for Travis CI to load up a list of all your GitHub repos. If you do not see your new repo, log out and log in again.

   3. Click on your new repo.

   4. Click on “Activate repository”.

2. In PyCharm's terminal, ensure that you are in the directory of your project and:

   travis encrypt --add deploy.password "My PyPI password" --com
   

   (replace with your actual password, in quotation marks).

3. Open the file .travis.yml, which is in the root of your project (you can do so in PyCharm). Check that deploy.password.secure is encoded.

Set up Pyup

1. On Pyup's website:

   1. Click on the green Add Repo button and select the repo you created.

   2. A pop up appears. Personally, I checked the first item and unchecked the two others.

   Within a few minutes, you will probably receive a pull request in GitHub (and in your email).

2. On GitHub's website, open the pull request and:

   1. Merge pull request.

   2. Confirm merge.

   3. Delete branch.

3. In PyCharm, menu VCS → Update project. This does a git update (to get the modifications done by Pyup).

Check That Everything Is Working

1. In PyCharm, make a slight modification, e.g. in the file README.rst. Then commit/push:

   1. Menu VCS → Commit.

   2. Enter a commit message.

   3. Commit → Commit and push.

   4. Push.

2. On Travis' website:

   1. Go to Build History.

   2. The build should be a success (it may take several minutes).

3. On Codecov's website:

   1. Log in with your GitHub account.

   2. Click on your name.

   3. Select your package.

   4. Once Travis has finished building, you can navigate in your project to see what parts of the code are covered by the tests.

4. On ReadTheDocs' website:

   1. In Compilations, the doc should be transmis.

   2. Open the documentation.

   3. In the table of contents, click on the first page (e.g. My Toy Package). Depending on your initial choice of options, you should have three to five badges:

      1. PyPI: package or version not found (there will be the version number after your first release).

      2. Build: passing.

      3. Docs: passing.

      4. Codecov: with a percentage.

      5. Pyup: up-to-date.

   4. In the table of contents, click on Reference. You should see the documentation of the classes provided by the template.

If you wish, you are now ready to release your first version (cf. below).

Develop and Maintain Your Package

Release a Version

Test your package:

1. Run the tests (in PyCharm and/or in Travis).

2. Check your coverage on Codecov and add tests if necessary.

In PyCharm:

1. Check that all new classes are in the file __init__.py and in the Reference section of the documentation.

2. Generate the documentation locally in order to check that it is working.

3. If you were working on a secondary branch (which is desirable), do what you have to do: pull request to master, etc.

4. Ensure that you are now in the master branch.

5. Update the file HISTORY.rst. Stick to pure .rst syntax: never use Sphinx' specific directives such as :class:`MyClass`.

6. In PyCharm's terminal, do one of the following:

   • bumpversion patch (version x.y.z → x.y.(z+1)) when you made a backwards-compatible modification (such as a bug fix).

   • bumpversion minor (version x.y.z → x.(y+1).0) when you added a functionality.

   • bumpversion major (version x.y.z → (x+1).0.0) when you changed the API. Note: in versions 0.y.z, the API is not expected to be stable anyway.

7. Commit/push.

8. Check that the documentation compiles on ReadTheDocs.

On GitHub's website:

1. Go to “releases”.

2. Select “Draft a new release”.

3. Add a tag name (e.g. v0.1.0) and a message (e.g. First stable version).

4. Select “Publish release”.

After a few minutes, Travis CI has finished the built and it is deployed on PyPI.

If the Deployment on PyPI Fails...

Check that the readme will be correctly rendered on PyPI.

In a terminal:

python setup.py bdist
twine check dist/the_name_of_the_file.zip

where the_name_of_the_file must be replaced by the relevant file name.

Add a Module (= a File)

Typically, this is a file SubPackage\MyClass, containing the class MyClass.

1. In the file __init__.py: add the shortcut.

2. In the file reference.rst: add the auto-documentation.

Use a Third-Party Package

For example, you want to use Numpy in your module.

1. Open the file setup.py.

2. In the list requirements, add the name of the package (e.g. 'numpy').

When You Receive a Pull Request from Pyup

You use some third-party packages for the development, such as sphinx or pytest. These packages and their versions are listed in the file requirements_dev.txt of your package. Pyup informs you when a new version of these third-party packages is released: you receive a pull request in GitHub and you just have to accept it.

1. On GitHub's website:

   1. Open the pull request.

   2. If necessary, wait until Travis CI has finished the build, so that you know there is no problem.

   3. Merge pull request.

   4. Confirm merge.

   5. Delete branch.

   6. In the front page, you Pyup badge should be up-to-date. If not, this is probably just a matter of time. You can go to Pyup's website, click on the gear → reload.

2. In PyCharm, Menu VCS → Update project.