
Developer Guide
***************


Table of Contents
^^^^^^^^^^^^^^^^^

* Hacking

  * Running a local copy of the client

  * Find issues to work on

  * Testing

    * Integration testing with the boulder CA

* Code components and layout

  * Plugin-architecture

  * Authenticators

  * Installer

  * Installer Development

    * Display

* Writing your own plugin

* Coding style

* Submitting a pull request

* Updating the documentation

* Other methods for running the client

  * Vagrant

  * Docker

* Notes on OS dependencies

  * Debian

  * FreeBSD


Hacking
=======


Running a local copy of the client
----------------------------------

Running the client in developer mode from your local tree is a little
different than running "letsencrypt-auto".  To get set up, do these
things once:

   git clone https://github.com/certbot/certbot
   cd certbot
   ./letsencrypt-auto-source/letsencrypt-auto --os-packages-only
   ./tools/venv.sh

Then in each shell where you're working on the client, do:

   source ./venv/bin/activate

After that, your shell will be using the virtual environment, and you
run the client by typing:

   certbot

Activating a shell in this way makes it easier to run unit tests with
"tox" and integration tests, as described below. To reverse this, you
can type "deactivate".  More information can be found in the
virtualenv docs.


Find issues to work on
----------------------

You can find the open issues in the github issue tracker.
Comparatively easy ones are marked Good Volunteer Task.  If you're
starting work on something, post a comment to let others know and seek
feedback on your plan where appropriate.

Once you've got a working branch, you can open a pull request.  All
changes in your pull request must have thorough unit test coverage,
pass our integration tests, and be compliant with the coding style.


Testing
-------

The following tools are there to help you:

* "tox" starts a full set of tests. Please note that it includes
  apacheconftest, which uses the system's Apache install to test
  config file parsing, so it should only be run on systems that have
  an experimental, non-production Apache2 install on them.  "tox -e
  apacheconftest" can be used to run those specific Apache conf tests.

* "tox -e py27", "tox -e py26" etc, run unit tests for specific
  Python versions.

* "tox -e cover" checks the test coverage only. Calling the
  "./tox.cover.sh" script directly (or even "./tox.cover.sh $pkg1
  $pkg2 ..." for any subpackages) might be a bit quicker, though.

* "tox -e lint" checks the style of the whole project, while "pylint
  --rcfile=.pylintrc path" will check a single file or specific
  directory only.

* For debugging, we recommend "pip install ipdb" and putting "import
  ipdb; ipdb.set_trace()" statement inside the source code.
  Alternatively, you can use Python's standard library "pdb", but you
  won't get TAB completion...


Integration testing with the boulder CA
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Generally it is sufficient to open a pull request and let Github and
Travis run integration tests for you.

However, if you prefer to run tests, you can use Vagrant, using the
Vagrantfile in Certbot's repository. To execute the tests on a Vagrant
box, the only command you are required to run is:

   ./tests/boulder-integration.sh

Otherwise, please follow the following instructions.

Mac OS X users: Run "./tests/mac-bootstrap.sh" instead of "boulder-
start.sh" to install dependencies, configure the environment, and
start boulder.

Otherwise, install Go 1.5, "libtool-ltdl", "mariadb-server" and
"rabbitmq-server" and then start Boulder, an ACME CA server.

If you can't get packages of Go 1.5 for your Linux system, you can
execute the following commands to install it:

   wget https://storage.googleapis.com/golang/go1.5.3.linux-amd64.tar.gz -P /tmp/
   sudo tar -C /usr/local -xzf /tmp/go1.5.3.linux-amd64.tar.gz
   if ! grep -Fxq "export GOROOT=/usr/local/go" ~/.profile ; then echo "export GOROOT=/usr/local/go" >> ~/.profile; fi
   if ! grep -Fxq "export PATH=\\$GOROOT/bin:\\$PATH" ~/.profile ; then echo "export PATH=\\$GOROOT/bin:\\$PATH" >> ~/.profile; fi

These commands download Go 1.5.3 to "/tmp/", extracts to "/usr/local",
and then adds the export lines required to execute "boulder-start.sh"
to "~/.profile" if they were not previously added

Make sure you execute the following command after Go finishes
installing:

   if ! grep -Fxq "export GOPATH=\\$HOME/go" ~/.profile ; then echo "export GOPATH=\\$HOME/go" >> ~/.profile; fi

Afterwards, you'd be able to start Boulder using the following
command:

   ./tests/boulder-start.sh

The script will download, compile and run the executable; please be
patient - it will take some time... Once its ready, you will see
"Server running, listening on 127.0.0.1:4000...". Add "/etc/hosts"
entries pointing "le.wtf", "le1.wtf", "le2.wtf", "le3.wtf" and
"nginx.wtf" to 127.0.0.1.  You may now run (in a separate terminal):

   ./tests/boulder-integration.sh && echo OK || echo FAIL

If you would like to test "certbot_nginx" plugin (highly encouraged)
make sure to install prerequisites as listed in "certbot-nginx/tests
/boulder-integration.sh" and rerun the integration tests suite.


Code components and layout
==========================

acme
   contains all protocol specific code

certbot
   all client code


Plugin-architecture
-------------------

Certbot has a plugin architecture to facilitate support for different
webservers, other TLS servers, and operating systems. The interfaces
available for plugins to implement are defined in interfaces.py and
plugins/common.py.

The most common kind of plugin is a "Configurator", which is likely to
implement the "IAuthenticator" and "IInstaller" interfaces (though
some Configurators may implement just one of those).

There are also "IDisplay" plugins, which implement bindings to
alternative UI libraries.


Authenticators
--------------

Authenticators are plugins designed to prove that this client deserves
a certificate for some domain name by solving challenges received from
the ACME server. From the protocol, there are essentially two
different types of challenges. Challenges that must be solved by
individual plugins in order to satisfy domain validation (subclasses
of "DVChallenge", i.e. "TLSSNI01", "HTTP01", "DNS") and continuity
specific challenges (subclasses of "ContinuityChallenge", i.e.
"RecoveryToken", "RecoveryContact", "ProofOfPossession"). Continuity
challenges are always handled by the "ContinuityAuthenticator", while
plugins are expected to handle "DVChallenge" types. Right now, we have
two authenticator plugins, the "ApacheConfigurator" and the
"StandaloneAuthenticator". The Standalone and Apache authenticators
only solve the "TLSSNI01" challenge currently. (You can set which
challenges your authenticator can handle through the
"get_chall_pref()".

(FYI: We also have a partial implementation for a "DNSAuthenticator"
in a separate branch).


Installer
---------

Installers plugins exist to actually setup the certificate in a
server, possibly tweak the security configuration to make it more
correct and secure (Fix some mixed content problems, turn on HSTS,
redirect to HTTPS, etc). Installer plugins tell the main client about
their abilities to do the latter via the "supported_enhancements()"
call. We currently have two Installers in the tree, the
"ApacheConfigurator". and the "NginxConfigurator".  External projects
have made some progress toward support for IIS, Icecast and Plesk.

Installers and Authenticators will oftentimes be the same class/object
(because for instance both tasks can be performed by a webserver like
nginx) though this is not always the case (the standalone plugin is an
authenticator that listens on port 443, but it cannot install certs; a
postfix plugin would be an installer but not an authenticator).

Installers and Authenticators are kept separate because it should be
possible to use the "StandaloneAuthenticator" (it sets up its own
Python server to perform challenges) with a program that cannot solve
challenges itself (Such as MTA installers).


Installer Development
---------------------

There are a few existing classes that may be beneficial while
developing a new "IInstaller". Installers aimed to reconfigure UNIX
servers may use Augeas for configuration parsing and can inherit from
"AugeasConfigurator" class to handle much of the interface. Installers
that are unable to use Augeas may still find the "Reverter" class
helpful in handling configuration checkpoints and rollback.


Display
~~~~~~~

We currently offer a pythondialog and "text" mode for displays.
Display plugins implement the "IDisplay" interface.


Writing your own plugin
=======================

Certbot client supports dynamic discovery of plugins through the
setuptools entry points. This way you can, for example, create a
custom implementation of "IAuthenticator" or the "IInstaller" without
having to merge it with the core upstream source code. An example is
provided in "examples/plugins/" directory.

Warning: Please be aware though that as this client is still in a
  developer- preview stage, the API may undergo a few changes. If you
  believe the plugin will be beneficial to the community, please
  consider submitting a pull request to the repo and we will update it
  with any necessary API changes.


Coding style
============

Please:

1. **Be consistent with the rest of the code**.

2. Read PEP 8 - Style Guide for Python Code.

3. Follow the Google Python Style Guide, with the exception that we
   use Sphinx-style documentation:

      def foo(arg):
          """Short description.

          :param int arg: Some number.

          :returns: Argument
          :rtype: int

          """
          return arg

4. Remember to use "pylint".


Submitting a pull request
=========================

Steps:

1. Write your code!

2. Make sure your environment is set up properly and that you're in
   your virtualenv. You can do this by running "./tools/venv.sh".
   (this is a **very important** step)

3. Run "./pep8.travis.sh" to do a cursory check of your code style.
   Fix any errors.

4. Run "tox -e lint" to check for pylint errors. Fix any errors.

5. Run "tox" to run the entire test suite including coverage. Fix
   any errors.

6. If your code touches communication with an ACME server/Boulder,
   you should run the integration tests, see integration. See Known
   Issues for some common failures that have nothing to do with your
   code.

7. Submit the PR.

8. Did your tests pass on Travis? If they didn't, it might not be
   your fault! See Known Issues. If it's not a known issue, fix any
   errors.


Updating the documentation
==========================

In order to generate the Sphinx documentation, run the following
commands:

   make -C docs clean html man

This should generate documentation in the "docs/_build/html"
directory.


Other methods for running the client
====================================


Vagrant
-------

If you are a Vagrant user, Certbot comes with a Vagrantfile that
automates setting up a development environment in an Ubuntu 14.04 LTS
VM. To set it up, simply run "vagrant up". The repository is synced to
"/vagrant", so you can get started with:

   vagrant ssh
   cd /vagrant
   sudo ./venv/bin/certbot

Support for other Linux distributions coming soon.

Note: Unfortunately, Python distutils and, by extension, setup.py
  and tox, use hard linking quite extensively. Hard linking is not
  supported by the default sync filesystem in Vagrant. As a result,
  all actions with these commands are *significantly slower* in
  Vagrant. One potential fix is to use NFS (related issue).


Docker
------

OSX users will probably find it easiest to set up a Docker container
for development. Certbot comes with a Dockerfile ("Dockerfile-dev")
for doing so. To use Docker on OSX, install and setup docker-machine
using the instructions at https://docs.docker.com/installation/mac/.

To build the development Docker image:

   docker build -t certbot -f Dockerfile-dev .

Now run tests inside the Docker image:

   docker run -it certbot bash
   cd src
   tox -e py27


Notes on OS dependencies
========================

OS-level dependencies can be installed like so:

   letsencrypt-auto-source/letsencrypt-auto --os-packages-only

In general...

* "sudo" is required as a suggested way of running privileged
  process

* Python 2.6/2.7 is required

* Augeas is required for the Python bindings

* "virtualenv" and "pip" are used for managing other python library
  dependencies


Debian
------

For squeeze you will need to:

* Use "virtualenv --no-site-packages -p python" instead of "-p
  python2".


FreeBSD
-------

Package installation for FreeBSD uses "pkg", not ports.

FreeBSD by default uses "tcsh". In order to activate virtualenv (see
below), you will need a compatible shell, e.g. "pkg install bash &&
bash".
