IMP logo
Developer Guide

This page presents instructions on how to develop code using IMP. Developers who wish to contribute code back to IMP or distribute their code should also read the Contributing code to IMP page.

  1. Getting around
  2. Writing new functions and classes
  3. Debugging and testing your code
  4. Coding conventions
  5. Documenting your code
  6. Useful scripts
  7. Contributing code back to the repository
  8. Good programming practices
  9. Where to go next

1. Getting around

The input files in the IMP directory are structured as follows:

When IMP is built, the build directory is created and filled with the results of the build. build contains a number of subdirectories. They are

Unfortunately, various intermediate files from the build are scattered throughout the module and kernel hierarchies. This messiness is part of the reason we strongly recommend doing an out of source build.

When IMP is installed, the structure from the build directory is moved over more or less intact except that the C++ and Python libraries are put in the (different) appropriate locations.

Writing new functions and classes

The easiest way to start writing new functions and classes is to create a new module using the make-module script. This creates a new module in the modules directory, complete with example code.

We highly recommend using a revision control system such as Subversion or GIT to keep track of changes to your module.

If, instead, you choose to add code to an existing module you need to consult with the person who people who control that module. Their names can be found on the module main page.

When designing the interface for your new code, you should

You may want to read the design example for some suggestions on how to go about implementing your functionality in IMP.

Managing your own module

When there is a significant group of new functionality, a new set of authors, or code that is dependent on a new external dependency, it is probably a good idea to put that code in its own module. To create a new module, run the make_module script from the main IMP source directory, passing the name of your new module. The module name should consist of lower case characters and numbers and the name should not start with a number. In addition the name "local" is special and is reserved to modules that are internal to code for handling a particular biological system or application. eg 

./tools/make_module mymodule

The script adds a number of example classes to the new module, which can be read and deleted.

The next step is to update the information about the module stored in modules/mymodule/doc/SConscript. This includes the names of the authors and descriptions of what the module is supposed to do.

If the module makes use of external libraries, you can add code to check for the external library to modules/mymodule/SConscript. Add a check for a library called libextern.so (or libextern.dylib on a mac) which is accessed using the header extern.h, using a function call call_extern(1.0)

Each module has an auto-generated header called modulename_config.h. This header contains basic definitions needed for the module and should be included (first) in each header file in the module. In addition, there is a header module_version.h which contains the version info as preprocessor symbols. This should not be included in module headers or cpp files as doing so will force frequent recompilations.

Debugging and testing your code

Ensuring that your code is correct can be very difficult, so IMP provides a number of tools to help you out.

The first set are assert-style macros:

See Error reporting/checking page for more details. As a general guideline, any improper usage to produce at least a warning all return values should be checked by such code.

The second is logging macros such as:

Finally, each module has a set of unit tests. These are Python scripts which test the behavior of a particular piece of the module's API. The scripts are located in the modules/modulename/test directory. These tests should try, as much as possible to provide independent verification of the correctness of the C++ code. The command 

 scons test

or 

 scons modulename-test

run all modules unit tests or only those for the module modulename, respectively. Any file in that directory or a subdirectory whose name matches test_*.py is considered a test. The Python files are scanned for classes which inherit from IMP.test.TestCase. For each such class found, any method whose name starts with test_ is run.

Some tests will require input files or temporary files. Input files should be placed in a directory called input in the test directory. The test script should then call 

 self.get_input_file_name(file_name)

to get the true path to the file. Likewise, appropriate names for temporary files should be found by calling 

 self.get_tmp_file_name(file_name)

. Temporary files will be located in build/tmp. The test should remove temporary files after using them.

Coding conventions

Make sure you read the API conventions page first.

To ensure code consistency and readability, certain conventions must be adhered to when writing code for IMP. Some of these conventions are automatically checked for by source control before allowing a new commit, and can also be checked yourself in new code by running 

 scons standards

Indentation

All C++ headers and code should be indented in 'Linux' style, with 2-space indents. Do not use tabs. This is roughly the output of Artistic Style run like 

 astyle --convert-tabs --style=linux --indent=spaces=2 --unpad=paren --pad=oper

. Split lines if necessary to ensure that no line is longer than 80 characters.

Rationale: Different users have different-sized windows or terminals, and different tab settings, but everybody can read 80 column output without tabs.

All Python code should conform to the Python style guide. In essence this translates to 4-space indents, no tabs, and similar class, method and variable naming to the C++ code. You can ensure that your Python code is correctly indented by using the 

 tools/reindent.py

script, available as part of the IMP distribution.

Names in IMP

See the names section of the IMP conventions page. In addition, developers should be aware that

Rationale: This makes it easier to tell between class names and function names where this is ambiguous (particularly an issue with the Python interface). The Python guys also mandate CamelCase for their class names, so this avoids any need to rename classes between C++ and Python to ensure clean Python code. Good naming is especially important with preprocessor symbols since these have file scope and so can change the meaning of other people's code.

Passing and storing data

Display

All classes must have a show method which takes an optional std::ostream and prints information about the object (see IMP::Object::show() for an example). The helper macros, such as IMP_RESTRAINT() define such a method. In addition they must have operator<< defined. This can be easily done using the IMP_OUTPUT_OPERATOR() macro once the show method is defined. Note that operator<< writes human readable information. Add a write method if you want to provide output that can be read back in.

Errors

Classes and methods should use IMP exceptions to report errors. See IMP::Exception for a list of existing exceptions. See a list of functions to aid in error reporting and detection.

Namespaces

Use the provided IMPMODULE_BEGIN_NAMESPACE, IMPMODULE_END_NAMESPACE, IMPMODULE_BEGIN_INTERNAL_NAMESPACE and IMPMODULE_END_INTERNAL_NAMESPACE macros to put declarations in a namespace appropriate for module MODULE.

Each module has an internal namespace, module_name::internal and an internal include directory modulename/internal. Any function which is

should be declared in an internal header and placed in the internal namespace.

The functionality in such internal headers is

As a result, such functions do not need to obey all the coding conventions (but we recommend that they do).

Documenting your code

IMP is documented using Doxygen. See documenting source code with doxygen to get started. We use //! and /** ... * / blocks for documentation.

Python code should provide Python doc strings.

All headers not in internal directories are parsed through Doxygen. Any function that you do not want documented (for example, because it is not well tested), hide by surrounding with 

#ifndef IMP_DOXYGEN
void messy_poorly_thought_out_function();
#endif

We provide a number of extra Doxygen commands to aid in producing nice IMP documentation. The commands are used by writing \commandname{args} or \commandname if there are no arguments.

Scoring

Restraints take the current conformation of the particles and return a score and, if requested, add to the derivatives of each of the particles used. Evaluation can be done each of two ways

In whole model evaluation, each restraint is called one at a time and given a change to computes its score based on the current conformation of the particles and adds to each particles derivatives. That is, if $R(P_i)$ is the score of the restraint on particle conformation $i$ and $R'(P_i)$ and there are no other restraints:

Stage Score for R Particle attribute Particle derivative
before model evaluation undefined $P_0$ undefined
before restraint evaluation 0 $P_0$ 0
after restraint evaluation $R(P_0)$ $P_0$ $R'(P_0)$

Incremental evaluation is more complicated. The model first sets up for incremental evaluation by creating a shadow particle for each real particle. The "prechange" particles have all the same attributes and values as the particle from the last time step. They are accessed using IMP::Particle::get_prechange_particle().

During incremental evaluation in the model

  1. the derivatives are cleared on all the particles (but not the prechange particles)
  2. incremental restraints are evaluated. They need to make sure that the the change in the sum of the particle and prechange particle derivatives is equal to the change in derivatives and that the actual score is returned.
  3. derivatives are added to the prechange derivatives and then cleared
  4. the non-incremental restraints are then evaluated
  5. the derivatives of the prechange particles are added to the particle derivatives
  6. after scoring, all the particles are marked as clean and prechange particles are updated to reflect the current attributes of the particles

To put it another way

Stage Score for R Particle attribute Particle derivative Prechange particle attribute Prechange particle derivative
at start of first incremental model evaluation 0 $P_0$ undefined undefined undefined
at start of first restraint evaluation (non-incremental function called) 0 $P_0$ 0 undefined undefined
at end of first restraint evaluation (non-incremental function called on incremental restraint) $R(P_0)$ $P_0$ $R'(P_0)$ undefined undefined
at end of first model evaluation $R(P_0)$ $P_0$ undefined $P_0$ $R'(P_0)$
at start of second model evaluation 0 $P_1$ undefined $P_0$ $R'(P_0)$
at start of second restraint evaluation (incremental called) 0 $P_1$ 0 $P_0$ $R'(P_0)$
at end of second restraint evaluation (incremental called) $R(P_1)$ $P_1$ $R'(P_1)$ $P_0$ 0
at end of second model evaluation $R(P_1)$ $P_1$ undefined $P_1$ $R'(P_1)$

Note that the restraints is free to put anything into to particle derivative and the prechange derivative as long as the sum is $R'(P_i)$. Particularly, if the derivative has not changed, it does not have to touch either of the derivatives.

A IMP::Restraint is an incremental restraint if IMP::Restraint::get_is_incremental() returns true. For such restraints, IMP::Restraint::incremental_evaluate() is called instead of IMP::Restraint::evaluate(). IMP restraints which implement incremental evaluation tend to do it by:

Whenever a particle is changed is marked as dirty, so that returns true.

A (perhaps partial) list of classes which benefit from incremental evaluation is:

Writing Examples

Writing examples is very important part of being an IMP developer and one of the best ways to help people use your code. To write a (Python) example, create a file myexample.py in the example directory of an appropriate module, along with a file myexample.readme. The readme should provide a brief overview of what the code in the module is trying to accomplish as well as key pieces of IMP functionality that it uses.

When writing examples, one should try (as appropriate) to do the following:

  1. begin the example with import lines for the IMP modules used
  2. have parameters describing the process taking place. These include names of PDB files, the resolution to perform computations at etc.
  3. define a function create_representating which creates and returns the model with the needed particles along with a data structure so that key particles can be located. It should define nested functions as needed to encapsulate commonly used code
  4. define a function create_restraints which creates the restraints to score conformations of the representation
  5. define a function get_conformations to perform the sampling
  6. define a function analyze_conformations to perform some sort of clustering and analysis of the resulting conformations
  7. finally do the actual work of calling the create_representation and create_restraints functions and performing samping and analysis and displaying the solutions.

Obvious, not all examples need all of the above parts. See Nup84 CG for an canonical example.

The example should have enough comments that the reasoning behind each line of code is clear to someone who roughly understands how IMP in general works.

Useful Scripts

IMP provides a variety of scripts to aid the lives of developers.

Generate SConscripts

The SConscripts in a number of the modules list all of the header and cpp files which are part of the module (those of other modules automatically build this list at compile time). These lists can be generated using the make-sconscripts script. To run it to rebuild the SConscripts for the module modulename do 

 ./tools/make-sconscripts modulename

Making a module

Creating such a module is the easiest way to get started developing code for IMP. First, choose a name for the module. The name should only contain letters, numbers and underscores as it needs to be a valid file name as well as an identifier in Python and C++.

To create the module do 

 ./tools/make-module my_module

Then, if you run scons with localmodules=True, your new module will be built. The new module includes a number of examples and comments to help you add code to the module.

You can use your new module in a variety of ways:

If you feel your module is of interest to other IMP users and developers, see the contributing code to IMP section.

If you document your code, running 

 scons doc

will build documentation of all of the modules including yours. To access the documentation, open doc/html/index.html.

Contributing code back to the repository

In order to be shared with others as part of the IMP distribution, code needs to be of higher quality and more thoroughly vetted than typical research code. As a result, it may make sense to keep the code as part of a private module until you better understand what capabilities can be cleanly offered to others.

The first set of questions to answer are

You are encouraged to post to the imp-dev to find help answering these questions as it can be hard to grasp all the various pieces of functionality already in the repository.

All code contributed to IMP

The next suggestions provide more details about the above choices and how to implement them.

Submitting to a module

Small pieces of functionality or extensions to existing functionality probably should be submitted to an existing module. Please contact the authors of the appropriate module and discuss the submission and how the code will be maintained.

A list of all current modules in the IMP repository can be found in the modules list or from the modules tab at the top of this page.

As always, if in doubt, post to imp-dev.

Patches to modules for which you have write access can be submitted directly by doing: 

 svn commit -m "message describing the patch" files or directories to submit

Submitting to a module

If you have a large group of related functionality to submit, it may make sense to create a new module in svn. Please post to imp-dev to discuss your plans.

Once you have submitted code

Once you have submitted code, you should monitor the Nightly build status to make sure that your code builds on all platforms and passes the unit tests. Please fix all build problems as fast as possible.

The following sorts of changes must be announced on the imp-dev mailing list before being made

We recommend that changes be posted to the list a day or two before they are made so as to give everyone adequate time to comment.

In addition to monitoring the imp-dev list, developers who have a module or are committing patches to svn may want to subscribe to the imp-commits email list which receives notices of all changes made to the IMP SVN repository.

Cross platform compatibility

IMP is designed to run on a wide variety of platforms. To detect problems on other platforms we provide nightly test runs on the supported platforms for code that is part of the IMP SVN repository.

In order to make it more likely that your code works on all the supported platforms:

Good programming practices

The contents of this page are aimed at C++ programmers, but most apply also to Python.

General resources

Two excellent sources for general C++ coding guidelines are

IMP endeavors to follow all the of the guidelines published in those books. The Sali lab owns copies of both of these books that you are free to borrow.

IMP gotchas

Below are a suggestions prompted by bugs found in code submitted to IMP.

    #include <IMP/mymodule/mymodule_exports.h>
    #include <IMP/mymodule/MyRestraint.h>
    #include <IMP/Restraint.h>
    #include <vector>

Exporting code to Python

IMP uses SWIG to wrap code C++ code and export it to Python. Since SWIG is relatively complicated, we provide a number of helper macros and an example file (see modules/example/pyext/swig.i-in). The key bits are

Generated on Fri Feb 10 2012 23:36:20 for IMP by doxygen 1.7.5.1