Difference between revisions of "Developer's Corner"

From Eigen
Jump to: navigation, search
(Generating a patch)
(Configuring Mercurial)
Line 86: Line 86:
 
  username = John Smith <johnny@something.com>
 
  username = John Smith <johnny@something.com>
  
Note that it is '''very important''' to follow this convention (RealName <email>). First, we want to know the real names and e-mail of people who contribute to our project; second, our scripts, such as the one that generates the credits section on the main page, assume this convention.
+
It's important to provide your name and a valid email address. We need to be able to contact you in case of a problem.
  
 
On Windows systems, the global config file is located in your home directory (%USERPROFILE%) and is called mercurial.ini.
 
On Windows systems, the global config file is located in your home directory (%USERPROFILE%) and is called mercurial.ini.

Revision as of 04:13, 3 June 2010

Communication between developers

The eigen mailing list and IRC (#eigen on freenode) are where live discussions happen.

For reference, there are various places in this wiki where other developers expect you to update information about what you are doing.

Code Quality

Nightly build reports:eigen

Using CMake

We have a little CMake tutorial here.

Unit tests and dashboard

Everything about unit tests and the dashboard, is on the Tests page.

Eigen hacking

Coding rules

We are not very strict about coding rules, however the trends are:

  • indentation with 2 spaces (no tabs !)
  • the best is to show a typical example of a public class:
 template<typename MatrixType
 class NameOfTheClass
 {
     typedef typename MatrixType::Scalar Scalar;
   public:
     // public functions start with lower case:
     Scalar nameOfTheFunction(int i) const;
     // public static functions start with upper case:
     static NameOfTheClass* Create(const MatrixType& mat);
   protected:
     // member attributes start with m_:
     Scalar m_attributeName;
     MatrixType m_matrix;
 };
 
 template<typename MatrixType> typename MatrixType::Scalar
 NameOfTheClass<MatrixType>::nameOfTheFunction(int i) const
 {
   Scalar res = 0;
   if(i>0)
   {
     for(int j=0; i<m_matrix.cols(); ++j,++i)
       res = ei_hypot(res, m_matrix.coeff(i,j));
   }
   else
   {
     j = 0;
     while (i<m_matrix.rows())
     {
       res = ei_hypot(res, m_matrix.coeff(i,j));
       ++i;
       ++j;
     }
   }
   return res;
 }
  • global functions and class/struct reserved for internal use start with ei_, eg:
 struct ei_assign_selector;
  • favor explicit names rather than short ones

Mercurial workflow

Before starting, let's mention these links:

 http://www.selenic.com/mercurial/wiki/UnderstandingMercurial
 http://www.selenic.com/mercurial/wiki/Tutorial

Basic usage

Configuring Mercurial

Before you start comitting to the Eigen respository, it's important that you have configured your identity, so that you will be credited in the hg history. To do so, edit your mercurial config file. The global config file is ~/.hgrc (in your home directory), and the clone specific config file is .hg/hgrc inside the clone directory. We recommend to use the global config file unless you really know what you are doing. You need to add a line like this:

[ui]
username = John Smith <johnny@something.com>

It's important to provide your name and a valid email address. We need to be able to contact you in case of a problem.

On Windows systems, the global config file is located in your home directory (%USERPROFILE%) and is called mercurial.ini.

In particular for Windows users, it is furthermore important to follow the instructions given here (http://mercurial.selenic.com/wiki/Win32TextExtension) in order to prevent comitting files with Windows line endings to the repository. In the end you should add something like

[extensions]
hgext.win32text=
[encode]
** = cleverencode:
[decode]
** = cleverdecode:
[patch]
eol = crlf
[hooks]
pretxncommit.crlf = python:hgext.win32text.forbidcrlf

to your config file.

Getting the source

Initial checkout:

 $ hg clone http://bitbucket.org/eigen/eigen

or if you have a write access to the repository you have two options:

 $ hg clone ssh://hg@bitbucket.org/eigen/eigen/        # See [1] to setup ssh
 $ hg clone https://login@bitbucket.org/eigen/eigen/   # password login


After that, to update your checkout (get the new changes made to the repository):

 $ hg in            # check what's coming in
 $ hg pull -u       # hg pull brings the changes into your store, and -u updates
                    # your working directory to the latest revision.
                    # 'hg pull -u' is the same as 'hg pull && hg update'.

Branches

By default, what you now see is the development branch, called the 'default branch'. This is what was called 'trunk' in SVN.

 $ hg branch        # show the name the current branch
 $ hg branches      # show the list of branches
 $ hg up 2.0        # switch to 2.0 branch

Local commit

First of all, like with SVN, make sure you have put any new files under revision control:

 $ hg add <filename> # Put a file under revision control

Then you can examine before committing:

 $ hg status        # see what files are affected (same as 'svn status')
 $ hg diff          # show differences that are to be committed (same as 'svn diff')

To proceed with the local commit:

 $ hg commit        # take care, this will commit changes for the whole repository
                    # even if you are in a sub-directory (this is not as SVN does)
 $ hg commit .      # if you want to commit only the local directory

Undoing a local commit

In case you want to undo a local commit, e.g. to deal with typos in the commit log or similar, do:

 $ hg rollback      # roll back the last transaction

I suggest to only use this technique for local commits. Otherwise, it makes no sense, since it cannot be guaranteed that other users already pulled in public changes.

Generating a patch

Once you have made a local commit, you can generate a patch from it:

 $ hg export tip > somefile

Here, 'tip' means the latest revision. Try "hg help export" to see other options. This generates a file that you can then attach to an e-mail to the Eigen mailing list. If you have properly configured your identity as above before committing, then importing your patch will properly credit you.

Pushing the local changes to the central repository

Once you have made a local commit, you can push it to the central repository.

 $ hg out           # to check what you are actually pushing
 $ hg push          # no need to give the repository if this is the one cloned

You may see this error message:

 abort: push creates new remote heads!
 (did you forget to merge? use push -f to force)

This means that one of your modified files has been modified meanwhile in the repository. You need to merge before you can push: see next section. Do not use push -f as mentioned in that message, as there may be a conflict to resolve manually.

Merge changes from another repository

 $ hg in <repo>     # check what's coming in
 $ hg pull <repo>   # pull changesets from <repo>
 $ hg merge         # merge the new tip from <repo> into our working directory
 $ hg parents       # see the revisions that have been merged into the working directory
 $ hg commit        # commit the result of the merge

After 'hg merge', you may see this error message:

 warning: conflicts during merge.
 merging <filename> failed!
 0 files updated, 0 files merged, 0 files removed, x files unresolved

In that case, read the section "Resolving Conflicts" below.

Resolving conflicts

This is very similar to SVN. After a 'hg merge' has found conflicts, the files are left with special markers at the locations of conflicts, like:

 <<<<<<< local
 Hello Sir
 =======
 Hello Madam
 >>>>>>> other

You have to edit these files manually to resolve the conflict. Once a file <filename> is fixed, do:

 hg resolve -m <filename>   # mark the file as resolved

Once all the files are marked as resolved, you can proceed with committing.

Viewing history

The basic command here is 'hg log'. If you want to see full information about a certain revision, you can do:

 $ hg log --style changelog -r 1000  # examine revision 1000

But you can also enable a couple of extensions that provide fancier views. Edit (or create) your global .hgrc file (in your home directory) and add these lines:

[extensions]
hgk=
hgext.graphlog =

You can now enjoy a graphical view of the commit history:

 $ hgk   # or try 'hg view' if that doesn't work

Or if you prefer a plain text output of the graph:

 $ hg glog

Finally, if you want something more powerful, have a look at the following external programs: hgtk and hgview. Hgtk allows many things, like viewing only as specific branch, viewing only merges, etc. Do:

 $ hgtk log

Hgview is simpler but still allows viewing a specific branch:

 $ hgview

Pushing many local commits as a single one

In case you love abusing of local commits and finally want to record only one, then here is the recipe:

 $ hg clone <central-repo> hackrepo
 $ cd hackrepo
 $ hg log | head                  # record the last rev id (=id0)
 # hack/commit many times
 $ cd ..
 $ hg clone <central-repo> main
 $ cd main
 $ (cd ../hackrepo ; hg export id0:) | hg import - -m "unified commit message"
 $ hg push

Backporting bugfixes

 $ hg clone <central-repo>
 $ hg up 2.0              # switch your working copy to the 2.0 stable branch
 $ hg log | less             # find the changeset ids you want to backport
 $ hg transplant --log REV

The last command automatically commit the change using the same message than the exported commit appended with the

 (transplanted from REV)

line. If the merge failed, then you have to resolve the conflict and commit as usual. See also the transplant extension doc for more details.

Instead of the transplant command, you might also do:

 $ hg export REV | hg import -

or even:

 $ hg export REV | hg import - -m "backporting REV : short explanation"

Commit review

Unless we do not find the "optimal" solution for commit reviews here is a possibility using bitbucket.org forks.

Assuming you have some changes in your Eigen tip you want somebody to review, you may proceed as follows.

First, you locally commit your changes in your cloned working copy of Eigen

 $ cd eigen # the cloned working copy
 $ hg commit -m "My meaningful commit message."
 $ cd ..

Afterwards, you need to create a fork of Eigen. Once the fork is created you can clone it

 $ hg clone https://login@bitbucket.org/login/eigen/ eigen_clone

The next step is to pull the changes from the local clone of your Eigen tip into the forked version.

 $ cd eigen_clone
 $ hg pull -u -r tip ../eigen

Finally, you can push back the potential commit to the Eigen fork via

 $ hg push

Your commit is now publicly visible and the developers can review it.

Receiving commit notification

The Bitbucket web interface offers RSS and Atom feeds.

If you prefer email notification, subscribe to the eigen-commits mailing list. Instructions are the same as for the eigen mailing list, just replace "eigen" by "eigen-commits" everywhere.

Search and replace on whole directory

If you do a search-and-replace operation on a whole directory, you probably do not want it to affect Mercurial's own internal files! So you must avoid all .hg directories. You can do:

 $ find . -type f | grep -v \\.hg | xargs sed -i 's/oldstring/newstring/g'

Make a new release

These days it's very easy, as the docs are automatically generated and uploaded by Thomas' machine, and the tarballs are auto-generated by Bitbucket.

So here's what it takes to make a new release (a new minor version):

  • Announce a week in advance on the eigen mailing list the upcoming release (give a precise date) so that people can test and/or speak up if they know of an issue. Of course, in case of an emergency, no need to do that :)
  • Concert with other devs, and/or use CDash, and/or test yourself, to make sure that the test suite passes at the very least for:
    • latest stable GCC
    • some older GCC 4.x (ok, it's hard to check them all)
    • oldest GCC we're supposed to support (currently 3.3)
    • latest MSVC
  • Also do think to check "make install"! Remember how it spoiled the 2.0.7 release!
    • Supreme refinement would be to check building the test suite against the installed Eigen.
  • Update the version number (EIGEN_MINOR_VERSION, etc) in the file Eigen/src/Core/util/Macros.h.
  • In the 2.0 branch, also update the version number in the root CMakeLists.txt. That doesn't apply to the default branch.
  • commit that:
$ hg commit
  • Make a tag:
$ hg tag 4.5.6-beta7
  • recommended: check that everything is in order.
$ hg view
  • push the release:
$ hg push
  • For a minor version: make the full changelog. It's not just a dump of the commit messages: it must be readable and interesting for Eigen users. You can use "hg view" again, try to put the grave bug fixes first, try to group together what's related.
  • Update the Main Page on the wiki, that is the announcement at the top and the Download section. Make sure that the tarball links point to the new release, and that any example of how to get the latest tag also refers to the new tag.
  • Add the release notes into the ChangeLog page.
  • Write a mail to the eigen mailing list, paste the changelog into it.
  • Update the #eigen channel topic on IRC.

Debugging tips

Debugging under Visual Studio

"Examplary view of the enhanced debugger output."

The Visual Studio IDE allows to add specialized debug visualizers for C/C++ code. The way in which specific classes are displayed in the debug output is defined by the file

   %VS90COMNTOOLS%\..\Packages\Debugger\autoexp.dat 

(note that the environment variable may change depending on your IDE version, e.g. VS80COMMONTOOLS).

Adding the data found in this file (it is also in the Mercurial repository under debug/msvc/) right after the [Visualizer] section activates new visualization rules for some Eigen types. Currently, all kinds of Matrix types are covered wheras e.g. Eigen::SparseMatrix and Eigen::Transform are still missing. Any new visualizers are welcome and will be added with a note referencing the author to our current file.

Before working on the autoexp.dat file please ensure to create a backup copy!

Note: The file has been designed to work with the default branch. Currently, the 2.0.x versions do not work unless the constant 'Dynamic' is changed to 10000 as opposed to 33331.

Studying assembly output

Using GCC

We have a macro EIGEN_ASM_COMMENT (currently defined only on GCC! please port it, it is in Macros.h):

You can use assembly comments to make it much easier to find the asm code corresponding to a particular portion of your code:

EIGEN_ASM_COMMENT("begin");
Vector4f u = v + 3*w;
EIGEN_ASM_COMMENT("end");

Then tell your compiler to output assembly code, for example with GCC it is the -S option, so you'd do:

 g++ myprogram.cpp -O2 -S -o myprogram.s

Very important: GCC doesn't seem to optimize correctly code inside the main() function. So make sure to put your critical code in another, non-inlined, function. You can use EIGEN_DONT_INLINE to prevent inlining of a function. But actually, the easiest might be to put your code in a function foo() taking references to the Eigen objects as arguments and not putting any main() function, and then compiling with "g++ -c". So your code could look like:

#include<Eigen/Core>
using namespace Eigen;
void foo(Vector4f& u, Vector4f& v, Vector4f& w)
{
  EIGEN_ASM_COMMENT("begin");
  u = v + 3*w;
  EIGEN_ASM_COMMENT("end");
}

And you then compile it with:

 g++ myprogram.cpp -O2 -c -S -o myprogram.s

The resulting asm code is now very easy to find, just search for "begin":

 #APP
 # 5 "x.cpp" 1
 	#begin
 # 0 "" 2
 #NO_APP
 	movss	.LC2(%rip), %xmm0
 	shufps	$0, %xmm0, %xmm0
 	mulps	(%rdx), %xmm0
 	addps	(%rsi), %xmm0
 	movaps	%xmm0, (%rdi)
 #APP
 # 7 "x.cpp" 1
 	#end
 # 0 "" 2
 #NO_APP

Using Visual Studio

In a first step you need to compile your program with the correct compiler and linker switches as given below:

 * C/C++ -> General -> Debug Information Format -> Program Database /Zi
 * C/C++ -> General -> Optimization -> Optimization -> Maximize Speed /O2
 * C/C++ -> General -> Code Generation -> Basic Runtime Checks -> Default
 * C/C++ -> General -> Code Generation -> Enable Enhanced Instruction Set -> Streaming SIMD Extensions 2 /arch:SSE2
 * Linker -> Debugging -> Generate Debug Info -> Yes /DEBUG

Optional settings are:

 * C/C++ -> Output Files -> Assembler Output -> Assembly With Source Code /FAs
 * Linker -> Advanced -> Randomized Base Address -> Disable Image Randomization /DYNAMICBASE:NO
 * Linker -> Advanced -> Fixed Base Address -> Generate a relocation section /FIXED:NO

The second last entry (Radmonized Base Address) is useful when you actually need to debug assembly code since it guarantees that each time you start your program it will occupy the same address space. The very last option (Fixed Base Address) is essential when you are working with Intel's VTune for performance profiling.

The remaining part is to correctly setup your test-bed in order to prevent the rather intelligent compiler from converting your code into NOP's when it reallized the code is unused. One way is to return the result of your computations - you won't see the assembly of the assignment since we are explicitly preventing inlining of the code via the macro EIGEN_DONT_INLINE. Using this macro is actually one of the more important aspects in order to encapsulate the code you are interested in, since we are lacking GCC's comment capabilities. A sample test-bed could look as follows:

 template <typename VectorType>
 struct VectorAddition
 {
   typedef VectorType ReturnType;
   EIGEN_DONT_INLINE static VectorType run(int)
   {
     VectorType a,b,c,d;
     return a+b+c+d;
   }
 };
 int main()
 {
   Vector4f res = VectorAddition<Vector4f>::run();
 }

Now, when you have correctly configured your project, you should be able to place a break point within the run method. When running your program by hitting F5 (Start Debugging) you should end up at your breakpoint in the run method. The final step is to hit ALT-8 and you're welcome in the wonderful world of assembly code.

Profiling tips

Using timers

In the file bench/BenchTimer.h we have a little timer that can be used for that.

Documentation tips

We use Doxygen, see the commands. See existing source code for example usage. Here are some best practices:

  • Make generous use of "see also" tags: \sa
  • Always use tags for parameters and return values (e.g. \param and \returns), that makes things easier to find in a glance.
  • If possible, add a code snippet. All you have is to add it in doc/snippets (see existing files there). CMake will complete your snippet into working form, compile it, run it, and record its output into a .out file. Then in your doxygen comment, just put:
 \include mysnippet.cpp
 Output: \verbinclude mysnippet.out
  • Alternatively, if you want to add a full self-compilable example (so you dont want CMake to add stuff to make it compilable), put your example file in doc/examples.

Meetings

On separate Meetings page.