Developer's Corner
Contents
- 1 Communication between developers
- 2 Code Quality
- 3 Using CMake
- 4 Unit tests and dashboard
- 5 Eigen hacking
- 6 Mercurial workflow
- 7 Debugging tips
- 8 Profiling tips
- 9 Documentation tips
Communication between developers
The eigen mailing list and IRC (#eigen on freenode) are where live discussions happen.
For reference, there are various places in this wiki where other developers expect you to update information about what you are doing.
- Todo : general TODO
- Todo for 3.0 : our roadmap for 3.0, it only contains the stuff remaining to be done.
- Status of unsupported modules : This page gives the status of the modules currently under unsupported/.
Code Quality
Nightly build reports:eigen
- Dashboards:
- For the devel branch: http://eigen.tuxfamily.org/CDash/index.php?project=Eigen
- For the stable 2.0 branch: http://eigen.tuxfamily.org/CDash/index.php?project=Eigen+2.0
- Ohloh statistics: http://www.ohloh.net/p/5393
Using CMake
We have a little CMake tutorial here.
Unit tests and dashboard
Everything about unit tests and the dashboard, is on the Tests page.
Eigen hacking
Coding rules
We are not very strict about coding rules, however the trends are:
- indentation with 2 spaces (no tabs !)
- the best is to show a typical example of a public class:
template<typename MatrixType class NameOfTheClass { typedef typename MatrixType::Scalar Scalar; public: // public functions start with lower case: Scalar nameOfTheFunction(int i) const; // public static functions start with upper case: static NameOfTheClass* Create(const MatrixType& mat); protected: // member attributes start with m_: Scalar m_attributeName; MatrixType m_matrix; }; template<typename MatrixType> typename MatrixType::Scalar NameOfTheClass<MatrixType>::nameOfTheFunction(int i) const { Scalar res = 0; if(i>0) { for(int j=0; i<m_matrix.cols(); ++j,++i) res = ei_hypot(res, m_matrix.coeff(i,j)); } else { j = 0; while (i<m_matrix.rows()) { res = ei_hypot(res, m_matrix.coeff(i,j)); ++i; ++j; } } return res; }
- global functions and class/struct reserved for internal use start with ei_, eg:
struct ei_assign_selector;
- favor explicit names rather than short ones
Mercurial workflow
Before starting, let's mention these links:
http://www.selenic.com/mercurial/wiki/UnderstandingMercurial http://www.selenic.com/mercurial/wiki/Tutorial
Basic usage
Configuring Mercurial
Before you start comitting to the Eigen respository, it's important that you have configured your identity, so that you will be credited in the hg history. To do so, edit your mercurial config file. The global config file is ~/.hgrc (in your home directory), and the clone specific config file is .hg/hgrc inside the clone directory. We recommend to use the global config file unless you really know what you are doing. You need to add a line like this:
[ui] username = John Smith <johnny@something.com>
Note that it is very important to follow this convention (RealName <email>). First, we want to know the real names and e-mail of people who contribute to our project; second, our scripts, such as the one that generates the credits section on the main page, assume this convention.
On Windows systems, the global config file is located in your home directory (%USERPROFILE%) and is called mercurial.ini.
In particular for Windows users, it is furthermore important to follow the instructions given here (http://mercurial.selenic.com/wiki/Win32TextExtension) in order to prevent comitting files with Windows line endings to the repository. In the end you should add something like
[extensions] hgext.win32text= [encode] ** = cleverencode: [decode] ** = cleverdecode: [patch] eol = crlf [hooks] pretxncommit.crlf = python:hgext.win32text.forbidcrlf
to your config file.
Getting the source
Initial checkout:
$ hg clone http://bitbucket.org/eigen/eigen
or if you have a write access to the repository you have two options:
$ hg clone ssh://hg@bitbucket.org/eigen/eigen/ # See [1] to setup ssh $ hg clone https://login@bitbucket.org/eigen/eigen/ # password login
After that, to update your checkout (get the new changes made to the repository):
$ hg in # check what's coming in $ hg pull -u # hg pull brings the changes into your store, and -u updates # your working directory to the latest revision. # 'hg pull -u' is the same as 'hg pull && hg update'.
Branches
By default, what you now see is the development branch, called the 'default branch'. This is what was called 'trunk' in SVN.
$ hg branch # show the name the current branch $ hg branches # show the list of branches $ hg up 2.0 # switch to 2.0 branch
Local commit
First of all, like with SVN, make sure you have put any new files under revision control:
$ hg add <filename> # Put a file under revision control
Then you can examine before committing:
$ hg status # see what files are affected (same as 'svn status') $ hg diff # show differences that are to be committed (same as 'svn diff')
To proceed with the local commit:
$ hg commit # take care, this will commit changes for the whole repository # even if you are in a sub-directory (this is not as SVN does) $ hg commit . # if you want to commit only the local directory
Undoing a local commit
In case you want to undo a local commit, e.g. to deal with typos in the commit log or similar, do:
$ hg rollback # roll back the last transaction
I suggest to only use this technique for local commits. Otherwise, it makes no sense, since it cannot be guaranteed that other users already pulled in public changes.
Generating a patch
Once you have made a local commit, you can generate a patch from it:
$ hg export tip > somefile
Here, 'tip' means the latest revision. Try "hg help export" to see other options. This generates a file 'somefile' that you can then attach to an e-mail to the Eigen eigen mailing list. If you have properly configured your identity as above before committing, then importing your patch will properly credit you.
Pushing the local changes to the central repository
Once you have made a local commit, you can push it to the central repository.
$ hg out # to check what you are actually pushing $ hg push # no need to give the repository if this is the one cloned
You may see this error message:
abort: push creates new remote heads! (did you forget to merge? use push -f to force)
This means that one of your modified files has been modified meanwhile in the repository. You need to merge before you can push: see next section. Do not use push -f as mentioned in that message, as there may be a conflict to resolve manually.
Merge changes from another repository
$ hg in <repo> # check what's coming in $ hg pull <repo> # pull changesets from <repo> $ hg merge # merge the new tip from <repo> into our working directory $ hg parents # see the revisions that have been merged into the working directory $ hg commit # commit the result of the merge
After 'hg merge', you may see this error message:
warning: conflicts during merge. merging <filename> failed! 0 files updated, 0 files merged, 0 files removed, x files unresolved
In that case, read the section "Resolving Conflicts" below.
Resolving conflicts
This is very similar to SVN. After a 'hg merge' has found conflicts, the files are left with special markers at the locations of conflicts, like:
<<<<<<< local Hello Sir ======= Hello Madam >>>>>>> other
You have to edit these files manually to resolve the conflict. Once a file <filename> is fixed, do:
hg resolve -m <filename> # mark the file as resolved
Once all the files are marked as resolved, you can proceed with committing.
Viewing history
The basic command here is 'hg log'. If you want to see full information about a certain revision, you can do:
$ hg log --style changelog -r 1000 # examine revision 1000
But you can also enable a couple of extensions that provide fancier views. Edit (or create) your global .hgrc file (in your home directory) and add these lines:
[extensions] hgk= hgext.graphlog =
You can now enjoy a graphical view of the commit history:
$ hgk # or try 'hg view' if that doesn't work
Or if you prefer a plain text output of the graph:
$ hg glog
Finally, if you want something more powerful, have a look at the following external programs: hgtk and hgview. Hgtk allows many things, like viewing only as specific branch, viewing only merges, etc. Do:
$ hgtk log
Hgview is simpler but still allows viewing a specific branch:
$ hgview
Pushing many local commits as a single one
In case you love abusing of local commits and finally want to record only one, then here is the recipe:
$ hg clone <central-repo> hackrepo $ cd hackrepo $ hg log | head # record the last rev id (=id0) # hack/commit many times $ cd .. $ hg clone <central-repo> main $ cd main $ (cd ../hackrepo ; hg export id0:) | hg import - -m "unified commit message" $ hg push
Backporting bugfixes
$ hg clone <central-repo> $ hg up 2.0 # switch your working copy to the 2.0 stable branch $ hg log | less # find the changeset ids you want to backport $ hg transplant --log REV
The last command automatically commit the change using the same message than the exported commit appended with the
(transplanted from REV)
line. If the merge failed, then you have to resolve the conflict and commit as usual. See also the transplant extension doc for more details.
Instead of the transplant command, you might also do:
$ hg export REV | hg import -
or even:
$ hg export REV | hg import - -m "backporting REV : short explanation"
Commit review
Unless we do not find the "optimal" solution for commit reviews here is a possibility using bitbucket.org forks.
Assuming you have some changes in your Eigen tip you want somebody to review, you may proceed as follows.
First, you locally commit your changes in your cloned working copy of Eigen
$ cd eigen # the cloned working copy $ hg commit -m "My meaningful commit message." $ cd ..
Afterwards, you need to create a fork of Eigen. Once the fork is created you can clone it
$ hg clone https://login@bitbucket.org/login/eigen/ eigen_clone
The next step is to pull the changes from the local clone of your Eigen tip into the forked version.
$ cd eigen_clone $ hg pull -u -r tip ../eigen
Finally, you can push back the potential commit to the Eigen fork via
$ hg push
Your commit is now publicly visible and the developers can review it.
Receiving commit notification
The Bitbucket web interface offers RSS and Atom feeds.
If you prefer email notification, subscribe to the eigen-commits mailing list. Instructions are the same as for the eigen mailing list, just replace "eigen" by "eigen-commits" everywhere.
Search and replace on whole directory
If you do a search-and-replace operation on a whole directory, you probably do not want it to affect Mercurial's own internal files! So you must avoid all .hg directories. You can do:
$ find . -type f | grep -v .hg | xargs sed -i 's/oldstring/newstring/g'
Make a new release
These days it's very easy, as the docs are automatically generated and uploaded by Thomas' machine, and the tarballs are auto-generated by Bitbucket.
So here's what it takes to make a new release (a new minor version):
- Announce a week in advance on the eigen mailing list the upcoming release (give a precise date) so that people can test and/or speak up if they know of an issue. Of course, in case of an emergency, no need to do that :)
- Concert with other devs, and/or use CDash, and/or test yourself, to make sure that the test suite passes at the very least for:
- latest stable GCC
- some older GCC 4.x (ok, it's hard to check them all)
- oldest GCC we're supposed to support (currently 3.3)
- latest MSVC
- Also do think to check "make install"! Remember how it spoiled the 2.0.7 release!
- Supreme refinement would be to check building the test suite against the installed Eigen.
- Update the version number (EIGEN_MINOR_VERSION, etc) in the file Eigen/src/Core/util/Macros.h.
- In the 2.0 branch, also update the version number in the root CMakeLists.txt. That doesn't apply to the default branch.
- commit that:
$ hg commit
- Make a tag:
$ hg tag 4.5.6-beta7
- recommended: check that everything is in order.
$ hg view
- push the release:
$ hg push
- For a minor version: make the full changelog. It's not just a dump of the commit messages: it must be readable and interesting for Eigen users. You can use "hg view" again, try to put the grave bug fixes first, try to group together what's related.
- Update the Main Page on the wiki, that is the announcement at the top and the Download section. Make sure that the tarball links point to the new release, and that any example of how to get the latest tag also refers to the new tag.
- Add the release notes into the ChangeLog page.
- Write a mail to the eigen mailing list, paste the changelog into it.
- Update the #eigen channel topic on IRC.
Debugging tips
Debugging under Visual Studio
The Visual Studio IDE allows to add specialized debug visualizers for C/C++ code. The way in which specific classes are displayed in the debug output is defined by the file
%VS90COMNTOOLS%\..\Packages\Debugger\autoexp.dat
(note that the environment variable may change depending on your IDE version, e.g. VS80COMMONTOOLS).
Adding the data found in this file right after the [Visualizer] section activates new visualization rules for some Eigen types. Currently, all kinds of Matrix types are covered wheras e.g. Eigen::SparseMatrix and Eigen::Transform are still missing. Any new visualizers are welcome and will be added with a note referencing the author to our current file.
Before working on the autoexp.dat file please ensure to create a backup copy!
Studying assembly output
Using GCC
We have a macro EIGEN_ASM_COMMENT (currently defined only on GCC! please port it, it is in Macros.h):
You can use assembly comments to make it much easier to find the asm code corresponding to a particular portion of your code:
EIGEN_ASM_COMMENT("begin"); Vector4f u = v + 3*w; EIGEN_ASM_COMMENT("end");
Then tell your compiler to output assembly code, for example with GCC it is the -S option, so you'd do:
g++ myprogram.cpp -O2 -S -o myprogram.s
The resulting asm code is now very easy to find, just search for "begin":
#APP # 5 "x.cpp" 1 #begin # 0 "" 2 #NO_APP movss .LC2(%rip), %xmm0 shufps $0, %xmm0, %xmm0 mulps (%rdx), %xmm0 addps (%rsi), %xmm0 movaps %xmm0, (%rdi) #APP # 7 "x.cpp" 1 #end # 0 "" 2 #NO_APP
Using Visual Studio
In a first step you need to compile your program with the correct compiler and linker switches as given below:
* C/C++ -> General -> Debug Information Format -> Program Database /Zi * C/C++ -> General -> Optimization -> Optimization -> Maximize Speed /O2 * C/C++ -> General -> Code Generation -> Basic Runtime Checks -> Default * C/C++ -> General -> Code Generation -> Enable Enhanced Instruction Set -> Streaming SIMD Extensions 2 /arch:SSE2 * Linker -> Debugging -> Generate Debug Info -> Yes /DEBUG
Optional settings are:
* C/C++ -> Output Files -> Assembler Output -> Assembly With Source Code /FAs * Linker -> Advanced -> Randomized Base Address -> Disable Image Randomization /DYNAMICBASE:NO * Linker -> Advanced -> Fixed Base Address -> Generate a relocation section /FIXED:NO
The second last entry (Radmonized Base Address) is really nice, when you actually need to debug assembly code since it guarantees that each time you start your program it will occupy the same address space. The very last option (Fixed Base Address) is essential when you are working with Intel's VTune for performance profiling.
The remaining part is to correctly setup your test-bed in order to prevent the rather intelligent compiler from converting your code into NOP's when it reallized the code is unused. One way is to return the result of your computations - you won't see the assembly of the assignment since we are explicitly preventing inlining of the code via the macro EIGEN_DONT_INLINE. Using this macro is actually one of the more important aspects in order to encapsulate the code you are interested in, since we are lacking GCC's comment capabilities. A sample test-bed could look as follows:
template <typename VectorType> struct VectorAddition { typedef VectorType ReturnType; EIGEN_DONT_INLINE static VectorType run(int) { VectorType a,b,c,d; return a+b+c+d; } }; int main() { Vector4f res = VectorAddition<Vector4f>::run(); }
Now, when you have correctly configured your project, you should be able to place a break point within the run method. When running your program by hitting F5 (Start Debugging) you should end up at your breakpoint in the run method. The final step is to hit ALT-8 and you're welcome in the wonderful world of assembly code.
Profiling tips
Using timers
In the file bench/BenchTimer.h we have a little timer that can be used for that.
Documentation tips
We use Doxygen, see the commands. See existing source code for example usage. Here are some best practices:
- Make generous use of "see also" tags: \sa
- Always use tags for parameters and return values (e.g. \param and \returns), that makes things easier to find in a glance.
- If possible, add a code snippet. All you have is to add it in doc/snippets (see existing files there). CMake will complete your snippet into working form, compile it, run it, and record its output into a .out file. Then in your doxygen comment, just put:
\include mysnippet.cpp Output: \verbinclude mysnippet.out
- Alternatively, if you want to add a full self-compilable example (so you dont want CMake to add stuff to make it compilable), put your example file in doc/examples.