Difference between revisions of "Developer's Corner"

From Eigen
Jump to: navigation, search
m (Debugging under Visual Studio)
(Mercurial workflow)
Line 76: Line 76:
 
= Mercurial workflow =
 
= Mercurial workflow =
  
Before starting, let's mention these links:
+
The contents of this section have moved to [[Mercurial]].
  http://www.selenic.com/mercurial/wiki/UnderstandingMercurial
+
  http://www.selenic.com/mercurial/wiki/Tutorial
+
 
+
== Basic usage ==
+
 
+
===Configuring Mercurial===
+
 
+
Before you start comitting to the Eigen respository, it's important that you have configured your identity, so that you will be credited in the hg history. To do so, edit your mercurial config file. The global config file is ~/.hgrc (in your home directory), and the clone specific config file is .hg/hgrc inside the clone directory. We recommend to use the global config file unless you really know what you are doing. You need to add a line like this:
+
 
+
[ui]
+
username = John Smith <johnny@something.com>
+
 
+
It's important to provide your name and a valid email address. We need to be able to contact you in case of a problem.
+
 
+
On Windows systems, the global config file is located in your home directory (%USERPROFILE%) and is called mercurial.ini.
+
 
+
In particular for Windows users, it is furthermore important to follow the instructions given here (http://mercurial.selenic.com/wiki/Win32TextExtension) in order to prevent comitting files with Windows line endings to the repository. In the end you should add something like
+
 
+
[extensions]
+
hgext.win32text=
+
[encode]
+
** = cleverencode:
+
[decode]
+
** = cleverdecode:
+
[patch]
+
eol = crlf
+
[hooks]
+
pretxncommit.crlf = python:hgext.win32text.forbidcrlf
+
 
+
to your config file.
+
 
+
===Getting the source===
+
 
+
Initial checkout:
+
 
+
  $ hg clone http://bitbucket.org/eigen/eigen
+
 
+
or if you have a write access to the repository you have two options:
+
 
+
  $ hg clone ssh://hg@bitbucket.org/eigen/eigen/        # See [http://bitbucket.org/help/using-ssh/] to setup ssh
+
  $ hg clone https://login@bitbucket.org/eigen/eigen/  # password login
+
 
+
 
+
After that, to update your checkout (get the new changes made to the repository):
+
 
+
  $ hg in            # check what's coming in
+
  $ hg pull -u      # hg pull brings the changes into your store, and -u updates
+
                    # your working directory to the latest revision.
+
                    # 'hg pull -u' is the same as 'hg pull && hg update'.
+
 
+
===Branches===
+
 
+
By default, what you now see is the development branch, called the 'default branch'. This is what was called 'trunk' in SVN.
+
 
+
  $ hg branch        # show the name the current branch
+
  $ hg branches      # show the list of branches
+
  $ hg up 2.0        # switch to 2.0 branch
+
 
+
===Local commit===
+
 
+
First of all, like with SVN, make sure you have put any new files under revision control:
+
  $ hg add <filename> # Put a file under revision control
+
 
+
Then you can examine before committing:
+
  $ hg status        # see what files are affected (same as 'svn status')
+
  $ hg diff          # show differences that are to be committed (same as 'svn diff')
+
 
+
To proceed with the local commit:
+
  $ hg commit        # take care, this will commit changes for the whole repository
+
                    # even if you are in a sub-directory (this is '''not''' as SVN does)
+
  $ hg commit .      # if you want to commit only the local directory
+
 
+
===Undoing a local commit===
+
In case you want to undo a local commit, e.g. to deal with typos in the commit log or similar, do:
+
  $ hg rollback      # roll back the last transaction
+
 
+
I suggest to '''only''' use this technique for local commits. Otherwise, it makes no sense, since it cannot be guaranteed that other users already pulled in public changes.
+
 
+
===Generating a patch===
+
 
+
Once you have made a local commit, you can generate a patch from it:
+
 
+
  $ hg export tip > somefile
+
 
+
Here, 'tip' means the latest revision. Try "hg help export" to see other options. This generates a file that you can then attach to an e-mail to the [[Main_Page#Mailing_list|Eigen mailing list]]. If you have properly configured your identity as above before committing, then importing your patch will properly credit you.
+
 
+
===Pushing the local changes to the central repository===
+
 
+
Once you have made a local commit, you can push it to the central repository.
+
  $ hg out          # to check what you are actually pushing
+
  $ hg push          # no need to give the repository if this is the one cloned
+
You may see this error message:
+
  abort: push creates new remote heads!
+
  (did you forget to merge? use push -f to force)
+
This means that one of your modified files has been modified meanwhile in the repository. You need to merge before you can push: see next section. Do '''not''' use push -f as mentioned in that message, as there may be a conflict to resolve manually.
+
 
+
===Merge changes from another repository===
+
  $ hg in <repo>    # check what's coming in
+
  $ hg pull <repo>  # pull changesets from <repo>
+
  $ hg merge        # merge the new tip from <repo> into our working directory
+
  $ hg parents      # see the revisions that have been merged into the working directory
+
  $ hg commit        # commit the result of the merge
+
After 'hg merge', you may see this error message:
+
  warning: conflicts during merge.
+
  merging <filename> failed!
+
  0 files updated, 0 files merged, 0 files removed, x files unresolved
+
In that case, read the section "Resolving Conflicts" below.
+
 
+
===Resolving conflicts===
+
This is very similar to SVN. After a 'hg merge' has found conflicts, the files are left with special markers at the locations of conflicts, like:
+
  <<<<<<< local
+
  Hello Sir
+
  =======
+
  Hello Madam
+
  >>>>>>> other
+
You have to edit these files manually to resolve the conflict. Once a file <filename> is fixed, do:
+
  hg resolve -m <filename>  # mark the file as resolved
+
Once all the files are marked as resolved, you can proceed with committing.
+
 
+
===Viewing history===
+
The basic command here is 'hg log'. If you want to see full information about a certain revision, you can do:
+
 
+
  $ hg log --style changelog -r 1000  # examine revision 1000
+
 
+
But you can also enable a couple of extensions that provide fancier views. Edit (or create) your global .hgrc file (in your home directory) and add these lines:
+
 
+
[extensions]
+
hgk=
+
hgext.graphlog =
+
 
+
You can now enjoy a graphical view of the commit history:
+
 
+
  $ hgk  # or try 'hg view' if that doesn't work
+
 
+
Or if you prefer a plain text output of the graph:
+
 
+
  $ hg glog
+
 
+
Finally, if you want something more powerful, have a look at the following external programs: hgtk and hgview. [http://bitbucket.org/tortoisehg/stable/wiki/hgtk Hgtk] allows many things, like viewing only as specific branch, viewing only merges, etc. Do:
+
 
+
  $ hgtk log
+
 
+
[http://www.logilab.org/project/name/hgview Hgview] is simpler but still allows viewing a specific branch:
+
 
+
  $ hgview
+
 
+
== Pushing many local commits as a single one ==
+
 
+
In case you love abusing of local commits and finally want to record only one, then here is the recipe:
+
  $ hg clone <central-repo> hackrepo
+
  $ cd hackrepo
+
  $ hg log | head                  # record the last rev id (=id0)
+
  # hack/commit many times
+
  $ cd ..
+
  $ hg clone <central-repo> main
+
  $ cd main
+
  $ (cd ../hackrepo ; hg export id0:) | hg import - -m "unified commit message"
+
  $ hg push
+
 
+
== Backporting bugfixes ==
+
 
+
  $ hg clone <central-repo>
+
  $ hg up 2.0              # switch your working copy to the 2.0 stable branch
+
  $ hg log | less            # find the changeset ids you want to backport
+
  $ hg transplant --log REV
+
 
+
The last command automatically commit the change using the same message than the exported commit appended with the
+
  (transplanted from REV)
+
line. If the merge failed, then you have to resolve the conflict and commit as usual. See also the [http://www.selenic.com/mercurial/wiki/TransplantExtension transplant extension] doc for more details.
+
 
+
Instead of the transplant command, you might also do:
+
  $ hg export REV | hg import -
+
or even:
+
  $ hg export REV | hg import - -m "backporting REV : short explanation"
+
 
+
== Commit review ==
+
 
+
Unless we do not find the "optimal" solution for commit reviews here is a possibility using bitbucket.org forks.
+
 
+
Assuming you have some changes in your Eigen tip you want somebody to review, you may proceed as follows.
+
 
+
First, you locally commit your changes in your cloned working copy of Eigen
+
  $ cd eigen # the cloned working copy
+
  $ hg commit -m "My meaningful commit message."
+
  $ cd ..
+
 
+
Afterwards, you need to create a fork of Eigen. Once the fork is created you can clone it
+
  $ hg clone https://login@bitbucket.org/login/eigen/ eigen_clone
+
 
+
The next step is to pull the changes from the local clone of your Eigen tip into the forked version.
+
  $ cd eigen_clone
+
  $ hg pull -u -r tip ../eigen
+
 
+
Finally, you can push back the potential commit to the Eigen fork via
+
  $ hg push
+
 
+
Your commit is now publicly visible and the developers can review it.
+
 
+
== Receiving commit notification ==
+
 
+
The Bitbucket web interface offers RSS and Atom feeds.
+
 
+
If you prefer email notification, subscribe to the eigen-commits mailing list. Instructions are the same as for the [[Main_Page#Mailing_list|eigen mailing list]], just replace "eigen" by "eigen-commits" everywhere.
+
 
+
== Search and replace on whole directory ==
+
 
+
If you do a search-and-replace operation on a whole directory, you probably do not want it to affect Mercurial's own internal files! So you must avoid all .hg directories. You can do:
+
 
+
  $ find . -type f | grep -v \\.hg | xargs sed -i 's/oldstring/newstring/g'
+
 
+
== Make a new release ==
+
 
+
These days it's very easy, as the docs are automatically generated and uploaded by Thomas' machine, and the tarballs are auto-generated by Bitbucket.
+
 
+
So here's what it takes to make a new release (a new minor version):
+
* Announce a week in advance on the [[Main_Page#Mailing_list|eigen mailing list]] the upcoming release (give a precise date) so that people can test and/or speak up if they know of an issue. Of course, in case of an emergency, no need to do that :)
+
* Concert with other devs, and/or use CDash, and/or test yourself, to make sure that the test suite passes at the very least for:
+
** latest stable GCC
+
** some older GCC 4.x (ok, it's hard to check them all)
+
** oldest GCC we're supposed to support (currently 3.3)
+
** latest MSVC
+
* Also do think to check "make install"! Remember how it spoiled the 2.0.7 release!
+
** Supreme refinement would be to check building the test suite against the installed Eigen.
+
* Update the version number (EIGEN_MINOR_VERSION, etc) in the file Eigen/src/Core/util/Macros.h.
+
* In the 2.0 branch, also update the version number in the root CMakeLists.txt. That doesn't apply to the default branch.
+
* commit that:
+
$ hg commit
+
* Make a tag:
+
$ hg tag 4.5.6-beta7
+
* recommended: check that everything is in order.
+
$ hg view
+
* push the release:
+
$ hg push
+
* For a minor version: make the full changelog. It's not just a dump of the commit messages: it must be readable and interesting for Eigen users. You can use "hg view" again, try to put the grave bug fixes first, try to group together what's related.
+
* Update the [[Main_Page|Main Page]] on the wiki, that is the announcement at the top and the [[Main_Page#Download|Download]] section. Make sure that the tarball links point to the new release, and that any example of how to get the latest tag also refers to the new tag.
+
* Add the release notes into the [[ChangeLog]] page.
+
* Write a mail to the [[Main_Page#Mailing_list|eigen mailing list]], paste the changelog into it.
+
* Update the #eigen channel topic on IRC.
+
  
 
= Debugging tips =
 
= Debugging tips =

Revision as of 00:23, 4 October 2010

Communication between developers

The eigen mailing list and IRC (#eigen on freenode) are where live discussions happen.

For reference, there are various places in this wiki where other developers expect you to update information about what you are doing.

Code Quality

Nightly build reports:eigen

Using CMake

We have a little CMake tutorial here.

Unit tests and dashboard

Everything about unit tests and the dashboard, is on the Tests page.

Eigen hacking

Coding rules

We are not very strict about coding rules, however the trends are:

  • indentation with 2 spaces (no tabs !)
  • the best is to show a typical example of a public class:
 template<typename MatrixType
 class NameOfTheClass
 {
     typedef typename MatrixType::Scalar Scalar;
   public:
     // public functions start with lower case:
     Scalar nameOfTheFunction(int i) const;
     // public static functions start with upper case:
     static NameOfTheClass* Create(const MatrixType& mat);
   protected:
     // member attributes start with m_:
     Scalar m_attributeName;
     MatrixType m_matrix;
 };
 
 template<typename MatrixType> typename MatrixType::Scalar
 NameOfTheClass<MatrixType>::nameOfTheFunction(int i) const
 {
   Scalar res = 0;
   if(i>0)
   {
     for(int j=0; i<m_matrix.cols(); ++j,++i)
       res = ei_hypot(res, m_matrix.coeff(i,j));
   }
   else
   {
     j = 0;
     while (i<m_matrix.rows())
     {
       res = ei_hypot(res, m_matrix.coeff(i,j));
       ++i;
       ++j;
     }
   }
   return res;
 }
  • global functions and class/struct reserved for internal use start with ei_, eg:
 struct ei_assign_selector;
  • favor explicit names rather than short ones

Developer Documentation

Notes explaining aspects of the architecture of Eigen 3.x are available at the Eigen3 Developer Documentation page

Mercurial workflow

The contents of this section have moved to Mercurial.

Debugging tips

Debugging under Visual Studio

"Examplary view of the enhanced debugger output."

The Visual Studio IDE allows to add specialized debug visualizers for C/C++ code. The way in which specific classes are displayed in the debug output is defined by the file

   %VS90COMNTOOLS%\..\Packages\Debugger\autoexp.dat 

(note that the environment variable may change depending on your IDE version, e.g. VS80COMMONTOOLS).

Adding the data found in this file (it is also in the Mercurial repository under debug/msvc/) right after the [Visualizer] section activates new visualization rules for some Eigen types. Currently, all kinds of Matrix types are covered wheras e.g. Eigen::SparseMatrix and Eigen::Transform are still missing. Any new visualizers are welcome and will be added with a note referencing the author to our current file.

Before working on the autoexp.dat file please ensure to create a backup copy!

Note: The file has been designed to work with the default branch. Currently, the 2.0.x versions do not work unless the constant 'Dynamic' is changed to 10000 as opposed to 33331.

Note: Since Visual Studio 2010 there is a new possibility available. You can take the file as is, copy it anywhere on your disc and create an environment variable called _vcee_autoexp which is pointing to that file. Visual Studio will read and merge it with the existing autoexp.dat.

Studying assembly output

Using GCC

We have a macro EIGEN_ASM_COMMENT (currently defined only on GCC! please port it, it is in Macros.h):

You can use assembly comments to make it much easier to find the asm code corresponding to a particular portion of your code:

EIGEN_ASM_COMMENT("begin");
Vector4f u = v + 3*w;
EIGEN_ASM_COMMENT("end");

Then tell your compiler to output assembly code, for example with GCC it is the -S option, so you'd do:

 g++ myprogram.cpp -O2 -S -o myprogram.s

Very important: GCC doesn't seem to optimize correctly code inside the main() function. So make sure to put your critical code in another, non-inlined, function. You can use EIGEN_DONT_INLINE to prevent inlining of a function. But actually, the easiest might be to put your code in a function foo() taking references to the Eigen objects as arguments and not putting any main() function, and then compiling with "g++ -c". So your code could look like:

#include<Eigen/Core>
using namespace Eigen;
void foo(Vector4f& u, Vector4f& v, Vector4f& w)
{
  EIGEN_ASM_COMMENT("begin");
  u = v + 3*w;
  EIGEN_ASM_COMMENT("end");
}

And you then compile it with:

 g++ myprogram.cpp -O2 -c -S -o myprogram.s

The resulting asm code is now very easy to find, just search for "begin":

 #APP
 # 5 "x.cpp" 1
 	#begin
 # 0 "" 2
 #NO_APP
 	movss	.LC2(%rip), %xmm0
 	shufps	$0, %xmm0, %xmm0
 	mulps	(%rdx), %xmm0
 	addps	(%rsi), %xmm0
 	movaps	%xmm0, (%rdi)
 #APP
 # 7 "x.cpp" 1
 	#end
 # 0 "" 2
 #NO_APP

Using Visual Studio

In a first step you need to compile your program with the correct compiler and linker switches as given below:

 * C/C++ -> General -> Debug Information Format -> Program Database /Zi
 * C/C++ -> General -> Optimization -> Optimization -> Maximize Speed /O2
 * C/C++ -> General -> Code Generation -> Basic Runtime Checks -> Default
 * C/C++ -> General -> Code Generation -> Enable Enhanced Instruction Set -> Streaming SIMD Extensions 2 /arch:SSE2
 * Linker -> Debugging -> Generate Debug Info -> Yes /DEBUG

Optional settings are:

 * C/C++ -> Output Files -> Assembler Output -> Assembly With Source Code /FAs
 * Linker -> Advanced -> Randomized Base Address -> Disable Image Randomization /DYNAMICBASE:NO
 * Linker -> Advanced -> Fixed Base Address -> Generate a relocation section /FIXED:NO

The second last entry (Radmonized Base Address) is useful when you actually need to debug assembly code since it guarantees that each time you start your program it will occupy the same address space. The very last option (Fixed Base Address) is essential when you are working with Intel's VTune for performance profiling.

The remaining part is to correctly setup your test-bed in order to prevent the rather intelligent compiler from converting your code into NOP's when it reallized the code is unused. One way is to return the result of your computations - you won't see the assembly of the assignment since we are explicitly preventing inlining of the code via the macro EIGEN_DONT_INLINE. Using this macro is actually one of the more important aspects in order to encapsulate the code you are interested in, since we are lacking GCC's comment capabilities. A sample test-bed could look as follows:

 template <typename VectorType>
 struct VectorAddition
 {
   typedef VectorType ReturnType;
   EIGEN_DONT_INLINE static VectorType run(int)
   {
     VectorType a,b,c,d;
     return a+b+c+d;
   }
 };
 int main()
 {
   Vector4f res = VectorAddition<Vector4f>::run();
 }

Now, when you have correctly configured your project, you should be able to place a break point within the run method. When running your program by hitting F5 (Start Debugging) you should end up at your breakpoint in the run method. The final step is to hit ALT-8 and you're welcome in the wonderful world of assembly code.

Profiling tips

Using timers

In the file bench/BenchTimer.h we have a little timer that can be used for that.

Documentation tips

We use Doxygen, see the commands. See existing source code for example usage. Here are some best practices:

  • Make generous use of "see also" tags: \sa
  • Always use tags for parameters and return values (e.g. \param and \returns), that makes things easier to find in a glance.
  • If possible, add a code snippet. All you have is to add it in doc/snippets (see existing files there). CMake will complete your snippet into working form, compile it, run it, and record its output into a .out file. Then in your doxygen comment, just put:
 \include mysnippet.cpp
 Output: \verbinclude mysnippet.out
  • Alternatively, if you want to add a full self-compilable example (so you dont want CMake to add stuff to make it compilable), put your example file in doc/examples.

Resources

Linear algebra

Matrix products

CPU level optimizations

Meetings

On separate Meetings page.