Difference between revisions of "Developer's Corner"

From Eigen
Jump to: navigation, search
(Contributors crediting and copyright)
(25 intermediate revisions by 6 users not shown)
Line 3: Line 3:
 
= Communication between developers =
 
= Communication between developers =
  
The [[Main_Page#Mailing_list|eigen mailing list]] and IRC (#eigen on freenode) are where live discussions happen.
+
[[Bugzilla]], the [[Main_Page#Mailing_list|mailing list]] and IRC (#eigen on freenode) are where live discussions happen.
  
For reference, there are various places in this wiki where other developers expect you to update information about what you are doing.
+
We are transitioning to using Bugzilla for as many things as possible: for feature development, for tracking future Eigen versions and for TODO items. However, the mailing list remains a good communication channel for general discussion.
  
* [[Todo]] : general TODO
+
= Contributors crediting and copyright =
* [[Todo for 3.0]] : our roadmap for 3.0, it only contains the stuff remaining to be done.
+
 
* [[Status of unsupported modules]] : This page gives the status of the modules currently under unsupported/.
+
Everyone who made even the smallest contibution should add himself/herself to the [[Main Page#Credits|Credits]].
 +
 
 +
Everyone keeps copyright on his contributions. There is no copyright assignment.
 +
 
 +
When making a contribution on behalf of your employer, please try making it in your own name, not in your employer's name, if possible.
 +
 
 +
When you make large improvements to a source file, it is a good idea to add yourself a copyright line in it, although this is by no means necessary: you do implictly get copyright on the code you wrote.
 +
 
 +
Every commit should be made using the author's identity. We explain on the [[Git]] page how to generate patches in such a way that you get credited when we import them.
 +
 
 +
Copying Permissively-Licensed Code into an MPL 2 File: [http://www.mozilla.org/MPL/2.0/permissive-code-into-mpl.html Guidelines for Developers].
  
 
= Code Quality =
 
= Code Quality =
  
Nightly build reports:eigen
+
* '''Dashboards''' for unit tests with nightly build reports:
* '''Dashboards''':
+
** For the devel branch: http://manao.inria.fr/CDash/index.php?project=Eigen&display=project
** For the devel branch: http://eigen.tuxfamily.org/CDash/index.php?project=Eigen
+
** For the stable 3.3 branch: http://manao.inria.fr/CDash/index.php?project=Eigen3.3
** For the stable 2.0 branch: http://eigen.tuxfamily.org/CDash/index.php?project=Eigen+2.0
+
* '''[[Performance monitoring]]'''
* '''Ohloh''' statistics: http://www.ohloh.net/p/5393
+
* '''Openhub''' statistics: https://www.openhub.net/p/eigen
  
 
= Using CMake =
 
= Using CMake =
Line 76: Line 86:
 
Notes explaining aspects of the architecture of Eigen 3.x are available at the [[Eigen3 Developer Documentation]] page
 
Notes explaining aspects of the architecture of Eigen 3.x are available at the [[Eigen3 Developer Documentation]] page
  
= Mercurial workflow =
+
= Git workflow =
  
The contents of this section have moved to [[Mercurial]].
+
Please read [[Git|this page]] to learn more about our Git workflow.
 +
 
 +
= Make a new release =
 +
 
 +
These days it's very easy, as the docs are automatically generated and uploaded, and the tarballs are auto-generated by GitLab.
 +
 
 +
So here's what it takes to make a new release (a new minor version):
 +
* Announce a week in advance on the [[Main_Page#Mailing_list|eigen mailing list]] the upcoming release (give a precise date) so that people can test and/or speak up if they know of an issue. Of course, in case of an emergency, no need to do that :)
 +
* Concert with other devs, and/or use CDash, and/or test yourself, to make sure that the test suite passes at the very least for:
 +
** latest stable GCC
 +
** some older GCC 4.x (ok, it's hard to check them all), try to cover the oldest GCC we're supposed to support (see main page)
 +
** latest MSVC
 +
** latest Clang
 +
* Also do think to check "make install"! Remember how it spoiled the 2.0.7 release!
 +
** Supreme refinement would be to check building the test suite against the installed Eigen.
 +
* Update the version number (EIGEN_MINOR_VERSION, etc) in the file Eigen/src/Core/util/Macros.h.
 +
* commit that:
 +
git add Eigen/src/Core/util/Macros.h
 +
git commit
 +
* Make a tag:
 +
git tag TAGNAME
 +
* recommended: check that everything is in order.
 +
git show
 +
* push the release:
 +
git push --tags REMOTENAME BRANCHNAME
 +
e.g. to release a 3.3.X version
 +
git push --tags origin 3.3
 +
* For a major version, write and keep up-to-date a dedicated page on the wiki, e.g. see: http://eigen.tuxfamily.org/index.php?title=3.4
 +
* For a minor version: make the full changelog. It's not just a dump of the commit messages: it must be readable and interesting for Eigen users. You can use "git log" again, try to put the grafted/cherry-picked bug fixes first, try to group together what's related. You can also use git and sed to generate an initial raw dump of the changesets:
 +
  git --no-pager log --pretty=format:"* <nowiki>[https://gitlab.com/libeigen/eigen/-/commit/%H Commit %h]:</nowiki> %s" HEAD...PREVIOUSTAGNAME \
 +
    | sed 's/\([Bb]ug\) #\([0-9]*\)/[https:\/\/gitlab.com\/libeigen\/eigen\/-\/issues\/\2 Issue #\2]/g'
 +
to get the tag of the previous release you can check
 +
# Get all tags
 +
git tag
 +
# Filter for tags
 +
git tag --list 'PATTERN'
 +
* Update the [[Main_Page|Main Page]] on the wiki, that is the announcement at the top and the [[Main_Page#Download|Download]] section. Make sure that the tarball links point to the new release, and that any example of how to get the latest tag also refers to the new tag.
 +
** To add a news, simply create a new page in the "News" namespace:
 +
# enter <nowiki>"News:Title of my great announcement!"</nowiki> in the search box,
 +
# press search, and click "create a new page",
 +
# write your news as usual, preview, save... ([http://eigen.tuxfamily.org/index.php?title=Special%3AAllPages&from=&to=&namespace=100 examples])
 +
# you might have to enforce the regeneration of the mainpage by editing/saving it (because of mediawiki's cache)
 +
* Add the release notes into the [[ChangeLog]] page.
 +
* Write a mail to the [[Main_Page#Mailing_list|eigen mailing list]], paste the changelog into it.
 +
* Update the #eigen channel topic on IRC.
  
 
= Debugging tips =
 
= Debugging tips =
Line 84: Line 138:
 
=== Debugging under Visual Studio ===
 
=== Debugging under Visual Studio ===
 
[[Image:Eigen_debug.jpg|thumb|left|"Examplary view of the enhanced debugger output."]]
 
[[Image:Eigen_debug.jpg|thumb|left|"Examplary view of the enhanced debugger output."]]
The Visual Studio IDE allows to add specialized debug visualizers for C/C++ code. The way in which specific classes are displayed in the debug output is defined by the file
+
The Visual Studio IDE supports specialized debug visualizers for custom C/C++ types.
 
+
    %VS90COMNTOOLS%\..\Packages\Debugger\autoexp.dat
+
 
+
(note that the environment variable may change depending on your IDE version, e.g. VS80COMMONTOOLS).
+
 
+
Adding the data found in [http://eigen.tuxfamily.org/misc/eigen_autoexp_part.dat this] file (it is also in the Mercurial repository under debug/msvc/) right after the [Visualizer] section activates new visualization rules for some Eigen types. Currently, all kinds of Matrix types are covered wheras e.g. Eigen::SparseMatrix and Eigen::Transform are still missing. Any new visualizers are welcome and will be added with a note referencing the author to our current file.
+
 
+
Before working on the autoexp.dat file please ensure to create a backup copy!
+
  
'''Note:''' The file has been designed to work with the default branch. Currently, the 2.0.x versions do not work unless the constant 'Dynamic' is changed to 10000 as opposed to 33331.
+
Eigen debug visualizers for Visual Studio 2012 and newer can be downloaded from [https://bitbucket.org/eigen/eigen/src/tip/debug/msvc/eigen.natvis eigen.natvis]. The file is typically copied to <code>%VSINSTALLDIR%\Common7\Packages\Debugger\Visualizers</code>, where <code>%VSINSTALLDIR%</code> is the Visual Studio installation folder. More info can be found in [https://msdn.microsoft.com/en-us/library/jj620914.aspx Create Custom Views of Native Objects] on MSDN.
  
'''Note:''' Since Visual Studio 2010 there is a new possibility available. You can take the file as is, copy it anywhere on your disc and create an environment variable called _vcee_autoexp which is pointing to that file. Visual Studio will read and merge it with the existing autoexp.dat.
+
For Visual Studio 2010 and older debug visualizers are defined in <code>%VSINSTALLDIR%\..\Packages\Debugger\autoexp.dat</code>. Adding the data found in [http://bitbucket.org/eigen/eigen/src/tip/debug/msvc/eigen_autoexp_part.dat this file] right after the [Visualizer] section activates new visualization rules for some Eigen types (for Eigen2, use [http://eigen.tuxfamily.org/bz/attachment.cgi?id=245 this file]).
  
 
=== Studying assembly output ===
 
=== Studying assembly output ===
Line 201: Line 247:
  
 
== Linear algebra ==
 
== Linear algebra ==
* [http://www.netlib.org/linalg/html_templates/Templates.html Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods]
+
* General
* [http://www.cs.utk.edu/~dongarra/etemplates/index.html Templates for the Solution of Algebraic Eigenvalue Problems: a Practical Guide]
+
** [http://www.mit.edu/~wingated/stuff_i_use/matrix_cookbook.pdf The Matrix Cookbook]
* [http://dx.doi.org/10.1109/38.486688 Consider the lowly 2x2 matrix] (requires an IEEE subscription)
+
** [http://www.netlib.org/lapack/lawns Lapack working notes]
* [http://www.cds.caltech.edu/~andrea/research/sw/2x2_eigen.html Eigen decomposition for a 2x2 matrix]
+
** [http://dx.doi.org/10.1109/38.486688 Consider the lowly 2x2 matrix] (requires an IEEE subscription)
* [http://www.geometrictools.com/Documentation/EigenSymmetric3x3.pdf Direct eigen decomposition for a 3x3 symmetric matrix]
+
* Linear iterative solvers
 +
** [http://www.netlib.org/linalg/html_templates/Templates.html Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods]
 +
** [http://www-users.cs.umn.edu/~saad/books.html Iterative methods for sparse linear systems]
 +
* Eigenvalue problem
 +
** [http://www.cs.utk.edu/~dongarra/etemplates/index.html Templates for the Solution of Algebraic Eigenvalue Problems: a Practical Guide]
 +
** [http://www-users.cs.umn.edu/~saad/books.html Numerical Methods for Large Eigenvalue Problems]
 +
** [http://www.cds.caltech.edu/~andrea/research/sw/2x2_eigen.html Eigen decomposition for a 2x2 matrix]
 +
** [http://www.geometrictools.com/Documentation/EigenSymmetric3x3.pdf Direct eigen decomposition for a 3x3 symmetric matrix]
 +
** [http://www.cs.utexas.edu/users/flame/pubs/flawn60.pdf Restructuring the QR Algorithm for High-Performance Application of Givens Rotations]
  
 
== Matrix products ==
 
== Matrix products ==
Line 215: Line 269:
 
* CPUID
 
* CPUID
 
** [http://www.sandpile.org/ia32/cpuid.htm IA-32 architecture]
 
** [http://www.sandpile.org/ia32/cpuid.htm IA-32 architecture]
 +
* [http://www.research.scea.com/gdc2003/fast-math-functions_p1.pdf fast-math-functions 1], [http://www.research.scea.com/gdc2003/fast-math-functions_p2.pdf fast-math-functions 2]
  
 
= Meetings =
 
= Meetings =
  
 
On separate [[Meetings]] page.
 
On separate [[Meetings]] page.

Revision as of 18:58, 2 March 2021


Communication between developers

Bugzilla, the mailing list and IRC (#eigen on freenode) are where live discussions happen.

We are transitioning to using Bugzilla for as many things as possible: for feature development, for tracking future Eigen versions and for TODO items. However, the mailing list remains a good communication channel for general discussion.

Contributors crediting and copyright

Everyone who made even the smallest contibution should add himself/herself to the Credits.

Everyone keeps copyright on his contributions. There is no copyright assignment.

When making a contribution on behalf of your employer, please try making it in your own name, not in your employer's name, if possible.

When you make large improvements to a source file, it is a good idea to add yourself a copyright line in it, although this is by no means necessary: you do implictly get copyright on the code you wrote.

Every commit should be made using the author's identity. We explain on the Git page how to generate patches in such a way that you get credited when we import them.

Copying Permissively-Licensed Code into an MPL 2 File: Guidelines for Developers.

Code Quality

Using CMake

We have a little CMake tutorial here.

Unit tests and dashboard

Everything about unit tests and the dashboard, is on the Tests page.

Eigen hacking

Coding rules

We are not very strict about coding rules, however the trends are:

  • indentation with 2 spaces (no tabs !)
  • the best is to show a typical example of a public class:
 template<typename MatrixType
 class NameOfTheClass
 {
     typedef typename MatrixType::Scalar Scalar;
   public:
     // public functions start with lower case:
     Scalar nameOfTheFunction(int i) const;
     // public static functions start with upper case:
     static NameOfTheClass* Create(const MatrixType& mat);
   protected:
     // member attributes start with m_:
     Scalar m_attributeName;
     MatrixType m_matrix;
 };
 
 template<typename MatrixType> typename MatrixType::Scalar
 NameOfTheClass<MatrixType>::nameOfTheFunction(int i) const
 {
   Scalar res = 0;
   if(i>0)
   {
     for(int j=0; i<m_matrix.cols(); ++j,++i)
       res = ei_hypot(res, m_matrix.coeff(i,j));
   }
   else
   {
     j = 0;
     while (i<m_matrix.rows())
     {
       res = ei_hypot(res, m_matrix.coeff(i,j));
       ++i;
       ++j;
     }
   }
   return res;
 }
  • global functions and class/struct reserved for internal use start with ei_, eg:
 struct ei_assign_selector;
  • favor explicit names rather than short ones

Developer Documentation

Notes explaining aspects of the architecture of Eigen 3.x are available at the Eigen3 Developer Documentation page

Git workflow

Please read this page to learn more about our Git workflow.

Make a new release

These days it's very easy, as the docs are automatically generated and uploaded, and the tarballs are auto-generated by GitLab.

So here's what it takes to make a new release (a new minor version):

  • Announce a week in advance on the eigen mailing list the upcoming release (give a precise date) so that people can test and/or speak up if they know of an issue. Of course, in case of an emergency, no need to do that :)
  • Concert with other devs, and/or use CDash, and/or test yourself, to make sure that the test suite passes at the very least for:
    • latest stable GCC
    • some older GCC 4.x (ok, it's hard to check them all), try to cover the oldest GCC we're supposed to support (see main page)
    • latest MSVC
    • latest Clang
  • Also do think to check "make install"! Remember how it spoiled the 2.0.7 release!
    • Supreme refinement would be to check building the test suite against the installed Eigen.
  • Update the version number (EIGEN_MINOR_VERSION, etc) in the file Eigen/src/Core/util/Macros.h.
  • commit that:
git add Eigen/src/Core/util/Macros.h
git commit
  • Make a tag:
git tag TAGNAME
  • recommended: check that everything is in order.
git show
  • push the release:
git push --tags REMOTENAME BRANCHNAME

e.g. to release a 3.3.X version

git push --tags origin 3.3
  • For a major version, write and keep up-to-date a dedicated page on the wiki, e.g. see: http://eigen.tuxfamily.org/index.php?title=3.4
  • For a minor version: make the full changelog. It's not just a dump of the commit messages: it must be readable and interesting for Eigen users. You can use "git log" again, try to put the grafted/cherry-picked bug fixes first, try to group together what's related. You can also use git and sed to generate an initial raw dump of the changesets:
 git --no-pager log --pretty=format:"* [https://gitlab.com/libeigen/eigen/-/commit/%H Commit %h]: %s" HEAD...PREVIOUSTAGNAME \
   | sed 's/\([Bb]ug\) #\([0-9]*\)/[https:\/\/gitlab.com\/libeigen\/eigen\/-\/issues\/\2 Issue #\2]/g'

to get the tag of the previous release you can check

# Get all tags
git tag
# Filter for tags
git tag --list 'PATTERN'
  • Update the Main Page on the wiki, that is the announcement at the top and the Download section. Make sure that the tarball links point to the new release, and that any example of how to get the latest tag also refers to the new tag.
    • To add a news, simply create a new page in the "News" namespace:
  1. enter "News:Title of my great announcement!" in the search box,
  2. press search, and click "create a new page",
  3. write your news as usual, preview, save... (examples)
  4. you might have to enforce the regeneration of the mainpage by editing/saving it (because of mediawiki's cache)
  • Add the release notes into the ChangeLog page.
  • Write a mail to the eigen mailing list, paste the changelog into it.
  • Update the #eigen channel topic on IRC.

Debugging tips

Debugging under Visual Studio

"Examplary view of the enhanced debugger output."

The Visual Studio IDE supports specialized debug visualizers for custom C/C++ types.

Eigen debug visualizers for Visual Studio 2012 and newer can be downloaded from eigen.natvis. The file is typically copied to %VSINSTALLDIR%\Common7\Packages\Debugger\Visualizers, where %VSINSTALLDIR% is the Visual Studio installation folder. More info can be found in Create Custom Views of Native Objects on MSDN.

For Visual Studio 2010 and older debug visualizers are defined in %VSINSTALLDIR%\..\Packages\Debugger\autoexp.dat. Adding the data found in this file right after the [Visualizer] section activates new visualization rules for some Eigen types (for Eigen2, use this file).

Studying assembly output

Using GCC

We have a macro EIGEN_ASM_COMMENT (currently defined only on GCC! please port it, it is in Macros.h):

You can use assembly comments to make it much easier to find the asm code corresponding to a particular portion of your code:

EIGEN_ASM_COMMENT("begin");
Vector4f u = v + 3*w;
EIGEN_ASM_COMMENT("end");

Then tell your compiler to output assembly code, for example with GCC it is the -S option, so you'd do:

 g++ myprogram.cpp -O2 -S -o myprogram.s

Very important: GCC doesn't seem to optimize correctly code inside the main() function. So make sure to put your critical code in another, non-inlined, function. You can use EIGEN_DONT_INLINE to prevent inlining of a function. But actually, the easiest might be to put your code in a function foo() taking references to the Eigen objects as arguments and not putting any main() function, and then compiling with "g++ -c". So your code could look like:

#include<Eigen/Core>
using namespace Eigen;
void foo(Vector4f& u, Vector4f& v, Vector4f& w)
{
  EIGEN_ASM_COMMENT("begin");
  u = v + 3*w;
  EIGEN_ASM_COMMENT("end");
}

And you then compile it with:

 g++ myprogram.cpp -O2 -c -S -o myprogram.s

The resulting asm code is now very easy to find, just search for "begin":

 #APP
 # 5 "x.cpp" 1
 	#begin
 # 0 "" 2
 #NO_APP
 	movss	.LC2(%rip), %xmm0
 	shufps	$0, %xmm0, %xmm0
 	mulps	(%rdx), %xmm0
 	addps	(%rsi), %xmm0
 	movaps	%xmm0, (%rdi)
 #APP
 # 7 "x.cpp" 1
 	#end
 # 0 "" 2
 #NO_APP

Using Visual Studio

In a first step you need to compile your program with the correct compiler and linker switches as given below:

 * C/C++ -> General -> Debug Information Format -> Program Database /Zi
 * C/C++ -> General -> Optimization -> Optimization -> Maximize Speed /O2
 * C/C++ -> General -> Code Generation -> Basic Runtime Checks -> Default
 * C/C++ -> General -> Code Generation -> Enable Enhanced Instruction Set -> Streaming SIMD Extensions 2 /arch:SSE2
 * Linker -> Debugging -> Generate Debug Info -> Yes /DEBUG

Optional settings are:

 * C/C++ -> Output Files -> Assembler Output -> Assembly With Source Code /FAs
 * Linker -> Advanced -> Randomized Base Address -> Disable Image Randomization /DYNAMICBASE:NO
 * Linker -> Advanced -> Fixed Base Address -> Generate a relocation section /FIXED:NO

The second last entry (Radmonized Base Address) is useful when you actually need to debug assembly code since it guarantees that each time you start your program it will occupy the same address space. The very last option (Fixed Base Address) is essential when you are working with Intel's VTune for performance profiling.

The remaining part is to correctly setup your test-bed in order to prevent the rather intelligent compiler from converting your code into NOP's when it reallized the code is unused. One way is to return the result of your computations - you won't see the assembly of the assignment since we are explicitly preventing inlining of the code via the macro EIGEN_DONT_INLINE. Using this macro is actually one of the more important aspects in order to encapsulate the code you are interested in, since we are lacking GCC's comment capabilities. A sample test-bed could look as follows:

 template <typename VectorType>
 struct VectorAddition
 {
   typedef VectorType ReturnType;
   EIGEN_DONT_INLINE static VectorType run(int)
   {
     VectorType a,b,c,d;
     return a+b+c+d;
   }
 };
 int main()
 {
   Vector4f res = VectorAddition<Vector4f>::run();
 }

Now, when you have correctly configured your project, you should be able to place a break point within the run method. When running your program by hitting F5 (Start Debugging) you should end up at your breakpoint in the run method. The final step is to hit ALT-8 and you're welcome in the wonderful world of assembly code.

Profiling tips

Using timers

In the file bench/BenchTimer.h we have a little timer that can be used for that.

Documentation tips

We use Doxygen, see the commands. See existing source code for example usage. Here are some best practices:

  • Make generous use of "see also" tags: \sa
  • Always use tags for parameters and return values (e.g. \param and \returns), that makes things easier to find in a glance.
  • If possible, add a code snippet. All you have is to add it in doc/snippets (see existing files there). CMake will complete your snippet into working form, compile it, run it, and record its output into a .out file. Then in your doxygen comment, just put:
 \include mysnippet.cpp
 Output: \verbinclude mysnippet.out
  • Alternatively, if you want to add a full self-compilable example (so you dont want CMake to add stuff to make it compilable), put your example file in doc/examples.

Resources

Linear algebra

Matrix products

CPU level optimizations

Meetings

On separate Meetings page.