#387 Bundling exception: Heimdal bundles libtommath
Closed: Fixed None Opened 5 years ago by ktdreyer.

'''High-level summary:''' Heimdal Kerberos bundles the "libtommath" library. The upstream libtommath project has a specific code optimization that the Heimdal developers do not want to include in their code.

== Work so far ==

I've discussed this with the libtommath developer (Steffen Jaeckel) and two of the primary Heimdal developers (Jeffrey Altman and Love Hörnquist Åstrand). Jeffrey and Love are aware of Fedora's bundled libs policy, and they have recently removed all the differences in Heimdal's copy of libtommath, except for this one change: https://github.com/libtom/libtommath/commit/921be35779f7d71080ad85c27ed58671602d59b3

The Heimdal developers are concerned that this changes the function from constant computation time to variable computation time. Jeffrey and Love are concerned that the distinction might be used by an attacker to perform a time measurement attack on the computation to reduce the size of a key space.

The Heimdal developers have communicated this to the libtommath developer, and the libtommath developer agrees that libtommath should support the constant time use-case. The sticking point is in how to support this. It is not in libtommath's interest to simply revert the commit. The best way forward would be to support two APIs: a "slow, secure" one that Heimdal could use, and a fast one that other libtommath consumers could use.

== History with libtom ==

The LibTom project experienced a dry spell between 2007 and 2010. The project's original maintainer Tom St Denis had abandoned the project. It was at this time that Heimdal bundled the libtommath library in their tree.

Fast forward a couple years, and a new maintainer (Steffen Jaeckel) has assumed maintenance on libtommath.

== History in Fedora ==

Heimdal packaging for Fedora has been a work-in-progress for years (see https://bugzilla.redhat.com/613001). Alexander Boström, Orion Poplawski, Rok Papez, and myself have contributed to the packaging.

Orion, Alexander and I have worked over the past six months to communicate with Heimdal and libtommath upstream. We've been able to remove all other Heimdal bundled libraries besides libtommath, and the libtommath gap has closed significantly over the past couple of months, so we certainly consider this FPC ticket to be a last resort. We are continuing to push Heimdal's smaller changes into libtommath, and I've kept in regular contact with Steffen Jaeckel and the Heimdal developers on IRC and email.

== Bundled library Q&A ==

Below are the standard form questions.

'''Has the library behaviour been modified? If so, how has it been modified?'''

The short answer is "Yes, it has been modified."

The long answer is that libtommath 0.41 was copied into Heimdal's tree because the libtom project had died. After a couple years, new developers resumed libtommath maintenance. The Heimdal developers do not prefer to build against the current upstream libtommath because they disagree with a change to optimize the square-multiply algorithm (https://github.com/libtom/libtommath/commit/921be35779f7d71080ad85c27ed58671602d59b3). If this difference could be resolved, then Heimdal could stop bundling and build against a system version.

'''Why haven't the changes been pushed to the upstream library?'''

At a general level, there have been several attempts to clear up Heimdal's bundled dependencies, and we have cleared up all of them except this one remaining issue.

Regarding libtommath in particular, both Heimdal master and libtommath upstream have come a lot closer as a result of a concentrated effort. The problem is the "optimization" commit that I referred to above.

Both Heimdal upstream and I myself have had conversations with the libtom maintainer, Steffen Jaeckel, about this change. Steffen agrees that the default behavior of the function should be the slow, secure method of calculation. However, libtom is a volunteer project, and Steffen has not settled on what the API would look like to support both the fast version and the secure version.

It would be hard to construe this forking as a matter of simple neglect. Jeffrey Altman, Alexander, and I have worked hard to reduce the differences as much as we can based on the Git development branches for both projects. In fact Steffen has even done some work on this extra branch in Git here: https://github.com/libtom/libtommath/tree/feature/timing_resist . The sticking point is that libtommath is a volunteer project and Steffen's work is not yet ready to be merged into the mainline branch, let alone released in a stable libtommath version.

'''Have the changes been proposed to the Fedora package maintainer for the library? In some cases it may make sense for our package to take the changes despite upstream not taking them (for instance, if upstream for the library is dead).'''

In August I talked with the maintainer of libtommath (Simone Caronni, FAS slaanesh), so he is aware of the basics of the situation.

In theory Fedora could revert upstream's optimization commit, but my current opinion is that Simone shouldn't do this for three reasons.
1. The first reason is that if we reverted libtom's upstream commit in Fedora, we would be diverging from upstream there, going against our policy of "staying close to upstream".
2. The second reason we should leave libtommath as-is in Fedora is that Steffen seems amenable to solving this upstream eventually with a new API.
3. The third reason is that this does not solve the whole problem - in order to completely unbundle libtommath from Heimdal, we need additional patches that are only on libtom's "develop" branch, and are not available in a released libtommath version. In other words, not only would Simone need to revert upstream's decision, but he would also have to package a Git snapshot.

It is more sensible to bundle in Heimdal until the ongoing work in libtom's git repository can peter down into a stable libtommath release.

'''Could we make the forked version the canonical version within Fedora? For instance, if upstream for the library is dead, is the package we're working on that bundles willing to make their fork a library that others can link against?'''

I believe there is light at the end of the tunnel and eventually libtommath will support a cryptographically-secure version of mp_expt_d() that Heimdal can use.

'''Are the changes useful to consumers other than the bundling application? If so why aren't we proposing that the library be released as a fork of the upstream library?'''

Since we've been able to work over the past year to narrow the gap down to a single commit, I think that we should wait for libtommath's author to just incorporate this with a new API, rather than forking the project off entirely into its own namespace with an unspecified end date.

I'm not very familiar with any specific consumers of libtommath other than Heimdal. The commit log for the offending commit mentions Rubinius, so I have a hunch that some consumers are going to want speed, and some will want security.

'''Is upstream keeping the base library updated or are they continuously one or more versions behind the latest upstream release?'''

See "History" for an explanation of the death and rebirth of libtom. Additionally, there has been renewed interest over the past five months in making this all work.

'''What is the attitude of upstream towards bundling? (Are they eager to remove the bundled version? are they engaged with the upstream for the library? Do they have a history of bundling? Are they argumentative?)'''

The Heimdal developers are eager to remove the differences between their copy and the revived libtommath upstream. They have demonstrated this willingness by accepting patches to synchronize their copies, or in some cases, even remove bundled libraries from the tree altogether.

Orion and I have both brought up the bundling issues over the years. Upstream has patiently explained the particular reasons for bundling, and has unbundled libraries in the master branch in Git.

If the Heimdal developers were satisfied that upstream libtommath's function was cryptographically secure, they would not bundle the library, and they would just require libtommath as an external dependency as they do for other libraries.

'''Overview of the security ramifications of bundling'''

As explained above, the Heimdal developers deem the difference in the bundled copy to be cryptographically relevant to the operation of their software. Due to the nature of the project, Heimdal's codebase receives regular review by security experts. In fact Heimdal has ended up fixing a couple uninitialized variables and compiler warnings that were not yet in libtom's codebase. Alexander and I have since proposed those changes to libtom.

'''Does the maintainer of the Fedora package of the library being bundled have any comments about this?'''

Back in August I reached out to Simone Caronni over personal email, explaining the situation, and advising him that Alexander and I might file a bundling exception. I am adding Simone to the CC on this ticket now.

'''Is there a plan for unbundling the library at a later time? Include things like what features would need to be added to the upstream library, a timeline for when those features would be merged, how we're helping to meet those goals, etc.'''

Yes, the libtom maintainer plans to add a new API to choose between "fast" and "secure" methods of operation.

There is no concrete timeline since libtom is a volunteer project.

I am helping to meet the goal of unbundling by conversing regularly with Steffen and Heimdal upstream, and synchronizing the simpler parts of the code as I can (I'm not a C coder).

'''Please include any relevant documentation -- mailing list links, bug reports for upstream or the bundled library, etc.'''

Here's the Heimdal review request: https://bugzilla.redhat.com/613001 . I would like to draw attention to the fact that the list of bundled libraries has decreased considerably from the beginning till now.

At a general level, here's a mailing list discussion that conveys Heimdal's willingness to remove bundled libraries: https://list.sics.se/sympa/arc/heimdal-discuss/2011-01/msg00025.html . Note this was from three years ago, which I take to mean that the situation is slowly improving.

Here is Alexander's first attempt to get Heimdal's changes to libtom upstream:
https://github.com/libtom/libtommath/pull/14 . Upstream asked for the changes to be broken out into logical commits.

Here's Alexander's second try, broken into small patches: https://github.com/libtom/libtommath/pull/18 . After this, Alexander was pulled onto other projects so he hasn't had much time to address the comments. I've been breaking these up further into distinct pull requests so that Steffen can merge them more easily.

During Alexander's second try, Steffen pointed out some differences and duplicated functionality. As a result, Heimdal was able to recently simplify the differences on their end: https://github.com/heimdal/heimdal/commits/master/lib/hcrypto/libtommath . However, as explained above, Heimdal does not want to merge this final problematic commit in libtom:

== Conclusion ==

When libtom provides a cryptographically-secure version of mp_expt_d() that Heimdal can use, then the Fedora Heimdal packages can drop the bundled libtommath. Until then, the least-worst option is to permit a bundling exception.

Temporary exception for Heimdal to bundle libtommath until Fedora 22 granted. FPC expressed a willingness to look at granting an extension of another two releases if this one expires without the changes being merged upstream. To grant an extension we'd want to see that it was still being worked on, just not completed. (+1:5, 0:0, -1:0)

Please use virtual Provides: bundled(libtommath) = [ESTIMATE OF VERSION THE BUNDLE IS BASED ON]

Login to comment on this ticket.