#350 Bundled library exception for codimension-parser
Closed: Fixed None Opened 10 years ago by cicku.

Hola FPC.

This is my first time to ask for an exception from FPC to get package codimension-parser apporved by Fedora.

Review request is here: https://bugzilla.redhat.com/show_bug.cgi?id=1012215


Looking into the source tarball, we can see a directory named "libantlr3c-3.2". Yes, this is already packaged in antlr3-C-devel.

But after talking with upstream we decide to request an exception because:


Q. Has the library behaviour been modified? If so, how has it been modified?
A. Yes, it has been modified by upstream because they want to use modified code to match their actual needs. Codimension parser uses ANTLR generated grammar. C-runtime library however was patched for codimension python parser specifically and the original library will not work.

Q. Why haven't the changes been pushed to the upstream library?
A. Good question. At the moment the libraries are incompatible to each other. It is impossible to take the main stream library and use it in the codimension python parser. And vice verse, there is no way to use the modified version instead of the mainstream one in other projects.

Q. If no attempt has been made to push the changes upstream, we shouldn't be supporting people forking out of laziness.
Have the changes been proposed to the Fedora package maintainer for the library? In some cases it may make sense for our package to take the changes despite upstream not taking them (for instance, if upstream for the library is dead).
A. Upstream is not dead though, however this is not proposed for Fedora only.

'''See [1] for details.'''

Q. Are the changes useful to consumers other than the bundling application?
A. No. Only useful for this parser.

Q. Is upstream keeping the base library updated or are they continuously one or more versions behind the latest upstream release?
A. Upstream is developing antlr constantly.

Q. Does the maintainer of the Fedora package of the library being bundled have any comments about this?
A. Well, we have no other choices.

Q. Is there a plan for unbundling the library at a later time?
A. No, bundling forever.


Additional note:

[1]. Upstream explanation(Sergey Satskiy, MAIN UPSTREAM DEVELOPER):

I do not think that my changes could be accepted as an upstream feature. This is why I link my version statically. It guarantees that there will be no conflicts with the mainstream library.

Meantime, the ANTLR C runtime library is not actively developed. Some promised features are not implemented for years. The library structure is complicated (I'd even say over engineered) and I doubt that somebody will make any significant changes for it in mainstream. The patched library is linked statically into the parser so there will be no problem even if the original library is installed on the user system as well.

[2]. Packager notes(Christopher Meng, FEDORA PACKAGER):

I don't know know if codimension upstream can merge their changes to 3.5 antlr. Now codimension use 3.2-based modified code. Is it a MUST to let bundled libs get rebased from latest original upstream verison?


FPC voted on granting an exception for codimension-parser to use this antlr fork (as a static lib only), but could only get to (+1:4, 0:0, -1:0).

Additional votes from FPC members are needed.

I'm a bit uncomfortable with this one. Upstream codimension IDE + parser seems to only have a single committer. The bundled library is on 3.2 but the upstream version is 3.5. upstream for antlr is alive and well. From a truly brief look at the code, my best guess is that the parser deals with user-supplied data (Text files that the user could have written, downloaded, or etc).

I think my feeling is that upstream is bundling and modifying but has stopped short of forking and building a community to care for their changes to the code. That makes me -1 this unless we relax the bundled library guidelines to handle this case generally.

I concur with Toshio: -1

Colleagues,
let me clarify how the things are working in codimension python parser - it may help to make a better judgement if the exception should be granted.

There is a huge framework ANTLRv3 (http://www.antlr3.org/). It is comprised of many pieces - code generator for many languages including C/C++, java, python etc., GUI ANTLRWorks for interactive editing/testing/debugging grammars, runtime libraries for the languages the generator supports etc. The codimension python parser uses only a small fraction of this framework - the C-runtime library which is in question in this ticket.

It seems to me that mr. toshio is referring to the ANTLR generator in his comment while the ticket is about the ANTLR C-runtime library.

Let me describe the ANTLR involved building process in a few more details. A user provides a grammar definition (.g file) for a certain input. The ANTLR generator takes the grammar and produces source code for parsing this input. In our case a C code is generated. The generated code relies on a support provided by a corresponding runtime library. libantlr3c is the C runtime library for the generated C code. In order to build an application usually the following components are required:
- generated C code
- C runtime library
- some glue code to stick together the parsing part with an application area
Typically C runtime library is linked dynamically.

Unfortunately this generic approach did not work well for codimension python parser. The fresh version of ANTLRv3 (3.5.1) generates uncompilable C code for the grammar we have. Earlier versions produce a correct compilable code. So we use an older version (and include the generated code into the source distribution (together with the grammar file) to reduce dependencies for those who want to build everything from source). The most up to date version of the C runtime library is 3.4 (http://www.antlr3.org/download/C/) however the version which corresponds to the generator which produces a correct code is 3.2 so we use 3.2. Side note here is that there is a chance that 3.4 will work too however it is still safer to use 3.2.
It is also unfortunate that the C runtime library is not flexible enough. It produces output only on stdout/stderr which does not suit the codimension python parser. So the C runtime library 3.2 was altered to be able to collect output in memory and to avoid any potential interference to potentially installed other software the library is linked statically.

Here is a short summary:
Why do we need our own version of the C runtime library?
Because the most up to date version of ANTLRv3 generates uncompilable code from the grammar we have and because we need project specific changes in the library.
Why did we patch our version of the C runtime library?
To be able to collect output in memory instead of printing in stdout/stderr.
* May our version of library interfere with something and break somebody's code?
Nope. It's statically linked.

I hope this helps.

At the meeting toshio asked the question: "How does this differ from the zlib case?" for which there were currently no good answers. geppetto stated that he'd be willing to change his vote to -1.

That puts the current vote at (+1:4, 0:0, -1:3)

  • Current +1's: spot, SmootherFr0gZ, Remi|Fedora, limburgher
  • Current -1's: abadger1999, geppetto, racor
  • Currently haven't voted: Rathann, tibbs

I'm -1 here.

Unfortunately this generic approach did not work well for codimension python parser. The fresh version of ANTLRv3 (3.5.1) generates uncompilable C code for the grammar we have. Earlier versions produce a correct compilable code. So we use an older version (and include the generated code into the source distribution (together with the grammar file) to reduce dependencies for those who want to build everything from source).

This looks similar to cases where projects recommended compiling them using an older gcc instead of fixing bugs in their code. Have you reported the uncompilable code issue to antlr upstream?

You complain that

It is also unfortunate that the C runtime library is not flexible enough. It produces output only on stdout/stderr which does not suit the codimension python parser. So the C runtime library 3.2 was altered to be able to collect output in memory and to avoid any potential interference to potentially installed other software the library is linked statically.

but again you don't say if you tried to work this out with antlr upstream and what the results were.

I checked antlr3 github repo (https://github.com/antlr/antlr3/issues?sort=created&direction=desc&state=open) and haven't found either of those issues reported.

Well, this is a mess. It never passed, and it looks like I was designated by fate to be the tiebreaker. But that was over a year ago, the committee composition has changed, and we've never really even considered whether a pending vote this old is still even valid.

Maybe cicku could tell us if this is still relevant, and we could talk about it again if necessary.

No response; closing. Please feel free to reopen if you have additional information to add.

Login to comment on this ticket.

Metadata