I personally want to see everything here. In the Fedora Rust SIG, doing so has made it much easier to keep track of everything.

Additionally, having everything in Pagure will give us the ability to wire up CentOS CI to test everything with Fedora, openSUSE, and even Mageia if they're interested (cc @akien ).

Things are unfortunately a tad more complex that just moving a single gofed repo.

Right now the go packaging macros we use in fedora exercise the following bits:

1. forge handling macros:
   non Go specific
   srpm: redhat-rpm-config (should be cloned in epel-rpm-config eventually)
   coding langage: lua
   source project: https://src.fedoraproject.org/rpms/redhat-rpm-config/

2. dynamic buildrequires installation in mock
   non Go specific
   srpm: mock-install https://bugzilla.redhat.com/show_bug.cgi?id=1629371
   coding langage: go
   source project: https://github.com/nim-nim/mock-install

3. Go macros used at the srpm stage
   Go specific
   srpm: go-srpm macros
   coding language: lua
   source project: https://github.com/gofed/go-macros
   still missing several PRs I posted months ago

4. Go macros used at the rpm stage
   Go specific
   srpm: go-compilers
   coding language: lua+shell
   source project: https://github.com/gofed/go-macros

5. Go source code analyser
   Go specific
   srpm: go-compilers
   coding language: go
   source project: https://github.com/gofed/symbols-extractor
   uses vendoring
   uses obsolete deps
   is not built according to our own guidelines due to the forceful injection of go source code in a shell/lua macro package

(I'll spare you the dependency and subpackage map, it's yet more complex)

So it is pretty awful, any change has to be synchronised over 5 projects in 3 languages where the srpm layout does not map cleanly to the project layout, with some srpms duplicating the same project or trying to mix different projects in a single package

That is wasting a lot more time and energy than pagure vs non pagure. So +1 to move it under one hosting service, if we spend the necessary time to untangle the srpms and make sure one srpm = one upstream project, and go code is actually compiled inside golang packages that follow our own guidelines. Otherwise that will be a lot of churn for few wins.

Strictly speaking, the forge macros can stay in redhat-rpm-config, or we can make a separate dist-git package called forge-rpm-macros and make redhat-rpm-config/epel-rpm-macros require it.

Strictly speaking, the forge macros can stay in redhat-rpm-config, or we can make a separate dist-git package called forge-rpm-macros and make redhat-rpm-config/epel-rpm-macros require it.

Well this part is rather sane if annoying. Even if we used a separate forge-rpm-macros, as soon as something using it ended up in EL, we'd need to override its content in epel-rpm-macros so no big win in splitting it up.

No, whats awful is how we split go-macros in two srpms, one of those embedding another project (symbols extractor. Which is itself vendoring yet more code).

@nim there are some areas that could be improved, sure, but I see your post as "just" purism rant, many of the point we have already discussed in the past and we have told you why it is the way it is and it seems that you are ignoring it altogether. TBH you have also contributed to this split significantly by introducing lua in to the mix.
IMO we should discuss this at out next meeting.

Let's see the consequences of this "impurity" on my workflow

1. every time you or or someone else FPC asks for a change or a fix I need to update one macro project on github (should be pagure). So far so good standard way of working

2. and then I need to modify two different specs for no good technical reason, just 'we used to do it like this' (and no srpm deps do not bleed in subpackages deps)

3. and then I need to rebuild those two specs in coordinated manner (or bad things ensue).

4. and one of those will build a complete unrelated go binary, with sources pulled from another github project, so I take the hit of a binary build when all I've changed is some rpm macros

5. need i say that this binary build is non standard that it does not follow our own guidelines and the go compiler settings you put in the macros because they're not deployed yet?

6. and then rinse and repeat with two other specs for epel.

7. and do it all in mock, then copr, then koji

Contrast it with:

1. I do the macro change in the macro project

2. I build one macro spec with not go binary dumped in the middle of things and I don't need to worry about package coordination

3. I adapt to epel

And when someone makes a fix in golist

1. he rebuilds the golist binary package, which is a normal fedora go package that follows normal fedora go conventions and uses the go compiler flags you've defined in the macros

2. this does not have to happen every time the macro change, nor do the macro package needs rebuilding every time there is a fix in the golist code

And BTW some of this stuff has already been rebuilt several hundred of times during dev and qa process so this 'purism rant' has already made me waste several man-days (probably weeks) of time I could have better spent fixing and enhancing Go stuff in Fedora, just dealing with a spec layout that is completely in-adapted to the technical needs of the things it packages.

@nim You have still not convinced my or the whole point is totally missing me(as this seems to me that you are saying/implying "that you are not doing this my way or how I say so everything is bad and you are incompetent"). From your list you are omitting the time cost of filing bodhi updates for each of the simple rebuilds. Initial overhead of baby sitting package review and new-package process. Neither of those unfortunately happen without any maintainer intervention.

To be honest I believe that one of the mistakes that lead to this is not having due diligence in taking in the changes that you have been proposing. As we are in the situation that some of your proposals(IMO very helpful in their base principles) got implemented but as you have resigned on finishing the docs side of the things packaging guidelines are even in worse state than in the beginning as they are not only incomplete they are now split too.

@jcajka I’m not saying you are incompetent, I’m saying you are focusing on the bit of the process you use, and completely forgetting the bit of process I use. So everything you dismiss as “purism” because it does not impact too much your workflow, makes me waste time every single day I try to do Go stuff for Fedora.

And yes BTW the current setup you defend so much means submitting two separate packages to Bodhi for every change, and if the result in not deployed synchronously you get potential build failures. More hilariously it means submitting existing macros as bodhi changes every single time you need to rebuild golist for another Go compiler version.

And I’m not given up on guidelining. I even have a ticker FCP-side on how to do it best. Based on how things have evolved this past year, and based on how it has evolved in the 10 years I've maintained Fedora packaging guidelines for fonts packages, the next version will skip the wiki wall no one reads and just move to example templates.

I'm sick of wasting hours formatting pages of wiki, just so people don't bother to read them and ask me directly “what’s on the wiki again” ? They want directly applicable templates with no long explanations, they'll get those, and the templates are actually less work to create and maintain, than the wiki page is.

@nim I don't defend anything, I'm not convinced that the changes that you are vaguely proposing will make any positive difference to general use case(and not hurt it), not only yours. Could you please specify what is your use case, so I don't get it wrong.

My use case is just:
1. make Go macro fixes or enhancements
2. push the result for review or directly to koji in a single spec rebuild, that generates synchronised subpackages
3. do not have any Go code builds in the pathway when not a single line of Go code changed

And conversely
1. make golist fixes or enhancements, or rebuild for new Go compiler version
2. push the result for review or directly to koji in a single spec rebuild, that generates a binary golist command package
3. do not have any macro deployment in the pathway when not a single line of macro code changed

And there’s nothing vague in the proposed changed, it's just one project = one spec file which has always been the Fedora best practice

And the end-result for people using the macros or golist is the same, they get the same files on-system, with the same packages, and the same package inter-dependencies. Except golist is split, which should have been the case from day one, and the srpm and rpm Go macro packages are synchronized cleanly, which again should have been the case from day one.

My use case is just:
1. make Go macro fixes or enhancements
2. push the result for review or directly to koji in a single spec rebuild, that generates synchronised subpackages
3. do not have any Go code builds in the pathway when not a single line of Go code changed

This point is pure purism, it is good to aim for it, but it might be very negligible to rebuild some code, actually matter of seconds(incomparable to the regular koji build overhead). Only issue might be "unnecessary" dependencies.

And conversely
1. make golist fixes or enhancements, or rebuild for new Go compiler version

Rebuilds in general are taken care by rel-eng(on the compiler maintainers request) in automated fashion. IMO there should be some level of the involvement of the SIG in future, I would really appreciate that.

1. push the result for review or directly to koji in a single spec rebuild, that generates a binary golist command package
2. do not have any macro deployment in the pathway when not a single line of macro code changed

Again purism, it doesn't matter in most cases(we shouldn't have any generated macros that would change with each rebuild).

And there’s nothing vague in the proposed changed, it's just one project = one spec file which has always been the Fedora best practice

I'm not saying that it is vague, I'm saying it is not worth the effort as we have more pressing issues to work on than increasing level of theoretical perfection.

And the end-result for people using the macros or golist is the same, they get the same files on-system, with the same packages, and the same package inter-dependencies. Except golist is split, which should have been the case from day one, and the srpm and rpm Go macro packages are synchronized cleanly, which again should have been the case from day one.

I think that this would be good change, golist might have its application on its own. And I don't believe anything prevents you on working on this, especially if this breaks your use case so much. I will be happy do do the reviews.

@jcajka I’m not saying you are incompetent, I’m saying you are focusing on the bit of the process you use, and completely forgetting the bit of process I use. So everything you dismiss as “purism” because it does not impact too much your workflow, makes me waste time every single day I try to do Go stuff for Fedora.
And yes BTW the current setup you defend so much means submitting two separate packages to Bodhi for every change, and if the result in not deployed synchronously you get potential build failures. More hilariously it means submitting existing macros as bodhi changes every single time you need to rebuild golist for another Go compiler version.
And I’m not given up on guidelining. I even have a ticker FCP-side on how to do it best. Based on how things have evolved this past year, and based on how it has evolved in the 10 years I've maintained Fedora packaging guidelines for fonts packages, the next version will skip the wiki wall no one reads and just move to example templates.
I'm sick of wasting hours formatting pages of wiki, just so people don't bother to read them and ask me directly “what’s on the wiki again” ? They want directly applicable templates with no long explanations, they'll get those, and the templates are actually less work to create and maintain, than the wiki page is.

One more note, if the issue is just the formatting, I will be happy to format what you will write. But I'm more afraid that they are asking you, because the split created more FUD and nobody is really sure what the hell is going on(it is hard to follow even for me).

My use case is just:
1. make Go macro fixes or enhancements
2. push the result for review or directly to koji in a single spec rebuild, that generates synchronised subpackages
3. do not have any Go code builds in the pathway when not a single line of Go code changed

This point is pure purism, it is good to aim for it, but it might be very negligible to rebuild some code, actually matter of seconds(incomparable to the regular koji build overhead).

Unfortunately, no, the code Jan wrote does not make mere seconds to build.
Plus, all the go compilation lines abscure what actually is changed in the spec.

Only issue might be "unnecessary" dependencies.

And conversely
1. make golist fixes or enhancements, or rebuild for new Go compiler version

Rebuilds in general are taken care by rel-eng(on the compiler maintainers request) in automated fashion. IMO there should be some level of the involvement of the SIG in future, I would really appreciate that.

golist does not work that way, Jan had to force a manual macro rebuild to get golist ready for Go 1.11. (that build is the latest one for the macro package in Koji)

push the result for review or directly to koji in a single spec rebuild, that generates a binary golist command package
do not have any macro deployment in the pathway when not a single line of macro code changed

Again purism

What you call purism is wasted time. And ince it's my time I care about it.

I'm not saying that it is vague, I'm saying it is not worth the effort as we have more pressing issues to work on than increasing level of theoretical perfection.

I’m saying it has already wasted me (so, not theorical purim qualifier) way more time than doing things properly. So let's do them properly.

My use case is just:
1. make Go macro fixes or enhancements
2. push the result for review or directly to koji in a single spec rebuild, that generates synchronised subpackages
3. do not have any Go code builds in the pathway when not a single line of Go code changed
This point is pure purism, it is good to aim for it, but it might be very negligible to rebuild some code, actually matter of seconds(incomparable to the regular koji build overhead).

Unfortunately, no, the code Jan wrote does not make mere seconds to build.
Plus, all the go compilation lines abscure what actually is changed in the spec.
Only issue might be "unnecessary" dependencies.

Point of the log is to preserve all the information if the Go compiler's output is some how removing the information from the log it is definitively bug, but if you are building something then you are expecting to see it recorded in the log. And I don't believe(I will be happy to checkout the log of your build) that that the Go build takes more than the minute as it has been AFAIK fairly minimal code base(compiler itself takes 10 min to build including the downloads and test suite run).

And conversely
1. make golist fixes or enhancements, or rebuild for new Go compiler version
Rebuilds in general are taken care by rel-eng(on the compiler maintainers request) in automated fashion. IMO there should be some level of the involvement of the SIG in future, I would really appreciate that.

golist does not work that way, Jan had to force a manual macro rebuild to get golist ready for Go 1.11. (that build is the latest one for the macro package in Koji)

IIRC because those have been last moment changes after the mass rebuild. So in general exception to the usual workflow to land big changes prior to it.

push the result for review or directly to koji in a single spec rebuild, that generates a binary golist command package
do not have any macro deployment in the pathway when not a single line of macro code changed
Again purism

What you call purism is wasted time. And ince it's my time I care about it.

I'm not saying that it is vague, I'm saying it is not worth the effort as we have more pressing issues to work on than increasing level of theoretical perfection.

I’m saying it has already wasted me (so, not theorical purim qualifier) way more time than doing things properly. So let's do them properly.

I still don't see how it is wasting your/our time in the context of fedora. Kind of don't understand why we are having this discussion if it it wasting you so much time, you could used that time to fix/propose fixes(actual changes/PRs) instead.

I already did it once. It was not applied ("oh, purism, no need to bother")

I will do it once more for the next macro level.

I already did it once. It was not applied ("oh, purism, no need to bother")
I will do it once more for the next macro level.

Link?

I don't think that this discussion will lead anywhere productive. Let's refocus. @nim could you please come up with the repo layout that you would prefer with mapping to the current upstream repos(or their parts)?

Hello Nicolas and the rest of the audience.

Topic of this issue is about moving relevant gofed repos (or their subsets) under pagure. Please, let's keep it this way and avoid unnecessary deviations from the topic.

In short, there is a lot of historical reasons for the current implementation of the symbols extractor. I don't see a point of dragging that history here and/or into the go packaging effort. Neither using it as supportive arguments for negative claims about the current situation. In case someone wants to fork it and make it better, feel free to do it. It's open source so you can treat it as such.

From my PoV I can reduce the conversation into discussion about golist. I am speaking only as a representative of the gofed project. I see Nicolas opened new issue [1] to separate the golist from go-compilers package. If that's going to help to improve the current situation, I have no objections against that. I would just suggest to create a new repo that will contain the golist implementation and will be maintained by this Go-sig community. It does not make sense to mirror entire https://github.com/gofed/symbols-extractor repo.

From my experience Nicolas was the driving force to introduce better macros including application of the go list command. I would be happy to see him as the main driving force for making the current golist implementation community driven. Nicolas, feel free to go ahead.

[1] https://src.fedoraproject.org/rpms/go-compilers/pull-request/3

@jchaloup Is there a reason we shouldn't consider moving gofed and the rest of that stuff that is part of what we use in Fedora Go to the Go SIG Pagure group?

I did not see much interested from anyone to continue with development of entire gofed stack. So I would assume it would be better to migrate/mirror only the relevant pieces. Which, at this point is only the go list functionality.

There is still go specfile generator under gofed/gofed. Though, the packaging guidelines are now in a quantum state and the repo has not been maintained for a while. Unless we get at least two folks on the board and restart the development, mirroring abandoned repo will not be practical.

@ngompa I think we should agree(as a SIG) if we want to continue on using and developing the gofed(IMO even just for the record, so we know where to put our efforts and focus).

@jchaloup Hi; nice to see you with us again

I've answered your questions in the PR. Basically, you need a separate golist subpackage to have a simple upgrade path to a separate golist srpm, otherwise this srpm will conflict with the macros package as soon as it is build.

So, basically:
1. make golist its own subpackage
2. create a separate golist package that obsoletes the subpackage
3. build it
4. remove golist build traces from the macro package

@jcajka

My preference is just

1. one pagure project to host all the Go macro code and centralize issues fixes and prs.
2. one project, on pagure if jchaloup is ok with it, to host the golist source and centralize issues fixes and prs.

And then:

1. one Fedora srpm, to package the first project
2. one Fedora srpm, to package the second project

I've no major opinion on the second project, I was never a golist developer and I'm not going to give advice on how to do it to golist developers.

I'd like the first project to be done according to Fedora best practices so the rehosting and spec-in can be done in a single operation, without arguing for months on naming and layout, and without redoing it as soon as it is finished because someone finds a problem.

My current understanding of Fedora naming and best practices is here:
https://copr.fedorainfracloud.org/coprs/nim/go-macros-3/build/812419/
(not finished yet I need to add a rpm spec template subpackage)

It's probably imperfect because a lot of this stuff has never been written down, and you need loads of Fedora know-how to avoid making mistakes. Therefore, I've asked FPC to take a stance on the ideal layout Fedora macro writers should apply when they are not satisfied with their macro package state and want to clean it up.
https://pagure.io/packaging-committee/issue/813

I hope they take the time to clarify things.

My preference is just

• one pagure project to host all the Go macro code and centralize issues fixes and prs.
• one project, on pagure if jchaloup is ok with it, to host the golist source and centralize issues > fixes and prs.

SGTM. Let's start with that. Once the projects exist, let's flip the current go packages in question to get the source tarball from the about-to-be created pagure repos.

I quickly read the rest of your plans. Let's start with something without unnecessary arguing about the rest. One iteration, once we flip to it, we can re-iterate and move to the next (to-be) item.

Anyway, that is done now:

• https://pagure.io/go-rpm-macros
• https://pagure.io/golist

That required quite a lot of git surgery to untangle the parts that didn't belong in the target projects, and to graft the histories of all the repositories involved in the right place. Some of the branches conflicted with one another (commits in already deprecated repos and so on). Plus, manual copying of all the issues scattered right and left.

Anyone interested: please check I didn't forget anything important (commits or issues). Once the repositories start seeing some use, it will be very hard to rework git history again.

I’ll open a separate ticket to track how to shift the existing Fedora packages to the new clean pagure repositories, without breaking anything in the process. The macro state is more recent than the one in Fedora right now, it has seen lots of fixes en enhancing, so it can not be plopped in the distro just like that.

@nim Can we move these under the GoSIG namespace? That makes it easier to keep track of.

@ngompa I don't know how to do that in Pagure :( I don't see any change namespace option in the wall of project settings (but I may have missed it).

And to be honest, I quite hate all the namespacing people do on GitHub, the namespaces are not stable and their changes break the tooling all the time. And what the IETF RFCs say about casing, and what tool writers often do, differs so cased URL are a pit of snakes.

@ngompa If you go to GoSIG pagure will list all the projects associated with the SIG. Is that not sufficient to keep track of things?

Since you’re part of the SIG you have a direct link to this page in groups.

@nim We'd like to have everything under the folder namespace of the GoSIG, too, similar to how the Rust SIG does: https://pagure.io/projects/fedora-rust/%2A

@ngompa: apart from the fact that will leak ACLs in urls, create cased URLs that won’t work in all tools, break links in issues right and left, and adds nothing over GoSIG, where is the control in pagure to change a project namespace?

It leaks no ACLs, and if cased URLs aren't going to work, perhaps we should revisit having a cased Pagure group in the first place. The whole point was for everything to be under the SIG's namespace, so that it's all collected in one spot.

Well, it is all collected in one spot here. It has all the elements you pointed on the rust sig page, and more.

But at the risk of repeating myself, assuming the SIG wanted to go go further and do what you propose: where are the settings in pagure to do so?

Because killing the pagure group and projects, and redoing everything again by hand, from repos to issues, without losing too much history, it a least a full day work, and will lose all the existing issue exchanges and links.

Let's close it, since no one provided the magic to move a project inside a SIG namespace, and the main objective (get it all in pagure) was finished a long time ago