#82 Add a first sketch of how to build modules in Copr
Merged 3 years ago by mcurlej. Opened 3 years ago by frostyx.
fedora-docs/ frostyx/modularity building-modules-in-copr  into  master

empty or binary file added
empty or binary file added
empty or binary file added
file modified
+13 -11
@@ -12,17 +12,19 @@ 

  ** xref:installing-modules.adoc[Installation and Discovery]

  ** xref:using-modules-switching-streams.adoc[Switching Streams]

  ** xref:removing-modules.adoc[Removing Modules]

- * xref:making-modules.adoc[Making Modules — Guides]

- ** xref:making-modules/adding-new-modules.adoc[Adding New Modules]

- ** xref:making-modules/updating-modules.adoc[Updating Existing Modules]

- ** xref:making-modules/defining-modules.adoc[Defining Modules in modulemd]

- ** xref:making-modules/building-modules.adoc[Building Modules]

- ** xref:making-modules/managing-defaults.adoc[Managing Defaults]

- ** xref:making-modules/inspecting-build-failures.adoc[Inspecting Build Failures]

- ** xref:making-modules/building-modules-locally.adoc[Building Modules Locally]

- * xref:making-modules-policies.adoc[Making Modules — Policies]

- ** xref:making-modules/packaging-guidelines.adoc[Packaging Policy]

- ** xref:making-modules/naming-guidelines.adoc[Naming Policy]

+ * xref:building-modules/index.adoc[Building Modules]

+ ** xref:building-modules/fedora/index.adoc[Fedora]

+ *** xref:building-modules/fedora/adding-new-modules.adoc[Adding New Modules]

+ *** xref:building-modules/fedora/updating-modules.adoc[Updating Existing Modules]

+ *** xref:building-modules/fedora/defining-modules.adoc[Defining Modules in modulemd]

+ *** xref:building-modules/fedora/building-modules.adoc[Building Modules]

+ *** xref:building-modules/fedora/managing-defaults.adoc[Managing Defaults]

+ *** xref:building-modules/fedora/inspecting-build-failures.adoc[Inspecting Build Failures]

+ ** xref:building-modules/copr/building-modules-in-copr.adoc[COPR]

+ ** xref:building-modules/local/building-modules-locally.adoc[Localhost]

+ * xref:policies/index.adoc[Policies]

+ ** xref:policies/packaging-guidelines.adoc[Packaging Policy]

+ ** xref:policies/naming-guidelines.adoc[Naming Policy]

  * xref:faq.adoc[FAQ]

  

  * **Advanced**

@@ -0,0 +1,201 @@ 

+ = Building modules in Copr

+ 

+ 

+ == Introduction

+ 

+ Copr ("Community projects") is a service that builds your open-source projects and creates your own

+ RPM repositories. Read the https://docs.pagure.org/copr.copr/user_documentation.html[documentation]

+ to find out more or see it in action https://copr.fedorainfracloud.org/[here].

+ 

+ There are multiple ways how to build modules in Copr and it is up to each user to choose the most

+ suitable one for their specific use-case.

+ 

+ 1. Do you know what modulemd yaml file is and do you have it written for your module?

+    Then <<Submit an existing modulemd yaml,submit an existing modulemd yaml>> to Copr.

+ 2. Do you have a Copr project providing some RPM packages and want to make module from them?

+    Then don't write a modulemd yaml file manually and

+     <<Generate modulemd from a Copr project,generate it from a Copr project>>.

+ 

+ Discover more advanced topics such as <<Modules with Copr packages,requiring Copr packages>> from an existing modulemd

+ yaml file, <<Modules in Copr as dependencies,depending on modules from Copr>>, and

+ <<Override module packages,overriding module packages by regular ones>>.

+ 

+ 

+ == Disclaimer

+ 

+ Copr is a third-party service with its own implementation of the module build process. It does not

+ rely on the https://pagure.io/fm-orchestrator[Module Build Service^] and it is not governed by xref:community.adoc[Fedora Modularity team].

+ As a consequence, some features might behave differently.

+ 

+ Some features are also in rather experimental or proof-of-concept state (e.g. depending on other

+ Copr modules) and therefore might be cumbersome to use. Feedback from users is more than welcome and

+  it is the deciding factor of what is going to be implemented or improved. Please submit your

+ requests to https://pagure.io/copr/copr/issues[pagure issue tracker].

+ 

+ Additionally, only versions 1 and 2 of the modulemd format are supported.

+ Version 3 is not supported yet.

+ 

+ 

+ == Submit an existing modulemd yaml

+ 

+ Currently, there is no web UI for submitting a module build from an existing modulemd yaml file.

+ The only way is to use `copr-cli` package. Make sure it is installed.

+ 

+ ----

+ dnf install copr-cli

+ ----

+ 

+ And make sure that an https://copr.fedorainfracloud.org/api/[API token is properly configured].

+ 

+ To submit a build from locally stored modulemd yaml file, use:

+ 

+ ----

+ copr-cli build-module --yaml /some/path/testmodule.yaml <COPR project name>

+ ----

+ 

+ If the modulemd yaml file is available on a URL accessible for public, use following:

+ 

+ ----

+ copr-cli build-module --url https://example/some/path/testmodule.yaml <COPR project name>

+ ----

+ 

+ To specify a different project owner (e.g. when building into a group project), use:

+ 

+ ----

+ copr-cli build-module --yaml /some/path/testmodule.yaml @mygroup/testmodule

+ ----

+ 

+ For more information, see `copr-cli build-module --help` or `man copr-cli`.

+ 

+ 

+ == Generate modulemd from a Copr project

+ 

+ This method aims to provide modularity features with the lowest entry barrier possible. It doesn't

+ demand a user to know either how to write modulemd yaml file nor using the command line. The only

+ prerequisite is to have a Copr project with some package successfully built in it.

+ 

+ Open your project, click on `Modules`, then click to `New Module`. The input page looks like this.

+ 

+ image::copr-build-new-module.png[,100%,]

+ 

+ It provides a list of all successfully built packages in the project. By default, they are all

+ selected to become part of the module. Uncheck those that are not wanted. Optionally specify

+ https://pagure.io/modulemd/blob/ade28f3f3b39fcddcb626ca915df1a6ce35c14fd/f/spec.yaml#_137[a module API]

+ or its https://pagure.io/modulemd/blob/ade28f3f3b39fcddcb626ca915df1a6ce35c14fd/f/spec.yaml#_90[profiles].

+ 

+ 

+ == Modules with Copr packages

+ 

+ It is trivial to achieve when generating modulemd from Copr project. This section

+ describes a special case of how to add a Copr package into an existing custom modulemd yaml file.

+ 

+ Update `components.rpms` section and add a new package definition.

+ 

+ ----

+ hello:

+     rationale: Showing how to use a package from Copr

+     ref: ...

+     repository: ...

+ ----

+ 

+ The `...` placeholders need to be replaced with real values. Open a web browser and navigate

+ to a build detail whose results should be used by the module. The build needs to be finished

+ successfully. Scroll down to the `Results` section.

+ 

+ image::copr-build-result.png[,100%,]

+ 

+ - Use the hash value from the `Dist Git Source` column as a `ref` in the modulemd.

+ - Follow the link of the `Dist Git Source` value to a Copr distgit instance. Use the URL as

+   a `repository` in modulemd - with some small adjustments.

+     * Trim everything after `.git`

+     * Replace `/cgit/` with `/git/`

+ 

+ image::copr-disgit.png[,100%,]

+ 

+ To avoid confusion, here are some example values.

+ 

+ ----

+ hello:

+     rationale: Showing how to use a package from Copr

+     ref: 9d1ced1

+     repository: http://copr-dist-git.fedorainfracloud.org/git/frostyx/hello/hello.git

+ ----

+ 

+ 

+ == Modules in Copr as dependencies

+ 

+ At this moment, Copr doesn't properly parse module dependencies from modulemd yaml file. They need

+ to be explicitly enabled in the project settings - more precisely chroot settings within a project.

+ 

+ When the dependency comes from Fedora repositories, simply edit `Enable module` field and append

+ the module in `name:stream` format.

+ 

+ If the dependency was built in Copr, it is necessary to edit also the `Repos` field and provide a

+ repository URL from which the module can be installed. See the <<Installing modules from Copr>>

+ section to learn how to find a repofile for a module (in this case the module that is used

+ as a dependency). Use it's `baseurl` as the value for `Repos` field.

+ 

+ 

+ == Installing modules from Copr

+ 

+ Currently `dnf copr` plugin does not support enabling module repositories and therefore it needs to

+ be done manually. Navigate to a Copr project in a web browser, see the `Modules` page, and open

+ detail of a module that is to be installed. If the build is still running, wait until it

+ finishes. For successfully built modules, a "How to use" section is displayed. Follow its

+ instructions. After that, standard DNF commands for manipulation with modules can be used.

+ 

+ See https://docs.fedoraproject.org/en-US/modularity/installing-modules/

+ 

+ 

+ == Override module packages

+ 

+ The package version resolution was straightforward in the pre-modularity era. If multiple

+ repositories provided the same package, the one with the highest version was preferred.

+ It is https://dnf.readthedocs.io/en/latest/modularity.html#package-filtering[more complicated now].

+ If there is a module stream enabled within the system and this stream provides a package, it is

+ preferred over its non-modular variant regardless of its version.

+ 

+ To suppress this behavior, `module_hotfixes` repositories were invented. For them, DNF upgrades to a

+ higher version of a package regardless of where it comes from. Enable them in Copr by going to

+ project settings and turning on this checkbox.

+ 

+ ----

+ [ ] This repository contains module hotfixes

+     This will make packages from this project available on along with

+     packages from the active module streams.

+ ----

+ 

+ Or in a command line

+ 

+ ----

+ copr-cli modify frostyx/foo --module-hotfixes on

+ ----

+ 

+ 

+ == Examples

+ 

+ Build a module from a modulemd yaml file hosted online:

+ 

+ ----

+ copr-cli create testproject1 --chroot fedora-rawhide-x86_64

+ copr-cli build-module testproject1 \

+     --url https://src.fedoraproject.org/modules/httpd/raw/2.4/f/httpd.yaml

+ ----

+ 

+ Alternatively, build a module from a localy stored modulemd yaml file:

+ 

+ ----

+ fedpkg clone -a modules/httpd

+ cd httpd

+ copr-cli create testproject2 --chroot fedora-rawhide-x86_64

+ copr-cli build-module testproject2 --yaml httpd.yaml

+ ----

+ 

+ 

+ == External resources

+ 

+ - http://frostyx.cz/posts/how-to-build-modules-in-copr

+ - http://frostyx.cz/posts/modules-with-copr-packages

+ - http://frostyx.cz/posts/copr-loves-modularity

+ - http://frostyx.cz/posts/module-hotfixes-in-copr

+ - http://frostyx.cz/posts/copr-modularity-in-retrospect

modules/ROOT/pages/building-modules/fedora/adding-new-modules.adoc modules/ROOT/pages/making-modules/adding-new-modules.adoc
file renamed
+2 -2
@@ -70,7 +70,7 @@ 

  

  === Write and push the modulemd

  

- Writing a modulemd is covered in detail in the xref:making-modules/defining-modules.adoc[Defining Modules in modulemd] section.

+ Writing a modulemd is covered in detail in the xref:building-modules/fedora/defining-modules.adoc[Defining Modules in modulemd] section.

  

  NOTE: The name of the modulemd must match the name of the repository. Remember to replace every `NAME` in the following example.

  
@@ -91,7 +91,7 @@ 

  

  NOTE: NOTE: With Modularity, you no longer build individual packages. Instead, you need to submit a module build.

  

- Submitting module builds is done using `fedpkg` and is covered in the xref:making-modules/building-modules.adoc[Building modules] section.

+ Submitting module builds is done using `fedpkg` and is covered in the xref:building-modules/fedora/building-modules.adoc[Building modules] section.

  

  == Publishing the Module

  

modules/ROOT/pages/building-modules/fedora/building-modules.adoc modules/ROOT/pages/making-modules/building-modules.adoc
file renamed
+1 -1
@@ -53,7 +53,7 @@ 

  

  * ``all`` — All packages in the module get rebuilt.

  * ``only-changed`` — Only packages that have changed since the last successful build get rebuilt. This is the **default** in Fedora.

- * ``changed-and-after`` — This one leverages xref:making-modules/defining-modules.adoc#_building_in_a_specific_order_optional[build groups (buildorder)]. Packages that have changed since the last successful build get rebuilt, and also all packages with a buildorder higher than any of the changed ones get rebuilt as well.

+ * ``changed-and-after`` — This one leverages xref:building-modules/fedora/defining-modules.adoc#_building_in_a_specific_order_optional[build groups (buildorder)]. Packages that have changed since the last successful build get rebuilt, and also all packages with a buildorder higher than any of the changed ones get rebuilt as well.

  

  For more detailed information, please see the

  https://pagure.io/fm-orchestrator/blob/master/f/docs/REBUILD_STRATEGIES.rst[MBS rebuild strategies documentation].

modules/ROOT/pages/building-modules/fedora/defining-modules.adoc modules/ROOT/pages/making-modules/defining-modules.adoc
file renamed
file was moved with no change to the file
@@ -0,0 +1,5 @@ 

+ = Building Modules for Fedora

+ 

+ If you wish to maintain an additional version of an application, language stack, or any other piece of software, modules might be the way to go for you.

+ 

+ The xref:building-modules/fedora/adding-new-modules.adoc[Adding New Modules] covers the whole process of adding a new module to Fedora. 

\ No newline at end of file

modules/ROOT/pages/building-modules/fedora/inspecting-build-failures.adoc modules/ROOT/pages/making-modules/inspecting-build-failures.adoc
file renamed
file was moved with no change to the file
modules/ROOT/pages/building-modules/fedora/managing-defaults.adoc modules/ROOT/pages/making-modules/managing-defaults.adoc
file renamed
file was moved with no change to the file
modules/ROOT/pages/building-modules/fedora/updating-modules.adoc modules/ROOT/pages/making-modules/updating-modules.adoc
file renamed
+1 -1
@@ -35,7 +35,7 @@ 

  

  NOTE: NOTE: With Modularity, you no longer build individual packages. Instead, you need to submit a module build.

  

- Submitting module builds is done using `fedpkg` and is covered in the xref:making-modules/building-modules.adoc[Building modules] section.

+ Submitting module builds is done using `fedpkg` and is covered in the xref:building-modules/fedora/building-modules.adoc[Building modules] section.

  

  == Publishing the Module

  

@@ -0,0 +1,5 @@ 

+ = Building Modules

+ 

+ 1. **xref::building-modules/fedora/index.adoc[Fedora]**

+ 2. **xref::building-modules/copr/building-modules-in-copr.adoc[COPR]**

+ 3. **xref::building-modules/local/building-modules-locally.adoc[Localhost]** 

\ No newline at end of file

modules/ROOT/pages/building-modules/local/building-modules-locally.adoc modules/ROOT/pages/making-modules/building-modules-locally.adoc
file renamed
file was moved with no change to the file
@@ -1,5 +0,0 @@ 

- = Making Modules for Fedora

- 

- If you wish to maintain an additional version of an application, language stack, or any other piece of software, modules might be the way to go for you.

- 

- The xref:making-modules/adding-new-modules.adoc[Adding New Modules] covers the whole process of adding a new module to Fedora. 

\ No newline at end of file

modules/ROOT/pages/policies/index.adoc modules/ROOT/pages/making-modules-policies.adoc
file renamed
file was moved with no change to the file
modules/ROOT/pages/policies/naming-guidelines.adoc modules/ROOT/pages/making-modules/naming-guidelines.adoc
file renamed
file was moved with no change to the file
modules/ROOT/pages/policies/packaging-guidelines.adoc modules/ROOT/pages/making-modules/packaging-guidelines.adoc
file renamed
file was moved with no change to the file

This is not a finished text.
@mcurlej asked me for a sketch that he can test and polish into a final version.
So, here it is.

I am adding you as a collaborator to my fork @mcurlej, so you can make changes to this PR.

Adding and inviting @msuchy to review this PR. Related issue modularity#127

FYI, if you're looking for some inspiration, I wrote https://sgallagh.wordpress.com/2019/08/14/sausage-factory-modules-fake-it-till-you-make-it/ a while back that may be of interest.

1 new commit added

  • Update COPR tutorial and organization of the wiki
3 years ago

+1, thank you for finishing it @mcurlej

rebased onto 4402dbe

3 years ago

Pull-Request has been merged by mcurlej

3 years ago
Metadata
Changes Summary 18
+0
file added
modules/ROOT/assets/images/copr-build-new-module.png
+0
file added
modules/ROOT/assets/images/copr-build-result.png
+0
file added
modules/ROOT/assets/images/copr-disgit.png
+13 -11
file changed
modules/ROOT/nav.adoc
+201
file added
modules/ROOT/pages/building-modules/copr/building-modules-in-copr.adoc
+2 -2
file renamed
modules/ROOT/pages/making-modules/adding-new-modules.adoc
modules/ROOT/pages/building-modules/fedora/adding-new-modules.adoc
+1 -1
file renamed
modules/ROOT/pages/making-modules/building-modules.adoc
modules/ROOT/pages/building-modules/fedora/building-modules.adoc
+0 -0
file renamed
modules/ROOT/pages/making-modules/defining-modules.adoc
modules/ROOT/pages/building-modules/fedora/defining-modules.adoc
+5
file added
modules/ROOT/pages/building-modules/fedora/index.adoc
+0 -0
file renamed
modules/ROOT/pages/making-modules/inspecting-build-failures.adoc
modules/ROOT/pages/building-modules/fedora/inspecting-build-failures.adoc
+0 -0
file renamed
modules/ROOT/pages/making-modules/managing-defaults.adoc
modules/ROOT/pages/building-modules/fedora/managing-defaults.adoc
+1 -1
file renamed
modules/ROOT/pages/making-modules/updating-modules.adoc
modules/ROOT/pages/building-modules/fedora/updating-modules.adoc
+5
file added
modules/ROOT/pages/building-modules/index.adoc
+0 -0
file renamed
modules/ROOT/pages/making-modules/building-modules-locally.adoc
modules/ROOT/pages/building-modules/local/building-modules-locally.adoc
-5
file removed
modules/ROOT/pages/making-modules.adoc
+0 -0
file renamed
modules/ROOT/pages/making-modules-policies.adoc
modules/ROOT/pages/policies/index.adoc
+0 -0
file renamed
modules/ROOT/pages/making-modules/naming-guidelines.adoc
modules/ROOT/pages/policies/naming-guidelines.adoc
+0 -0
file renamed
modules/ROOT/pages/making-modules/packaging-guidelines.adoc
modules/ROOT/pages/policies/packaging-guidelines.adoc