#1216 Updates to SourceURL page
Merged 2 years ago by tibbs. Opened 2 years ago by oturpe.
oturpe/packaging-committee fix-issue-1215  into  master

@@ -1,8 +1,14 @@ 

  = Referencing Source

  

- One of the design goals of rpm is to cleanly separate upstream source from vendor modifications. For the Fedora packager, this means that sources used to build a package should be the vanilla sources available from upstream. To help reviewers and QA scripts verify this, the packager needs to indicate where a reviewer can find the source that was used to make the rpm.

+ One of the design goals of rpm is to cleanly separate upstream source from vendor modifications.

+ For the Fedora packager, this means that sources used to build a package should be the vanilla sources available from upstream.

+ To help reviewers and QA scripts verify this,

+ the packager needs to indicate where a reviewer can find the source that was used to make the rpm.

  

- The most common case is where upstream distributes source as a tar.gz, tar.bz2 or zip archive that we can download from an upstream website. In these cases you must use a full URL to the package in the SourceX: line. For example:

+ The most common case is where upstream distributes source as a `tar.gz`, `tar.bz2` or `zip` archive

+ that we can download from an upstream website.

+ In these cases you must use a full URL to the package in the `Source:` line.

+ For example:

  

  ....

  Source: https://downloads.sourceforge.net/%{name}/%{name}-%{version}.tar.gz
@@ -13,24 +19,29 @@ 

  [NOTE]

  .Smallest Compressed Archive

  ======

- If the upstream source archive is available in multiple compressed formats that our tools can decompress it's best to use the one that is smallest in size.

+ If the upstream source archive is available in multiple compressed formats that our tools can decompress,

+ it's best to use the one that is smallest in size.

  This ensures the smallest source rpm to save space on the mirrors and downloads of source RPM packages.

  ======

  

- 

- There are several cases where upstream is not providing the source to you in an upstream tarball. In these cases you must document how to generate the tarball used in the rpm either through a spec file comment or a script included as a separate SourceX:.

- 

- Here are some specific examples:

- 

  == Using Forges (Hosted Revision Control)

  

- Any software publishing website, permitting the download of source archives via normalized URLs, that can be deduced from a project root URL and version, commit, tag, scm, extension… values is a _“forge”_ that can be supported by the `redhat-rpm-config` `+%{forgemeta}+` macro. Common Forge examples are _GitLab_ and _GitHub_. `+%{forgemeta}+` centralizes and abstracts our knowledge about those forges, so packagers do not have to handle download quirks manually.

+ Any software publishing website, permitting the download of source archives via normalized URLs,

+ that can be deduced from a project root URL and version, commit, tag, scm, extension… values is a _“forge”_

+ that can be supported by the `redhat-rpm-config` `+%{forgemeta}+` macro.

+ Common Forge examples are _GitLab_ and _GitHub_.

+ `+%{forgemeta}+` centralizes and abstracts our knowledge about those forges,

+ so packagers do not have to handle download quirks manually.

  

- `+%{forgemeta}+` makes it easy to switch from release to tag to commit source archives. Using `+%{forgemeta}+`, forge download URLs or guideline changes are propagated to spec files without manual refactoring.

+ `+%{forgemeta}+` makes it easy to switch from release to tag to commit source archives.

+ Using `+%{forgemeta}+`,

+ forge download URLs or guideline changes are propagated to spec files without manual refactoring.

  

- When those changes result in a different naming or structure of the source archive, the source file needs to be uploaded to the build system before rebuilding existing spec files. That is the main drawback of using `+%{forgemeta}+`.

+ When those changes result in a different naming or structure of the source archive,

+ the source file needs to be uploaded to the build system before rebuilding existing spec files.

+ That is the main drawback of using `+%{forgemeta}+`.

  

- The following examples are taken from the `redhat-rpm-templates` package, that provides their latest version.

+ The following examples show how to use the Forge macros in various situations.

  

  === Release Example

  
@@ -74,7 +85,10 @@ 

  

  == Using Revision Control

  

- In some cases you may want to pull sources from upstream's revision control system because there have been many changes since the last release and you think that a tarball that you generate from there will more accurately show how the package relates to upstream's development. Here's how you can use a comment to show where the source came from:

+ In some cases you may want to pull sources from upstream's revision control system

+ because there have been many changes since the last release

+ and you think that a tarball that you generate from there will more accurately show how the package relates to upstream's development.

+ Here's how you can use a comment to show where the source came from:

  

  ....

  # The source for this package was pulled from upstream's vcs.  Use the
@@ -84,12 +98,16 @@ 

  Source: foo-20070221.tar.xz

  ....

  

- When pulling from revision control, please remember to use a Name-version-release compatible with the xref:Versioning.adoc[Versioning] Guidelines. In particular, check the section on xref:Versioning.adoc#_complex_versioning[Complex versioning].

+ When pulling from revision control,

+ please remember to use a Name-version-release compatible with the xref:Versioning.adoc[Versioning] Guidelines.

+ In particular, check the section on xref:Versioning.adoc#_complex_versioning[Complex versioning].

  

  [#when-upstream-uses-prohibited-code]

  == When Upstream uses Prohibited Code

  

- Some upstream packages include patents or trademarks that we are not allowed to ship even as source code. In these cases you have to modify the source tarball to remove this code before you even upload it to the build system. Here's an example of using a script to document how you went from the upstream tarball to the one included in the package:

+ Some upstream packages include patents or trademarks that we are not allowed to ship even as source code.

+ In these cases you have to modify the source tarball to remove this code before you even upload it to the build system.

+ Here's an example of using a script to document how you went from the upstream tarball to the one included in the package:

  

  From the spec:

  
@@ -120,7 +138,7 @@ 

  == Python Packages (pypi)

  

  As PyPI has moved to storing files in directories which change depending on the file being stored,

- it is rather unpleasant to use in a Source: URL.

+ it is rather unpleasant to use in a `Source:` URL.

  Instead, files.pythonhosted.org can be used trough the `+%{pypi_source}+` macro.

  

  ....
@@ -137,15 +155,21 @@ 

  Source: https://downloads.sourceforge.net/%{name}/%{name}-%{version}.tar.gz

  ....

  

- changing ".tar.gz" to whatever matches the upstream distribution. Note that we are using downloads.sourceforge.net instead of an arbitrarily chosen mirror. You may use the package name/package version instead of the %\{name} and %\{version} macros, of course.

+ changing `.tar.gz` to whatever matches the upstream distribution.

+ Note that we are using `downloads.sourceforge.net` instead of an arbitrarily chosen mirror.

+ You may use the package name/package version instead of the `+%{name}+` and `+%{version}+` macros, of course.

  

  Please note that the correct url is `+downloads.sourceforge.net+`, and *NOT* `+download.sourceforge.net+`.

  

  == Git Hosting Services

  

- If the upstream *does* create tarballs you should use them as tarballs provide an easier trail for people auditing the packages.

+ If the upstream *does* create tarballs you should use them

+ as tarballs provide an easier trail for people auditing the packages.

  

- Git web-based hosting services provide a mechanism to create tarballs on demand, either from a specific commit revision, or from a specific tag. If the upstream does not create tarballs for releases, you can use this mechanism to produce them.

+ Git web-based hosting services provide a mechanism to create tarballs on demand,

+ either from a specific commit revision, or from a specific tag.

+ If the upstream does not create tarballs for releases,

+ you can use this mechanism to produce them.

  

  The full 40-character hash and associated git tag may be obtained by issuing the following git command:

  
@@ -157,7 +181,8 @@ 

  PROJECT:          upstream project name (if it's identical to the package name, use %{name} instead)

  ....

  

- You may also obtain the 40-character hash and associated git tag via the web-interface of the HOSTING-SERVICE, or by cloning the repository and issuing the https://git-scm.com/docs/git-show-ref[git show-ref] command.

+ You may also obtain the 40-character hash and associated git tag via the web-interface of the HOSTING-SERVICE,

+ or by cloning the repository and issuing the https://git-scm.com/docs/git-show-ref[git show-ref] command.

  

  Once the commit hash and git tag are known, you can define them in your spec file as follows:

  
@@ -185,15 +210,28 @@ 

  %autosetup -n PROJECT-%{commit}              [GitLab]

  ....

  

- If the release corresponds to a git tag with a sane numeric version, you must use that version to populate the Version: tag in the spec file. If it does not, look at the source code to see if a version is indicated there, and use that value. If no numeric version is indicated in the code, you may set Version to 0, and treat the package as a "pre-release" package (and make use of the %\{shortcommit} macro). See xref:Versioning.adoc#_prerelease_versions[Prerelease versions] for details.

+ If the release corresponds to a git tag with a sane numeric version,

+ you must use that version to populate the `Version:` tag in the spec file.

+ If it does not, look at the source code to see if a version is indicated there,

+ and use that value.

+ If no numeric version is indicated in the code, you may set `Version` to 0,

+ and treat the package as a "pre-release" package

+ (and make use of the `+%{shortcommit}+` macro).

+ See xref:Versioning.adoc#_prerelease_versions[Prerelease versions] for details.

  

- Alternately, if you are using a specific revision that is either a pre-release revision or a post-release revision, you must follow the "snapshot" guidelines. They are documented here: xref:Versioning.adoc#_snapshots[Snapshots]. You can substitute %\{shortcommit} for the git hash in %\{checkout} in that section.

+ Alternately, if you are using a specific revision that is either a pre-release revision or a post-release revision,

+ you must follow the "snapshot" guidelines.

+ They are documented here: xref:Versioning.adoc#_snapshots[Snapshots].

+ You can substitute `+%{shortcommit}+` for the git hash in `+%{checkout}+` in that section.

  

  === Git Tags

  

- https://git-scm.com/docs/git-tag[Git tags] represent a particular code point that upstream deems important; and are typically used to mark release points.

+ https://git-scm.com/docs/git-tag[Git tags] represent a particular code point that upstream deems important;

+ and are typically used to mark release points.

  

- Bitbucket uses the %\{shortcommit} identifier as part of the archive directory structure; regardless of whether you use git tag or Commit Revision to retrieve it. This is shown in the %prep section example.

+ Bitbucket uses the `+%{shortcommit}+` identifier as part of the archive directory structure;

+ regardless of whether you use git tag or Commit Revision to retrieve it.

+ This is shown in the %prep section example.

  

  For the source tarball, you can use the following syntax:

  
@@ -211,19 +249,30 @@ 

  

  == Using %\{version}

  

- Using %\{version} in the SourceX: makes it easier for you to bump the version of a package, because most of the time you do not need to edit SourceX: when editing the spec file for the new package.

+ Using `+%{version}+` in the `Source:` makes it easier for you to bump the version of a package,

+ because most of the time you do not need to edit `Source:` when editing the spec file for the new package.

  

  == Troublesome URLs

  

- When upstream has URLs for the download that do not end with the tarball name rpm will be unable to parse the tarball out of the source URL. One workaround for many cases is to construct a URL where the tarball is listed in a "URL fragment":

+ When upstream has URLs for the download that do not end with the tarball name

+ rpm will be unable to parse the tarball out of the source URL.

+ One workaround for many cases is to construct a URL where the tarball is listed in a "URL fragment":

  

  ....

  Source: https://example.com/foo/1.0/download.cgi#/%{name}-%{version}.tar.gz

  ....

  

- rpm will then use %\{name}-%\{version}.tar.gz as the tarball name. If you use `+spectool -g foo.spec+` to download the tarball, it will rename the tarball for you.

+ rpm will then use `+%{name}-%{version}.tar.gz+` as the tarball name.

+ If you use `+spectool -g foo.spec+` to download the tarball,

+ it will rename the tarball for you.

  

- Sometimes this does not work because the upstream cgi tries to parse the fragment or because you need to login or fill in a form to access the tarball. In these cases, you have to put just the tarball's filename into the Source: tag. To make clear where you got the tarball, you should leave notes in comments above the Source: line to explain the situation to reviewers and future packagers. For example:

+ Sometimes this does not work because the upstream cgi tries to parse the fragment

+ or because you need to login or fill in a form to access the tarball.

+ In these cases, you have to put just the tarball's filename into the `Source:` tag.

+ To make clear where you got the tarball,

+ you should leave notes in comments above the `Source:` line

+ to explain the situation to reviewers and future packagers.

+ For example:

  

  ....

   # Mysql has a mirror redirector for its downloads
@@ -234,7 +283,9 @@ 

  

  == Do not conditionalize Sources

  

- When building an SRPM, it is critical that all of the sources are present, irrespective of the platform on which the SRPM is generated. For example, the following code  *does not work* (the way you might expect):

+ When building an SRPM, it is critical that all of the sources are present,

+ irrespective of the platform on which the SRPM is generated.

+ For example, the following code  *does not work* (the way you might expect):

  

  ....

  %if 0%{?fedora} < 35
@@ -246,9 +297,13 @@ 

  %endif

  ....

  

- If you were to build this SRPM from a Fedora 34 host (`rpmbuild -bs mysource.spec`), the resulting SRPM would carry `mysource-old.tar.bz2` and `oldpatch.patch`. However, if you then try to use this SRPM to build for Fedora 35, the operation would fail because `mysource-ng.tar.bz2` is not available.

+ If you were to build this SRPM from a Fedora 34 host (`rpmbuild -bs mysource.spec`),

+ the resulting SRPM would carry `mysource-old.tar.bz2` and `oldpatch.patch`.

+ However, if you then try to use this SRPM to build for Fedora 35,

+ the operation would fail because `mysource-ng.tar.bz2` is not available.

  

- The correct behavior is to always carry all of the sources that might be used in the build and then conditionalize the *usage* of them instead. For example:

+ The correct behavior is to always carry all of the sources that might be used in the build

+ and then conditionalize the *usage* of them instead. For example:

  

  ```

  Source1: mysource-old.tar.bz2
@@ -266,4 +321,4 @@ 

  tar xf %{SOURCE2}

  %patch2 -p1

  %endif

- ``` 

\ No newline at end of file

+ ```

Following changes are each in their own commit:

  • Apply semantic linebreaks
  • Improve formatting
  • Remove reference to redhat-rpm-templates
  • Remove incorrect description of following content

Resolves #1215

Pull-Request has been merged by tibbs

2 years ago

I wonder whether we should actually remove (or at least de-emphasize) the documentation for "forge" macros given that they've been basically unmaintained for years.

Metadata