modularity / fedmod

Clone
Source Code
GIT

#70 Use a Jinja2 template to format the output
Closed 6 years ago by karsten. Opened 6 years ago by otaylor.
modularity/ otaylor/fedmod jinja2-template  into  master

file modified
+1 
@@ -3,3 +3,4 @@ 
 

  include LICENSE
 

  include modtools
 

  recursive-include modularity *
 

+ include _fedmod/templates/*.j2

file modified
+22 -2 
@@ -1,5 +1,8 @@ 
 

+ import jinja2
 

  import sys
 

  import modulemd
 

+ import pkg_resources
 

+ import yaml
 

  import logging
 

  from . import _depchase, _repodata
 

  
@@ -18,12 +21,21 @@ 
 

              remaining_rpm_deps.add(pkgname)
 

      return module_deps, remaining_rpm_deps
 

  

+ def sortitems_filter(items):
 

+     return sorted(items, key=lambda item: item[0])
 

+ 

+ def key_filter(obj):
 

+     return yaml.dump({obj: 'x'}, default_flow_style=False).strip()[:-3]
 

+ def value_filter(obj):
 

+     return yaml.dump({'x': obj}, default_flow_style=False).strip()[3:]
 

+ 

  class ModuleGenerator(object):
 

  

      def __init__(self, pkgs):
 

          self.pkgs = pkgs
 

          self.core_srpm = None
 

          self.mmd = modulemd.ModuleMetadata()
 

+         self.template = 'module.yaml.j2'
 

          self._pool = _depchase.make_pool("x86_64")
 

  

      def _calculate_dependencies(self, build_deps_iterations):
 
@@ -74,11 +86,19 @@ 
 

          Function writes modulemd to either stdout or the given filename
 

          :return:
 

          """
 

+ 

+         env = jinja2.Environment(loader=jinja2.PackageLoader('_fedmod', 'templates'),
 

+                                  autoescape=False)
 

+         env.filters['sortitems'] = sortitems_filter
 

+         env.filters['key'] = key_filter
 

+         env.filters['value'] = value_filter
 

+         template = env.get_template('module.yaml.j2')
 

+ 

          if output_fname is not None:
 

-             self.mmd.dump(output_fname)
 

+             template.stream(mmd=self.mmd).dump(output_fname)
 

              print('Generated modulemd file: %r' % output_fname)
 

          else:
 

-             print(self.mmd.dumps())
 

+             print(template.render(mmd=self.mmd))
 

          return True
 

  

      def _update_module_md(self):

file added
+48 
@@ -0,0 +1,48 @@ 
 

+ document: modulemd
 

+ version: 1
 

+ data:
 

+     summary: {{ mmd.summary|value }}
 

+     description:>
 

+         {{ mmd.description|indent(8) }}
 

+     license:
 

+         module:
 

+         - MIT
 

+     {%- if mmd.profiles %}
 

+     profiles:
 

+     {%- for name, profile in mmd.profiles.items() %}
 

+         {{ name|key }}:
 

+             rpms:
 

+             {%- for name in profile.rpms %}
 

+             - {{ name|value }}
 

+             {%- endfor %}
 

+     {%- endfor %}
 

+     {%- endif %}
 

+     api:
 

+         rpms:
 

+         {%- for name in mmd.api.rpms %}
 

+         - {{ name|value }}
 

+         {%- endfor %}
 

+     {%- if mmd.buildrequires or mmd.requires %}
 

+     dependencies:
 

+         {%- if mmd.buildrequires %}
 

+         buildrequires:
 

+         {%- for module, stream in mmd.buildrequires.items()|sortitems %}
 

+             {{ module|key }}: {{ stream|value }}
 

+         {%- endfor %}
 

+         {%- endif %}
 

+         {%- if mmd.requires %}
 

+         requires:
 

+         {%- for module, stream in mmd.requires.items()|sortitems %}
 

+             {{ module|key }}: {{ stream|value }}
 

+         {%- endfor %}
 

+         {%- endif %}
 

+     {%- endif %}
 

+     components:
 

+         rpms:
 

+             {%- for name, rpm in mmd.components.rpms.items()|sortitems %}
 

+             {{ name|key }}:
 

+                 {%- if rpm.buildorder %}
 

+                 buildorder: {{ rpm.buildorder|value }}
 

+                 {%- endif %}
 

+                 rationale: {{ rpm.rationale|value }}
 

+             {%- endfor -%}

file modified
+2 
@@ -26,6 +26,7 @@ 
 

      install_requires=[
 

          'modulemd',
 

          'click',
 

+         'jinja2',
 

          'requests',
 

          'requests-toolbelt',
 

          'lxml',
 
@@ -33,4 +34,5 @@ 
 

          'PyYAML',
 

      ],
 

      packages=find_packages(),
 

+     include_package_data=True,
 

  )

Instead of depending on modulemd functionality, format the output using
a Jinja2 template. While this is somewhat more cumbersome, it allows for
ordering and indenting the output as we want, and will allow comments
to be added in the future.

See: https://pagure.io/modularity/fedmod/issue/8

Have you looked at the behavior of libmodulemd with regards to output? It doesn't support comments, however it does enforce the ordering and indentation of the rest of the content in a human-understandable way. (Specifically, it guarantees that the ordering matches the specification).

If this patch is intended to resolve https://pagure.io/modularity/fedmod/issue/8, I think it may be redundant at this point.

My main concern was the unreadable ordering, which moving to libmodulemd would fix. Tried that (will file a PR once someone reviews #69) - seems to basically work. One thing is a bit harder is that when I added auto-generated description metadata based on the RPM description, as a hack I did:

    description:>
        {{ mmd.description|indent(8) }}

I took the the literal RPM description, wrapped as in the RPM metadata, indented it, and let the yaml rules unwrap paragraphs.... a hack, but leaves the yaml file and the result both pretty readable for normal RPM descriptions, and the module maintainer could fine tune. Getting the same result here will be harder - will require unwrapping the description in the fedmod code, or libyaml will escape the embedded newlines in the RPM description and produce an unreadable mess.

Original:

data:
  api:
    rpms:
    - eog
  components:
    rpms:
      eog:
        buildorder: 10
        rationale: Package in api
  dependencies:
    buildrequires:
      platform: f28
    requires:
      platform: f28
  description: Module auto-generated by fedmod
  license:
    module:
    - MIT
  summary: Generated module for eog
document: modulemd
version: 1

Jinja2 template:

document: modulemd
version: 1
data:
    summary: Generated module for eog
    description:>
        Module auto-generated by fedmod
    license:
        module:
        - MIT
    api:
        rpms:
        - eog
    dependencies:
        buildrequires:
            platform: f28
        requires:
            platform: f28
    components:
        rpms:
            eog:
                buildorder: 10
                rationale: Package in api

libmodulemd:

---
document: modulemd
version: 2
data:
  summary: Generated module for eog
  description: >-
    Module auto-generated by fedmod
  license:
    module:
    - MIT
  dependencies:
  - buildrequires:
      platform: [f28]
    requires:
      platform: [f28]
  api:
    rpms:
    - eog
  components:
    rpms:
      eog:
        rationale: Package in api
        buildorder: 10
...

My main concern was the unreadable ordering, which moving to libmodulemd would fix. Tried that (will file a PR once someone reviews #69) - seems to basically work. One thing is a bit harder is that when I added auto-generated description metadata based on the RPM description, as a hack I did:
description:>
{{ mmd.description|indent(8) }}

I took the the literal RPM description, wrapped as in the RPM metadata, indented it, and let the yaml rules unwrap paragraphs.... a hack, but leaves the yaml file and the result both pretty readable for normal RPM descriptions, and the module maintainer could fine tune. Getting the same result here will be harder - will require unwrapping the description in the fedmod code, or libyaml will escape the embedded newlines in the RPM description and produce an unreadable mess.

Could you post an example of this? It shouldn't escape the newlines and if it does, I made a mistake on the format for the description field, which I can fix. Right now the emitter uses YAML_FOLDED_SCALAR_STYLE (which is what was used in the spec.yaml) and that has the effect of turning newlines into spaces, but I could switch it to YAML_LITERAL_SCALAR_STYLE which would preserve the newlines.

See http://www.yaml.org/spec/1.2/spec.html#id2760844 for all the ways scalars can be represented.

libmodulemd is doing exactly what it's asked to - it's just preventing my hack from working - as it should.

data:
  summary: Eye of GNOME image viewer
  description: >-
    The Eye of GNOME image viewer (eog) is the official image viewer for the

    GNOME desktop. It can view single image files in a variety of formats, as

    well as large image collections.


    eog is extensible through a plugin system.

Using the literal style would look better in the file:

data:
  summary: Eye of GNOME image viewer
  description: |-
    The Eye of GNOME image viewer (eog) is the official image viewer for the
    GNOME desktop. It can view single image files in a variety of formats, as
    well as large image collections.

    eog is extensible through a plugin system.

But I think we don't want to say that description fields in modulemd files have random width wrapping in them! - let's take advantage of yaml's flexibility here and leave pre-wrapped descriptions for RPM.
Edited 6 years ago by otaylor

Do we still need this patch when we switch to libmodulemd ? This looks to me like libmodulemd needs just some minor fixes to output yaml with correct indentation and text wrapping.

karsten commented on the pull-request: Use a Jinja2 template to format the output that you are following:
Do we still need this patch when we switch to libmodulemd ? This looks to me like libmodulemd needs just some minor fixes to output yaml with correct indentation and text wrapping.

No, I think libmodulemd is fine without this. Now that you have the f28
update merged, I'll rebase my libmodulemd patch and post a PR for that.

Thanks for the clarification. I'll close this, then.

Pull-Request has been closed by karsten
6 years ago
Metadata
None
No Tags
Changes Summary 4
+1 -0
file changed
src/MANIFEST.in
+22 -2
file changed
src/_fedmod/module_generator.py
+48
file added
src/_fedmod/templates/module.yaml.j2
+2 -0
file changed
src/setup.py
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.