#506 Guideline Draft: Service First-Time Setup
Closed None Opened 9 years ago by sgallagh.

I've been working on a new package guidelines draft[1] for dealing with packages that provide a service and need some level of first-time configuration before the service can run.

One of the issues we're dealing with in the world of Fedora Atomic and other environments where VMs or systems are cloned is the issue of keeping system-specific data out of those clones. In particular, we want to make sure that clones of a system don't have the same private keys or certificates as its siblings.

Classically, the way that many services set up this configuration is during the %post phase of RPM installation; they create whatever certificates, etc. they need at this time and then the service will run when it is started. Admins will set up their systems with the packages they want and then run a tool like {{{virt-sysprep}}} to clear out system-specific information. The problem with this approach is that in many cases, this results in a system that cannot run many of its services without additional steps being taken on the new cloned VM to re-generate these components.

This proposed set of guidelines provides two major new changes to this process:
1. It requires that all system-specific generated files are moved into the service start itself and out of %post. This means that any time the files needed are not present, they are generated at service start time.
1. It provides a detailed description of a secure process to produce "self-signed" service certificates for bootstrapping the services. This follows a newer approach to generating certificates that allows safe importing of the certificates for use on the local system (and even for sharing that certificate with other machines in the event that a proper certificate chain is unavailable, such as many non-production environments).

Once these guidelines are approved, I will also be developing helper scripts to accomplish the certificate generation so that packagers will have an easier time following this guideline.

The OpenSSL portions of this guideline were written by me and reviewed by Kai Engert and Miloslav Trmac. The NSS portions were written by Kai Engert and reviewed by myself and Miloslav Trmac.

[1] https://fedoraproject.org/wiki/User:Sgallagh/FirstTimeSetupDraft


I'm having trouble understanding how the admittedly useful tutorials on the use of OpenSSL and NSS belong in the packaging guidelines.

It is of course reasonable to document what packagers need to do to get these initial setup things into their packages. But I'd suggest that the certificate stuff be pushed off into some helper scripts somewhere and only examples showing the use of those scripts actually placed into the guidelines.

Replying to [comment:1 tibbs]:

I'm having trouble understanding how the admittedly useful tutorials on the use of OpenSSL and NSS belong in the packaging guidelines.

It is of course reasonable to document what packagers need to do to get these initial setup things into their packages. But I'd suggest that the certificate stuff be pushed off into some helper scripts somewhere and only examples showing the use of those scripts actually placed into the guidelines.

Well, I'm working on said scripts as well, but I wanted the methodology approved as a guideline, mostly because then there's a design to point at in the reviews as opposed to "just use this script that we're telling you to trust", which doesn't go over well in security circles.

I'm amenable to splitting these into two separate pages if you feel it's confusing. That being said, it was certain packages' treatment of certificates that prompted the creation of this proposal in the first place. (Basically, there are a bunch of packages whose self-signed certificates and private keys are often getting quietly duplicated around, leading to potential attack vectors). I wrote the guideline to be more general, since there are other situations that can benefit from it, but ultimately the most important one is that of generated private keys.

We discussed this at today's meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-03-05/fpc.2015-03-05-17.00.txt):

  • 506 Guideline Draft: Service First-Time Setup (geppetto, 18:12:26)

  • LINK: https://fedorahosted.org/fpc/ticket/506 (geppetto, 18:12:30)
  • ACTION: sgallagh Need to solve the localhost problem (geppetto,
    18:36:46)
  • ACTION: sgallagh include a better "Here's what you have to do"
    section. (geppetto, 18:41:17)

Proposed guidelines require creation of a script. Systemd provides the various ConditionXXXX= settings which can be used to skip execution of units. Implementing setup as a separate unit with some ConditionXXXX= settings has some advantages:
- no need to shell out and spawn processes.
- status is better reported. systemctl status will say 'ConditionXXXX not met at time and date'.
- no need for a custom protocol in most cases, so nothing new to learn for administrators.
- systemctl mask / systemctl preset can be used to disable the setup unit if necessary

It is true that for some services some more complex setup is necessary, but in many cases a few ExecStart= lines will be enough, so no separate script will be needed in addition to the unit file.

If a helper script to create self-signed certificates is implemented, which would be extremely useful in general, I'm sure that many such setup units will be basically a call to this script, fitting in one line in a unit file.

Replying to [comment:5 zbyszek]:

Proposed guidelines require creation of a script. Systemd provides the various ConditionXXXX= settings which can be used to skip execution of units. Implementing setup as a separate unit with some ConditionXXXX= settings has some advantages:
- no need to shell out and spawn processes.
- status is better reported. systemctl status will say 'ConditionXXXX not met at time and date'.
- no need for a custom protocol in most cases, so nothing new to learn for administrators.
- systemctl mask / systemctl preset can be used to disable the setup unit if necessary

It is true that for some services some more complex setup is necessary, but in many cases a few ExecStart= lines will be enough, so no separate script will be needed in addition to the unit file.

So are you suggesting that instead of creating an extension unit file with ExecStartPre, we should instead create an entirely new service unit that has Before= and RequiredBy= for the original service it applies to?

My concern with this is that these scripts REALLY aren't services. They should not (must not!) continue to be running after they perform this setup. That to me seems perfectly like the duty of ExecStartPre. The only advantage I see to this approach is to avoid the fork (which is not a huge issue during startup for a short-lived process) and the ability to mask the startup with systemctl, but this is not something I expect anyone to do often enough to make it worth doing.

I don't understand what this line means: "no need for a custom protocol in most cases, so nothing new to learn for administrators". What protocol are you talking about?

If a helper script to create self-signed certificates is implemented, which would be extremely useful in general, I'm sure that many such setup units will be basically a call to this script, fitting in one line in a unit file.

I've been working on a script to create the self-signed certificates: https://github.com/sgallagher/sscg

It's currently going through a security audit by the Red Hat security team before I update the draft to include it. It's also up for package review at https://bugzilla.redhat.com/show_bug.cgi?id=1202604

Replying to [comment:7 sgallagh]:

So are you suggesting that instead of creating an extension unit file with ExecStartPre, we should instead create an entirely new service unit that has Before= and RequiredBy= for the original service it applies to?

Yes.

My concern with this is that these scripts REALLY aren't services. They should not (must not!) continue to be running after they perform this setup. That to me seems perfectly like the duty of ExecStartPre.

We already have a number of precedents. systemd-fsck@.service is a good example: something needs to be done before the real thing (mounting the fs) happens. The unit has Type=oneshot, RemainAfterExit=yes. The helper does its job and exits, and the service is in state active/exited. This also means that if the main unit is stopped and started again, the preparatory unit is not run again.

The question whether something should be a separate service or just a ExecStartPre line doesn't have a clear cut answer. If it is something trivial and fast, like copying a file or creating a directory, then ExecStartPre is fine. If it is something that can fail on it's own, or something that the administrator might want to disable/re-run/tweak on its own, then probably a separate script is in order. It seems that things like generating certificates fall into this latter category.

I don't understand what this line means: "no need for a custom protocol in most cases, so nothing new to learn for administrators". What protocol are you talking about?

By "protocol" I meant the rules where to place the helper script, what return value it should have, etc. Not a big deal, but just a bit more convention to learn.

Replying to [comment:8 zbyszek]:

Replying to [comment:7 sgallagh]:

So are you suggesting that instead of creating an extension unit file with ExecStartPre, we should instead create an entirely new service unit that has Before= and RequiredBy= for the original service it applies to?

Yes.

My concern with this is that these scripts REALLY aren't services. They should not (must not!) continue to be running after they perform this setup. That to me seems perfectly like the duty of ExecStartPre.

We already have a number of precedents. systemd-fsck@.service is a good example: something needs to be done before the real thing (mounting the fs) happens. The unit has Type=oneshot, RemainAfterExit=yes. The helper does its job and exits, and the service is in state active/exited. This also means that if the main unit is stopped and started again, the preparatory unit is not run again.

That's actually exactly the opposite of what we want here. We really do want it to check on each startup for the presence of these files. The goal here is that we need to be able to operate stably in the presence of tools like virt-sysprep. The use-case is this: I build up a VM to my needs, then run virt-sysprep and snapshot the VM in its clean state, then distribute this VM all over my environment. I need to be assured that if I start up the services, they will always generate their machine-private information at that time. If I have to manually stop/restart the initial-config pre-service, I've gained nothing here. (Of course, this is a bit of an edge-case; hopefully people would normally snapshot the system while it was powered off, but I've seen many cases where they don't, so I'd prefer to account for it).

The question whether something should be a separate service or just a ExecStartPre line doesn't have a clear cut answer. If it is something trivial and fast, like copying a file or creating a directory, then ExecStartPre is fine. If it is something that can fail on it's own, or something that the administrator might want to disable/re-run/tweak on its own, then probably a separate script is in order. It seems that things like generating certificates fall into this latter category.

Well, a script will be created to do the work here; I wouldn't recommend putting a huge amount of logic into the unit file itself. That's the wrong place for it. That's why the document describes dropping such a script in a well-known location ({{{/usr/lib/systemd/system-init/}}}, location subject to change at the FPC's recommendation)

I don't understand what this line means: "no need for a custom protocol in most cases, so nothing new to learn for administrators". What protocol are you talking about?

By "protocol" I meant the rules where to place the helper script, what return value it should have, etc. Not a big deal, but just a bit more convention to learn.

Well, the location doesn't have to be fixed (though my current draft strongly recommends a location, see above). As for the return value, it just needs to be zero (success) or nonzero (failure). I think that's a pretty well-established convention already.

Replying to [comment:9 sgallagh]:

My concern with this is that these scripts REALLY aren't services. They should not (must not!) continue to be running after they perform this setup. That to me seems perfectly like the duty of ExecStartPre.

We already have a number of precedents. systemd-fsck@.service is a good example: something needs to be done before the real thing (mounting the fs) happens. The unit has Type=oneshot, RemainAfterExit=yes. The helper does its job and exits, and the service is in state active/exited. This also means that if the main unit is stopped and started again, the preparatory unit is not run again.

That's actually exactly the opposite of what we want here. We really do want it to check on each startup for the presence of these files. The goal here is that we need to be able to operate stably in the presence of tools like virt-sysprep. The use-case is this: I build up a VM to my needs, then run virt-sysprep and snapshot the VM in its clean state, then distribute this VM all over my environment. I need to be assured that if I start up the services, they will always generate their machine-private information at that time. If I have to manually stop/restart the initial-config pre-service, I've gained nothing here. (Of course, this is a bit of an edge-case; hopefully people would normally snapshot the system while it was powered off, but I've seen many cases where they don't, so I'd prefer to account for it).

If you want the preparatory service to be restarted every time the main service is started, set RemainAfterExit=no.

The question whether something should be a separate service or just a ExecStartPre line doesn't have a clear cut answer. If it is something trivial and fast, like copying a file or creating a directory, then ExecStartPre is fine. If it is something that can fail on it's own, or something that the administrator might want to disable/re-run/tweak on its own, then probably a separate script is in order. It seems that things like generating certificates fall into this latter category.

Well, a script will be created to do the work here; I wouldn't recommend putting a huge amount of logic into the unit file itself. That's the wrong place for it.

There are two parts of logic: one is whether to run the script at all, and the second is the actual setup done by the script. The first belongs in the unit file. The second may be complicated enough to merit its own script, but it may also be a single call to sscg or another helper, in which case wrapping it in a script is overkill.

What I don't like about currently proposed approach is that it mandates that a script must be run to check the state at every launch of the service. This is going back to sysvinit, and a bad pattern.

Replying to [comment:10 zbyszek]:

If you want the preparatory service to be restarted every time the main service is started, set RemainAfterExit=no.

OK, that might work. I'll do a little experimentation on that.

What I don't like about currently proposed approach is that it mandates that a script must be run to check the state at every launch of the service. This is going back to sysvinit, and a bad pattern.

I'm not married to the current approach, but it seemed to do what I wanted with the least amount of packager effort. I'll run a few experiments with your recommendations above and see if I can get them to work in the same way. I'll update this ticket once I have done so.

So there was a whole lot of discussion above, and some relatively recent changes to the draft but I'm not sure if this is ready for FPC to consider yet. Please let us know.

Yeah, this work is ongoing. I'm going to rewrite some portions of it based on zbyszek's suggestions and I've also scrapped my plans for the sscg for generating certificates. Instead, we're enhancing certmonger (a standard piece of our certificate story for FreeIPA, SSSD and AD integration) to be able to generate certificates in this style (and also monitor them).

So for now, please leave this in NEEDINFO until I update this ticket indicating that it should be discussed again. Thanks for following up.

So it's been about three months now; I'm going to go ahead and close this out to get it out of our work list but feel free to reopen if you would like to pursue this further.

Sorry for the long delay on this. Just finally got back to it today (closing this ticket helped me get myself in gear...)

I've basically rewritten the guidelines from the ground up, taking Zbigniew's recommendations:

Current version:
https://fedoraproject.org/wiki/User:Sgallagh/FirstTimeSetupDraft

Changes:
https://fedoraproject.org/w/index.php?title=User%3ASgallagh%2FFirstTimeSetupDraft&diff=HEAD&oldid=410351

The last bit of that draft links to another draft. My memory may be faulty, but I don't think we've talked about that one. We do have https://fedoraproject.org/wiki/Packaging:SSLCertificateHandling but that's about something completely different.

Replying to [comment:18 tibbs]:

The last bit of that draft links to another draft. My memory may be faulty, but I don't think we've talked about that one. We do have https://fedoraproject.org/wiki/Packaging:SSLCertificateHandling but that's about something completely different.

That was previously included in this document (see the change diff). I pulled it out and just left a link to it to explain the sscg approach. If you prefer, I'll drop it entirely.

+1 to the guidelines. Seems reasonable to me.

We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-07-16/fpc.2015-07-16-16.00.txt):

  • 506 Guideline Draft: Service First-Time Setup (geppetto, 16:07:32)

  • LINK: https://fedorahosted.org/fpc/ticket/506 (geppetto, 16:07:32)
  • ACTION: sgallagh sscg should be in F22 before we publish policy
    saying to use it. (geppetto, 17:07:52)

Those guidelines look very reasonable :)

Some nitpicking:
- I think the conditions are described wrong. From the manpage:

If multiple conditions are specified, the unit will be executed if all of them apply
(i.e. a logical AND is applied). Condition checks can be prefixed with a pipe symbol (|)
in which case a condition becomes a triggering condition. If at least one triggering
condition is defined for a unit, then the unit will be executed if at least one of the
triggering conditions apply and all of the non-triggering conditions.

  • After=syslog.target is obsolete, and should be removed from the example.

/path/to/config/script is not explained anywhere. In the example the command is embedded
inline, which is certainly preferable, but probably will not be possible in all cases.
I think the guidelines should say that inline commands are preferred.
Shouldn't the guidelines also suggest a place for those scripts?

Replying to [comment:23 zbyszek]:

Those guidelines look very reasonable :)

Some nitpicking:
- I think the conditions are described wrong. From the manpage:

If multiple conditions are specified, the unit will be executed if all of them apply
(i.e. a logical AND is applied). Condition checks can be prefixed with a pipe symbol (|)
in which case a condition becomes a triggering condition. If at least one triggering
condition is defined for a unit, then the unit will be executed if at least one of the
triggering conditions apply and all of the non-triggering conditions.

I read the manpage. I thought that's exactly how I described it above. If not, could you suggest an alternative? What I wrote seems to work as far as I can tell.

  • After=syslog.target is obsolete, and should be removed from the example.

Done.

/path/to/config/script is not explained anywhere. In the example the command is embedded
inline, which is certainly preferable, but probably will not be possible in all cases.
I think the guidelines should say that inline commands are preferred.
Shouldn't the guidelines also suggest a place for those scripts?

I added a few additional sentences to provide additional explanation of the script.

Changes: https://fedoraproject.org/w/index.php?title=User%3ASgallagh%2FFirstTimeSetupDraft&diff=418294&oldid=417884

Your wording seems to imply that Condition=a Condition=|b results in an OR between them. But it's an AND between all conditions without-|, and an OR between all conditions with-|, and finally and AND between the two groups if they are non-empty. I took the liberty to directly update your text: https://fedoraproject.org/w/index.php?title=User%3ASgallagh%2FFirstTimeSetupDraft&diff=418304&oldid=418294.

Ahh, I understand now. Thanks for the sharp eyes!

We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-07-23/fpc.2015-07-23-16.01.txt):

  • 506 Guideline Draft: Service First-Time Setup (geppetto, 16:32:13)

  • LINK: https://fedorahosted.org/fpc/ticket/506 (geppetto, 16:32:13)
  • ACTION: Service First-Time Setup guildlines (+1:5, 0:0, -1:0)
    (geppetto, 17:04:52)
  • Some edits were made during discussions (geppetto, 17:05:11)

I was on PTO during the meeting. The edits look good (and thanks for clearing up that typo in the Pegasus example). I haven't written the manpage for sscg yet; I will try to do that today or tomorrow and will provide a link.

Guidelines have been added covering services which need to perform setup when they are first started (including self-signed certificate generation).

The guideline seems silent about who is responsible for the cleanup of the system specific setup files. Is it still a responsibility of RPM? Or, would there be a systemd way of doing the cleanup?

Replying to [comment:31 ravindrakumar]:

The guideline seems silent about who is responsible for the cleanup of the system specific setup files. Is it still a responsibility of RPM? Or, would there be a systemd way of doing the cleanup?

Could you be more specific? What cleanup are you referring to?

The idea behind this was more to enable tools that can clear system-specific content in order to reset to something like a "factory defaults" or "VM template". But the tools to actually wipe that content aren't important to this proposal; it just enables them by requiring packages that need auto-generated system-specific content to use a mechanism that can tolerate such a wipe.

Metadata Update from @sgallagh:
- Issue assigned to tibbs

7 years ago

Sorry for delay, I missed the update. Looks like there are no email notifications for the updates?

Could you be more specific? What cleanup are you referring to?

I meant to say, whatever is configured by first time setup needs to be cleaned up, e.g. if there are configuration files generated, those should be deleted. I believe right time to cleanup would be the RPM uninstall. When RPMs are no longer managing such configuration data directly, who will cleanup this one time setup once RPM is uninstalled? Doing it in RPM uninstall section will require hard coding assumptions in the RPM to do the cleanup because RPM would not know what needs to be cleaned up.

rpm has left any modified config files in place for. . .I'm not even sure how long. This, alongside generating .rpmnew versions of config noreplace, has been the expected behaviour for a very long time, and changing them would likely require a long process of expectation setting, changes to dnf, etc. What benefit would this bring, aside from saving a few K of disk space?

Metadata Update from @limb:
- Issue close_status updated to: None (was: Fixed)

7 years ago

Thanks @limb

I'm not making a recommendation to change anything. I'm just asking to better understand the processes/policies regarding 'first time setup'.

@limb I think what @ravindrakumar was getting at is that the files created by initial service setup should be created as %ghost %config(noreplace), so that if the package is completely removed, the associated generated configuration goes with it.

@ravindrakumar I omitted that from this particular page mostly because I figured it was covered by other packaging guidelines regarding how configuration files are packaged in Fedora, but I could see an argument for mentioning it on this page as well to reinforce it.

@limb, @tibbs If you think that's valuable, let me know and I'll draft something up.

Login to comment on this ticket.

Metadata