#20 Add a configurable syslog handler.
Merged 7 years ago by bowlofeggs. Opened 7 years ago by bowlofeggs.
bowlofeggs/fegistry 4  into  master

@@ -10,6 +10,7 @@ 

        - python3-mock

        - python3-nose

        - python3-nose-cov

+       - python3-PyYAML

        - python3-sphinx

  

  - name: Install the .bashrc

file modified
+2 -2
@@ -51,8 +51,8 @@ 

  

  # General information about the project.

  project = 'fegistry'

- copyright = '2016, Jeremy Cline'

- author = 'Jeremy Cline'

+ copyright = '2016, 2017 Red Hat, Inc.'

+ author = 'Red Hat, Inc.'

  

  # The version info for the project you're documenting, acts as replacement for

  # |version| and |release|, also used in various other places throughout the

@@ -0,0 +1,17 @@ 

+ Configuration

+ =============

+ 

+ fegistry can be configured through a `YAML <http://yaml.org/>`_ config file. By default, fegistry

+ will load its configuration from ``/etc/fegistry/fegistry.yaml``. You can override the configuration

+ path by setting its ``FEGISTRY_CONFIG`` environment variable to an alternate path, if you like.

+ fegistry has built-in defaults for all of its configuration items and will use the default for any

+ parameters not found in its config file, or if no config file is available for it to use.

+ 

+ As fegistry is a `Flask <http://flask.pocoo.org>`_ application, there is a set of

+ `built-in Flask settings <http://flask.pocoo.org/docs/0.12/config/#builtin-configuration-values>`_

+ that you can use in the config file. In addition to these settings, fegistry defines the following

+ settings:

+ 

+ * ``LOG_LEVEL``: The logging level that fegistry should use when logging to syslog. This accepts any

+   of the standard Python `logging levels <https://docs.python.org/3/library/logging.html#levels>`_,

+   and it will even upper case for you if you don't like shouting.

file modified
+1
@@ -13,6 +13,7 @@ 

     :maxdepth: 2

     :caption: Contents:

  

+    configuration

     contributing

     reference/index

  

@@ -0,0 +1,2 @@ 

+ ---

+ # LOG_LEVEL: warning

file added
+59
@@ -0,0 +1,59 @@ 

+ # Copyright ⓒ 2017 Red Hat, Inc.

+ # This file is part of fegistry.

+ #

+ # fegistry is free software: you can redistribute it and/or modify

+ # it under the terms of the GNU General Public License as published by

+ # the Free Software Foundation, either version 3 of the License, or

+ # (at your option) any later version.

+ #

+ # fegistry is distributed in the hope that it will be useful,

+ # but WITHOUT ANY WARRANTY; without even the implied warranty of

+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

+ # GNU General Public License for more details.

+ #

+ # You should have received a copy of the GNU General Public License

+ # along with fegistry.  If not, see <http://www.gnu.org/licenses/>.

+ """

+ This module contains code to initialize fegistry's app.config.

+ """

+ import logging.handlers

+ import os

+ 

+ import yaml

+ 

+ 

+ _DEFAULT_CONFIG = {'LOG_LEVEL': 'WARNING'}

+ 

+ 

+ def load(app):

+     """

+     Load fegistry's configuration into the given app's config attribute. It will first load the

+     default values and then will use either /etc/fegistry/fegistry.yaml, or alternatively a path

+     specified by the environment variable FEGISTRY_CONFIG_FILE, if provided.

+     """

+     app.config.from_mapping(_DEFAULT_CONFIG)

+ 

+     if 'FEGISTRY_CONFIG' in os.environ:

+         config_path = os.environ['FEGISTRY_CONFIG']

+     else:

+         config_path = '/etc/fegistry/fegistry.yaml'

+ 

+     if os.path.exists(config_path):

+         app.logger.info('Reading config from {}'.format(config_path))

+         with open(config_path) as config_file:

+             config = config_file.read()

+             config = yaml.safe_load(config)

+             if isinstance(config, dict):

+                 app.config.from_mapping(config)

+             else:

+                 app.logger.info(('Config file does not map to an associative array. Using default'

+                                  ' config only'))

+     else:

+         app.logger.info('Config file {} not found. Using default config'.format(config_path))

+ 

+     # Set up logging

+     app.config['LOG_LEVEL'] = app.config['LOG_LEVEL'].upper()

+     app.logger.info('Configuring a syslog handler at level: {}'.format(app.config['LOG_LEVEL']))

+     syslog_handler = logging.handlers.SysLogHandler(address='/dev/log')

+     syslog_handler.setLevel(getattr(logging, app.config['LOG_LEVEL']))

+     app.logger.addHandler(syslog_handler)

@@ -0,0 +1,132 @@ 

+ # Copyright ⓒ 2017 Red Hat, Inc.

+ # This file is part of fegistry.

+ #

+ # fegistry is free software: you can redistribute it and/or modify

+ # it under the terms of the GNU General Public License as published by

+ # the Free Software Foundation, either version 3 of the License, or

+ # (at your option) any later version.

+ #

+ # fegistry is distributed in the hope that it will be useful,

+ # but WITHOUT ANY WARRANTY; without even the implied warranty of

+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

+ # GNU General Public License for more details.

+ #

+ # You should have received a copy of the GNU General Public License

+ # along with fegistry.  If not, see <http://www.gnu.org/licenses/>.

+ """This test suite contains tests on fegistry.config."""

+ import logging

+ import mock

+ import unittest

+ 

+ import flask

+ 

+ from fegistry import config

+ 

+ 

+ class LoadTestCase(unittest.TestCase):

+     """A test class for testing the load() function."""

+     @mock.patch('fegistry.config.open',

+                 mock.mock_open(read_data='# Some comment\n---\n# LOG_LEVEL: ERROR'))

+     @mock.patch('fegistry.config.os.path.exists', return_value=True)

+     def test_empty_config_file(self, exists):

+         """Ensure that fegistry can handle an empty config file, and uses defaults in this case."""

+         app = flask.Flask('test')

+ 

+         with mock.patch.object(app.logger, 'info'):

+             with mock.patch.object(app.logger, 'addHandler'):

+                 config.load(app)

+ 

+                 self.assertEqual(app.logger.addHandler.call_count, 1)

+                 handler = app.logger.addHandler.mock_calls[0][1][0]

+ 

+             info_logs = '\n'.join(c[1][0] for c in app.logger.info.mock_calls)

+             self.assertEqual(

+                 info_logs,

+                 ('Reading config from /etc/fegistry/fegistry.yaml\nConfig file does not map to an '

+                  'associative array. Using default config only\nConfiguring a syslog handler at '

+                  'level: WARNING'))

+ 

+         exists.assert_called_once_with('/etc/fegistry/fegistry.yaml')

+         self.assertTrue(isinstance(handler, logging.handlers.SysLogHandler))

+         self.assertEqual(handler.address, '/dev/log')

+         # The default value of WARNING should have been used since the config file was just comments

+         self.assertEqual(handler.level, logging.WARNING)

+ 

+     @mock.patch('fegistry.config.open',

+                 mock.mock_open(read_data='# Some comment\n---\nLOG_LEVEL: ERROR'))

+     @mock.patch('fegistry.config.os.path.exists', return_value=True)

+     @mock.patch.dict('fegistry.config.os.environ', {'FEGISTRY_CONFIG': '/some/custom.yaml'})

+     def test_fegistry_config_env_var_provided(self, exists):

+         """Ensure correct operation when the FEGISTRY_CONFIG environment variable is provided."""

+         app = flask.Flask('test')

+ 

+         with mock.patch.object(app.logger, 'info'):

+             with mock.patch.object(app.logger, 'addHandler'):

+                 config.load(app)

+ 

+                 self.assertEqual(app.logger.addHandler.call_count, 1)

+                 handler = app.logger.addHandler.mock_calls[0][1][0]

+ 

+             info_logs = '\n'.join(c[1][0] for c in app.logger.info.mock_calls)

+             self.assertEqual(

+                 info_logs,

+                 ('Reading config from /some/custom.yaml\nConfiguring a syslog handler at level: '

+                  'ERROR'))

+ 

+         exists.assert_called_once_with('/some/custom.yaml')

+         self.assertTrue(isinstance(handler, logging.handlers.SysLogHandler))

+         self.assertEqual(handler.address, '/dev/log')

+         # The ERROR log level should have been used since the config file was configured as such.

+         self.assertEqual(handler.level, logging.ERROR)

+ 

+     @mock.patch('fegistry.config.os.path.exists', return_value=False)

+     def test_missing_config_file(self, exists):

+         """Ensure that a missing config file is handled gracefully, with a log message and the use

+            of defaults."""

+         app = flask.Flask('test')

+ 

+         with mock.patch.object(app.logger, 'info'):

+             with mock.patch.object(app.logger, 'addHandler'):

+                 config.load(app)

+ 

+                 self.assertEqual(app.logger.addHandler.call_count, 1)

+                 handler = app.logger.addHandler.mock_calls[0][1][0]

+ 

+             info_logs = '\n'.join(c[1][0] for c in app.logger.info.mock_calls)

+             self.assertEqual(

+                 info_logs,

+                 ('Config file /etc/fegistry/fegistry.yaml not found. Using default config\n'

+                  'Configuring a syslog handler at level: WARNING'))

+ 

+         exists.assert_called_once_with('/etc/fegistry/fegistry.yaml')

+         self.assertTrue(isinstance(handler, logging.handlers.SysLogHandler))

+         self.assertEqual(handler.address, '/dev/log')

+         # The default value of WARNING should have been used since the config file didn't exist

+         self.assertEqual(handler.level, logging.WARNING)

+ 

+     @mock.patch('fegistry.config.open',

+                 mock.mock_open(read_data='# Some comment\n---\nLOG_LEVEL: DEBUG'))

+     @mock.patch('fegistry.config.os.path.exists', return_value=True)

+     def test_with_config_file(self, exists):

+         """Ensure correct operation when the FEGISTRY_CONFIG environment variable is not

+            provided and the config file is not empty."""

+         app = flask.Flask('test')

+ 

+         with mock.patch.object(app.logger, 'info'):

+             with mock.patch.object(app.logger, 'addHandler'):

+                 config.load(app)

+ 

+                 self.assertEqual(app.logger.addHandler.call_count, 1)

+                 handler = app.logger.addHandler.mock_calls[0][1][0]

+ 

+             info_logs = '\n'.join(c[1][0] for c in app.logger.info.mock_calls)

+             self.assertEqual(

+                 info_logs,

+                 ('Reading config from /etc/fegistry/fegistry.yaml\nConfiguring a syslog handler at '

+                  'level: DEBUG'))

+ 

+         exists.assert_called_once_with('/etc/fegistry/fegistry.yaml')

+         self.assertTrue(isinstance(handler, logging.handlers.SysLogHandler))

+         self.assertEqual(handler.address, '/dev/log')

+         # The DEBUG log level should have been used since the config file was configured as such.

+         self.assertEqual(handler.level, logging.DEBUG)

file modified
+4 -1
@@ -1,4 +1,4 @@ 

- # Copyright ⓒ 2016 Red Hat, Inc.

+ # Copyright ⓒ 2016, 2017 Red Hat, Inc.

  # This file is part of fegistry.

  #

  # fegistry is free software: you can redistribute it and/or modify
@@ -15,8 +15,11 @@ 

  # along with fegistry.  If not, see <http://www.gnu.org/licenses/>.

  import flask

  

+ from fegistry import config

+ 

  

  app = flask.Flask(__name__)

+ config.load(app)

  

  

  @app.after_request

file modified
+2 -2
@@ -1,5 +1,5 @@ 

  #!/usr/bin/env python3

- # Copyright ⓒ 2016 Red Hat, Inc.

+ # Copyright ⓒ 2016, 2017 Red Hat, Inc.

  # This file is part of fegistry.

  #

  # fegistry is free software: you can redistribute it and/or modify
@@ -47,6 +47,6 @@ 

      long_description=README, classifiers=CLASSIFIERS, license=LICENSE, maintainer=MAINTAINER,

      maintainer_email=MAINTAINER_EMAIL, platforms=PLATFORMS, url=URL, keywords='fedora',

      packages=find_packages(exclude=('fegistry.tests', 'fegistry.tests.*')),

-     include_package_data=True, zip_safe=False, install_requires=['flask'],

+     include_package_data=True, zip_safe=False, install_requires=['flask', 'PyYAML'],

      tests_require=['flake8', 'mock', 'nose', 'nose-cov'],

      test_suite="nose.collector")

This commit adds a syslog handler to fegistry. It also adds a
simple YAML configuration system to fegistry, and uses it to allow
the user to configure the syslog handler's log level.

Signed-off-by: Randy Barlow randy@electronsweatshop.com

FYI this can be used to call Python functions if the YAML is malicious. Obviously if someone has written malicious YAML to /etc/fegistry/ you're in trouble anyway, but the path is configurable via an environment variable...

yaml.safe_load recognizes only standard YAML tags and cannot construct an arbitrary Python object so you might want to use it here.

I'm sure it is a super class, but is it a superclass?

So this happens with the default Flask logging configuration, right?

The docs say that it logs to stderr if the application is in debug mode. Does it log to anything if the application isn't in debug mode?

Out of curiosity, why syslog? I've been logging to stdout/err and letting systemd handle logging that.

Yeah at this point we haven't set up our own logger yet. I could set up a logger at the INFO level before loading the config, and then change the logger's log level after we've read the config and know what the user wanted it set to if you think that would be better. However, if a user is having trouble getting it to load a config file it might make sense to load it in debug mode anyway. What do you think?

There are a few reasons I think logging to syslog directly is still better:

  1. Logging to stdout/err only allows you two log levels, but logging to syslog gives us quite a few (debug, info, warning, error), which lets users filter out fairly granularly after the fact if they want. Of course, this also makes configuring the log level a little moot, but the setting will still control how much data is sent to syslog so I think it's still a useful setting,
  2. I don't want to assume all users will be using systemd, even though I love systemd ☺
  3. Sending logs to syslog can also be more flexible for using centralized log collectors, or rsyslog (for users who prefer alternate loggers).
  4. I'm not totally sure that all web servers would pass stdout/err on to their parent process (maybe they do?)

rebased

7 years ago

@jcline I've fixed the copy/paste docblock error (oops) and I used yaml.safe_load() (good catch, I didn't know about that!) I also left some responses to some of your other comments.

How does this look now?

Looks good to me :thumbsup:

Pull-Request has been merged by bowlofeggs

7 years ago