PR#99: Add several improvements to kiskadee - kiskadee

		`@@ -6,7 +6,9 @@`
		`stage('Create Virtualenv') {`
		`steps {`
		`sh 'virtualenv -p /usr/bin/python3 .'`
		`- sh 'source bin/activate && pip install -e .'`
		`+ sh 'source bin/activate && pip install -e .'`
		`+ sh "source bin/activate && pip install fedmsg[consumers]"`
		`+ sh "source bin/activate && pip install -r test_requirements.txt"`
		`}`
		`}`

		`@@ -18,7 +20,7 @@`

		`stage('Run Tests') {`
		`steps {`
		`- sh "source bin/activate && pip install coverage && make check"`
		`+ sh "source bin/activate && make test"`
		`}`
		`post {`
		`success {`

Makefile

file modified

+15 -3

		`@@ -2,12 +2,24 @@`
		`SHELL:=/bin/bash`

		`help:`
		`- @printf "Available targets: check, analyzers, clean\n\n"`
		`+ @printf "Available targets: test, analyzers, clean\n\n"`

		`- check:`
		`- coverage run --omit="lib/","setup.py","kiskadee/tests/",".eggs/",".venv/" ./setup.py test`
		`+ test:`
		`+ coverage run --omit="lib/","setup.py","kiskadee/tests/",".eggs/",".venv/","/usr/" -m unittest kiskadee/tests//*.py`
		`coverage html`

		`+ test_units:`
		`+ python3 -m unittest kiskadee/tests/units/*.py`
		`+`
		`+ test_integration:`
		`+ python3 -m unittest kiskadee/tests/integration/*.py`
		`+`
		`+ test_api:`
		`+ python3 -m unittest kiskadee/tests/api/*.py`
		`+`
		`+ test_plugins:`
		`+ python3 -m unittest kiskadee/tests/plugins/*.py`
		`+`
		`analyzers:`
		`docker ps 2> /dev/null; \`
		`if [ $$? -ne 1 ]; then \`

README.md

file modified

+13

		`@@ -11,6 +11,14 @@`
		`environment setup, refer to the full [kiskadee`
		`documentation](https://docs.pagure.org/kiskadee).`

		`+ To build the documentation locally:`
		`+`
		`+ pip install -U sphinx sphinx\_rtd\_theme`
		`+ cd doc`
		`+ make html`
		`+`
		`+ Open the index.html file, inside the _build/html directory.`
		`+`
		`## Repositories`

		`kiskadee core and API development are hosted at [pagure](https://pagure.io/kiskadee).`
		`@@ -34,6 +42,11 @@`

		`Join us on IRC: #kiskadee @ freenode`

		+ Mailing list: `kiskadee@googlegroups.com`
		`+`
		`+ You can subscribe to our mailing lists by sending an email to`
		+ `kiskadee+subscribe@googlegroups.com`.
		`+`
		`## License`
		`Copyright (C) 2017 the AUTHORS (see the AUTHORS file)`

Vagrantfile

file modified

+3 -1

		`@@ -29,7 +29,9 @@`
		`# https://developer.fedoraproject.org/tools/vagrant/vagrant-nfs.html`
		`# After enable nfs-server, you will need to start the service.`
		`core.vm.synced_folder "./", "/home/vagrant/kiskadee", type: "nfs", nfs_udp: false`
		`- core.vm.network "public_network", ip: ips[:core]`
		`+ core.vm.network "public_network", ip: ips[:core],`
		`+ :bridge => 'virbr0', :dev => 'virbr0'`
		`+ core.vm.network "forwarded_port", guest: 5000, host: 5000`
		`core.vm.provider "libvirt" do \|v\|`
		`v.memory = 2048`
		`end`

doc/_static/kiskadee_arch.png

file modified

doc/architecture.rst

file modified

+1 -1

		`@@ -37,7 +37,7 @@`


		`.. autoclass:: kiskadee.monitor.Monitor()`
		`- :members: monitor`
		`+ :members:`

		`Runner - Module to run static analysis`
		`--------------------------------------`

doc/development.rst

file modified

+4 -4

		`@@ -1,10 +1,10 @@`
		`kiskadee development`
		`====================`

		`- This section is a guide for new developers willing to`
		`- set up a development environemnt to start contributing to`
		`- kiskadee development. If you have any doubt, please contact`
		`- us on IRC in #kiskadee at freenode.`
		`+ This section is a guide for new developers willing to set up a development`
		`+ environemnt to start contributing to kiskadee development. If you have any`
		`+ doubt, please contact us on IRC in #kiskadee at freenode or join the`
		`+ development discussion in our mailing list at kiskadee@googlegroups.com.`

		`**kiskadee development only suports python versions >= 3. Assume such versions`
		`for all commands run along this documentation.**`

doc/fetchers.rst

file modified

+28 -24

		`@@ -21,30 +21,34 @@`

		`.. code-block:: python`

		`- import kiskadee`
		`- import sys`
		`- import kiskadee.queue`
		`- class Fetcher(kiskadee.fetchers.Fetcher):`
		`- def get_sources(self, source_data):`
		`- return 'kiskadee/tests/test_source/test_source.tar.gz'`
		`-`
		`- @kiskadee.queue.package_enqueuer`
		`- def watch(self):`
		`- """There is no proper API to inspect new example versions.`
		`- It should not matter, since example will not receive updates.`
		`- """`
		`- example = {}`
		`- example['fetcher'] = sys.modules[__name__]`
		`- example['version'] = '0.1'`
		`- example['name'] = 'example'`
		`- return example`
		`-`
		`- def compare_versions(self, new, old):`
		`- """Example has only one version`
		`-`
		`- This method does not matter here, let's just pass`
		`- """`
		`- return 0`
		`+ """Example kiskadee fetcher implementation."""`
		`+`
		`+ import kiskadee`
		`+`
		`+ class Fetcher(kiskadee.fetchers.Fetcher):`
		`+ """Example kiskadee fetcher implementation."""`
		`+`
		`+ def get_sources(self, source_data):`
		`+ """Use sources from test suite."""`
		`+ return 'kiskadee/tests/test_source/test_source.tar.gz'`
		`+`
		`+ def watch(self, **kwargs):`
		`+ """There is no proper API to inspect new example versions.`
		`+ It should not matter, since example will not receive updates.`
		`+ """`
		`+ example = {}`
		`+ example['fetcher'] = __name__`
		`+ example['version'] = '0.1'`
		`+ example['name'] = 'example'`
		`+ super().watch(**example)`
		`+`
		`+ def compare_versions(self, new, old):`
		`+ """Compare package versions.`
		`+`
		`+ Example has only one version. This method does not matter here, let's`
		`+ just pass.`
		`+ """`
		`+ return 0`

		`List of kiskadee fetchers`
		`----------------------------`

doc/installing.rst

file modified

+24 -3

		`@@ -1,7 +1,28 @@`
		`Installing kiskadee`
		`===================`

		`- Nothing to show here yet :(`
		`+ Development`
		`+ -----------`
		`+ First, make shure that ansible will be able to login on the vm. ansible will`
		`+ use the root user to do this, so you will have to add your public ssh`
		`+ key inside the root ~/.ssh/authorized_keys file. You can also create the host`
		`+ user inside the vm, in order to be able to test the vm access with the ping`
		`+ command.`

		`- We are still packaging kiskadee and writing proper ansible playbooks for fully`
		`- automated deployment :)`
		`+ To check if ansible can access the machine:`
		`+`
		`+ .. code-block:: bash`
		`+`
		`+ ansible -i playbook/hosts.local all -m ping`
		`+`
		`+ To deploy kiskadee locally:`
		`+`
		`+ .. code-block:: bash`
		`+`
		`+ ansible-playbook -c paramiko -i playbook/hosts.local playbook/local.yml`
		`+`
		`+ Production`
		`+ ----------`
		`+`
		`+ We are still working on a rpm package to install kiskadee on production mode.`
		+ You can check the latest build `here <https://copr.fedorainfracloud.org/coprs/davidcarlos/kiskadee/>`_.

kiskadee/api/app.py

file modified

+16 -41

		`@@ -14,13 +14,13 @@`

		`CORS(kiskadee)`

		`+ db = Database()`

		`@kiskadee.route('/fetchers')`
		`def index():`
		`"""Get the list of available fetchers."""`
		`if request.method == 'GET':`
		`- db_session = kiskadee_db_session()`
		`- fetchers = db_session.query(Fetcher).all()`
		`+ fetchers = db.session.query(Fetcher).all()`
		`fetcher_schema = FetcherSchema(many=True)`
		`result = fetcher_schema.dump(fetchers)`
		`return jsonify({'fetchers': result.data})`
		`@@ -30,8 +30,7 @@`
		`def packages():`
		`"""Get the list of analyzed packages."""`
		`if request.method == 'GET':`
		`- db_session = kiskadee_db_session()`
		`- packages = db_session.query(Package).all()`
		`+ packages = db.session.query(Package).all()`
		`package_schema = PackageSchema(`
		`many=True,`
		`exclude=['versions.analysis', 'versions.package_id']`
		`@@ -42,28 +41,20 @@`

		`@kiskadee.route('/analysis/<pkg_name>/<version>', methods=['GET'])`
		`def package_analysis_overview(pkg_name, version):`
		`- """Get the a analysis list of some package version."""`
		`- db_session = kiskadee_db_session()`
		`+ """Get a analysis list of some package version."""`

		`#TODO: This can be a simple inner join between package, version and analysis`
		`- package_id = (`
		`- db_session.query(Package).filter_by(name = pkg_name).id`
		`- )`
		`- version_id = (`
		`- db_session.query(Version)`
		`- .filter(Version.number == version)`
		`- .filter(Version.package_id == package_id).first().id`
		`- )`
		`+ _package_id = db.filter_by_name(Package, pkg_name).id`
		`+ version_id = db.session.query(Version)\`
		`+ .filter_by(number = version, package_id = _package_id ).first().id`
		`analysis = (`
		`- db_session.query(Analysis)`
		`+ db.session.query(Analysis)`
		`.options(`
		`eagerload(Analysis.analyzers, innerjoin=True)`
		`)`
		`.filter(Analysis.version_id == version_id)`
		`.all()`
		`)`
		`- kiskadee.logger.debug("haaaaaaa")`
		`- kiskadee.logger.debug(analysis)`
		`analysis_schema = AnalysisSchema(many=True, exclude=['raw', 'report'])`
		`data, errors = analysis_schema.dump(analysis)`
		`return jsonify(data)`
		`@@ -75,11 +66,7 @@`
		`)`
		`def analysis_results(pkg_name, version, analysis_id):`
		`"""Get the analysis results from a specific analyzer."""`
		`- db_session = kiskadee_db_session()`
		`- analysis = (`
		`- db_session.query(Analysis)`
		`- .get(analysis_id)`
		`- )`
		`+ analysis = db.get(Analysis, analysis_id)`
		`analysis_schema = AnalysisSchema(only=['raw'])`
		`data, errors = analysis_schema.dump(analysis)`
		`response = data['raw']['results']`
		`@@ -92,26 +79,14 @@`
		`)`
		`def analysis_reports(pkg_name, version, analysis_id):`
		`"""Get the analysis reports from a specific analyzer."""`
		`- db_session = kiskadee_db_session()`
		`- analysis = (`
		`- db_session.query(Analysis)`
		`- .get(analysis_id)`
		`- )`
		`- analysis_schema = AnalysisSchema(only=['report'])`
		`- data, errors = analysis_schema.dump(analysis)`
		`+ analysis = db.get(Analysis, analysis_id)`
		`+ data, errors = AnalysisSchema(only=['report']).dump(analysis)`
		`report = data['report']`
		`- if (report is not None) and\`
		`- ('results' in report.keys()) and\`
		`- report['results'] is not None:`
		`- report['results'] = json\`
		`- .loads(report['results'])`
		`- return jsonify({'analysis_report': report})`
		`-`
		`-`
		`- def kiskadee_db_session():`
		`- """Return a kiskadee database session."""`
		`- return Database().session`
		`-`
		`+ try:`
		`+ report['results'] = json.loads(report['results'])`
		`+ return jsonify({'analysis_report': report})`
		`+ except Exception as err:`
		`+ return jsonify({'analysis_report': {}})`

		`def main():`
		`"""Initialize the kiskadee API."""`

kiskadee/database.py

file modified

+8 -2

		`@@ -1,9 +1,10 @@`
		`"""Provide kiskadee Database operations."""`

		`- import kiskadee`
		`- from kiskadee.model import Base`
		`from sqlalchemy import create_engine, orm`

		`+ import kiskadee`
		`+ from kiskadee.model import Base`
		`+ from kiskadee.model import Package, Fetcher, Version, Report, Analysis`

		`class Database:`
		`"""kiskadee Database class."""`
		`@@ -22,6 +23,11 @@`
		`DBSession = orm.sessionmaker(bind=engine)`
		`return DBSession()`

		`+ def filter_by_name(self, model, _name):`
		`+ return self.session.query(model).filter_by(name = _name).first()`
		`+`
		`+ def get(self, model, id):`
		`+ return self.session.query(model).get(id)`

		`def get_database_uri(db):`
		`"""Return the Database URI of the current session."""`

kiskadee/fetchers/__init__.py

file modified

+8 -9

		`@@ -43,18 +43,14 @@`
		`This method will be called as a thread, and will run concurrently with`
		`the main kiskadee thread. The monitoring process must be done by`
		`classes that heritage from this one. This method is only responsible`
		`- to enqueue new monitored projects, so must be called by it childrens.`
		`+ to enqueue new monitored packages, so must be called by it childrens.`
		`"""`
		`- project = kwargs`
		`- if not project:`
		`+ package = kwargs`
		`+ if not package:`
		`raise NotImplementedError('fetcher must call parent watch method ')`

		`- kiskadee.queue.Queues().enqueue_project(project)`
		`- fetcher = project['fetcher']`
		`- kiskadee.logger.debug(`
		`- "{} fetcher: sending package {}_{} for monitor"`
		`- .format(fetcher, project['name'], project['version'])`
		`- )`
		`+ fetcher = package['fetcher'].split('.')[-1]`
		`+ kiskadee.queue.Queues().enqueue_package(package, fetcher)`

		`@abc.abstractmethod`
		`def compare_versions(self, new, old):`
		`@@ -93,6 +89,9 @@`
		`download_path = os.path.join(path, file_name)`
		`with open(download_path, 'wb') as info:`
		`info.write(data)`
		`+ kiskadee.logger.debug(`
		`+ "Downloaded {} source in {} path".format(file_name, path)`
		`+ )`
		`return download_path`
		`except Exception as err:`
		`kiskadee.logger.debug(`

kiskadee/fetchers/anitya.py

file modified

+13 -14

		`@@ -17,9 +17,9 @@`
		`def watch(self):`
		`"""Start the monitoring process for Anitya reports.`

		`- Each project monitored by the fetcher will be`
		`+ Each package monitored by the fetcher will be`
		`queued by calling the watch parent method,`
		`- passing the project data as argument.`
		`+ passing the package data as argument.`

		`The fetcher will use zmq as messaging protocol to receive`
		`the fedmsg-hub events. kiskadee and fedmsg-hub runs in different`
		`@@ -36,7 +36,7 @@`
		`if socket:`
		`while True:`
		`msg = socket.recv_string()`
		`- project = self.project_to_enqueue(msg)`
		`+ package = self.package_to_enqueue(msg)`

		`def get_sources(self, source_data):`
		`"""Download packages from some Anitya Backend."""`
		`@@ -77,23 +77,22 @@`
		`kiskadee.logger.debug(err)`
		`return False`

		`- def project_to_enqueue(self, fedmsg_event):`
		`+ def package_to_enqueue(self, fedmsg_event):`
		`event = self._event_to_dict(fedmsg_event)`
		`if event:`
		`- project = event.get('body').get('msg').get('project')`
		`+ package = event.get('body').get('msg').get('package')`
		`source_dict = {}`
		`- if project:`
		`- kiskadee_project = {`
		`- 'name': project.get('name'),`
		`- 'version': project.get('version'),`
		`+ if package:`
		`+ kiskadee_package = {`
		`+ 'name': package.get('name'),`
		`+ 'version': package.get('version'),`
		`'fetcher': __name__,`
		`'meta': {`
		`- 'backend': project.get('backend'),`
		`- 'homepage': project.get('homepage')`
		`+ 'backend': package.get('backend'),`
		`+ 'homepage': package.get('homepage')`
		`}`
		`}`
		`- super().watch(**kiskadee_project)`
		`-`
		`+ super().watch(**kiskadee_package)`

		`def _event_to_dict(self, msg):`
		`msg = msg[msg.find(" ")+1::]`
		`@@ -126,7 +125,7 @@`
		`class AnityaConsumer(fedmsg.consumers.FedmsgConsumer):`
		`"""Consumer used by fedmsg-hub to subscribe to fedmsg bus."""`

		`- topic = 'org.release-monitoring.prod.anitya.project.version.update'`
		`+ topic = 'org.release-monitoring.prod.anitya.package.version.update'`
		`config_key = 'anityaconsumer'`
		`validate_signatures = False`

kiskadee/fetchers/debian.py

file modified

+7 -7

		`@@ -1,4 +1,4 @@`
		`- """Fetcher to monitor Debian Project Packages."""`
		`+ """Fetcher to monitor Debian package Packages."""`

		`import os`
		`import tempfile`
		`@@ -16,14 +16,14 @@`


		`class Fetcher(kiskadee.fetchers.Fetcher):`
		`- """Fetcher to monitor Debian Project Packages."""`
		`+ """Fetcher to monitor Debian Packages."""`

		`def watch(self):`
		`"""Start the monitoring process for Debian Repositories.`

		`- Each project monitored by the fetcher will be`
		`+ Each package monitored by the fetcher will be`
		`queued by calling the watch parent method,`
		`- passing the project data as argument.`
		`+ passing the package data as argument.`
		`"""`
		`kiskadee.logger.debug("Starting Debian fetcher")`
		`while RUNNING:`
		`@@ -74,10 +74,10 @@`
		`sources = os.path.join(path, 'Sources')`
		`with open(sources) as sources_file:`
		`for src in Sources.iter_paragraphs(sources_file):`
		`- project = self.project_to_enqueue(src)`
		`- super().watch(**project)`
		`+ package = self.package_to_enqueue(src)`
		`+ super().watch(**package)`

		`- def project_to_enqueue(self, src):`
		`+ def package_to_enqueue(self, src):`
		`return {'name': src["Package"],`
		`'version': self._parse_version(src["Version"]),`
		`'fetcher': __name__,`

kiskadee/initializer.py

file added

+22

		`@@ -0,0 +1,22 @@`
		`+ """Initializes kiskadee components.`
		`+`
		`+ This module will run monitor and runner as processes.`
		`+ """`
		`+`
		`+ from multiprocessing import Process`
		`+`
		`+ import kiskadee.model`
		`+ import kiskadee.queue`
		`+ import kiskadee.database`
		`+ from kiskadee.runner import Runner`
		`+ from kiskadee.monitor import Monitor`
		`+`
		`+ def init():`
		`+ queues = kiskadee.queue.Queues()`
		`+ db = kiskadee.database.Database()`
		`+ monitor = Monitor(db, queues)`
		`+ runner = Runner(queues)`
		`+ Process(target=monitor.run,).start()`
		`+ runner_process = Process(target=runner.run,)`
		`+ runner_process.start()`
		`+ runner_process.join()`

kiskadee/model.py

file modified

+128 -17

		`@@ -3,7 +3,10 @@`
		`from sqlalchemy.ext.declarative import declarative_base`
		`from sqlalchemy import Column, Integer, UnicodeText, UniqueConstraint,\`
		`Sequence, Unicode, ForeignKey, orm, JSON`
		`+ import json`
		`+`
		`import kiskadee`
		`+ from kiskadee.report import CppcheckReport, FlawfinderReport`

		`Base = declarative_base()`

		`@@ -11,7 +14,7 @@`
		`class Package(Base):`
		`"""Software packages abstraction.`

		`- A software package is the source code for a software project. It may be`
		`+ A software package is the source code for a software package. It may be`
		`upstream's distribution or the sources provided by some other source, like`
		`a linux distribution.`
		`"""`
		`@@ -27,6 +30,47 @@`
		`UniqueConstraint('name', 'fetcher_id'),`
		`)`

		`+ @staticmethod`
		`+ def save(db, data):`
		`+ homepage = None`
		`+ if ('meta' in data) and ('homepage' in data['meta']):`
		`+ homepage = data['meta']['homepage']`
		`+`
		`+ _package = Package(`
		`+ name=data['name'],`
		`+ homepage=homepage,`
		`+ fetcher_id=data['fetcher_id']`
		`+ )`
		`+ db.session.add(_package)`
		`+ db.session.commit()`
		`+ _version = Version(number=data['version'],`
		`+ package_id=_package.id)`
		`+ db.session.add(_version)`
		`+ db.session.commit()`
		`+ return _package`
		`+`
		`+ @staticmethod`
		`+ def update(db, package, data):`
		`+`
		`+ if(package.versions[-1].number == data['version']):`
		`+ return package`
		`+ try:`
		`+ _new_version = Version(`
		`+ number=data['version'],`
		`+ package_id=package.id`
		`+ )`
		`+ package.versions.append(_new_version)`
		`+ db.session.add(package)`
		`+ db.session.commit()`
		`+ kiskadee.logger.debug(`
		`+ "MONITOR: Sending package {}_{}"`
		`+ "for analysis".format(data['name'], data['version'])`
		`+ )`
		`+ return package`
		`+ except ValueError:`
		`+ kiskadee.logger.debug("MONITOR: Could not compare versions")`
		`+ return None`
		`+`

		`class Fetcher(Base):`
		`"""kiskadee fetcher abstraction."""`
		`@@ -64,6 +108,23 @@`
		`version = Column(Unicode(255), nullable=True)`
		`analysis = orm.relationship('Analysis', backref='analyzers')`

		`+ @staticmethod`
		`+ def create_analyzers(db):`
		`+ """Create the analyzers on database.`
		`+`
		+ The kiskadee analyzers are defined on the section `analyzers` of the
		+ kiskadee.conf file. The `_session` argument represents a sqlalchemy
		`+ session.`
		`+ """`
		`+ list_of_analyzers = dict(kiskadee.config._sections["analyzers"])`
		`+ for _name, _version in list_of_analyzers.items():`
		`+ if not db.session.query(Analyzer)\`
		`+ .filter_by(name = _name, version = _version )\`
		`+ .first():`
		`+ new_analyzer = kiskadee.model.Analyzer(name = _name,`
		`+ version = _version)`
		`+ db.session.add(new_analyzer)`
		`+ db.session.commit()`

		`class Analysis(Base):`
		`"""Abstraction of a package analysis."""`
		`@@ -77,10 +138,46 @@`
		`report = orm.relationship('Report',`
		`uselist=False, back_populates='analysis')`

		`+ @staticmethod`
		`+ def save(db, analyzer, result, package):`
		`+ version = package.versions[-1]`
		`+ name = package.name`
		`+ _analysis = kiskadee.model.Analysis()`
		`+ try:`
		`+ _analyzer = db.session.query(kiskadee.model.Analyzer).\`
		`+ filter_by(name = analyzer).first()`
		`+ _analysis.analyzer_id = _analyzer.id`
		`+ _analysis.version_id = version.id`
		`+ _analysis.raw = json.loads(result)`
		`+ db.session.add(_analysis)`
		`+ db.session.commit()`
		`+ dict_analysis = {`
		`+ 'results': _analysis.raw['results'],`
		`+ 'id': _analysis.id`
		`+ }`
		`+ Report.save(db, dict_analysis, _analyzer.name, name)`
		`+ kiskadee.logger.debug(`
		`+ "MONITOR: Saved analysis done by {} for package: {}-{}"`
		`+ .format(analyzer, data["name"], data["version"])`
		`+ )`
		`+ return`
		`+ except Exception as err:`
		`+ kiskadee.logger.debug(`
		`+ "MONITOR: The required analyzer was " +`
		`+ "not registered in kiskadee"`
		`+ )`
		`+ kiskadee.logger.debug(err)`
		`+ return None`
		`+`

		`class Report(Base):`
		`"""Abstraction of a analysis report."""`

		`+ REPORTERS = {`
		`+ 'cppcheck': CppcheckReport,`
		`+ 'flawfinder': FlawfinderReport`
		`+ }`
		`+`
		`__tablename__ = 'reports'`
		`id = Column(Integer,`
		`Sequence('reports_id_seq', optional=True), primary_key=True)`
		`@@ -88,20 +185,34 @@`
		`results = Column(JSON)`
		`analysis = orm.relationship('Analysis', back_populates='report')`

		`+ @staticmethod`
		`+ def save(db, analysis, analyzer_name, package_name):`
		`+ try:`
		`+ results = analysis['results']`
		`+ analyzer_report = Report.REPORTERS[analyzer_name](results)`
		`+ _reports = Report()`
		`+ _reports.results = json.dumps(`
		`+ analyzer_report`
		`+ ._compute_reports(analyzer_name)`
		`+ )`
		`+ _reports.analysis_id = analysis['id']`
		`+ db.session.add(_reports)`
		`+ db.session.commit()`
		`+ kiskadee.logger.debug(`
		`+ "MONITOR: Saved analysis reports for {} package"`
		`+ .format(data["name"])`
		`+ )`
		`+ except KeyError as key:`
		`+ kiskadee.logger.debug(`
		`+ "ERROR: There's no reporter " +`
		`+ "to get reports from {} analyzer. ".format(key) +`
		`+ "Make shure to import or implement them."`
		`+ )`
		`+ except Exception as err:`
		`+ kiskadee.logger.debug(`
		`+ "MONITOR: Failed to get analysis reports to {} package"`
		`+ .format(data["name"])`
		`+ )`
		`+ kiskadee.logger.debug(err)`
		`+ return`

		`- def create_analyzers(_session):`
		`- """Create the analyzers on database.`
		`-`
		- The kiskadee analyzers are defined on the section `analyzers` of the
		- kiskadee.conf file. The `_session` argument represents a sqlalchemy
		`- session.`
		`- """`
		`- list_of_analyzers = dict(kiskadee.config._sections["analyzers"])`
		`- for name, version in list_of_analyzers.items():`
		`- if not (_session.query(Analyzer).filter(Analyzer.name == name).`
		`- filter(Analyzer.version == version).first()):`
		`- new_analyzer = kiskadee.model.Analyzer()`
		`- new_analyzer.name = name`
		`- new_analyzer.version = version`
		`- _session.add(new_analyzer)`
		`- _session.commit()`

kiskadee/monitor.py

file modified

+70 -215

		`@@ -4,238 +4,93 @@`
		`analyzed. This module provides such capabilities.`
		`"""`
		`import threading`
		`- from multiprocessing import Process`
		`import time`
		`import os`
		`- import json`
		`import importlib`

		`- import kiskadee.database`
		`- from kiskadee.report import CppcheckReport, FlawfinderReport`
		`- from kiskadee.runner import Runner`
		`- import kiskadee.queue`
		`- from kiskadee.model import Package, Fetcher, Version, Report`
		`+ import kiskadee`
		`+ from kiskadee.model import Package, Fetcher, Version, Report, Analysis, Analyzer`

		`RUNNING = True`

		`- REPORTERS = {`
		`- 'cppcheck': CppcheckReport,`
		`- 'flawfinder': FlawfinderReport`
		`- }`
		`-`
		`class Monitor:`
		`"""Provide kiskadee monitoring objects."""`

		`- def __init__(self, _session):`
		`+ def __init__(self, db, queues):`
		`"""Return a non initialized Monitor."""`
		`- self.session = _session`
		`- self.kiskadee_queue = None`
		`-`
		`- def monitor(self, queues):`
		`- """Dequeue packages and check if they need to be analyzed.`
		`+ self.db = db`
		`+ self.queues = queues`
		`+ Analyzer.create_analyzers(self.db)`

		- The packages are dequeued from the `package_queue`. When a package
		- needs to be analyzed, this package is enqueued in the `analyses_queue`
		`- so the runner component can trigger an analysis. Each fetcher must`
		- enqueue its packages in the `packages_queue`.
		`- """`
		`- kiskadee.logger.debug('kiskadee PID: {}'.format(os.getppid()))`
		`- kiskadee.logger.debug('Starting monitor subprocess')`
		`- kiskadee.logger.debug('monitor PID: {}'.format(os.getpid()))`
		`+ def run(self):`
		`+ """Run Monitor process. This method resumes Monitor behavior. It dequeue`
		`+ packages from fetchers, check if the package is valid and if so send it`
		`+ to Runner. When Runner ends a analysis, the Monitor dequeue the analysis`
		`+ and save it on database. """`
		`+ kiskadee.logger.debug('MONITOR PID: {}'.format(os.getpid()))`

		`for fetcher in kiskadee.load_fetchers():`
		`- _start_fetcher(fetcher.Fetcher().watch)`
		`+ thread = Monitor.start_fetcher(fetcher.Fetcher().watch)`

		`while RUNNING:`
		`- self.kiskadee_queue = queues`
		`- pkg = self.kiskadee_queue.dequeue_project()`
		`-`
		`- if pkg:`
		`- self._send_to_runner(pkg)`
		`- time.sleep(2)`
		`- analyzed_project = self.kiskadee_queue.dequeue_result()`
		`- self._save_analyzed_project(analyzed_project)`
		`-`
		`- def _send_to_runner(self, pkg):`
		`- _name = pkg['fetcher'].split('.')[-1]`
		`- _fetcher = self._query(Fetcher).filter_by(name = _name).first()`
		`- _package = (`
		`- self._query(Package)`
		`- .filter(Package.name == pkg['name']).first()`
		`- )`
		`-`
		`- if _fetcher:`
		`- pkg["fetcher_id"] = _fetcher.id`
		`- if not _package:`
		`- kiskadee.logger.debug(`
		`- "MONITOR: Sending package {}_{} "`
		`- " for analysis".format(pkg['name'], pkg['version'])`
		`- )`
		`- self.kiskadee_queue.enqueue_analysis(pkg)`
		`- else:`
		`- new_version = pkg['version']`
		`- analysed_version = _package.versions[-1].number`
		`- fetcher = importlib.import_module(pkg['fetcher']).Fetcher()`
		`- if (fetcher.compare_versions(new_version, analysed_version)):`
		`- self.kiskadee_queue.enqueue_analysis(pkg)`
		`-`
		`- # Move this to model.py`
		`- def _save_analyzed_project(self, data):`
		`+ kiskadee.logger.debug('MONITOR STATE: Idle'.format(os.getpid()))`
		`+ new_package = self.dequeue_package_from_fetchers()`
		`+ self.send_package_to_runner(new_package)`
		`+ analyzed_package = self.dequeue_analysis_from_runner()`
		`+ self.save_analyzed_package(analyzed_package)`
		`+`
		`+ def dequeue_package_from_fetchers(self):`
		`+ return self.queues.dequeue_package()`
		`+`
		`+ def dequeue_analysis_from_runner(self):`
		`+ return self.queues.dequeue_result()`
		`+`
		`+ def send_package_to_runner(self, data):`
		`+ if data:`
		`+ fetcher, package = self.get_fetcher_and_package(data)`
		`+ data["fetcher_id"] = fetcher.id if fetcher else ''`
		`+ if not self.is_a_new_package_version(package, data):`
		`+ kiskadee.logger.debug('MONITOR STATE: Package {} already'\`
		`+ ' analyzed'.format(data['name']))`
		`+ return`
		`+ self.queues.enqueue_analysis(data)`
		`+`
		`+`
		`+ def get_fetcher_and_package(self, data):`
		`+ fetcher_name = data['fetcher'].split('.')[-1]`
		`+ package_name = data['name']`
		`+ fetcher = self.db.filter_by_name(Fetcher, fetcher_name)`
		`+ package = self.db.filter_by_name(Package, package_name)`
		`+ return fetcher, package`
		`+`
		`+ def is_a_new_package_version(self, package, data):`
		`+ if package:`
		`+ package_version = data['version']`
		`+ analysed_version = package.versions[-1].number`
		`+ fetcher = importlib.import_module(data['fetcher']).Fetcher()`
		`+ return fetcher.compare_versions(package_version, analysed_version)`
		`+ else:`
		`+ return True`
		`+`
		`+ def save_analyzed_package(self, data):`
		`if not data:`
		`return {}`
		`- project = self._query(Package).filter_by(name = data['name']).first()`
		`- if not project:`
		`- project = self._save_project(data)`
		`- if project:`
		`- project = self._update_project(project, data)`
		`+ package = self.db.filter_by_name(Package, data['name'])`
		`+ if not package:`
		`+ Package.save(self.db, data)`
		`+ else:`
		`+ Package.update(self.db, package, data)`

		`+ def save_package_analysis(self, data, package):`
		`for analyzer, result in data['results'].items():`
		`- self._save_analysis(data, analyzer, result, project.versions[-1])`
		`-`
		`- # Move this to model.py`
		`- def _update_project(self, package, pkg):`
		`-`
		`- if(package.versions[-1].number == pkg['version']):`
		`- return package`
		`- try:`
		`- _new_version = Version(`
		`- number=pkg['version'],`
		`- package_id=package.id`
		`- )`
		`- package.versions.append(_new_version)`
		`- self.session.add(package)`
		`- self.session.commit()`
		`- kiskadee.logger.debug(`
		`- "MONITOR: Sending package {}_{}"`
		`- "for analysis".format(pkg['name'], pkg['version'])`
		`- )`
		`- return package`
		`- except ValueError:`
		`- kiskadee.logger.debug("MONITOR: Could not compare versions")`
		`- return None`
		`-`
		`- # Move this to model.py`
		`- def _save_project(self, pkg):`
		`- homepage = None`
		`- if ('meta' in pkg) and ('homepage' in pkg['meta']):`
		`- homepage = pkg['meta']['homepage']`
		`-`
		`- _package = Package(`
		`- name=pkg['name'],`
		`- homepage=homepage,`
		`- fetcher_id=pkg['fetcher_id']`
		`- )`
		`- self.session.add(_package)`
		`- self.session.commit()`
		`- _version = Version(number=pkg['version'],`
		`- package_id=_package.id)`
		`- self.session.add(_version)`
		`- self.session.commit()`
		`- return _package`
		`-`
		`- # Move this to model.py`
		`- def _save_reports(self, analysis, pkg, analyzer_name):`
		`- try:`
		`- results = analysis['results']`
		`- analyzer_report = REPORTERS[analyzer_name](results)`
		`- _reports = Report()`
		`- _reports.results = json.dumps(`
		`- analyzer_report`
		`- ._compute_reports(analyzer_name)`
		`- )`
		`- _reports.analysis_id = analysis['id']`
		`- self.session.add(_reports)`
		`- self.session.commit()`
		`- kiskadee.logger.debug(`
		`- "MONITOR: Saved analysis reports for {} package"`
		`- .format(pkg["name"])`
		`- )`
		`- except KeyError as key:`
		`- kiskadee.logger.debug(`
		`- "ERROR: There's no reporter " +`
		`- "to get reports from {} analyzer. ".format(key) +`
		`- "Make shure to import or implement them."`
		`- )`
		`- except Exception as err:`
		`- kiskadee.logger.debug(`
		`- "MONITOR: Failed to get analysis reports to {} package"`
		`- .format(pkg["name"])`
		`- )`
		`- kiskadee.logger.debug(err)`
		`- return`
		`-`
		`- # Move this to model.py`
		`- def _save_analysis(self, pkg, analyzer, result, version):`
		`- _analysis = kiskadee.model.Analysis()`
		`- try:`
		`- _analyzer = self._query(kiskadee.model.Analyzer).\`
		`- filter_by(name = analyzer).first()`
		`- _analysis.analyzer_id = _analyzer.id`
		`- _analysis.version_id = version.id`
		`- _analysis.raw = json.loads(result)`
		`- self.session.add(_analysis)`
		`- self.session.commit()`
		`- dict_analysis = {`
		`- 'results': _analysis.raw['results'],`
		`- 'id': _analysis.id`
		`- }`
		`- self._save_reports(dict_analysis, pkg, _analyzer.name)`
		`- kiskadee.logger.debug(`
		`- "MONITOR: Saved analysis done by {} for package: {}-{}"`
		`- .format(analyzer, pkg["name"], pkg["version"])`
		`- )`
		`- return`
		`- except Exception as err:`
		`- kiskadee.logger.debug(`
		`- "MONITOR: The required analyzer was " +`
		`- "not registered in kiskadee"`
		`- )`
		`- kiskadee.logger.debug(err)`
		`- return None`
		`-`
		`- # Move this to model.py`
		`- def _save_fetcher(self, fetcher):`
		`- name = fetcher.name`
		`- kiskadee.logger.debug(`
		`- "MONITOR: Saving {} fetcher in database".format(name)`
		`- )`
		`- if not self.session.query(Fetcher)\`
		`- .filter(Fetcher.name == name).first():`
		`- _fetcher = Fetcher(`
		`- name=name,`
		`- target=fetcher.config['target'],`
		`- description=fetcher.config['description']`
		`- )`
		`- self.session.add(_fetcher)`
		`- self.session.commit()`
		`-`
		`- def _query(self, arg):`
		`- return self.session.query(arg)`
		`-`
		`-`
		`- def _start_fetcher(module, joinable=False, timeout=None):`
		`- module_as_a_thread = threading.Thread(target=module)`
		`- module_as_a_thread.daemon = True`
		`- module_as_a_thread.start()`
		`- if joinable or timeout:`
		`- module_as_a_thread.join(timeout)`
		`-`
		`-`
		`- def daemon():`
		`- """Entry point to the monitor module."""`
		`- # TODO: improve with start/stop system`
		`- queues = kiskadee.queue.Queues()`
		`- session = kiskadee.database.Database().session`
		`- monitor = Monitor(session)`
		`- runner = Runner()`
		`- monitor_process = Process(`
		`- target=monitor.monitor,`
		`- args=(queues,)`
		`- )`
		`- runner_process = Process(`
		`- target=runner.runner,`
		`- args=(queues,)`
		`- )`
		`- monitor_process.start()`
		`- runner_process.start()`
		`- runner_process.join()`
		`+ Analysis.save(self.db, analyzer, result, package)`
		`+`
		`+`
		`+ @staticmethod`
		`+ def start_fetcher(module, joinable=False, timeout=None):`
		`+ """Start a fetcher as a thread of the Monitor process."""`
		`+ module_as_a_thread = threading.Thread(target=module)`
		`+ module_as_a_thread.daemon = True`
		`+ module_as_a_thread.start()`
		`+ if joinable or timeout:`
		`+ module_as_a_thread.join(timeout)`

kiskadee/queue.py

file modified

+45 -42

		`@@ -1,73 +1,76 @@`
		`"""Provide kiskadee queues and operations on them."""`
		`import time`
		`- import dill`
		`from multiprocessing import Queue`

		`import kiskadee`

		`- def empty_queue(func):`
		`- """Decorator that checks if retrieve data from a`
		`- queue will trigger a empty exception.`
		`- """`
		`- def wrapper(args, *kwargs):`
		`- try:`
		`- return func(args, *kwargs)`
		`- except Exception:`
		`- return {}`
		`- return wrapper`
		`-`
		`- def enqueue_project(fetcher_watch):`
		`- """Decorator to be used with watch fetchers method. Every project returned`
		`- by the watch method, will be enqueued on the queue projects.`
		`- :arg1: watch method`
		`- :returns: a wrapper variable`
		`- """`
		`- def wrapper(args, *kwargs):`
		`- project = fetcher_watch(args, *kwargs)`
		`- kiskadee.queue.queues.enqueue_project(project)`
		`- fetcher = project['fetcher'].name`
		`- kiskadee.logger.debug(`
		`- "{} fetcher: sending package {}_{} for monitor"`
		`- .format(fetcher, package['name'], package['version'])`
		`- )`
		`- time.sleep(2)`
		`- return wrapper`
		`-`
		`analysis = Queue()`
		`results = Queue()`
		`- projects = Queue()`
		`+ packages = Queue()`

		`class Queues():`
		`"""Provide kiskadee queues objects."""`

		`@staticmethod`
		`- def enqueue_analysis(project_to_analysis):`
		`+ def enqueue_analysis(package_to_analysis):`
		`"""Put a analysis on the analysis queue."""`
		`- analysis.put(project_to_analysis)`
		`+ log_msg = "MONITOR STATE: Sending package {}-{} for analysis"\`
		`+ .format(package_to_analysis['name'],`
		`+ package_to_analysis['version'])`
		`+ kiskadee.logger.debug(log_msg)`
		`+ analysis.put(package_to_analysis)`

		`@staticmethod`
		`def dequeue_analysis():`
		`"""Get a analysis from the analysis queue."""`
		`- return analysis.get()`
		`+ package_to_analysis = analysis.get()`
		`+ fetcher = package_to_analysis ['fetcher'].split('.')[-1]`
		`+ kiskadee.logger.debug(`
		`+ 'RUNNER STATE: dequeued {}-{} from {}'`
		`+ .format(package_to_analysis['name'],`
		`+ package_to_analysis['version'],`
		`+ fetcher)`
		`+ )`
		`+ return package_to_analysis`

		`@staticmethod`
		`- def enqueue_result(result):`
		`+ def enqueue_result(package):`
		`"""Put a result on the results queue."""`
		`- results.put(result)`
		`+ kiskadee.logger.debug(`
		`+ "RUNNER STATE: Sending {}-{} to Monitor"`
		`+ .format(package["name"],`
		`+ package["version"])`
		`+ )`
		`+ results.put(package)`

		`@staticmethod`
		`- @empty_queue`
		`def dequeue_result():`
		`"""Get a result from the results queue."""`
		`- return results.get(timeout=1)`
		`+ result = results.get()`
		`+ kiskadee.logger.debug(`
		`+ "MONITOR STATE: Pick Up analyzed package"`
		`+ .format(result["name"],`
		`+ result["version"])`
		`+ )`
		`+ return result`

		`@staticmethod`
		`- def enqueue_project(project):`
		`+ def enqueue_package(package, fetcher=None):`
		`"""Put a result on the results queue."""`
		`- projects.put(project)`
		`+ if fetcher:`
		`+ kiskadee.logger.debug(`
		`+ "FETCHER {}: sending package {}-{} for monitor"`
		`+ .format(fetcher, package['name'], package['version'])`
		`+ )`
		`+ packages.put(package)`

		`@staticmethod`
		`- @empty_queue`
		`- def dequeue_project():`
		`+ def dequeue_package():`
		`"""Get a result from the results queue."""`
		`- return projects.get(timeout=1)`
		`+ package = packages.get()`
		`+ kiskadee.logger.debug(`
		`+ "MONITOR STATE: Pick Up monitored package."`
		`+ .format(package["name"],`
		`+ package["version"])`
		`+ )`
		`+ return package`

kiskadee/report.py

file modified

-1

		`@@ -36,7 +36,6 @@`

		`def _compute_reports(self, analyzer):`
		`"""Compute every report type for cppcheck analyzer."""`
		`- print(analyzer)`
		`result_dict = {`
		`'warning': 0,`
		`'error': 0,`

kiskadee/runner.py

file modified

+75 -85

		`@@ -16,63 +16,72 @@`
		`class Runner:`
		`"""Provide kiskadee runner objects."""`

		`- def __init__(self):`
		`+ def __init__(self, queues):`
		`"""Return a non initialized Runner."""`
		`- self.queues = None`
		`+ self.queues = queues`
		`+ self.fetcher = None`
		`+ self.package = None`

		`- def runner(self, queues):`
		`+ def run(self):`
		`"""Run static analyzers.`

		Continuously dequeue packages from `analyses_queue` and call the
		- :func:`analyze` method, passing the dequeued package.
		`- After the analysis, updates the status of this package on the database.`
		+ :func:`analyze` method, passing the dequeued package as argument.
		`+ After the analysis, send the result back to Monitor.`
		`"""`
		`- kiskadee.logger.debug('Starting runner subprocess')`
		`- kiskadee.logger.debug('runner PID: {}'.format(os.getpid()))`
		`- session = kiskadee.database.Database().session`
		`- kiskadee.model.create_analyzers(session)`
		`- self.queues = queues`
		`+ kiskadee.logger.debug('RUNNER PID: {}'.format(os.getpid()))`
		`while RUNNING:`
		`- kiskadee.logger.debug('RUNNER: Waiting to dequeue'\`
		`- ' project to analysis...')`
		`- source_to_analysis = self.queues.dequeue_analysis()`
		`- kiskadee.logger.debug(`
		`- 'RUNNER: deqeued {}-{} from {}'`
		`- .format(source_to_analysis['name'],`
		`- source_to_analysis['version'],`
		`- source_to_analysis['fetcher'])`
		`+ kiskadee.logger.debug('RUNNER STATE: Idle.')`
		`+ self.package = self.queues.dequeue_analysis()`
		`+ self.call_analyzers()`
		`+`
		`+ def import_package_fetcher(self):`
		`+ try:`
		`+ return importlib.import_module(`
		`+ self.package['fetcher']`
		`+ ).Fetcher()`
		`+ except ModuleNotFoundError:`
		`+ kiskadee.logger.debug("RUNNER STATE: Fetcher {}\`
		`+ could not be loaded".format(self.package['fetcher'])`
		`)`
		`- self.call_analyzers(source_to_analysis)`
		`+ return {}`
		`+`
		`+ def run_analysis(self, analyzers, source_path):`
		`+ self.package['results'] = {}`
		`+ for analyzer in analyzers:`
		`+ firehose_report = self.analyze(analyzer, source_path)`
		`+ if firehose_report:`
		`+ self.package['results'][analyzer] = firehose_report`
		`+ return self.package`
		`+`
		`+`
		`+ def rmdtemp(self, temp_dir):`
		`+ # not delete the source code used on tests.`
		`+ if not temp_dir.find("kiskadee/tests") > -1:`
		`+ shutil.rmtree(temp_dir)`

		`- def call_analyzers(self, source_to_analysis):`
		`+ def call_analyzers(self):`
		`"""Iterate over the package analyzers.`

		`For each analyzer defined to analysis the source, call`
		the function :func:`analyze`, passing the source dict, the analyzer
		`to run the analysis, and the path to a compressed source.`
		`"""`
		`- fetcher = importlib.import_module(source_to_analysis['fetcher']).Fetcher()`
		`- source_path = self._path_to_uncompressed_source(`
		`- source_to_analysis, fetcher`
		`- )`
		`+ self.fetcher = self.import_package_fetcher()`
		`+ source_path = self.get_package_code_path()`
		`if not source_path:`
		`return None`

		`- analyzers = fetcher.analyzers()`
		`- source_to_analysis['results'] = {}`
		`- for analyzer in analyzers:`
		`- firehose_report = self.analyze(analyzer, source_path)`
		`- if firehose_report:`
		`- source_to_analysis['results'][analyzer] = firehose_report`
		`-`
		`- if source_to_analysis['results']:`
		`- kiskadee.logger.debug(`
		`- "RUNNER: Sending {}-{} to Monitor"`
		`- .format(source_to_analysis["name"],`
		`- source_to_analysis["version"])`
		`+ analyzers = self.fetcher.analyzers()`
		`+ analysis_result = self.run_analysis(`
		`+ analyzers, source_path`
		`)`
		`- self.queues.enqueue_result(source_to_analysis)`
		`- shutil.rmtree(source_path)`
		`+ self.enqueue_analysis_to_monitor(analysis_result)`
		`+ self.rmdtemp(source_path)`
		`+`
		`+ def enqueue_analysis_to_monitor(self, analysis_result):`
		`+ if analysis_result['results']:`
		`+ self.queues.enqueue_result(analysis_result)`

		`def analyze(self, analyzer, source_path):`
		`"""Run each analyzer on some sorce code.`
		`@@ -80,73 +89,54 @@`
		The `analyzer` is the name of a static analyzer already created on the
		`database.`
		The `source_path` is the directory to a uncompressed source, returned
		- by the :func:`_path_to_uncompressed_source`.
		+ by the :func:`get_package_code_path`.
		`"""`
		`if source_path is None:`
		`return None`

		`- kiskadee.logger.debug('ANALYSIS: running {} ...'.format(analyzer))`
		`+ kiskadee.logger.debug('RUNNER STATE: analysing with {} ...'\`
		`+ .format(analyzer))`
		`try:`
		`analysis = kiskadee.analyzers.run(analyzer, source_path)`
		`firehose_report = kiskadee.converter.to_firehose(`
		`analysis, analyzer`
		`)`
		`kiskadee.logger.debug(`
		`- 'ANALYSIS: DONE {} analysis'`
		`+ 'RUNNER STATE: DONE {} analysis'`
		`.format(analyzer)`
		`)`
		`return firehose_report`
		`except Exception as err:`
		`- kiskadee.logger.debug('RUNNER: could not generate analysis')`
		`+ kiskadee.logger.debug('RUNNER STATE: could not generate analysis')`
		`kiskadee.logger.debug(err)`
		`return None`

		`- def _path_to_uncompressed_source(self, package, fetcher):`
		`-`
		`- if not (fetcher and package):`
		`+ def get_package_code_path(self):`
		`+ """ Returns a string, representing the path of the uncompressed`
		`+ package source."""`
		`+ if not (self.fetcher and self.package):`
		`return None`

		`- kiskadee.logger.debug(`
		`- 'ANALYSIS: Downloading {} '`
		`- 'source...'.format(package['name'])`
		`- )`
		`-`
		`- compressed_source = fetcher.get_sources(package)`
		`-`
		`- if compressed_source:`
		`- kiskadee.logger.debug(`
		`- 'ANALYSIS: Downloaded {} source in {} path'`
		`- .format(`
		`- package['name'],`
		`- os.path.dirname(compressed_source)`
		`+ compressed_source_path = self.fetcher.get_sources(self.package)`
		`+ if compressed_source_path :`
		`+ uncompressed_source_path = self.uncompress_package_code(`
		`+ compressed_source_path`
		`)`
		`- )`
		`- tmp_path = tempfile.gettempdir()`
		`- uncompressed_source_path = tempfile.mkdtemp(dir=tmp_path)`
		`- try:`
		`- shutil.unpack_archive(`
		`- compressed_source,`
		`- uncompressed_source_path`
		`- )`
		`- kiskadee.logger.debug(`
		`- 'ANALYSIS: Unpacking {} source in {} path'`
		`- .format(package['name'], uncompressed_source_path)`
		`- )`
		`- # not delete the source code used on tests.`
		`- if not compressed_source.find("kiskadee/tests") > -1:`
		`- shutil.rmtree(os.path.dirname(compressed_source))`
		`- kiskadee.logger.debug(`
		`- 'ANALYSIS: Remove {} temp directory'`
		`- .format(os.path.dirname(compressed_source))`
		`- )`
		`- kiskadee.logger.debug(`
		`- 'ANALYSIS: Unpacked {} source'.format(package['name'])`
		`- )`
		`- return uncompressed_source_path`
		`- except Exception as err:`
		`- kiskadee.logger.debug('Something went wrong')`
		`- kiskadee.logger.debug(err)`
		`- return None`
		`+ self.rmdtemp(os.path.dirname(compressed_source_path))`
		`+ return uncompressed_source_path`
		`else:`
		`kiskadee.logger.debug('RUNNER: invalid compressed source')`
		`return None`
		`+`
		`+ def uncompress_package_code(self, compressed_source):`
		`+ dir_to_unpack_source = tempfile.mkdtemp()`
		`+ try:`
		`+ shutil.unpack_archive(compressed_source,`
		`+ dir_to_unpack_source)`
		`+ return dir_to_unpack_source`
		`+ except Exception as err:`
		`+ kiskadee.logger.debug('RUNNER STATE: Could not unpack'\`
		`+ 'package source')`
		`+ kiskadee.logger.debug(err)`
		`+ return {}`
		`+`

kiskadee/tests/__init__.py

file modified

+12

		`@@ -1,1 +1,13 @@`
		`"""kiskadee test suite."""`
		`+ from kiskadee import model`
		`+`
		`+`
		`+ def clean_test_db(db, metadata):`
		`+ """Clean test database to run tests independently."""`
		`+ # The number of openend sessions is limited by the operational system.`
		`+ # If we open too much connections, the database will lock the tests`
		`+ # execution.`
		`+ db.session.close_all()`
		`+ metadata.drop_all(db.engine)`
		`+ metadata.create_all(db.engine)`
		`+ model.Analyzer.create_analyzers(db)`

kiskadee/tests/api/test_api.py

file modified

+12 -28

		`@@ -3,9 +3,9 @@`
		`from unittest.mock import MagicMock`

		`import kiskadee`
		`- from kiskadee.monitor import Monitor`
		`import kiskadee.api.app`
		`import kiskadee.fetchers.example`
		`+ import kiskadee.database`


		`class ApiTestCase(unittest.TestCase):`
		`@@ -14,15 +14,14 @@`
		`def setUpClass(cls):`
		`kiskadee.api.app.kiskadee.testing = True`
		`cls.app = kiskadee.api.app.kiskadee.test_client()`
		`+ cls.db = kiskadee.api.app.db = MagicMock()`

		`def setUp(self):`
		`self.fetcher = kiskadee.model.Fetcher(name='kiskadee-fetcher')`
		`+ self.queues = kiskadee.queue.Queues()`

		`def test_get_fetchers(self):`
		`- kiskadee.api.app.kiskadee_db_session = MagicMock()`
		`- kiskadee.api.app.kiskadee_db_session().query().all = MagicMock(`
		`- return_value=[self.fetcher]`
		`- )`
		`+ self.db.session.query().all = MagicMock(return_value=[self.fetcher])`
		`response = self.app.get("/fetchers")`
		`response_data = json.loads(response.data.decode("utf-8"))`
		`self.assertIn("fetchers", response_data)`
		`@@ -32,30 +31,18 @@`
		`)`

		`def test_get_analysis_as_json(self):`
		`- package = kiskadee.model.Package(name='mocked-package', id=1)`
		`- version = kiskadee.model.Version(number='1.0.0', id=1, package_id=1)`
		`analysis = kiskadee.model.Analysis(`
		`id=1, analyzer_id=1, version_id=1,`
		`raw='analysis result: foo'`
		`)`

		`- def side_effect(klass):`
		`- mocked_objects = {`
		`- 'Package': package,`
		`- 'Version': version,`
		`- }`
		`- return mocked_objects[klass.__name__]`
		`-`
		`- kiskadee.api.app.kiskadee_db_session = MagicMock()`
		`- Monitor(kiskadee.api.app.kiskadee_db_session())`
		`- db_session = kiskadee.api.app.kiskadee_db_session()`
		`- db_session.query(kiskadee.model.Package)\`
		`- .filter_by().id = MagicMock(side_effect=side_effect)`
		`-`
		`- db_session.query(kiskadee.model.Version)\`
		`- .filter_by().id = MagicMock(side_effect=side_effect)`
		`- db_session.query().options()\`
		`- .filter().all = MagicMock(return_value=[analysis])`
		`+ self.db.filter_by_name = MagicMock()`
		`+ self.db.session.query(kiskadee.model.Package)\`
		`+ .filter_by().id = MagicMock()`
		`+ self.db.session.query(kiskadee.model.Version)\`
		`+ .filter_by().id = MagicMock()`
		`+ self.db.session.query().options()\`
		`+ .filter().all = MagicMock(return_value=[analysis])`

		`response = self.app.get("/analysis/mocked-package/1.0.0")`
		`response_data = json.loads(response.data.decode("utf-8"))`
		`@@ -67,10 +54,7 @@`
		`id=1, analyzer_id=1, version_id=1,`
		`raw={'results': 'first analysis results'}`
		`)`
		`-`
		`- kiskadee.api.app.kiskadee_db_session().query().get = MagicMock(`
		`- return_value=analysis`
		`- )`
		`+ self.db.get = MagicMock(return_value=analysis)`
		`response = self.app.get("/analysis/kiskadee-package/7.23/1/results")`
		`response_data = json.loads(response.data.decode("utf-8"))`
		`self.assertIn("analysis_results", response_data)`

kiskadee/tests/integration/test_monitor.py

file modified

+48 -144

		`@@ -1,27 +1,31 @@`
		`import unittest`
		`- from sqlalchemy.orm import sessionmaker`
		`+ from unittest.mock import MagicMock`

		`- from kiskadee import model`
		`from kiskadee.monitor import Monitor`
		`from kiskadee.queue import Queues`
		`- from kiskadee.model import Package, create_analyzers, Report, Analysis`
		`+ from kiskadee.model import Package, Fetcher`
		`import kiskadee.queue`
		`import kiskadee.fetchers.debian`
		`import kiskadee.fetchers.anitya`
		`- from kiskadee.database import Database`
		`+ import kiskadee.fetchers.example`


		`class MonitorTestCase(unittest.TestCase):`

		`def setUp(self):`
		`- self.engine = Database('db_test').engine`
		`- Session = sessionmaker(bind=self.engine)`
		`- self.session = Session()`
		`- self.monitor = Monitor(self.session)`
		`- self.queues = Queues()`
		`- model.Base.metadata.create_all(self.engine)`
		`- create_analyzers(self.session)`
		`- self.pkg1 = {`
		`+ def mocked_models(klass):`
		`+ class Package:`
		`+ def all(self):`
		`+ [self.pkg1, self.pkg2, self.pkg3, self.pkg4]`
		`+ klass()`
		`+`
		`+ self.example_fetcher = kiskadee.fetchers.example.Fetcher()`
		`+ self.db = MagicMock()`
		`+`
		`+ queues = Queues()`
		`+ self.monitor = Monitor(self.db, queues)`
		`+`
		`+ self.data1 = {`
		`'name': 'curl',`
		`'version': '7.52.1-5',`
		`'fetcher': kiskadee.fetchers.debian.__name__,`
		`@@ -31,137 +35,37 @@`
		`'flawfinder': '><'},`
		`'fetcher_id': 1}`

		`- self.pkg2 = {'name': 'urlscan',`
		`- 'version': '0.8.2',`
		`- 'fetcher': kiskadee.fetchers.debian.__name__,`
		`- 'meta': {'directory': 'pool/main/u/urlscan'},`
		`- 'results': {`
		`- 'cppcheck': '<>',`
		`- 'flawfinder': '><'},`
		`- 'fetcher_id': 1}`
		`-`
		`- self.pkg3 = {'name': 'curl',`
		`- 'version': '7.52.2-5',`
		`- 'fetcher': kiskadee.fetchers.debian.__name__,`
		`- 'meta': {'directory': 'pool/main/c/curl'},`
		`- 'results': {`
		`- 'cppcheck': '<>',`
		`- 'flawfinder': '><'},`
		`- 'fetcher_id': 1}`
		`- self.pkg4 = {'name': 'urlanitya',`
		`- 'version': '0.11',`
		`- 'fetcher': kiskadee.fetchers.anitya.__name__,`
		`- 'meta': {`
		`- 'backend': 'gitHub',`
		`- 'homepage': 'https://github.com/GesielFreitas/Cros'`
		`- },`
		`- 'results': {`
		`- 'cppcheck': '<>',`
		`- 'flawfinder': '><'},`
		`- 'fetcher_id': 1}`
		`- self.analysis = {`
		`- 'analyzer_id': 1,`
		`- 'id': 1,`
		`- 'raw': {`
		`- 'results': [`
		`- {'severity': 'warning'},`
		`- {'severity': 'style'},`
		`- {'severity': 'error'}`
		`- ]`
		`- }`
		`- }`
		`-`
		`- def tearDown(self):`
		`- self.session.close()`
		`- model.Base.metadata.drop_all()`
		`-`
		`- def test_return_fetcher_name(self):`
		`- fetcher = kiskadee.fetchers.debian.Fetcher()`
		`- self.assertEqual(fetcher.name, 'debian')`
		`-`
		`- def test_save_package(self):`
		`- self.monitor._save_fetcher(kiskadee.fetchers.debian.Fetcher())`
		`- self.monitor._save_analyzed_project(self.pkg1)`
		`- self.monitor._save_analyzed_project(self.pkg2)`
		`- _pkgs = self.monitor.session.query(Package).all()`
		`- self.assertEqual(len(_pkgs), 2)`
		`- self.assertEqual(_pkgs[0].name, self.pkg1['name'])`
		`- self.assertEqual(_pkgs[1].name, self.pkg2['name'])`
		`-`
		`- def test_save_reports(self):`
		`- _fetcher = model.Fetcher(`
		`- name='kiskadee-fetcher3', target='university'`
		`- )`
		`- _pkg = model.Package(`
		`- name='kiskadee-package', fetcher_id=1`
		`- )`
		`- _version = model.Version(`
		`- number='1.0-rc1', package_id=1`
		`+ def test_dequeue_package_from_fetcher(self):`
		`+ self.example_fetcher.watch()`
		`+ monitored_package = self.monitor.dequeue_package_from_fetchers()`
		`+ self.assertIn("version", monitored_package)`
		`+ self.assertIn("name", monitored_package)`
		`+ self.assertIn("fetcher", monitored_package)`
		`+ self.assertEqual(monitored_package['version'], '0.1')`
		`+`
		`+ def test_run_fetchers_as_threads(self):`
		`+ Monitor.start_fetcher(self.example_fetcher.watch)`
		`+ Monitor.start_fetcher(self.example_fetcher.watch)`
		`+ first_monitored_package = self.monitor.dequeue_package_from_fetchers()`
		`+ second_monitored_package = self.monitor.dequeue_package_from_fetchers()`
		`+ self.assertIn("version", first_monitored_package)`
		`+ self.assertIn("version", second_monitored_package)`
		`+ self.assertIn("name", first_monitored_package)`
		`+ self.assertIn("name", second_monitored_package)`
		`+ self.assertIn("fetcher", first_monitored_package)`
		`+ self.assertIn("fetcher", second_monitored_package)`
		`+`
		`+ def test_send_package_to_runner(self):`
		`+ fetcher = Fetcher(name='example')`
		`+ Package(name='package1', fetcher_id=fetcher.id)`
		`+ self.monitor.get_fetcher_and_package = MagicMock(`
		`+ return_value=[fetcher, {}]`
		`)`
		`- _raw = {`
		`- 'results': [`
		`- {'severity': 'warning'},`
		`- {'severity': 'style'},`
		`- {'severity': 'error'}`
		`- ]`
		`- }`
		`- _analysis = Analysis(`
		`- version_id=1,`
		`- analyzer_id=1,`
		`- id=1,`
		`- raw=_raw`
		`- )`
		`- self.session.add(_fetcher)`
		`- self.session.add(_pkg)`
		`- self.session.add(_version)`
		`- self.session.add(_analysis)`
		`- self.session.commit()`
		`- _dict_analysis = {`
		`- 'results': _raw['results'],`
		`- 'id': 1`
		`- }`
		`- self.monitor._save_reports(_dict_analysis, self.pkg1, 'cppcheck')`
		`- _report = self.monitor.session.query(Report).all()`
		`- self.assertEqual(len(_report), 1)`
		`-`
		`- def test_save_version(self):`
		`- self.monitor._save_fetcher(kiskadee.fetchers.debian.Fetcher())`
		`- self.monitor._save_analyzed_project(self.pkg1)`
		`- _pkgs = self.monitor.session.query(Package).all()`
		`- _version = _pkgs[0].versions[0].number`
		`- self.assertEqual(_version, self.pkg1['version'])`
		`-`
		`- def test_update_version(self):`
		`- self.monitor._save_fetcher(kiskadee.fetchers.debian.Fetcher())`
		`- self.queues.enqueue_project(self.pkg1)`
		`- self.queues.enqueue_project(self.pkg3)`
		`-`
		`- _pkg = self.queues.dequeue_project()`
		`- self.monitor._save_analyzed_project(_pkg)`
		`-`
		`- _pkg = self.queues.dequeue_project()`
		`- self.monitor._save_analyzed_project(_pkg)`
		`-`
		`- _pkgs = self.monitor.session.query(Package).all()`
		`- self.assertEqual(len(_pkgs), 1)`
		`-`
		`- _pkg_versions = self.monitor._query(Package).\`
		`- filter(Package.name == _pkg['name']).first().versions`
		`-`
		`- _first_version = _pkg_versions[0].number`
		`- _current_version = _pkg_versions[-1].number`
		`- self.assertEqual(self.pkg1['version'], _first_version)`
		`- self.assertEqual(_pkg['version'], _current_version)`
		`-`
		`- def test_save_package_anitya(self):`
		`- self.monitor._save_fetcher(kiskadee.fetchers.anitya.Fetcher())`
		`- self.queues.enqueue_project(self.pkg4)`
		`- _pkg = self.queues.dequeue_project()`
		`- self.monitor._save_analyzed_project(_pkg)`
		`- _pkgs = self.monitor.session.query(Package).all()`
		`- self.assertEqual(len(_pkgs), 1)`
		`- self.assertEqual(_pkgs[0].homepage, _pkg['meta']['homepage'])`
		`-`
		`+ self.monitor.is_a_new_package_version = MagicMock(`
		`+ return_value=True`
		`+ )`
		`+ self.monitor.send_package_to_runner(self.data1)`
		`+ self.assertEqual(self.monitor.queues.dequeue_analysis(), self.data1)`

		`- if __name__ == '__main__':`
		`- unittest.main()`
		`+ if __name__ == '__main__':`
		`+ unittest.main()`

kiskadee/tests/integration/test_runner.py

file modified

+38 -78

		`@@ -3,97 +3,57 @@`
		`by the user that is executing the tests. It also needs selinux running in`
		`a permissive mode (setenforce 0)`
		`"""`
		`+`
		`import unittest`
		`- import tempfile`
		`+ import re`
		`+ from unittest.mock import MagicMock`

		`- from kiskadee.runner import Runner`
		`+ import kiskadee.queue`
		`+ import kiskadee.runner`
		`+ import kiskadee.monitor`
		`import kiskadee.fetchers.example`
		`- import kiskadee.fetchers.debian`
		`- from sqlalchemy.orm import sessionmaker`
		`- from kiskadee import model`
		`- from kiskadee.queue import Queues`
		`- from kiskadee.database import Database`
		`-`
		`-`
		`- class AnalyzersTestCase(unittest.TestCase):`
		`- def setUp(self):`
		`- self.engine = Database('db_test').engine`
		`- Session = sessionmaker(bind=self.engine)`
		`- self.session = Session()`
		`- model.Base.metadata.create_all(self.engine)`
		`- model.create_analyzers(self.session)`
		`- self.fetcher = kiskadee.fetchers.debian.Fetcher()`
		`- self.deb_pkg = {'name': 'test',`
		`- 'version': '1.0.0',`
		`- 'fetcher': kiskadee.fetchers.debian.__name__`
		`- }`
		`- self.fetcher = model.Fetcher(`
		`- name='kiskadee-fetcher2', target='university'`
		`- )`
		`- self.session.add(self.fetcher)`
		`- self.session.commit()`
		`- self.runner = Runner()`
		`- self.runner.queues = Queues()`
		`+ import kiskadee.analyzers`

		`- def tearDown(self):`
		`- self.session.close()`
		`- model.Base.metadata.drop_all()`

		`- def test_run_analyzer(self):`
		`+ class RunnerTestCase(unittest.TestCase):`

		`- source_to_analysis = {`
		`- 'name': 'test',`
		`- 'version': '1.0.0',`
		`- 'fetcher': kiskadee.fetchers.example.Fetcher()`
		`- }`
		`+ """Docstring for AnalyzersTestCase. """`

		`- source_path = self.runner._path_to_uncompressed_source(`
		`- source_to_analysis, kiskadee.fetchers.example.Fetcher()`
		`- )`
		`- firehose_report = self.runner.analyze("cppcheck", source_path)`
		`- self.assertIsNotNone(firehose_report)`
		`+ def setUp(self):`
		`+ self.queues = kiskadee.queue.Queues()`
		`+ self.runner = kiskadee.runner.Runner(self.queues)`

		`- def test_generate_a_firehose_report(self):`
		`- source_to_analysis = {`
		`+ self.package = {`
		`'name': 'test',`
		`'version': '1.0.0',`
		`'fetcher': kiskadee.fetchers.example.__name__`
		`}`

		`- self.runner.call_analyzers(source_to_analysis)`
		`- analyzed_pkg = self.runner.queues.dequeue_result()`
		`- self.assertEqual(analyzed_pkg['name'], source_to_analysis['name'])`
		`- self.assertIn('cppcheck', analyzed_pkg['results'])`
		`- self.assertIn('flawfinder', analyzed_pkg['results'])`
		`-`
		`- def test_path_to_uncompressed_source(self):`
		`-`
		`- source_to_analysis = {`
		`- 'name': 'test',`
		`- 'version': '1.0.0',`
		`- 'fetcher': kiskadee.fetchers.example`
		`- }`
		`-`
		`- source_path = self.runner._path_to_uncompressed_source(`
		`- source_to_analysis, kiskadee.fetchers.example.Fetcher()`
		`- )`
		`- tmp_path = tempfile.gettempdir()`
		`- self.assertTrue(source_path.find(tmp_path) >= 0)`
		`- self.assertIsNotNone(source_path)`
		`-`
		`- def test_invalid_path_to_uncompressed_source(self):`
		`-`
		`- source_to_analysis = {`
		`- 'name': 'test',`
		`- 'version': '1.0.0',`
		`- 'fetcher': kiskadee.fetchers.example`
		`- }`
		`-`
		`- source_path = self.runner._path_to_uncompressed_source(`
		`- source_to_analysis, None`
		`- )`
		`-`
		`- self.assertIsNone(source_path)`
		`+ def tearDown(self):`
		`+ """TODO: to be defined1. """`
		`+`
		`+ def test_run_analyzers(self):`
		`+ self.runner.package = self.package`
		`+ self.runner.call_analyzers()`
		`+ result = self.queues.dequeue_result()`
		`+ self.assertEqual(result['name'], 'test')`
		`+ self.assertIn('results', result)`
		`+`
		`+ def test_run_a_single_analyzer(self):`
		`+ self.runner.package = self.package`
		`+ self.runner.fetcher = self.runner.import_package_fetcher()`
		`+ self.runner.get_package_code_path()`
		`+ analyzer = self.runner.fetcher.analyzers()[0]`
		`+ self.assertIsNotNone(re.search('.cppcheck.', analyzer))`
		`+ self.assertIsNone(re.search('.pylint.', analyzer))`
		`+`
		`+ def test_analysis_an_incoming_monitor_package(self):`
		`+ self.db = MagicMock()`
		`+ self.db.filter_by_name = MagicMock(return_value={})`
		`+ self.monitor = kiskadee.monitor.Monitor(self.db, self.queues)`
		`+ self.monitor.send_package_to_runner(self.package)`
		`+ incoming_package = self.runner.queues.dequeue_analysis()`
		`+ self.assertEqual(self.package, incoming_package)`


		`if __name__ == '__main__':`

kiskadee/tests/plugins/test_plugins.py

file modified

+9 -7

		`@@ -70,7 +70,7 @@`
		`self.debian_fetcher._queue_sources_gz_pkgs(temp_dir)`
		`shutil.rmtree(temp_dir)`

		`- some_pkg = kiskadee.queue.Queues.dequeue_project()`
		`+ some_pkg = kiskadee.queue.Queues.dequeue_package()`
		`self.assertTrue(isinstance(some_pkg, dict))`
		`self.assertIn('name', some_pkg)`
		`self.assertIn('version', some_pkg)`
		`@@ -112,11 +112,11 @@`
		`import kiskadee.fetchers.anitya`
		`self.anitya_fetcher = kiskadee.fetchers.anitya.Fetcher()`

		`- self.msg = "anitya {'body':{'msg':{'project':{name: 'urlscan',"\`
		`+ self.msg = "anitya {'body':{'msg':{'package':{name: 'urlscan',"\`
		`"'version':'0.8.5','backend':'GitHub',"\`
		`"'homepage':'https://github.com/firecat53/urlscan'}}}}"`

		`- self.msg1 = "{'body':{'msg':{'project':{name: 'urlscan',"\`
		`+ self.msg1 = "{'body':{'msg':{'package':{name: 'urlscan',"\`
		`"'version':'0.8.5','backend':'GitHub',"\`
		`"'homepage':'https://github.com/firecat53/urlscan'}}}}"`

		`@@ -180,18 +180,20 @@`

		`def test_get_sources(self):`

		`- def mock_github(self, source_data, path):`
		`+ def mock_github(self, fetcher, source_data, path):`
		`return 'kiskadee/tests/test_source/Sources.gz'`

		`kiskadee.fetchers.anitya.Backends.github = mock_github`
		`source_data = {'meta': {'backend': 'GitHub'}}`
		`source_path = self.anitya_fetcher.get_sources(source_data)`
		`- self.assertEqual(source_path, mock_github("self", "foo", "bla"))`
		`+ self.assertEqual(source_path, mock_github(`
		`+ "self", "example", "foo", "bla")`
		`+ )`

		`def test_create_package_dict(self):`

		`- self.anitya_fetcher.project_to_enqueue(self.msg)`
		`- _dict = kiskadee.queue.Queues().dequeue_project()`
		`+ self.anitya_fetcher.package_to_enqueue(self.msg)`
		`+ _dict = kiskadee.queue.Queues().dequeue_package()`
		`self.assertEqual(_dict['name'], 'urlscan')`
		`self.assertEqual(_dict['version'], '0.8.5')`
		`self.assertEqual(_dict['meta']['backend'], 'GitHub')`

kiskadee/tests/test_api.py

file removed

-118

		`@@ -1,118 +0,0 @@`
		`- import json`
		`- import unittest`
		`- from sqlalchemy.orm import sessionmaker`
		`-`
		`- import kiskadee`
		`- from kiskadee.runner import Runner`
		`- from kiskadee.monitor import Monitor`
		`- import kiskadee.api.app`
		`- import kiskadee.fetchers.example`
		`-`
		`-`
		`- class ApiTestCase(unittest.TestCase):`
		`-`
		`- def setUp(self):`
		`- kiskadee.api.app.kiskadee.testing = True`
		`- self.engine = kiskadee.database.Database('db_test').engine`
		`- Session = sessionmaker(bind=self.engine)`
		`- self.session = Session()`
		`- self.app = kiskadee.api.app.kiskadee.test_client()`
		`- kiskadee.model.create_analyzers(self.session)`
		`- kiskadee.model.Base.metadata.create_all(self.engine)`
		`- kiskadee.model.create_analyzers(self.session)`
		`- fetcher = kiskadee.model.Fetcher(`
		`- name='kiskadee-fetcher', target='university'`
		`- )`
		`- pkg = kiskadee.model.Package(`
		`- name='kiskadee-package', fetcher_id=1`
		`- )`
		`- version = kiskadee.model.Version(`
		`- number='7.23', package_id=1`
		`- )`
		`- analysis = kiskadee.model.Analysis(`
		`- version_id=1, analyzer_id=1,`
		`- raw={`
		`- 'results': [`
		`- {'severity': 'warning'},`
		`- {'severity': 'style'},`
		`- {'severity': 'error'}`
		`- ]`
		`- }`
		`- )`
		`- report = kiskadee.model.Report(`
		`- analysis_id=1`
		`- )`
		`- self.session.add(fetcher)`
		`- self.session.add(pkg)`
		`- self.session.add(version)`
		`- self.session.add(analysis)`
		`- self.session.add(report)`
		`- self.session.commit()`
		`- self.runner = Runner()`
		`- self.monitor = Monitor(self.session)`
		`- self.runner.kiskadee_queue = kiskadee.queue.KiskadeeQueue()`
		`-`
		`- def tearDown(self):`
		`- self.session.close()`
		`- kiskadee.model.Base.metadata.drop_all()`
		`-`
		`- def test_get_fetchers(self):`
		`- def mock_kiskadee_db_session():`
		`- return self.session`
		`-`
		`- kiskadee.api.app.kiskadee_db_session = mock_kiskadee_db_session`
		`- response = self.app.get("/fetchers")`
		`- self.assertIn("fetchers", json.loads(response.data.decode("utf-8")))`
		`-`
		`- def test_get_activated_fetcher(self):`
		`-`
		`- def mock_kiskadee_db_session():`
		`- return self.session`
		`-`
		`- kiskadee.api.app.kiskadee_db_session = mock_kiskadee_db_session`
		`- response = self.app.get("/fetchers")`
		`- response_as_json = json.loads(response.data.decode("utf-8"))`
		`- fetcher_name = response_as_json["fetchers"][0]["name"]`
		`- self.assertEqual("kiskadee-fetcher", fetcher_name)`
		`-`
		`- def test_get_analysis_as_json(self):`
		`- source_to_analysis = {`
		`- 'name': 'test',`
		`- 'version': '1.0.0',`
		`- 'fetcher_id': '1',`
		`- 'fetcher': kiskadee.fetchers.example.Fetcher()`
		`- }`
		`-`
		`- def mock_kiskadee_db_session():`
		`- return self.session`
		`-`
		`- self.runner.call_analyzers(source_to_analysis)`
		`- analyzed_pkg = self.runner.kiskadee_queue.dequeue_result()`
		`- self.monitor._save_analyzed_pkg(analyzed_pkg)`
		`- kiskadee.api.app.kiskadee_db_session = mock_kiskadee_db_session`
		`- response = self.app.get("/analysis/test/1.0.0")`
		`- response_data = json.loads(response.data.decode("utf-8"))`
		`- self.assertTrue(len(response_data) >= 1)`
		`-`
		`- def test_get_analysis_results(self):`
		`- def mock_kiskadee_db_session():`
		`- return self.session`
		`-`
		`- kiskadee.api.app.kiskadee_db_session = mock_kiskadee_db_session`
		`- response = self.app.get("/analysis/kiskadee-package/7.23/1/results")`
		`- response_data = json.loads(response.data.decode("utf-8"))`
		`- self.assertIn("analysis_results", response_data)`
		`- self.assertTrue(len(response_data["analysis_results"]) > 0)`
		`-`
		`- def test_get_analysis_reports(self):`
		`- def mock_kiskadee_db_session():`
		`- return self.session`
		`-`
		`- kiskadee.api.app.kiskadee_db_session = mock_kiskadee_db_session`
		`- response = self.app.get("/analysis/kiskadee-package/7.23/1/reports")`
		`- response_data = json.loads(response.data.decode("utf-8"))`
		`- self.assertIn("analysis_report", response_data)`
		`-`
		`-`
		`- if __name__ == '__main__':`
		`- unittest.main()`

kiskadee/tests/test_model.py

file removed

-179

		`@@ -1,179 +0,0 @@`
		`- import unittest`
		`- from sqlalchemy import exc`
		`- from sqlalchemy.orm import sessionmaker`
		`-`
		`- from kiskadee import model`
		`- from kiskadee.database import Database`
		`-`
		`-`
		`- class ModelTestCase(unittest.TestCase):`
		`-`
		`- def setUp(self):`
		`- self.engine = Database('db_test').engine`
		`- Session = sessionmaker(bind=self.engine)`
		`- self.session = Session()`
		`- model.Base.metadata.create_all(self.engine)`
		`- model.create_analyzers(self.session)`
		`- self.fetcher = model.Fetcher(`
		`- name='kiskadee-fetcher', target='university'`
		`- )`
		`- self.package = model.Package(name='python-kiskadee')`
		`- self.version = model.Version(number='1.0-rc1')`
		`- self.fetcher.packages.append(self.package)`
		`- self.package.versions.append(self.version)`
		`- self.session.add(self.package)`
		`- self.session.add(self.fetcher)`
		`- self.session.add(self.version)`
		`-`
		`- self.analysis = model.Analysis(`
		`- analyzer_id=1,`
		`- version_id=1,`
		`- raw=""`
		`- )`
		`- self.session.add(self.analysis)`
		`- self.report = model.Report(`
		`- analysis_id=1,`
		`- )`
		`- self.session.add(self.report)`
		`- self.session.commit()`
		`-`
		`- def tearDown(self):`
		`- self.session.close()`
		`- model.Base.metadata.drop_all()`
		`-`
		`- def test_query_fetcher(self):`
		`- fetchers = self.session.query(model.Fetcher).all()`
		`- self.assertEqual(fetchers, [self.fetcher])`
		`-`
		`- def test_query_package(self):`
		`- packages = self.session.query(model.Package).all()`
		`- self.assertEqual(packages, [self.package])`
		`-`
		`- def test_query_version(self):`
		`- versions = self.session.query(model.Version).all()`
		`- self.assertEqual(versions, [self.version])`
		`-`
		`- def test_query_report(self):`
		`- reports = self.session.query(model.Report).all()`
		`- self.assertEqual(reports, [self.report])`
		`-`
		`- def test_add_fetcher(self):`
		`- fetchers = self.session.query(model.Fetcher).all()`
		`- self.assertEqual(len(fetchers), 1)`
		`- self.session.add(model.Fetcher(name='foo', target='bar'))`
		`- fetchers = self.session.query(model.Fetcher).all()`
		`- self.assertEqual(len(fetchers), 2)`
		`-`
		`- def test_add_version_without_package(self):`
		`- version = model.Version(number='3.1')`
		`- self.session.add(version)`
		`- with self.assertRaises(exc.IntegrityError):`
		`- self.session.commit()`
		`-`
		`- def test_add_report_without_analysis(self):`
		`- report = model.Report(`
		`- )`
		`- self.session.add(report)`
		`- with self.assertRaises(exc.IntegrityError):`
		`- self.session.commit()`
		`-`
		`- def test_add_package_without_fetcher(self):`
		`- package = model.Package(name='foo-bar')`
		`- self.session.add(package)`
		`- with self.assertRaises(exc.IntegrityError):`
		`- self.session.commit()`
		`-`
		`- def test_unique_package_in_fetcher(self):`
		`- package_1 = model.Package(name='foo-bar')`
		`- package_2 = model.Package(name='foo-bar')`
		`- self.fetcher.packages.append(package_1)`
		`- self.fetcher.packages.append(package_2)`
		`- with self.assertRaises(exc.IntegrityError):`
		`- self.session.commit()`
		`-`
		`- def test_unique_version_for_package(self):`
		`- package_version_1 = model.Version(number='1.0')`
		`- package_version_2 = model.Version(number='1.0')`
		`- self.package.versions.append(package_version_1)`
		`- self.package.versions.append(package_version_2)`
		`- with self.assertRaises(exc.IntegrityError):`
		`- self.session.commit()`
		`-`
		`- def test_compose_kiskadee_source(self):`
		`- _analyzer = self.session.query(model.Analyzer)\`
		`- .filter(model.Analyzer.name == "cppcheck").first()`
		`- package = model.Package(`
		`- name='bla',`
		`- fetcher_id=self.fetcher.id`
		`- )`
		`- package_version = model.Version(`
		`- number='1.0.1',`
		`- package_id=package.id`
		`- )`
		`-`
		`- package_analysis = model.Analysis(`
		`- raw="<>",`
		`- analyzer_id=_analyzer.id,`
		`- version_id=package_version.id`
		`- )`
		`-`
		`- self.fetcher.packages.append(package)`
		`- package.versions.append(package_version)`
		`- package_version.analysis.append(package_analysis)`
		`-`
		`- self.assertEqual(package.versions[0].analysis[0].raw, "<>")`
		`-`
		`- def test_save_several_analysis(self):`
		`-`
		`- _analyzer1 = (`
		`- self.session.query(model.Analyzer)`
		`- .filter(model.Analyzer.name == "cppcheck").first()`
		`- )`
		`- _analyzer2 = (`
		`- self.session.query(model.Analyzer)`
		`- .filter(model.Analyzer.name == "flawfinder").first()`
		`- )`
		`-`
		`- package = model.Package(`
		`- name='bla',`
		`- fetcher_id=self.fetcher.id`
		`- )`
		`- package_version = model.Version(`
		`- number='1.0.1',`
		`- package_id=package.id`
		`- )`
		`-`
		`- self.fetcher.packages.append(package)`
		`- package.versions.append(package_version)`
		`-`
		`- self.session.add(package)`
		`- self.session.add(package_version)`
		`- self.session.commit()`
		`-`
		`- package_analysis1 = model.Analysis(`
		`- raw="<>",`
		`- analyzer_id=_analyzer1.id,`
		`- version_id=package_version.id`
		`- )`
		`- package_analysis2 = model.Analysis(`
		`- raw="><",`
		`- analyzer_id=_analyzer2.id,`
		`- version_id=package_version.id`
		`- )`
		`-`
		`- self.session.add(package_analysis1)`
		`- self.session.add(package_analysis2)`
		`- self.session.commit()`
		`-`
		`- saved_package = (`
		`- self.session.query(model.Package)`
		`- .filter(model.Package.name == 'bla').first()`
		`- )`
		`- analysis = saved_package.versions[-1].analysis`
		`- self.assertEqual(len(analysis), 2)`
		`- self.assertEqual(analysis[0].raw, "<>")`
		`- self.assertEqual(analysis[1].raw, "><")`
		`-`
		`-`
		`- if __name__ == '__main__':`
		`- unittest.main()`

kiskadee/tests/test_monitor.py

file removed

-190

		`@@ -1,190 +0,0 @@`
		`- import unittest`
		`- from sqlalchemy.orm import sessionmaker`
		`-`
		`- from kiskadee import model`
		`- from kiskadee.monitor import Monitor`
		`- from kiskadee.queue import packages_queue`
		`- from kiskadee.model import Package, Fetcher, create_analyzers, Report, Analysis`
		`- import kiskadee.queue`
		`- import kiskadee.fetchers.debian`
		`- import kiskadee.fetchers.anitya`
		`- from kiskadee.database import Database`
		`-`
		`-`
		`- class MonitorTestCase(unittest.TestCase):`
		`-`
		`- def setUp(self):`
		`- self.engine = Database('db_test').engine`
		`- Session = sessionmaker(bind=self.engine)`
		`- self.session = Session()`
		`- self.monitor = Monitor(self.session)`
		`- model.Base.metadata.create_all(self.engine)`
		`- create_analyzers(self.session)`
		`- self.pkg1 = {`
		`- 'name': 'curl',`
		`- 'version': '7.52.1-5',`
		`- 'fetcher': kiskadee.fetchers.debian,`
		`- 'meta': {'directory': 'pool/main/c/curl'},`
		`- 'results': {`
		`- 'cppcheck': '<>',`
		`- 'flawfinder': '><'},`
		`- 'fetcher_id': 1}`
		`-`
		`- self.pkg2 = {'name': 'urlscan',`
		`- 'version': '0.8.2',`
		`- 'fetcher': kiskadee.fetchers.debian,`
		`- 'meta': {'directory': 'pool/main/u/urlscan'},`
		`- 'results': {`
		`- 'cppcheck': '<>',`
		`- 'flawfinder': '><'},`
		`- 'fetcher_id': 1}`
		`-`
		`- self.pkg3 = {'name': 'curl',`
		`- 'version': '7.52.2-5',`
		`- 'fetcher': kiskadee.fetchers.debian,`
		`- 'meta': {'directory': 'pool/main/c/curl'},`
		`- 'results': {`
		`- 'cppcheck': '<>',`
		`- 'flawfinder': '><'},`
		`- 'fetcher_id': 1}`
		`- self.pkg4 = {'name': 'urlanitya',`
		`- 'version': '0.11',`
		`- 'fetcher': kiskadee.fetchers.anitya,`
		`- 'meta': {`
		`- 'backend': 'gitHub',`
		`- 'homepage': 'https://github.com/GesielFreitas/Cros'`
		`- },`
		`- 'results': {`
		`- 'cppcheck': '<>',`
		`- 'flawfinder': '><'},`
		`- 'fetcher_id': 1}`
		`- self.analysis = {`
		`- 'analyzer_id': 1,`
		`- 'id': 1,`
		`- 'raw': {`
		`- 'results': [`
		`- {'severity': 'warning'},`
		`- {'severity': 'style'},`
		`- {'severity': 'error'}`
		`- ]`
		`- }`
		`- }`
		`-`
		`- def tearDown(self):`
		`- self.session.close()`
		`- model.Base.metadata.drop_all()`
		`-`
		`- def test_dequeue_package(self):`
		`- packages_queue.put(self.pkg1)`
		`- _pkg = self.monitor.dequeue_package()`
		`- self.assertTrue(isinstance(_pkg, dict))`
		`-`
		`- def test_return_fetcher_name(self):`
		`- fetcher = kiskadee.fetchers.debian.Fetcher()`
		`- self.assertEqual(fetcher.name, 'debian')`
		`-`
		`- def test_save_some_fetcher(self):`
		`- fetcher = kiskadee.fetchers.debian.Fetcher()`
		`- self.monitor._save_fetcher(fetcher)`
		`- _fetchers = self.monitor.session.query(Fetcher).all()`
		`- self.assertEqual(len(_fetchers), 1)`
		`- self.assertEqual(_fetchers[0].name, 'debian')`
		`- self.assertEqual(_fetchers[0].description,`
		`- fetcher.config['description'])`
		`-`
		`- def test_save_package(self):`
		`- self.monitor._save_fetcher(kiskadee.fetchers.debian.Fetcher())`
		`- packages_queue.put(self.pkg1)`
		`- packages_queue.put(self.pkg2)`
		`-`
		`- _pkg = self.monitor.dequeue_package()`
		`- self.monitor._save_analyzed_pkg(_pkg)`
		`- _pkgs = self.monitor.session.query(Package).all()`
		`- self.assertEqual(len(_pkgs), 1)`
		`- self.assertEqual(_pkgs[0].name, _pkg['name'])`
		`-`
		`- _pkg = self.monitor.dequeue_package()`
		`- self.monitor._save_analyzed_pkg(_pkg)`
		`- _pkgs = self.monitor.session.query(Package).all()`
		`- self.assertEqual(len(_pkgs), 2)`
		`- self.assertEqual(_pkgs[1].name, _pkg['name'])`
		`-`
		`- def test_save_reports(self):`
		`- _fetcher = model.Fetcher(`
		`- name='kiskadee-fetcher', target='university'`
		`- )`
		`- _pkg = model.Package(`
		`- name='kiskadee-package', fetcher_id=1`
		`- )`
		`- _version = model.Version(`
		`- number='1.0-rc1', package_id=1`
		`- )`
		`- _raw = {`
		`- 'results': [`
		`- {'severity': 'warning'},`
		`- {'severity': 'style'},`
		`- {'severity': 'error'}`
		`- ]`
		`- }`
		`- _analysis = Analysis(`
		`- version_id=1,`
		`- analyzer_id=1,`
		`- id=1,`
		`- raw=_raw`
		`- )`
		`- self.session.add(_fetcher)`
		`- self.session.add(_pkg)`
		`- self.session.add(_version)`
		`- self.session.add(_analysis)`
		`- self.session.commit()`
		`- _dict_analysis = {`
		`- 'results': _raw['results'],`
		`- 'id': 1`
		`- }`
		`- self.monitor._save_reports(_dict_analysis, self.pkg1, 'cppcheck')`
		`- _report = self.monitor.session.query(Report).all()`
		`- self.assertEqual(len(_report), 1)`
		`-`
		`- def test_save_version(self):`
		`- self.monitor._save_fetcher(kiskadee.fetchers.debian.Fetcher())`
		`- self.monitor._save_analyzed_pkg(self.pkg1)`
		`- _pkgs = self.monitor.session.query(Package).all()`
		`- _version = _pkgs[0].versions[0].number`
		`- self.assertEqual(_version, self.pkg1['version'])`
		`-`
		`- def test_update_version(self):`
		`- self.monitor._save_fetcher(kiskadee.fetchers.debian.Fetcher())`
		`- packages_queue.put(self.pkg1)`
		`- packages_queue.put(self.pkg3)`
		`-`
		`- _pkg = self.monitor.dequeue_package()`
		`- self.monitor._save_analyzed_pkg(_pkg)`
		`-`
		`- _pkg = self.monitor.dequeue_package()`
		`- self.monitor._save_analyzed_pkg(_pkg)`
		`-`
		`- _pkgs = self.monitor.session.query(Package).all()`
		`- self.assertEqual(len(_pkgs), 1)`
		`-`
		`- _pkg_versions = self.monitor._query(Package).\`
		`- filter(Package.name == _pkg['name']).first().versions`
		`-`
		`- _first_version = _pkg_versions[0].number`
		`- _current_version = _pkg_versions[-1].number`
		`- self.assertEqual(self.pkg1['version'], _first_version)`
		`- self.assertEqual(_pkg['version'], _current_version)`
		`-`
		`- def test_save_package_anitya(self):`
		`- self.monitor._save_fetcher(kiskadee.fetchers.anitya.Fetcher())`
		`- packages_queue.put(self.pkg4)`
		`-`
		`- _pkg = self.monitor.dequeue_package()`
		`- self.monitor._save_analyzed_pkg(_pkg)`
		`-`
		`- _pkgs = self.monitor.session.query(Package).all()`
		`- self.assertEqual(len(_pkgs), 1)`
		`- self.assertEqual(_pkgs[0].homepage, _pkg['meta']['homepage'])`
		`-`
		`-`
		`- if __name__ == '__main__':`
		`- unittest.main()`

kiskadee/tests/test_plugins.py

file removed

-207

		`@@ -1,207 +0,0 @@`
		`- import unittest`
		`- import os`
		`- import shutil`
		`- import tempfile`
		`- import zmq`
		`- import threading`
		`- import time`
		`-`
		`- import kiskadee.queue`
		`- import kiskadee`
		`-`
		`-`
		`- class FetchersTestCase(unittest.TestCase):`
		`- def test_loading(self):`
		`- _config = kiskadee.config`
		`- _config['debian_fetcher'] = {'active': 'no'}`
		`- _config['juliet_fetcher'] = {'active': 'yes'}`
		`- _config['example_fetcher'] = {'active': 'yes'}`
		`- kiskadee.config = _config`
		`- fetchers = kiskadee.load_fetchers()`
		`- for fetcher in fetchers:`
		`- name_index = len(fetcher.__name__.split('.')) - 1`
		`- name = fetcher.__name__.split('.')[name_index]`
		`- self.assertTrue(name != 'debian')`
		`- kiskadee.config['example_fetcher'] = {`
		`- 'target': 'example',`
		`- 'description': 'SAMATE Juliet test suite',`
		`- 'analyzers': 'cppcheck flawfinder',`
		`- 'active': 'yes'`
		`- }`
		`-`
		`-`
		`- class DebianFetcherTestCase(unittest.TestCase):`
		`-`
		`- def setUp(self):`
		`- import kiskadee.fetchers.debian`
		`- self.debian_fetcher = kiskadee.fetchers.debian.Fetcher()`
		`- self.data = self.debian_fetcher.config`
		`-`
		`- def _download_sources_gz(self):`
		`- tmp_path = tempfile.gettempdir()`
		`- path = tempfile.mkdtemp(dir=tmp_path)`
		`- source = 'kiskadee/tests/test_source/Sources.gz'`
		`- shutil.copy2(source, path)`
		`- return path`
		`-`
		`- def test_mount_sources_gz_url(self):`
		`- mirror = self.data['target']`
		`- release = self.data['release']`
		`- url = self.debian_fetcher._sources_gz_url()`
		`- expected_url = "%s/dists/%s/main/source/Sources.gz" % (mirror, release)`
		`- self.assertEqual(url, expected_url)`
		`-`
		`- def test_uncompress_sources_gz(self):`
		`- tmp_path = tempfile.gettempdir()`
		`- temp_dir = tempfile.mkdtemp(dir=tmp_path)`
		`- self.debian_fetcher._download_sources_gz = self._download_sources_gz`
		`- temp_dir = self.debian_fetcher._download_sources_gz()`
		`- self.debian_fetcher._uncompress_gz(temp_dir)`
		`- files = os.listdir(temp_dir)`
		`- shutil.rmtree(temp_dir)`
		`- self.assertTrue('Sources' in files)`
		`-`
		`- def test_enqueue_a_valid_pkg(self):`
		`- tmp_path = tempfile.gettempdir()`
		`- temp_dir = tempfile.mkdtemp(dir=tmp_path)`
		`- self.debian_fetcher._download_sources_gz = self._download_sources_gz`
		`- temp_dir = self.debian_fetcher._download_sources_gz()`
		`- self.debian_fetcher._uncompress_gz(temp_dir)`
		`- self.debian_fetcher._queue_sources_gz_pkgs(temp_dir)`
		`- shutil.rmtree(temp_dir)`
		`-`
		`- some_pkg = kiskadee.queue.packages_queue.get()`
		`- self.assertTrue(isinstance(some_pkg, dict))`
		`- self.assertIn('name', some_pkg)`
		`- self.assertIn('version', some_pkg)`
		`- self.assertIn('fetcher', some_pkg)`
		`- self.assertIn('meta', some_pkg)`
		`- self.assertIn('directory', some_pkg['meta'])`
		`-`
		`- def test_mount_dsc_url(self):`
		`- expected_dsc_url = ("http://ftp.us.debian.org" +`
		`- "/debian/pool/main/0/0ad/0ad_0.0.21-2.dsc")`
		`- sample_package = {'name': '0ad',`
		`- 'version': '0.0.21-2',`
		`- 'meta': {'directory': 'pool/main/0/0ad'}}`
		`- url = self.debian_fetcher._dsc_url(sample_package)`
		`- self.assertEqual(expected_dsc_url, url)`
		`-`
		`- def test_compare_gt_version(self):`
		`- new = '1.1.1'`
		`- old = '1.1.0'`
		`- result = self.debian_fetcher.compare_versions(new, old)`
		`- self.assertTrue(result)`
		`-`
		`- def test_compare_smallest_version(self):`
		`- new = '8.5-2'`
		`- old = '8.6-0'`
		`- result = self.debian_fetcher.compare_versions(new, old)`
		`- self.assertFalse(result)`
		`-`
		`- def test_compare_equal_version(self):`
		`- new = '3.3.3-0'`
		`- old = '3.3.3-0'`
		`- result = self.debian_fetcher.compare_versions(new, old)`
		`- self.assertFalse(result)`
		`-`
		`-`
		`- class TestAnityaFetcher(unittest.TestCase):`
		`-`
		`- def setUp(self):`
		`- import kiskadee.fetchers.anitya`
		`- self.anitya_fetcher = kiskadee.fetchers.anitya.Fetcher()`
		`-`
		`- self.msg = "anitya {'body':{'msg':{'project':{name: 'urlscan',"\`
		`- "'version':'0.8.5','backend':'GitHub',"\`
		`- "'homepage':'https://github.com/firecat53/urlscan'}}}}"`
		`-`
		`- self.msg1 = "{'body':{'msg':{'project':{name: 'urlscan',"\`
		`- "'version':'0.8.5','backend':'GitHub',"\`
		`- "'homepage':'https://github.com/firecat53/urlscan'}}}}"`
		`-`
		`- def test_connect_to_zmq(self):`
		`-`
		`- def zmq_server():`
		`- context = zmq.Context()`
		`- socket = context.socket(zmq.PUB)`
		`- socket.bind("tcp://*:7776")`
		`-`
		`- zmq_server()`
		`- socket = self.anitya_fetcher._connect_to_zmq("7776", "anitya")`
		`- self.assertIsNotNone(socket)`
		`-`
		`- def test_receive_msg_from_zmq(self):`
		`- """definitely this is not a unit test, but is important to kiskadee`
		`- be able to interact correctly with ZeroMQ.`
		`- We need to define other test levels to kiskadee asap.`
		`- When we do that, we can move integration tests`
		`- to a proper place. For now we will maintain this test here"""`
		`-`
		`- def zmq_server():`
		`- context = zmq.Context()`
		`- socket = context.socket(zmq.PUB)`
		`- socket.bind("tcp://*:7776")`
		`- time.sleep(1)`
		`- socket.send_string("%s" % (self.msg))`
		`- time.sleep(1)`
		`-`
		`- def receive_msg_from_server():`
		`- client_socket = self.anitya_fetcher._connect_to_zmq(`
		`- "7776", "anitya")`
		`- if client_socket:`
		`- response = client_socket.recv_string()`
		`- results[0] = response[response.find(" ")+1::]`
		`- else:`
		`- results[0] = "invalid"`
		`-`
		`- results = [None]`
		`-`
		`- client_as_thread = threading.Thread(target=receive_msg_from_server)`
		`- server_as_thread = threading.Thread(`
		`- target=zmq_server)`
		`-`
		`- server_as_thread.start()`
		`- client_as_thread.start()`
		`- server_as_thread.join()`
		`- self.assertEqual(self.msg1, results[0])`
		`-`
		`- def test_compare_versions(self):`
		`- is_greater = self.anitya_fetcher.compare_versions('0.8.5-2', '0.8.5-1')`
		`- self.assertTrue(is_greater)`
		`-`
		`- def test_load_backend(self):`
		`- backend = self.anitya_fetcher._load_backend('github')`
		`- self.assertIsNotNone(backend)`
		`-`
		`- def test_not_load_backend(self):`
		`- backend = self.anitya_fetcher._load_backend('foo')`
		`- self.assertEqual(backend, {})`
		`-`
		`- def test_get_sources(self):`
		`-`
		`- def mock_github(self, fetcher, source_data, path):`
		`- return 'kiskadee/tests/test_source/Sources.gz'`
		`-`
		`- kiskadee.fetchers.anitya.Backends.github = mock_github`
		`- source_data = {'meta': {'backend': 'GitHub'}}`
		`- source_path = self.anitya_fetcher.get_sources(source_data)`
		`- self.assertEqual(source_path,`
		`- mock_github("self", "fetcher", "foo", "bla"))`
		`-`
		`- def test_create_package_dict(self):`
		`-`
		`- self.anitya_fetcher._create_package_dict(self.msg)`
		`- _dict = kiskadee.queue.packages_queue.get()`
		`- self.assertEqual(_dict['name'], 'urlscan')`
		`- self.assertEqual(_dict['version'], '0.8.5')`
		`- self.assertEqual(_dict['meta']['backend'], 'GitHub')`
		`- self.assertEqual(`
		`- _dict['meta']['homepage'],`
		`- 'https://github.com/firecat53/urlscan'`
		`- )`
		`- self.assertEqual(_dict['fetcher'].name, 'anitya')`
		`-`
		`-`
		`- if __name__ == '__main__':`
		`- unittest.main()`

kiskadee/tests/test_queue.py

file removed

-23

		`@@ -1,23 +0,0 @@`
		`- import unittest`
		`- import kiskadee.queue`
		`- import kiskadee.fetchers.example`
		`-`
		`-`
		`- class QueueTestCase(unittest.TestCase):`
		`-`
		`- def test_enqueue_wrapper_pkg(self):`
		`-`
		`- def my_dict():`
		`- return {`
		`- 'name': 'bar',`
		`- 'fetcher': kiskadee.fetchers.example.Fetcher(),`
		`- 'version': '1.0.0'`
		`- }`
		`-`
		`- enque_my_dict = kiskadee.queue.package_enqueuer(my_dict)`
		`- enque_my_dict()`
		`- self.assertTrue(isinstance(kiskadee.queue.packages_queue.get(), dict))`
		`-`
		`-`
		`- if __name__ == '__main__':`
		`- unittest.main()`

kiskadee/tests/test_report.py

file removed

-50

		`@@ -1,50 +0,0 @@`
		`- import unittest`
		`-`
		`- from sqlalchemy.orm import sessionmaker`
		`- from kiskadee.report import CppcheckReport, FlawfinderReport`
		`- from kiskadee.database import Database`
		`-`
		`-`
		`- class ReportTestCase(unittest.TestCase):`
		`-`
		`- def setUp(self):`
		`- self.engine = Database('db_test').engine`
		`- Session = sessionmaker(bind=self.engine)`
		`- self.session = Session()`
		`-`
		`- def tearDown(self):`
		`- self.session.close()`
		`-`
		`- def test_compute_cppcheck_reports(self):`
		`- _reports = [`
		`- {'severity': 'warning'},`
		`- {'severity': 'error'},`
		`- {'severity': 'style'},`
		`- {'some-attribute': None}`
		`- ]`
		`- _cpp_reporter = CppcheckReport(_reports)`
		`- result = _cpp_reporter._compute_reports('cppcheck')`
		`- self.assertEqual(len(result.keys()), 3)`
		`- self.assertEqual(result['warning'], 1)`
		`- self.assertEqual(result['style'], 1)`
		`- self.assertEqual(result['error'], 1)`
		`-`
		`- def test_compute_flawfinder_reports(self):`
		`- _reports = [`
		`- {'severity': '5'},`
		`- {'severity': '4'},`
		`- {'severity': '3'},`
		`- {'some-attribute': None}`
		`- ]`
		`- _flawfinder_reporter = FlawfinderReport(_reports)`
		`- result = _flawfinder_reporter._compute_reports('flawfinder')`
		`- self.assertEqual(len(result.keys()), 5)`
		`- self.assertEqual(result['severity_5'], 1)`
		`- self.assertEqual(result['severity_4'], 1)`
		`- self.assertEqual(result['severity_3'], 1)`
		`- self.assertEqual(result['severity_2'], 0)`
		`- self.assertEqual(result['severity_1'], 0)`
		`-`
		`-`
		`- if __name__ == '__main__':`
		`- unittest.main()`

kiskadee/tests/test_runner.py

file removed

-98

		`@@ -1,98 +0,0 @@`
		`- import unittest`
		`- import tempfile`
		`-`
		`- from kiskadee.runner import Runner`
		`- import kiskadee.fetchers.example`
		`- import kiskadee.fetchers.debian`
		`- from sqlalchemy.orm import sessionmaker`
		`- from kiskadee import model`
		`- from kiskadee.queue import KiskadeeQueue`
		`- from kiskadee.database import Database`
		`-`
		`-`
		`- class AnalyzersTestCase(unittest.TestCase):`
		`-`
		`- def setUp(self):`
		`- self.engine = Database('db_test').engine`
		`- Session = sessionmaker(bind=self.engine)`
		`- self.session = Session()`
		`- model.Base.metadata.create_all(self.engine)`
		`- model.create_analyzers(self.session)`
		`- self.fetcher = kiskadee.fetchers.debian.Fetcher()`
		`- self.deb_pkg = {'name': 'test',`
		`- 'version': '1.0.0',`
		`- 'fetcher': kiskadee.fetchers.debian.Fetcher()`
		`- }`
		`- self.fetcher = model.Fetcher(`
		`- name='kiskadee-fetcher', target='university'`
		`- )`
		`- self.session.add(self.fetcher)`
		`- self.session.commit()`
		`- kiskadee_queue = KiskadeeQueue()`
		`- self.runner = Runner()`
		`- self.runner.kiskadee_queue = kiskadee_queue`
		`-`
		`- def tearDown(self):`
		`- self.session.close()`
		`- model.Base.metadata.drop_all()`
		`-`
		`- def test_run_analyzer(self):`
		`-`
		`- source_to_analysis = {`
		`- 'name': 'test',`
		`- 'version': '1.0.0',`
		`- 'fetcher': kiskadee.fetchers.example.Fetcher()`
		`- }`
		`-`
		`- source_path = self.runner._path_to_uncompressed_source(`
		`- source_to_analysis, kiskadee.fetchers.example.Fetcher()`
		`- )`
		`- firehose_report = self.runner.analyze("cppcheck", source_path)`
		`- self.assertIsNotNone(firehose_report)`
		`-`
		`- def test_generate_a_firehose_report(self):`
		`- source_to_analysis = {`
		`- 'name': 'test',`
		`- 'version': '1.0.0',`
		`- 'fetcher': kiskadee.fetchers.example.Fetcher()`
		`- }`
		`-`
		`- self.runner.call_analyzers(source_to_analysis)`
		`- analyzed_pkg = self.runner.kiskadee_queue.dequeue_result()`
		`-`
		`- self.assertEqual(analyzed_pkg['name'], source_to_analysis['name'])`
		`- self.assertIn('cppcheck', analyzed_pkg['results'])`
		`- self.assertIn('flawfinder', analyzed_pkg['results'])`
		`-`
		`- def test_path_to_uncompressed_source(self):`
		`-`
		`- source_to_analysis = {`
		`- 'name': 'test',`
		`- 'version': '1.0.0',`
		`- 'fetcher': kiskadee.fetchers.example`
		`- }`
		`-`
		`- source_path = self.runner._path_to_uncompressed_source(`
		`- source_to_analysis, kiskadee.fetchers.example.Fetcher()`
		`- )`
		`- tmp_path = tempfile.gettempdir()`
		`- self.assertTrue(source_path.find(tmp_path) >= 0)`
		`- self.assertIsNotNone(source_path)`
		`-`
		`- def test_invalid_path_to_uncompressed_source(self):`
		`-`
		`- source_to_analysis = {`
		`- 'name': 'test',`
		`- 'version': '1.0.0',`
		`- 'fetcher': kiskadee.fetchers.example`
		`- }`
		`-`
		`- source_path = self.runner._path_to_uncompressed_source(`
		`- source_to_analysis, None`
		`- )`
		`-`
		`- self.assertIsNone(source_path)`
		`-`
		`-`
		`- if __name__ == '__main__':`
		`- unittest.main()`

kiskadee/tests/test_style.py

file removed

-20

		`@@ -1,20 +0,0 @@`
		`- # Thanks bodhi`
		`- # https://github.com/fedora-infra/bodhi/blob/develop/bodhi/tests/test_style.py`
		`-`
		`- import os`
		`- import subprocess`
		`- import unittest`
		`-`
		`- KISKADEE_PATH = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))`
		`-`
		`-`
		`- class TestStyle(unittest.TestCase):`
		`- def test_with_flake8(self):`
		`- """Enforce PEP-8"""`
		`- flake8_command = ['flake8', KISKADEE_PATH]`
		`- self.assertEqual(subprocess.call(flake8_command), 0)`
		`-`
		`- def test_with_pydocstyle(self):`
		`- """Enforce PEP-257"""`
		`- pydocstyle_command = ['pydocstyle', KISKADEE_PATH]`
		`- self.assertEqual(subprocess.call(pydocstyle_command), 0)`

kiskadee/tests/units/test_model.py

file modified

+41 -43

		`@@ -1,87 +1,85 @@`
		`import unittest`
		`from sqlalchemy import exc`
		`- from sqlalchemy.orm import sessionmaker`

		`+ import kiskadee`
		`from kiskadee import model`
		`from kiskadee.database import Database`


		`class ModelTestCase(unittest.TestCase):`

		`+ @classmethod`
		`+ def setUpClass(cls):`
		`+ cls.db = Database('db_test')`
		`+`
		`def setUp(self):`
		`- self.engine = Database('db_test').engine`
		`- Session = sessionmaker(bind=self.engine)`
		`- self.session = Session()`
		`- model.Base.metadata.create_all(self.engine)`
		`- model.create_analyzers(self.session)`
		`+ kiskadee.tests.clean_test_db(self.db, model.Base.metadata)`
		`+ self.package = model.Package(name='python-kiskadee')`
		`+ self.version = model.Version(number='1.0-rc1')`
		`self.fetcher = model.Fetcher(`
		`name='kiskadee-fetcher', target='university'`
		`)`
		`- self.package = model.Package(name='python-kiskadee')`
		`- self.version = model.Version(number='1.0-rc1')`
		`+ self.db.session.add(self.fetcher)`
		`+ self.db.session.commit()`
		`self.fetcher.packages.append(self.package)`
		`self.package.versions.append(self.version)`
		`- self.session.add(self.package)`
		`- self.session.add(self.fetcher)`
		`- self.session.add(self.version)`
		`+ self.db.session.add(self.package)`
		`+ self.db.session.add(self.fetcher)`
		`+ self.db.session.add(self.version)`

		`self.analysis = model.Analysis(`
		`analyzer_id=1,`
		`version_id=1,`
		`raw=""`
		`)`
		`- self.session.add(self.analysis)`
		`+ self.db.session.add(self.analysis)`
		`self.report = model.Report(`
		`analysis_id=1,`
		`)`
		`- self.session.add(self.report)`
		`- self.session.commit()`
		`-`
		`- def tearDown(self):`
		`- self.session.close()`
		`- model.Base.metadata.drop_all()`
		`+ self.db.session.add(self.report)`
		`+ self.db.session.commit()`

		`def test_query_fetcher(self):`
		`- fetchers = self.session.query(model.Fetcher).all()`
		`+ fetchers = self.db.session.query(model.Fetcher).all()`
		`self.assertEqual(fetchers, [self.fetcher])`

		`def test_query_package(self):`
		`- packages = self.session.query(model.Package).all()`
		`+ packages = self.db.session.query(model.Package).all()`
		`self.assertEqual(packages, [self.package])`

		`def test_query_version(self):`
		`- versions = self.session.query(model.Version).all()`
		`+ versions = self.db.session.query(model.Version).all()`
		`self.assertEqual(versions, [self.version])`

		`def test_query_report(self):`
		`- reports = self.session.query(model.Report).all()`
		`+ reports = self.db.session.query(model.Report).all()`
		`self.assertEqual(reports, [self.report])`

		`def test_add_fetcher(self):`
		`- fetchers = self.session.query(model.Fetcher).all()`
		`+ fetchers = self.db.session.query(model.Fetcher).all()`
		`self.assertEqual(len(fetchers), 1)`
		`- self.session.add(model.Fetcher(name='foo', target='bar'))`
		`- fetchers = self.session.query(model.Fetcher).all()`
		`+ self.db.session.add(model.Fetcher(name='foo', target='bar'))`
		`+ fetchers = self.db.session.query(model.Fetcher).all()`
		`self.assertEqual(len(fetchers), 2)`

		`def test_add_version_without_package(self):`
		`version = model.Version(number='3.1')`
		`- self.session.add(version)`
		`+ self.db.session.add(version)`
		`with self.assertRaises(exc.IntegrityError):`
		`- self.session.commit()`
		`+ self.db.session.commit()`

		`def test_add_report_without_analysis(self):`
		`report = model.Report(`
		`)`
		`- self.session.add(report)`
		`+ self.db.session.add(report)`
		`with self.assertRaises(exc.IntegrityError):`
		`- self.session.commit()`
		`+ self.db.session.commit()`

		`def test_add_package_without_fetcher(self):`
		`package = model.Package(name='foo-bar')`
		`- self.session.add(package)`
		`+ self.db.session.add(package)`
		`with self.assertRaises(exc.IntegrityError):`
		`- self.session.commit()`
		`+ self.db.session.commit()`

		`def test_unique_package_in_fetcher(self):`
		`package_1 = model.Package(name='foo-bar')`
		`@@ -89,7 +87,7 @@`
		`self.fetcher.packages.append(package_1)`
		`self.fetcher.packages.append(package_2)`
		`with self.assertRaises(exc.IntegrityError):`
		`- self.session.commit()`
		`+ self.db.session.commit()`

		`def test_unique_version_for_package(self):`
		`package_version_1 = model.Version(number='1.0')`
		`@@ -97,10 +95,10 @@`
		`self.package.versions.append(package_version_1)`
		`self.package.versions.append(package_version_2)`
		`with self.assertRaises(exc.IntegrityError):`
		`- self.session.commit()`
		`+ self.db.session.commit()`

		`def test_compose_kiskadee_source(self):`
		`- _analyzer = self.session.query(model.Analyzer)\`
		`+ _analyzer = self.db.session.query(model.Analyzer)\`
		`.filter(model.Analyzer.name == "cppcheck").first()`
		`package = model.Package(`
		`name='bla',`
		`@@ -126,11 +124,11 @@`
		`def test_save_several_analysis(self):`

		`_analyzer1 = (`
		`- self.session.query(model.Analyzer)`
		`+ self.db.session.query(model.Analyzer)`
		`.filter(model.Analyzer.name == "cppcheck").first()`
		`)`
		`_analyzer2 = (`
		`- self.session.query(model.Analyzer)`
		`+ self.db.session.query(model.Analyzer)`
		`.filter(model.Analyzer.name == "flawfinder").first()`
		`)`

		`@@ -146,9 +144,9 @@`
		`self.fetcher.packages.append(package)`
		`package.versions.append(package_version)`

		`- self.session.add(package)`
		`- self.session.add(package_version)`
		`- self.session.commit()`
		`+ self.db.session.add(package)`
		`+ self.db.session.add(package_version)`
		`+ self.db.session.commit()`

		`package_analysis1 = model.Analysis(`
		`raw="<>",`
		`@@ -161,12 +159,12 @@`
		`version_id=package_version.id`
		`)`

		`- self.session.add(package_analysis1)`
		`- self.session.add(package_analysis2)`
		`- self.session.commit()`
		`+ self.db.session.add(package_analysis1)`
		`+ self.db.session.add(package_analysis2)`
		`+ self.db.session.commit()`

		`saved_package = (`
		`- self.session.query(model.Package)`
		`+ self.db.session.query(model.Package)`
		`.filter(model.Package.name == 'bla').first()`
		`)`
		`analysis = saved_package.versions[-1].analysis`

kiskadee/tests/units/test_queue.py

file modified

+6 -6

		`@@ -7,17 +7,17 @@`

		`def setUp(self):`
		`self.queues = kiskadee.queue.Queues()`
		`- self.project = {`
		`+ self.package = {`
		`'name': 'bar',`
		`'fetcher': kiskadee.fetchers.example.Fetcher(),`
		`'version': '1.0.0'`
		`}`

		`- def test_enqueue_dequeue_project(self):`
		`- self.queues.enqueue_project(self.project)`
		`- _project = self.queues.dequeue_project()`
		`- self.assertTrue(isinstance(_project, dict))`
		`- self.assertEqual(_project['name'], 'bar')`
		`+ def test_enqueue_dequeue_package(self):`
		`+ self.queues.enqueue_package(self.package)`
		`+ _package = self.queues.dequeue_package()`
		`+ self.assertTrue(isinstance(_package, dict))`
		`+ self.assertEqual(_package['name'], 'bar')`


		`if __name__ == '__main__':`

kiskadee/tests/units/test_runner.py

file added

+18

		`@@ -0,0 +1,18 @@`
		`+ import unittest`
		`+`
		`+ import kiskadee.fetchers.example`
		`+ import kiskadee.runner`
		`+`
		`+`
		`+ class RunnerTestCase(unittest.TestCase):`
		`+`
		`+ def setUp(self):`
		`+ self.queues = kiskadee.queue.Queues()`
		`+ self.runner = kiskadee.runner.Runner(self.queues)`
		`+`
		`+ def test_path_to_uncompressed_source(self):`
		`+`
		`+ self.runner.fetcher = kiskadee.fetchers.example.Fetcher()`
		`+ self.runner.package = {'name': 'test'}`
		`+ uncompressed_source_path = self.runner.get_package_code_path()`
		`+ self.assertIsNotNone(uncompressed_source_path)`

playbook/hosts.local

file modified

+1 -1

		`@@ -1,5 +1,5 @@`
		`[kiskadee-core]`
		`- core-host ansible_host=10.18.0.115`
		`+ core-host ansible_host=192.168.121.6 ansible_user=root`

		`[kiskadee-frontend]`
		`frontend-host ansible_host=10.18.0.116`

playbook/roles/common/tasks/main.yml

file modified

		`@@ -12,4 +12,5 @@`
		`- postgresql-server`
		`- postgresql-contrib`
		`- libselinux-python`
		`+ - devscripts`
		`ignore_errors: yes`

playbook/roles/kiskadee/tasks/main.yml

file modified

-12

		`@@ -1,15 +1,3 @@`
		`- - name: Create kiskadee group`
		`- group:`
		`- name: kiskadee`
		`- state: present`
		`-`
		`- - name: Create kiskadee user`
		`- user:`
		`- name: kiskadee`
		`- groups: kiskadee, wheel`
		`- append: yes`
		`- shell: /bin/bash`
		`-`
		`# TODO: do not turn selinux off. Add rules instead`
		`- name: Turn off selinux`
		`command: setenforce 0`

setup.py

file modified

+1 -1

		`@@ -44,7 +44,7 @@`
		`packages=find_packages(),`
		`include_package_data=False,`
		`entry_points={'console_scripts': [`
		`- 'kiskadee = kiskadee.monitor:daemon',`
		`+ 'kiskadee = kiskadee.initializer:init',`
		`'kiskadee_api = kiskadee.api.app:main',],`
		`'moksha.consumer': (`
		`'anityaconsumer = kiskadee.fetchers.anitya:AnityaConsumer')},`

davidcarlos commented 7 years ago

From MES experience, I had notice some complaints from students to contribute to kiskadee code, and was really difficult to do that, because the main methods from monitor and runner were to coupled and bit coesive. Testing was another problem, because we had a lot of interdependence between then, and was not possible to run a specific test suite ( units, functional, integration). We were only able to run the entirely suite, or a single test. This MR propose to solve this problems implementing the following changes:

Refactor Monitor and Runner classes following a simple rule: Each method must have only one responsibility.
Refactor Monitor and Runner classes following a simple rule: Each method must have at most four arguments.
- I took this rules from this talk: https://www.youtube.com/watch?v=npOGOmkxuio, and I think we should use it to all future features that we will add to kiskadee.
All code that interacts with database was moved to the model.py module.
Creates test suites to different test levels. Now we have unit, functional, plugins and integration tests. We can run each one using make command.
- Mock all database calls on integration tests. Here I say why: http://davidcarlos.me/magickmock-and-unittest.html
Rename Package model to Project. We do not always analyze packages, but we always analyze projects.
Remove from Monitor the responsibility to start Runner and it self. Now we do that on initializer module.