Run the tests!

We have a test suite. You should run it before you start working on some code, to make sure that nothing is broken before you start. And you should run it after you finish your change, before you push, to verify that your change didn't inadvertently break something else.

You can run it with:

$ pip install -r test-requirements.txt
$ PYTHONPATH=. nosetests

Some credentials...

You need to add a new file to the fedmsg.d/ directory. Call it fedmsg.d/credentials.py. It's needed to query FAS for information about other users. When we deploy this thing to production, we'll use a system account for this but for now, just use your personal account. You don't want to accidentally share this with anyone else. so be sure to keep it private. It should have contents like this:

config = {
    # Put your FAS credentials here...
    # but leave them commented out.  Note that, if you leave your fas
    # credentials out, some widgets won't be as accurate as they could be..
    # but it also makes the app faster than it would otherwise be (which is
    # nice for development).
    #'fas_credentials': {
    #    'username': 'YOUR_FAS_USERNAME',
    #    'password': 'YOUR_FAS_PASSWORD',
    #},

    # Go and generate a token for this here:
    # https://github.com/settings/tokens
    'github.oauth_token': 'YOUR_GITHUB_TOKEN',
}

Make sure to leave the '' around you username and password - don't delete them! Some widgets will work without the above info being present.. but it is needed for a subset of them.

Feed Widget - the Extra Mile

One widget (the big tamale -- the feed widget) requires more legwork to stand up. If you just want to see how hubs works and you want to hack on other peripheral stuff around it, you don't need to bother with these steps.

The feed widget requires a direct DB connection to the datanommer database; it can't proxy through datagrepper because it needs more flexibility. To get this working, you're going to set up:

• a postgres db
• the datanommer daemon

Start with some required packages:

$ sudo dnf install postgresql-server python-datanommer-consumer datanommer-commands fedmsg-hub

And there are some support libraries you'll also need:

$ sudo dnf install python-psycopg2 python-fedmsg-meta-fedora-infrastructure

Now, with packages installed, you need to tell postgres to create its initial filesystem layout:

$ sudo postgresql-setup initdb

And then, we need to tweak its config to let us connect easily for development:

$ sudo vim /var/lib/pgsql/data/pg_hba.conf

Change two lines from ident to trust. These two:

# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
# IPv6 local connections:
host    all             all             ::1/128                 trust

Start that beast up:

$ sudo systemctl start postgresql

Now, with the db daemon configured and running, let's configure datanommer. Edit this file:

$ sudo vim /etc/fedmsg.d/datanommer.py

And do two things: 1) set enabled to True and 2) give it a real sqlalchemy url, like this:

config = {
    'datanommer.enabled': True,
    'datanommer.sqlalchemy.url': 'postgres://postgres:whatever@localhost/datanommer',
}

Tell postgres that it should create space for a 'datanommer' database with this command:

$ sudo -u postgres psql -c "CREATE DATABASE datanommer;"

And finally, tell datanommer to create all of its tables in that new db we just created:

$ datanommer-create-db

Tell the fedmsg-hub daemon to restart itself. It should pick up datanommer as a plugin and start handing messages to it:

$ sudo systemctl restart fedmsg-hub

And then, wait a few seconds for a message to get nommed, and then you can check that it's working by running sudo datanommer-stats. It should print out some kind of summary about what kinds of messages are in the db now -- it will just grow and grow over time:

$ datanommer-stats
[2015-07-01 14:33:21][    fedmsg    INFO] buildsys has 70 entries
[2015-07-01 14:33:21][    fedmsg    INFO] faf has 7 entries
[2015-07-01 14:33:21][    fedmsg    INFO] copr has 6 entries
[2015-07-01 14:33:21][    fedmsg    INFO] askbot has 2 entries

Lastly, (fingers crossed) start up the fedora-hubs webapp and load your profile page. Once there are some messages that get into your local database that should show up on your feed.. they should appear there. (At very least, you shouldn't get an error message about that widget being unable to be displayed).

A picture is worth...

Here are some more details on how the whole thing should work together.

A diagram of component interactions

Let's talk through how data will flow through the system by asking what happens when a user requsts their main hubs page:

• The user requests the page and the wsgi app responds with some barebones HTML and enough javascript to get off the ground.
• The user's browser runs javascript that subscribes it to the EventSource server.
• The user's browser runs that javascript, which requests data for each of the widgets defined on the page.
• The wsgi app receives those requests and checks to see if the data for any of them is cached in memcached. If it is, then it is returned. If not, then the wsgi app executes the data(...) function of that widget to get the response ready. It is stuffed in memcached for later access and returned.
• The client renders widgets as the data for each of its requests comes back.

Later, what happens when a trac ticket is filed that should show up in some widget on their page?

• The ticket is updated on fedorahosted.org and a fedmsg message is fired.
• That is received by the hubs backend, which looks up all the cached responses that should be invalidated by that event (there is a widget on mizmo's page, threebean's page, and on the design hub that should all get fresh data because of this change).
• All of those widgets get their cached data nuked.
• All of those widgets get their cached data rebuilt by calling data(...) on them.
• An EventSource event is fired off for any listening clients that new data is available for widgets X, Y, and Z. The data is included in the EventSource payload so the clients can immediately redraw without bothering to re-query the wsgi app.

What happens when the user is viewing the design team hub and simultaneously, an admin changes the configuration of a widget on that page?

• Changing the configuration results in a HTTP POST to the wsgi app.
• The configuration is changed accordingly in the postgres database.
• A fedmsg message is fired off indicating that the configuration for widget X has changed.
• The wsgi app responds 200 OK to the admin.
• Meanwhile, that fedmsg message is received by the backend which:
• ...looks up the cache key for widget X with the old configuration and nukes it the cached data.
• ...looks up the cache key for widget X with the new configuration and builds the cached data by calling data(...) on the widget.
• An EventSource event is fired off which gets recieved by everyone looking at the design team hub. The widget on their pages gets redrawn with data from the EventSource event.