wordcloudbot

systemd service that listens to fedmsg for IRC meeting logs, generates a word cloud, and tweets it to a Twitter account  |  https://twitter.com/fedobot

Commit b8b3bb7 Move the twitter-specific bits to the top; make image links absolute

1 file Authored and Committed by decause 2 years ago
Move the twitter-specific bits to the top; make image links absolute

    
README.rst +450 -533
   1 @@ -1,765 +1,682 @@
   2 - :title: fedmsg-flock14
   3 - :css: css/style-fedmsg.css
   4 - :data-transition-duration: 500
   5 - :skip-help: true
   6 - :hovercraft-path: m275,175 v-150 a150,150 0 0,0 -150,150 z
   7 + wordcloudbot
   8 + ============
   9   
  10 - ----
  11   
  12 - Make tools with fedmsg!
  13 - =======================
  14 + How
  15 + ===
  16 + it's going to work
  17 + ~~~~~~~~~~~~~~~~~~
  18   
  19 - Workshop at FLOCK14, Prague CZ, August 8th, 2014
  20 + .. image:: http://threebean.org/presentations/images/fedmsg-flock14-img/twitter-diagram.png
  21 +    :width: 900px
  22   
  23 - - Presented by Ralph Bean
  24 - - http://github.com/ralphbean
  25 - - http://twitter.com/ralphbean
  26 - - http://threebean.org
  27 - - ``2048R/971095FF 2012-12-06``
  28 + ----
  29   
  30 - Go sit in ``#fedora-fedmsg`` on ``irc.freenode.net``.
  31 + Take this
  32 + =========
  33 + It's dangerous out there
  34 + ~~~~~~~~~~~~~~~~~~~~~~~~
  35   
  36 - http://threebean.org/presentations/fedmsg-flock14/
  37 + .. code:: bash
  38   
  39 - .. image:: images/fedmsg-flock14-img/creative-commons.png
  40 +     sudo yum install fedmsg
  41 +     sudo yum install python-fedmsg-meta-fedora-infrastructure
  42 +     sudo yum install python-fabulous
  43 +     sudo yum install tweepy
  44   
  45   ----
  46   
  47 - fedmsg
  48 - ======
  49 - what it is
  50 - ~~~~~~~~~~
  51 + Your first
  52 + ==========
  53 + fedmsg script
  54 + ~~~~~~~~~~~~~
  55   
  56 - ----
  57 + .. code:: python
  58   
  59 - :data-x: r0
  60 - :data-y: r900
  61 +     import fedmsg
  62 +     import pprint
  63   
  64 - The `Fedora Infrastructure Message Bus <http://fedmsg.com>`_ is a
  65 - python package and API used around Fedora Infrastructure to send
  66 - and receive messages to and from applications.
  67 +     print "Posting up to listen on the fedmsg bus.  Waiting for a message..."
  68 +     for name, endpoint, topic, msg in fedmsg.tail_messages():
  69 +         pprint.pprint(msg)
  70 + 
  71 + Give it a run.
  72   
  73   ----
  74   
  75 - :data-x: r0
  76 - :data-y: r900
  77 + It's like a million voices cried out
  78 + ====================================
  79 + and then were silent
  80 + ~~~~~~~~~~~~~~~~~~~~
  81   
  82 - .. image:: images/fedmsg-flock14-img/topology.png
  83 -    :height: 485px
  84 + .. code:: python
  85 + 
  86 +     #topic_filter = 'fedbadges'     # We really want this, but its rare
  87 +     topic_filter = 'fedoratagger'   # This is much easier to test with
  88 + 
  89 +     for name, endpoint, topic, msg in fedmsg.tail_messages():
  90 +         if topic_filter not in topic:
  91 +             # Bail out if the topic doesn't match
  92 +             continue
  93 + 
  94 +         pprint.pprint(msg)
  95 + 
  96 + See http://fedmsg.com/en/latest/topics for more
  97   
  98   ----
  99   
 100 - :data-x: r1600
 101 - :data-y: 0
 102 + Some config
 103 + ===========
 104 + at the top
 105 + ~~~~~~~~~~
 106   
 107 - It is *publicly subscribable* -- hit up ``tcp://hub.fedoraproject.org:9940``
 108 - with a ``zmq.SUB`` socket.
 109 + .. code:: python
 110   
 111 - It has Fedora in the name, but `Debian Infrastructure started picking it up
 112 - <http://lists.debian.org/debian-qa/2013/04/msg00010.html>`_
 113 - last summer.  They've `made progress
 114 - <http://blog.olasd.eu/2013/07/bootstrapping-fedmsg-for-debian/>`_ to the point
 115 - that we had to change the name to mean the *FEDerated Message Bus* instead.
 116 +     import fedmsg.config
 117 +     import logging.config
 118   
 119 - `data.gouv.fr <https://data.gouv.fr>`_ is using it too.  Maybe others?  We get
 120 - questions and clarifications on the `deployment docs
 121 - <http://fedmsg.com/en/latest/deployment>`_ from time to time.
 122 +     # First, load the fedmsg config from fedmsg.d/
 123 +     config = fedmsg.config.load_config()
 124   
 125 - ----
 126 +     # Then, configure the python stdlib logging to use fedmsg's logging config
 127 +     logging.config.dictConfig(config.get('logging'))
 128   
 129 + ----
 130   
 131 - :data-x: r1600
 132 - :data-y: 0
 133 + So
 134 + ==
 135 + meta
 136 + ~~~~
 137   
 138 - fedmsg
 139 - ======
 140 - what do?
 141 - ~~~~~~~~
 142 + .. code:: python
 143   
 144 - There are two aspects to this workshop:
 145 +     import fedmsg.meta
 146   
 147 - - **A historical component**.  I want to show you briefly how to use
 148 -   `datagrepper <https://apps.fedoraproject.org/datagrepper>`_ which has been
 149 -   the most surprisingly useful piece of the fedmsg infrastructure.
 150 +     # Initialize fedmsg's "meta" module if you have the fedora infra plugin
 151 +     fedmsg.meta.make_processors(**config)
 152   
 153 - - **A realtime component**.  I want to go over some of the current applications
 154 -   of fedmsg briefly.  After that, I'll go into depth -- step-by-step -- to show
 155 -   you how to write your own script that connects to the live fedmsg stream and
 156 -   does something "useful" with it.
 157 +     for name, endpoint, topic, msg in fedmsg.tail_messages():
 158 +         if topic_filter not in topic:
 159 +             continue
 160   
 161 - Do you want me to cover?
 162 +         # Only act on your own messages -- things that *you* did.
 163 +         if 'YOUR_FAS_USERNAME' not in fedmsg.meta.msg2usernames(msg, **config):
 164 +             continue
 165   
 166 - - **Setting up your own local bus**.  It's really pretty easy and we can do it
 167 -   in time.  I'm just guessing that nobody here is interested in doing that.
 168 -   I'll touch on it but we can talk more about it later if you like.
 169 +         # Use it to make nice text and other things
 170 +         # See also: msg2icon, msg2link, msg2usernames, msg2packages...
 171 +         subtitle = fedmsg.meta.msg2subtitle(msg, **config)
 172 +         print subtitle
 173   
 174   ----
 175   
 176 - :data-x: r0
 177 - :data-y: r900
 178 + A picture
 179 + =========
 180 + is worth a thousand words
 181 + ~~~~~~~~~~~~~~~~~~~~~~~~~
 182   
 183 - first
 184 - =====
 185 - you should get it
 186 - ~~~~~~~~~~~~~~~~~
 187 + .. code:: python
 188   
 189 - .. code:: bash
 190 +     import tempfile
 191 +     import urllib
 192 +     import os
 193   
 194 -     sudo yum install fedmsg
 195 +     import fabulous.image
 196   
 197 - There's also a plugin that let's us render **Fedora Infrastructure** messages
 198 - nicely.  You should install that too:
 199 +     for name, endpoint, topic, msg in fedmsg.tail_messages():
 200 +         # This returns a URL (most of the time)
 201 +         icon = fedmsg.meta.msg2icon(msg, **config)
 202   
 203 - .. code:: bash
 204 +         _, filename = tempfile.mkstemp(suffix='.png')
 205 +         print "Downloading", icon, "to", filename
 206 +         urllib.urlretrieve(icon, filename)
 207   
 208 -     sudo yum install python-fedmsg-meta-fedora-infrastructure
 209 +         print fabulous.image.Image(filename)
 210 + 
 211 +         print "Cleaning up %r" % filename
 212 +         os.remove(filename)
 213   
 214   ----
 215   
 216   :data-x: r1600
 217   :data-y: 0
 218   
 219 - A taste
 220 - =======
 221 - of the bus
 222 - ~~~~~~~~~~
 223 + Intermezzo
 224 + ==========
 225   
 226 - Clone the repo from https://github.com/ralphbean/fedmsg2gource
 227 + We have a neat working script that gets fedmsg messages pushed to it.  It can
 228 + extract neato stuff and print it.
 229   
 230 - Run::
 231 + But... if we want to move to the next step, we have to take a break from our
 232 + happy hacking to go and deal with Twitter, its API, and API keys.
 233   
 234 -     python fedmsg2gource.py --days 14 > testing.log
 235 -     cat testing.log | \
 236 -         gource -i 10 \
 237 -             --user-image-dir ~/.cache/avatars/ \
 238 -             --log-format custom \
 239 -             --viewport 1024x730 \
 240 -             -
 241 + ----
 242   
 243 + :data-x: r1600
 244 + :data-y: 0
 245   
 246 - ----
 247 + The Twitter API
 248 + ===============
 249   
 250 - :data-x: r0
 251 - :data-y: r900
 252 + We're going to have to:
 253   
 254 - Explore
 255 - =======
 256 - the datagrepper API
 257 - ~~~~~~~~~~~~~~~~~~~
 258 + 1) Create our own "app".  Visit https://apps.twitter.com/app/new
 259 + 2) Modify that app's permission to include **"Read and Write"**.
 260 + 3) Authorize that app with our own account, which yields *oauth tokens*.
 261 +    To do this, click the **"Create my access token"** button at the bottom of
 262 +    your app's detail page.
 263   
 264 - https://apps.fedoraproject.org/datagrepper
 265 + We will keep those tokens a secret and our little bot will use them to login
 266 + and tweet on our behalf.  You'll get **four** secret strings.
 267   
 268   ----
 269   
 270 - :data-x: r1600
 271 - :data-y: 0
 272 + :data-x: r0
 273 + :data-y: r900
 274   
 275 - say
 276 - ===
 277 - you wanted your own local bus
 278 - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 279   
 280 - .. code:: bash
 281 + Storing
 282 + =======
 283 + those secrets
 284 + ~~~~~~~~~~~~~
 285   
 286 -     sudo yum install fedmsg-relay
 287 -     sudo systemctl start fedmsg-relay
 288 -     echo "Hello World." | fedmsg-logger --modname=git --topic=repo.update
 289 -     echo '{"a": 1}' | fedmsg-logger --json-input
 290 -     fedmsg-logger --message="This is a message."
 291 -     fedmsg-logger --message='{"a": 1}' --json-input
 292 + First, add a directory called ``fedmsg.d/`` to your current working directory.
 293   
 294 - or from python:
 295 + In it, put a file called ``fedmsg.d/twitter-secrets.py`` that looks like this:
 296   
 297   .. code:: python
 298   
 299 -     import fedmsg
 300 - 
 301 -     fedmsg.publish(
 302 -         topic='testing',
 303 -         msg={
 304 -             'test': 'Hello World',
 305 -             'foo': jsonifiable_objects,
 306 -             'bar': a_sqlalchemy_object,
 307 -         }
 308 +     config = dict(
 309 +         consumer_key        = "your api key goes here",
 310 +         consumer_secret     = "your api secret goes here",
 311 +         access_token_key    = "your access token goes here",
 312 +         access_token_secret = "your access token secret goes here",
 313       )
 314   
 315 - ----
 316 + Test that fedmsg can read in that new config file by looking for them in:
 317   
 318 - :data-x: r1600
 319 - :data-y: 0
 320 + .. code:: bash
 321   
 322 - if
 323 - ==
 324 - you want to consume
 325 - ~~~~~~~~~~~~~~~~~~~
 326 +     fedmsg-config | less
 327   
 328 - .. code:: bash
 329 + ----
 330   
 331 -     fedmsg-tail --really-pretty
 332 + Using
 333 + =====
 334 + those secrets
 335 + ~~~~~~~~~~~~~
 336 + 
 337 + Go back to ``badgebot.py`` and add the following:
 338   
 339   .. code:: python
 340   
 341 -     {
 342 -         "i": 1,
 343 -         "timestamp": 1344344053.2337201,
 344 -         "topic": "org.fedoraproject.prod.bodhi.update.comment",
 345 -         "msg": {
 346 -             "comment": {
 347 -                 "update_title": "nethack4-4.0.0-1.fc20",
 348 -                 "group": None,
 349 -                 "author": "ralph",
 350 -                 "text": "I'm so pumped to pwn those minotaurs!",
 351 -                 "karma": 1,
 352 -                 "anonymous": False,
 353 -                 "timestamp": 1344344050.0
 354 -             }
 355 -         }
 356 -     }
 357 - 
 358 - ----
 359 - 
 360 - :data-x: r0
 361 - :data-y: r900
 362 - 
 363 - consuming messages
 364 - ==================
 365 - from python
 366 - ~~~~~~~~~~~
 367 - 
 368 - .. code:: python
 369 +     import tweepy
 370   
 371 -     import fedmsg
 372 +     consumer_key        = config['consumer_key']
 373 +     consumer_secret     = config['consumer_secret']
 374 +     access_token_key    = config['access_token_key']
 375 +     access_token_secret = config['access_token_secret']
 376   
 377 -     for name, endpoint, topic, msg in fedmsg.tail_messages():
 378 -         print topic, msg
 379 +     auth_handler = tweepy.OAuthHandler(consumer_key, consumer_secret)
 380 +     auth_handler.set_access_token(access_token_key, access_token_secret)
 381 +     twitter_api = tweepy.API(auth_handler)
 382   
 383   ----
 384   
 385 - :data-x: r0
 386 - :data-y: r900
 387 - 
 388 - consuming messages
 389 - ==================
 390 - with a daemon
 391 - ~~~~~~~~~~~~~
 392 - 
 393 - ``fedmsg-hub`` is a daemon that can make writing your own
 394 - long-running consumers simpler.  There are `docs on fedmsg.com
 395 - <http://www.fedmsg.com/en/latest/consuming/#the-hub-consumer-approach>`_
 396 - for writing plugins, but they look like this:
 397 + And
 398 + ===
 399 + further down
 400 + ~~~~~~~~~~~~
 401   
 402   .. code:: python
 403   
 404 -     import pprint
 405 -     import fedmsg.consumers
 406 - 
 407 - 
 408 -     class MyConsumer(fedmsg.consumers.FedmsgConsumer):
 409 -         topic = "org.fedoraproject.*"
 410 -         config_key = 'myconsumer.enabled'
 411 - 
 412 -         def consume(self, message):
 413 -             pprint.pprint(message)
 414 - 
 415 - ----
 416 - 
 417 - :data-x: r0
 418 - :data-y: r900
 419 - :data-scale: 0.5
 420 +     for name, endpoint, topic, msg in fedmsg.tail_messages():
 421   
 422 - consuming messages
 423 - ==================
 424 - at the command line... an aside
 425 - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 426 +         subtitle = fedmsg.meta.msg2subtitle(msg, **config)
 427 +         link = fedmsg.meta.msg2link(msg, **config)
 428 +         icon = fedmsg.meta.msg2icon(msg, **config)
 429   
 430 - There are lots of fun options to ``fedmsg-tail`` like ``--terse``.
 431 +         _, filename = tempfile.mkstemp(suffix='.png')
 432 +         print "Downloading", icon, "to", filename
 433 +         urllib.urlretrieve(icon, filename)
 434   
 435 - .. code:: bash
 436 +         # Construct and post our tweet.
 437 +         #print fabulous.image.Image(filename)
 438 +         content = subtitle + " " + link
 439 +         print "Tweeting %r" % content
 440 +         twitter_api.update_with_media(filename, content)
 441   
 442 -    fedmsg-tail --terse
 443 +         print "Cleaning up %r" % filename
 444 +         os.remove(filename)
 445   
 446 - .. code:: text
 447 + ----
 448   
 449 -     buildsys.build.state.change -- ausil's tncfhh-0.8.3-14.fc20 completed
 450 -     http://koji.fedoraproject.org/koji/buildinfo?buildID=439734
 451 -     trac.ticket.update -- kevin closed a ticket on the Fedora Infrastructure trac instance as 'fixed'
 452 -     https://fedorahosted.org/fedora-infrastructure/ticket/3904
 453 -     bodhi.update.request.testing -- mmckinst submitted nawk-20121220-1.fc18 to testing
 454 -     https://admin.fedoraproject.org/updates/nawk-20121220-1.fc18
 455 -     wiki.article.edit -- Hguemar made a wiki edit to "Flock:Rideshare"
 456 -     https://fedoraproject.org/w/index.php?title=Flock:Rideshare&diff=prev&oldid=347430
 457 + Does it work?
 458 + =============
 459   
 460   ----
 461   
 462   :data-x: r1600
 463   :data-y: 0
 464   
 465 - things that use fedmsg
 466 - ======================
 467 - there's a lot of them at this point
 468 - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 469 - 
 470 - ----
 471 - 
 472 - :data-x: r0
 473 - :data-y: r900
 474 - 
 475 - koji
 476 - ====
 477 - stalk
 478 - ~~~~~
 479 + systemd
 480 + =======
 481 + for real
 482 + ~~~~~~~~
 483   
 484 - David Aquilina's (dwa's) `koji stalk
 485 - <http://dwa.fedorapeople.org/wip/koji-stalk.py>`_ monitors koji over fedmsg and
 486 - rebuilds packages for arm and ppc.
 487 + Make a new file called ``badgebot.service`` with these contents::
 488   
 489 - ----
 490 +     [Unit]
 491 +     Description=A Twitter bot for your Fedora Badges.  Wow.
 492 +     After=network.target
 493 +     Documentation=http://fedmsg.com
 494   
 495 - FAS2Trac (ftl)
 496 - ==============
 497 - (fama updater)
 498 - ~~~~~~~~~~~~~~
 499 +     [Service]
 500 +     ExecStart=/usr/local/bin/badgebot.py
 501 +     Type=simple
 502 +     User=fedmsg
 503 +     Group=fedmsg
 504   
 505 - herlo's `FAS2Trac fama updater (ftl)
 506 - <https://git.fedorahosted.org/cgit/ftl.git>`_ listens to messages indicating
 507 - that a user has applied for membership in the ambassadors group -- it then
 508 - files a ticket in the `ambassadors' trac instance
 509 - <https://fedorahosted.org/fama/>`_ for a potential sponsor via XMLRPC.
 510 +     [Install]
 511 +     WantedBy=multi-user.target
 512   
 513   ----
 514   
 515 - compose
 516 - =======
 517 - downloader
 518 - ~~~~~~~~~~
 519 + :data-x: r0
 520 + :data-y: r900
 521   
 522 - p3ck's `fedmsg-download <https://github.com/p3ck/fedmsg-download/>`_
 523 - listens for messages that the daily branched and rawhide compose
 524 - process has finished -- it then downloads the latest builds from
 525 - ``rsync://dl.fedoraproject.org/fedora-linux-development``
 526 + install.sh
 527 + ==========
 528   
 529 - ----
 530 + .. code:: bash
 531   
 532 - synchronization
 533 - ===============
 534 - of package ACLs
 535 - ~~~~~~~~~~~~~~~
 536 +     #!/bin/bash -x
 537 +     # install.sh - (re)install and (re)start the badgebot
 538   
 539 - So, it **used** to be that when someone was granted *commit* access to a
 540 - package in the `Fedora PackageDB (pkgdb)
 541 - <https://apps.fedoraproject.org/#PkgDB>`_, the webapp simply wrote to a
 542 - database table indicating the new relationship.  Every *hour*, a cronjob would
 543 - run that queried the state of that database and then re-wrote out the ACLs for
 544 - gitolite -- the software that manages access to our `package repositories
 545 - <http://pkgs.fedoraproject.org>`_.
 546 +     # Install our script
 547 +     cp badgebot.py /usr/local/bin/badgebot.py
 548   
 549 - Consequently, we had lots of *waiting*: you would request commit access to a
 550 - repository, then *wait* for an owner to grant you rights, then *wait* for that
 551 - cronjob to run before you could actually push.
 552 +     # Make sure no one else can read our secrets.
 553 +     cp fedmsg.d/twitter-secrets.py /etc/fedmsg.d/.
 554 +     chown fedmsg:fedmsg /etc/fedmsg.d/twitter-secrets.py
 555 +     chmod o-r /etc/fedmsg.d/twitter-secrets.py
 556   
 557 - With `a new fedmsg consumer
 558 - <https://github.com/fedora-infra/fedmsg-genacls/blob/develop/fedmsg_genacls.py>`_
 559 - that we have in place, those gitolite ACLs are re-written in response to
 560 - fedmsg messages from the pkgdb.  It is much faster.
 561 +     # Copy in service file for systemd
 562 +     cp badgebot.service /usr/lib/systemd/system/badgebot.service
 563 +     systemctl daemon-reload
 564 +     systemctl restart badgebot
 565   
 566   ----
 567   
 568 - notifications
 569 - =============
 570 - to email, irc, the desktop, and android
 571 - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 572 - 
 573 - There's the new `FMN system <https://apps.fedoraproject.org/>`_ that can
 574 - deliver notifications to you via irc, email, and android.
 575 - 
 576 - There's also lmacken's `fedmsg-notify <http://lewk.org/blog/fedmsg-notify>`_
 577 - which listens for messages and displays a filtered stream on your desktop with
 578 - ``libnotify``.
 579 - 
 580 - .. image:: images/fedmsg-flock14-img/fedmsg-notify-0-crop.png
 581 -    :height: 300px
 582 - 
 583 - ----
 584 + Watch the journal::
 585   
 586 - reports
 587 - =======
 588 - 10 ways from sunday
 589 - ~~~~~~~~~~~~~~~~~~~
 590 +     sudo journalctl -u badgebot --follow
 591   
 592 - Every week, pingou's `owner changes report tool
 593 - <https://lists.fedoraproject.org/pipermail/infrastructure/2013-June/013070.html>`_
 594 - emails the devel list with a report of what packages were orphaned, unorphaned
 595 - and retired.
 596 + - Presented by Ralph Bean
 597 + - http://github.com/ralphbean
 598 + - http://twitter.com/ralphbean
 599 + - http://threebean.org
 600 + - ``2048R/971095FF 2012-12-06``
 601   
 602 - .. image:: images/fedmsg-flock14-img/ownerchange-screenshot.png
 603 -    :height: 420px
 604 + Go sit in ``#fedora-fedmsg`` on ``irc.freenode.net``.
 605   
 606 - ----
 607 + http://threebean.org/presentations/fedmsg-flock14/
 608   
 609 - reports
 610 - =======
 611 - 10 ways from sunday
 612 - ~~~~~~~~~~~~~~~~~~~
 613 + .. image:: http://threebean.org/presentations/images/fedmsg-flock14-img/creative-commons.png
 614 + http://threebean.org/presentations/
 615   
 616 - There's also the `Release Engineering Dashboard
 617 - <https://apps.fedoraproject.org/releng-dash>`_ which grabs data from
 618 - datagrepper on all the latest updates syncs, composes, image builds, etc.. and
 619 - puts their status all in one place.  Pure HTML/javascript -- there's no
 620 - server-side app here.
 621 + fedmsg
 622 + ======
 623 + what it is
 624 + ~~~~~~~~~~
 625   
 626 - .. image:: images/fedmsg-flock14-img/releng-dash-screenshot.png
 627 -    :height: 350px
 628   
 629 - ----
 630 + The `Fedora Infrastructure Message Bus <http://fedmsg.com>`_ is a
 631 + python package and API used around Fedora Infrastructure to send
 632 + and receive messages to and from applications.
 633   
 634 - fedora badges
 635 - =============
 636 - for you, and you, and you
 637 - ~~~~~~~~~~~~~~~~~~~~~~~~~
 638   
 639 - `Fedora badges <https://badges.fedoraproject.org/>`_ launched last year at
 640 - Flock13.  It awards "badges" to Fedora contributors for their activity.
 641 + .. image:: http://threebean.org/presentations/images/fedmsg-flock14-img/topology.png
 642 +    :height: 485px
 643   
 644 - .. image:: images/fedmsg-flock14-img/badges_fan.png
 645   
 646 - Pretty fun.  ``:)``
 647 + It is *publicly subscribable* -- hit up ``tcp://hub.fedoraproject.org:9940``
 648 + with a ``zmq.SUB`` socket.
 649   
 650 - ----
 651 + It has Fedora in the name, but `Debian Infrastructure started picking it up
 652 + <http://lists.debian.org/debian-qa/2013/04/msg00010.html>`_
 653 + last summer.  They've `made progress
 654 + <http://blog.olasd.eu/2013/07/bootstrapping-fedmsg-for-debian/>`_ to the point
 655 + that we had to change the name to mean the *FEDerated Message Bus* instead.
 656   
 657 - To sum that up
 658 - ==============
 659 + `data.gouv.fr <https://data.gouv.fr>`_ is using it too.  Maybe others?  We get
 660 + questions and clarifications on the `deployment docs
 661 + <http://fedmsg.com/en/latest/deployment>`_ from time to time.
 662   
 663 - The assimilation of **message producing services** is nearly complete.
 664   
 665 - There are many **message consuming services** already in place.. but we can
 666 - likely make many more.  Which is why you're here, no?
 667 + fedmsg
 668 + ======
 669 + what do?
 670 + ~~~~~~~~
 671   
 672 - ----
 673 + There are two aspects to this workshop:
 674   
 675 - :data-x: r1600
 676 - :data-y: 0
 677 + - **A historical component**.  I want to show you briefly how to use
 678 +   `datagrepper <https://apps.fedoraproject.org/datagrepper>`_ which has been
 679 +   the most surprisingly useful piece of the fedmsg infrastructure.
 680   
 681 - Today's
 682 - =======
 683 - task
 684 - ~~~~
 685 + - **A realtime component**.  I want to go over some of the current applications
 686 +   of fedmsg briefly.  After that, I'll go into depth -- step-by-step -- to show
 687 +   you how to write your own script that connects to the live fedmsg stream and
 688 +   does something "useful" with it.
 689   
 690 - Surprise!  We're going to make a Twitter Bot!
 691 + Do you want me to cover?
 692   
 693 - ----
 694 + - **Setting up your own local bus**.  It's really pretty easy and we can do it
 695 +   in time.  I'm just guessing that nobody here is interested in doing that.
 696 +   I'll touch on it but we can talk more about it later if you like.
 697   
 698 - :data-x: r0
 699 - :data-y: r900
 700   
 701 - How
 702 - ===
 703 - it's going to work
 704 - ~~~~~~~~~~~~~~~~~~
 705 + first
 706 + =====
 707 + you should get it
 708 + ~~~~~~~~~~~~~~~~~
 709   
 710 - .. image:: images/fedmsg-flock14-img/twitter-diagram.png
 711 -    :width: 900px
 712 + .. code:: bash
 713   
 714 - ----
 715 +     sudo yum install fedmsg
 716   
 717 - Take this
 718 - =========
 719 - It's dangerous out there
 720 - ~~~~~~~~~~~~~~~~~~~~~~~~
 721 + There's also a plugin that let's us render **Fedora Infrastructure** messages
 722 + nicely.  You should install that too:
 723   
 724   .. code:: bash
 725   
 726 -     sudo yum install fedmsg
 727       sudo yum install python-fedmsg-meta-fedora-infrastructure
 728 -     sudo yum install python-fabulous
 729 -     sudo yum install tweepy
 730 - 
 731 - ----
 732   
 733 - Your first
 734 - ==========
 735 - fedmsg script
 736 - ~~~~~~~~~~~~~
 737   
 738 - .. code:: python
 739 + A taste
 740 + =======
 741 + of the bus
 742 + ~~~~~~~~~~
 743   
 744 -     import fedmsg
 745 -     import pprint
 746 + Clone the repo from https://github.com/ralphbean/fedmsg2gource
 747   
 748 -     print "Posting up to listen on the fedmsg bus.  Waiting for a message..."
 749 -     for name, endpoint, topic, msg in fedmsg.tail_messages():
 750 -         pprint.pprint(msg)
 751 + Run::
 752   
 753 - Give it a run.
 754 +     python fedmsg2gource.py --days 14 > testing.log
 755 +     cat testing.log | \
 756 +         gource -i 10 \
 757 +             --user-image-dir ~/.cache/avatars/ \
 758 +             --log-format custom \
 759 +             --viewport 1024x730 \
 760 +             -
 761   
 762 - ----
 763   
 764 - It's like a million voices cried out
 765 - ====================================
 766 - and then were silent
 767 - ~~~~~~~~~~~~~~~~~~~~
 768   
 769 - .. code:: python
 770 + Explore
 771 + =======
 772 + the datagrepper API
 773 + ~~~~~~~~~~~~~~~~~~~
 774   
 775 -     #topic_filter = 'fedbadges'     # We really want this, but its rare
 776 -     topic_filter = 'fedoratagger'   # This is much easier to test with
 777 + https://apps.fedoraproject.org/datagrepper
 778   
 779 -     for name, endpoint, topic, msg in fedmsg.tail_messages():
 780 -         if topic_filter not in topic:
 781 -             # Bail out if the topic doesn't match
 782 -             continue
 783   
 784 -         pprint.pprint(msg)
 785 + say
 786 + ===
 787 + you wanted your own local bus
 788 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 789   
 790 - See http://fedmsg.com/en/latest/topics for more
 791 + .. code:: bash
 792   
 793 - ----
 794 +     sudo yum install fedmsg-relay
 795 +     sudo systemctl start fedmsg-relay
 796 +     echo "Hello World." | fedmsg-logger --modname=git --topic=repo.update
 797 +     echo '{"a": 1}' | fedmsg-logger --json-input
 798 +     fedmsg-logger --message="This is a message."
 799 +     fedmsg-logger --message='{"a": 1}' --json-input
 800   
 801 - Some config
 802 - ===========
 803 - at the top
 804 - ~~~~~~~~~~
 805 + or from python:
 806   
 807   .. code:: python
 808   
 809 -     import fedmsg.config
 810 -     import logging.config
 811 - 
 812 -     # First, load the fedmsg config from fedmsg.d/
 813 -     config = fedmsg.config.load_config()
 814 +     import fedmsg
 815   
 816 -     # Then, configure the python stdlib logging to use fedmsg's logging config
 817 -     logging.config.dictConfig(config.get('logging'))
 818 +     fedmsg.publish(
 819 +         topic='testing',
 820 +         msg={
 821 +             'test': 'Hello World',
 822 +             'foo': jsonifiable_objects,
 823 +             'bar': a_sqlalchemy_object,
 824 +         }
 825 +     )
 826   
 827 - ----
 828   
 829 - So
 830 + if
 831   ==
 832 - meta
 833 - ~~~~
 834 - 
 835 - .. code:: python
 836 - 
 837 -     import fedmsg.meta
 838 - 
 839 -     # Initialize fedmsg's "meta" module if you have the fedora infra plugin
 840 -     fedmsg.meta.make_processors(**config)
 841 - 
 842 -     for name, endpoint, topic, msg in fedmsg.tail_messages():
 843 -         if topic_filter not in topic:
 844 -             continue
 845 - 
 846 -         # Only act on your own messages -- things that *you* did.
 847 -         if 'YOUR_FAS_USERNAME' not in fedmsg.meta.msg2usernames(msg, **config):
 848 -             continue
 849 - 
 850 -         # Use it to make nice text and other things
 851 -         # See also: msg2icon, msg2link, msg2usernames, msg2packages...
 852 -         subtitle = fedmsg.meta.msg2subtitle(msg, **config)
 853 -         print subtitle
 854 + you want to consume
 855 + ~~~~~~~~~~~~~~~~~~~
 856   
 857 - ----
 858 + .. code:: bash
 859   
 860 - A picture
 861 - =========
 862 - is worth a thousand words
 863 - ~~~~~~~~~~~~~~~~~~~~~~~~~
 864 +     fedmsg-tail --really-pretty
 865   
 866   .. code:: python
 867   
 868 -     import tempfile
 869 -     import urllib
 870 -     import os
 871 - 
 872 -     import fabulous.image
 873 - 
 874 -     for name, endpoint, topic, msg in fedmsg.tail_messages():
 875 -         # This returns a URL (most of the time)
 876 -         icon = fedmsg.meta.msg2icon(msg, **config)
 877 +     {
 878 +         "i": 1,
 879 +         "timestamp": 1344344053.2337201,
 880 +         "topic": "org.fedoraproject.prod.bodhi.update.comment",
 881 +         "msg": {
 882 +             "comment": {
 883 +                 "update_title": "nethack4-4.0.0-1.fc20",
 884 +                 "group": None,
 885 +                 "author": "ralph",
 886 +                 "text": "I'm so pumped to pwn those minotaurs!",
 887 +                 "karma": 1,
 888 +                 "anonymous": False,
 889 +                 "timestamp": 1344344050.0
 890 +             }
 891 +         }
 892 +     }
 893   
 894 -         _, filename = tempfile.mkstemp(suffix='.png')
 895 -         print "Downloading", icon, "to", filename
 896 -         urllib.urlretrieve(icon, filename)
 897   
 898 -         print fabulous.image.Image(filename)
 899 + consuming messages
 900 + ==================
 901 + from python
 902 + ~~~~~~~~~~~
 903   
 904 -         print "Cleaning up %r" % filename
 905 -         os.remove(filename)
 906 + .. code:: python
 907   
 908 - ----
 909 +     import fedmsg
 910   
 911 - :data-x: r1600
 912 - :data-y: 0
 913 +     for name, endpoint, topic, msg in fedmsg.tail_messages():
 914 +         print topic, msg
 915   
 916 - Intermezzo
 917 - ==========
 918   
 919 - We have a neat working script that gets fedmsg messages pushed to it.  It can
 920 - extract neato stuff and print it.
 921 + consuming messages
 922 + ==================
 923 + with a daemon
 924 + ~~~~~~~~~~~~~
 925   
 926 - But... if we want to move to the next step, we have to take a break from our
 927 - happy hacking to go and deal with Twitter, its API, and API keys.
 928 + ``fedmsg-hub`` is a daemon that can make writing your own
 929 + long-running consumers simpler.  There are `docs on fedmsg.com
 930 + <http://www.fedmsg.com/en/latest/consuming/#the-hub-consumer-approach>`_
 931 + for writing plugins, but they look like this:
 932   
 933 - ----
 934 + .. code:: python
 935   
 936 - :data-x: r1600
 937 - :data-y: 0
 938 +     import pprint
 939 +     import fedmsg.consumers
 940   
 941 - The Twitter API
 942 - ===============
 943   
 944 - We're going to have to:
 945 +     class MyConsumer(fedmsg.consumers.FedmsgConsumer):
 946 +         topic = "org.fedoraproject.*"
 947 +         config_key = 'myconsumer.enabled'
 948   
 949 - 1) Create our own "app".  Visit https://apps.twitter.com/app/new
 950 - 2) Modify that app's permission to include **"Read and Write"**.
 951 - 3) Authorize that app with our own account, which yields *oauth tokens*.
 952 -    To do this, click the **"Create my access token"** button at the bottom of
 953 -    your app's detail page.
 954 +         def consume(self, message):
 955 +             pprint.pprint(message)
 956   
 957 - We will keep those tokens a secret and our little bot will use them to login
 958 - and tweet on our behalf.  You'll get **four** secret strings.
 959   
 960 - ----
 961 + consuming messages
 962 + ==================
 963 + at the command line... an aside
 964 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 965   
 966 - :data-x: r0
 967 - :data-y: r900
 968 + There are lots of fun options to ``fedmsg-tail`` like ``--terse``.
 969   
 970 + .. code:: bash
 971   
 972 - Storing
 973 - =======
 974 - those secrets
 975 - ~~~~~~~~~~~~~
 976 +    fedmsg-tail --terse
 977   
 978 - First, add a directory called ``fedmsg.d/`` to your current working directory.
 979 + .. code:: text
 980   
 981 - In it, put a file called ``fedmsg.d/twitter-secrets.py`` that looks like this:
 982 +     buildsys.build.state.change -- ausil's tncfhh-0.8.3-14.fc20 completed
 983 +     http://koji.fedoraproject.org/koji/buildinfo?buildID=439734
 984 +     trac.ticket.update -- kevin closed a ticket on the Fedora Infrastructure trac instance as 'fixed'
 985 +     https://fedorahosted.org/fedora-infrastructure/ticket/3904
 986 +     bodhi.update.request.testing -- mmckinst submitted nawk-20121220-1.fc18 to testing
 987 +     https://admin.fedoraproject.org/updates/nawk-20121220-1.fc18
 988 +     wiki.article.edit -- Hguemar made a wiki edit to "Flock:Rideshare"
 989 +     https://fedoraproject.org/w/index.php?title=Flock:Rideshare&diff=prev&oldid=347430
 990   
 991 - .. code:: python
 992   
 993 -     config = dict(
 994 -         consumer_key        = "your api key goes here",
 995 -         consumer_secret     = "your api secret goes here",
 996 -         access_token_key    = "your access token goes here",
 997 -         access_token_secret = "your access token secret goes here",
 998 -     )
 999 + things that use fedmsg
1000 + ======================
1001 + there's a lot of them at this point
1002 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1003   
1004 - Test that fedmsg can read in that new config file by looking for them in:
1005   
1006 - .. code:: bash
1007 + koji
1008 + ====
1009 + stalk
1010 + ~~~~~
1011   
1012 -     fedmsg-config | less
1013 + David Aquilina's (dwa's) `koji stalk
1014 + <http://dwa.fedorapeople.org/wip/koji-stalk.py>`_ monitors koji over fedmsg and
1015 + rebuilds packages for arm and ppc.
1016   
1017   ----
1018   
1019 - Using
1020 - =====
1021 - those secrets
1022 - ~~~~~~~~~~~~~
1023 - 
1024 - Go back to ``badgebot.py`` and add the following:
1025 + FAS2Trac (ftl)
1026 + ==============
1027 + (fama updater)
1028 + ~~~~~~~~~~~~~~
1029   
1030 - .. code:: python
1031 + herlo's `FAS2Trac fama updater (ftl)
1032 + <https://git.fedorahosted.org/cgit/ftl.git>`_ listens to messages indicating
1033 + that a user has applied for membership in the ambassadors group -- it then
1034 + files a ticket in the `ambassadors' trac instance
1035 + <https://fedorahosted.org/fama/>`_ for a potential sponsor via XMLRPC.
1036   
1037 -     import tweepy
1038 + ----
1039   
1040 -     consumer_key        = config['consumer_key']
1041 -     consumer_secret     = config['consumer_secret']
1042 -     access_token_key    = config['access_token_key']
1043 -     access_token_secret = config['access_token_secret']
1044 + compose
1045 + =======
1046 + downloader
1047 + ~~~~~~~~~~
1048   
1049 -     auth_handler = tweepy.OAuthHandler(consumer_key, consumer_secret)
1050 -     auth_handler.set_access_token(access_token_key, access_token_secret)
1051 -     twitter_api = tweepy.API(auth_handler)
1052 + p3ck's `fedmsg-download <https://github.com/p3ck/fedmsg-download/>`_
1053 + listens for messages that the daily branched and rawhide compose
1054 + process has finished -- it then downloads the latest builds from
1055 + ``rsync://dl.fedoraproject.org/fedora-linux-development``
1056   
1057   ----
1058   
1059 - And
1060 - ===
1061 - further down
1062 - ~~~~~~~~~~~~
1063 + synchronization
1064 + ===============
1065 + of package ACLs
1066 + ~~~~~~~~~~~~~~~
1067   
1068 - .. code:: python
1069 + So, it **used** to be that when someone was granted *commit* access to a
1070 + package in the `Fedora PackageDB (pkgdb)
1071 + <https://apps.fedoraproject.org/#PkgDB>`_, the webapp simply wrote to a
1072 + database table indicating the new relationship.  Every *hour*, a cronjob would
1073 + run that queried the state of that database and then re-wrote out the ACLs for
1074 + gitolite -- the software that manages access to our `package repositories
1075 + <http://pkgs.fedoraproject.org>`_.
1076   
1077 -     for name, endpoint, topic, msg in fedmsg.tail_messages():
1078 + Consequently, we had lots of *waiting*: you would request commit access to a
1079 + repository, then *wait* for an owner to grant you rights, then *wait* for that
1080 + cronjob to run before you could actually push.
1081   
1082 -         subtitle = fedmsg.meta.msg2subtitle(msg, **config)
1083 -         link = fedmsg.meta.msg2link(msg, **config)
1084 -         icon = fedmsg.meta.msg2icon(msg, **config)
1085 + With `a new fedmsg consumer
1086 + <https://github.com/fedora-infra/fedmsg-genacls/blob/develop/fedmsg_genacls.py>`_
1087 + that we have in place, those gitolite ACLs are re-written in response to
1088 + fedmsg messages from the pkgdb.  It is much faster.
1089   
1090 -         _, filename = tempfile.mkstemp(suffix='.png')
1091 -         print "Downloading", icon, "to", filename
1092 -         urllib.urlretrieve(icon, filename)
1093 + ----
1094   
1095 -         # Construct and post our tweet.
1096 -         #print fabulous.image.Image(filename)
1097 -         content = subtitle + " " + link
1098 -         print "Tweeting %r" % content
1099 -         twitter_api.update_with_media(filename, content)
1100 + notifications
1101 + =============
1102 + to email, irc, the desktop, and android
1103 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1104   
1105 -         print "Cleaning up %r" % filename
1106 -         os.remove(filename)
1107 + There's the new `FMN system <https://apps.fedoraproject.org/>`_ that can
1108 + deliver notifications to you via irc, email, and android.
1109   
1110 - ----
1111 + There's also lmacken's `fedmsg-notify <http://lewk.org/blog/fedmsg-notify>`_
1112 + which listens for messages and displays a filtered stream on your desktop with
1113 + ``libnotify``.
1114   
1115 - Does it work?
1116 - =============
1117 + .. image:: http://threebean.org/presentations/images/fedmsg-flock14-img/fedmsg-notify-0-crop.png
1118 +    :height: 300px
1119   
1120   ----
1121   
1122 - :data-x: r1600
1123 - :data-y: 0
1124 - 
1125 - systemd
1126 + reports
1127   =======
1128 - for real
1129 - ~~~~~~~~
1130 + 10 ways from sunday
1131 + ~~~~~~~~~~~~~~~~~~~
1132   
1133 - Make a new file called ``badgebot.service`` with these contents::
1134 + Every week, pingou's `owner changes report tool
1135 + <https://lists.fedoraproject.org/pipermail/infrastructure/2013-June/013070.html>`_
1136 + emails the devel list with a report of what packages were orphaned, unorphaned
1137 + and retired.
1138   
1139 -     [Unit]
1140 -     Description=A Twitter bot for your Fedora Badges.  Wow.
1141 -     After=network.target
1142 -     Documentation=http://fedmsg.com
1143 + .. image:: http://threebean.org/presentations/images/fedmsg-flock14-img/ownerchange-screenshot.png
1144 +    :height: 420px
1145   
1146 -     [Service]
1147 -     ExecStart=/usr/local/bin/badgebot.py
1148 -     Type=simple
1149 -     User=fedmsg
1150 -     Group=fedmsg
1151 + ----
1152   
1153 -     [Install]
1154 -     WantedBy=multi-user.target
1155 + reports
1156 + =======
1157 + 10 ways from sunday
1158 + ~~~~~~~~~~~~~~~~~~~
1159 + 
1160 + There's also the `Release Engineering Dashboard
1161 + <https://apps.fedoraproject.org/releng-dash>`_ which grabs data from
1162 + datagrepper on all the latest updates syncs, composes, image builds, etc.. and
1163 + puts their status all in one place.  Pure HTML/javascript -- there's no
1164 + server-side app here.
1165 + 
1166 + .. image:: http://threebean.org/presentations/images/fedmsg-flock14-img/releng-dash-screenshot.png
1167 +    :height: 350px
1168   
1169   ----
1170   
1171 - :data-x: r0
1172 - :data-y: r900
1173 + fedora badges
1174 + =============
1175 + for you, and you, and you
1176 + ~~~~~~~~~~~~~~~~~~~~~~~~~
1177   
1178 - install.sh
1179 - ==========
1180 + `Fedora badges <https://badges.fedoraproject.org/>`_ launched last year at
1181 + Flock13.  It awards "badges" to Fedora contributors for their activity.
1182   
1183 - .. code:: bash
1184 + .. image:: http://threebean.org/presentations/images/fedmsg-flock14-img/badges_fan.png
1185   
1186 -     #!/bin/bash -x
1187 -     # install.sh - (re)install and (re)start the badgebot
1188 + Pretty fun.  ``:)``
1189   
1190 -     # Install our script
1191 -     cp badgebot.py /usr/local/bin/badgebot.py
1192 + ----
1193   
1194 -     # Make sure no one else can read our secrets.
1195 -     cp fedmsg.d/twitter-secrets.py /etc/fedmsg.d/.
1196 -     chown fedmsg:fedmsg /etc/fedmsg.d/twitter-secrets.py
1197 -     chmod o-r /etc/fedmsg.d/twitter-secrets.py
1198 + To sum that up
1199 + ==============
1200   
1201 -     # Copy in service file for systemd
1202 -     cp badgebot.service /usr/lib/systemd/system/badgebot.service
1203 -     systemctl daemon-reload
1204 -     systemctl restart badgebot
1205 + The assimilation of **message producing services** is nearly complete.
1206   
1207 - ----
1208 + There are many **message consuming services** already in place.. but we can
1209 + likely make many more.  Which is why you're here, no?
1210   
1211 - Watch the journal::
1212   
1213 -     sudo journalctl -u badgebot --follow
1214   
1215   Does it work?  Debug!
1216