daemon: don't allow worker manager interface to be unexported more than once
    
    if a conversation is stopped asynchronously and then synchronously, we
    will end up trying to unexport the bus interface more than once, leading
    to log spew.
    
    This commit protects against that.
    
    https://bugzilla.gnome.org/show_bug.cgi?id=737391