Wow. At first glance, I have no idea.

This seems unrelated to subscribing to mail (and seems more to do with oauth login). Are these emails sent directly to you or to the sysadmin-ask-members address?

You are right. These are all errors send to sysadmin-ask-members address (which I filter to the same folder and hence the confusion) and my subscriptions have stopped working entirely it appears (matches the recent feedback from a user)? I haven't looked into it more

Hm, I just tried a test of the mail backend in staging by:

• Changing the admin address to my personal address
• Running "python manage.py shell" and in that shell typing

{{{

from django.core.mail import mail_admins
mail_admins('test subject', 'hello world')
}}}

I successfully received the email. I'm going to try the other django.core.mail functions and see how they fare.

FYI, these are my settings

https://ask.fedoraproject.org/users/2/mether/?sort=email_subscriptions

I am still getting lots of errors like these:

Delivered-To: metherid@gmail.com
Received: by 10.194.34.103 with SMTP id y7csp11149wji;
Thu, 7 Mar 2013 01:06:50 -0800 (PST)
X-Received: by 10.67.10.45 with SMTP id dx13mr1853030pad.140.1362647210064;
Thu, 07 Mar 2013 01:06:50 -0800 (PST)
Return-Path: askfedora@fedoraproject.org
Received: from bastion.fedoraproject.org (bastion01.fedoraproject.org. [209.132.181.2])
by mx.google.com with ESMTP id tm6si842489pbc.235.2013.03.07.01.06.49;
Thu, 07 Mar 2013 01:06:50 -0800 (PST)
Received-SPF: pass (google.com: best guess record for domain of askfedora@fedoraproject.org designates 209.132.181.2 as permitted sender) client-ip=209.132.181.2;
Authentication-Results: mx.google.com;
spf=pass (google.com: best guess record for domain of askfedora@fedoraproject.org designates 209.132.181.2 as permitted sender) smtp.mail=askfedora@fedoraproject.org
Received: by bastion01.phx2.fedoraproject.org (Postfix)
id 5C81B20B0C; Thu, 7 Mar 2013 09:06:47 +0000 (UTC)
Delivered-To: sysadmin-ask-members@fedoraproject.org
Received: from ask01.fedoraproject.org (ask01.fedoraproject.org [10.5.126.79])
by bastion01.phx2.fedoraproject.org (Postfix) with ESMTP id 4EB6320A1F
for sysadmin-ask-members@fedoraproject.org; Thu, 7 Mar 2013 09:06:47 +0000 (UTC)
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Subject: ERROR (EXTERNAL IP): Internal Server Error: /users/
From: askfedora@fedoraproject.org
To: sysadmin-ask-members@fedoraproject.org
Date: Thu, 07 Mar 2013 09:06:48 -0000
Message-ID: 20130307090648.23632.22779@ask01.fedoraproject.org

Traceback (most recent call last):

File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py", lin=
e 113, in get_response
response =3D callback(request, callback_args, *callback_kwargs)

File "/usr/lib/python2.6/site-packages/askbot/views/users.py", line 159, =
in show_users
users_page =3D objects_list.page(page)

File "/usr/lib/python2.6/site-packages/django/core/paginator.py", line 37=
, in page
number =3D self.validate_number(number)

File "/usr/lib/python2.6/site-packages/django/core/paginator.py", line 28=
, in validate_number
if number > self.num_pages:

File "/usr/lib/python2.6/site-packages/django/core/paginator.py", line 60=
, in _get_num_pages
if self.count =3D=3D 0 and not self.allow_empty_first_page:

File "/usr/lib/python2.6/site-packages/django/core/paginator.py", line 48=
, in _get_count
self._count =3D self.object_list.count()

File "/usr/lib/python2.6/site-packages/django/db/models/query.py", line 3=
34, in count
return self.query.get_count(using=3Dself.db)

File "/usr/lib/python2.6/site-packages/django/db/models/sql/query.py", li=
ne 401, in get_count
number =3D obj.get_aggregation(using=3Dusing)[None]

File "/usr/lib/python2.6/site-packages/django/db/models/sql/query.py", li=
ne 367, in get_aggregation
result =3D query.get_compiler(using).execute_sql(SINGLE)

File "/usr/lib/python2.6/site-packages/django/db/models/sql/compiler.py",=
line 735, in execute_sql
cursor.execute(sql, params)

File "/usr/lib/python2.6/site-packages/django/db/backends/postgresql_psyc=
opg2/base.py", line 44, in execute
return self.cursor.execute(query, args)

DatabaseError: column "language_code" does not exist
LINE 1: ...search_vector @@ plainto_tsquery(E'praneeth') AND language_c...
^

<WSGIRequest GET:<QueryDict: {u'query': [u'praneeth'], u'search': [u''], u't': [u'user']= }>,
POST:<QueryDict: {}>,
COOKIES:{'ask.fedoraproject.org': '7d9b9180116ce33d8a7e7c76d8964a19',
'askbot_visitor': 'False',
'sessionid': 'b2b5a6c058de0b5700e7be6da6752526'},
META:{'DOCUMENT_ROOT': '/var/www/html',
'GATEWAY_INTERFACE': 'CGI/1.1',
'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=3D0.9,/=
;q=3D0.8',
'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=3D0.7,*;q=3D0.3',
'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch',
'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=3D0.8',
'HTTP_COOKIE': 'ask.fedoraproject.org=3D7d9b9180116ce33d8a7e7c76d8964a19; =
askbot_visitor=3DFalse; sessionid=3Db2b5a6c058de0b5700e7be6da6752526',
'HTTP_HOST': 'ask01',
'HTTP_REFERER': 'https://ask.fedoraproject.org/users/',
'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.22 (KHTM=
L, like Gecko) Chrome/25.0.1364.97 Safari/537.22',
'HTTP_X_FORWARDED_FOR': '117.192.186.208',
'HTTP_X_FORWARDED_HOST': 'ask.fedoraproject.org',
'HTTP_X_FORWARDED_SERVER': 'ask.fedoraproject.org',
'PATH_INFO': u'/users/',
'PATH_TRANSLATED': '/usr/sbin/askbot.wsgi/users/',
'QUERY_STRING': 'search=3D&t=3Duser&query=3Dpraneeth',
'REMOTE_ADDR': '192.168.1.7',
'REMOTE_PORT': '44621',
'REQUEST_METHOD': 'GET',
'REQUEST_URI': '/users/?search=3D&t=3Duser&query=3Dpraneeth',
'SCRIPT_FILENAME': '/usr/sbin/askbot.wsgi',
'SCRIPT_NAME': u'',
'SERVER_ADDR': '192.168.1.51',
'SERVER_ADMIN': 'admin@fedoraproject.org',
'SERVER_NAME': 'ask01',
'SERVER_PORT': '80',
'SERVER_PROTOCOL': 'HTTP/1.0',
'SERVER_SIGNATURE': '<address>Apache/2.2.15 (Red Hat) Server at ask01 Port=
80</address>\n',
'SERVER_SOFTWARE': 'Apache/2.2.15 (Red Hat)',
'mod_wsgi.application_group': '',
'mod_wsgi.callable_object': 'application',
'mod_wsgi.handler_script': '',
'mod_wsgi.input_chunked': '0',
'mod_wsgi.listener_host': '0.0.0.0',
'mod_wsgi.listener_port': '80',
'mod_wsgi.process_group': 'askbot',
'mod_wsgi.request_handler': 'wsgi-script',
'mod_wsgi.script_reloading': '1',
'mod_wsgi.version': (3, 2),
'wsgi.errors': <mod_wsgi.Log object at 0xb23398e0>,
'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter obj= ect at 0xb2695650>,
'wsgi.input': <mod_wsgi.Input object at 0xb2339ea8>,
'wsgi.multiprocess': True,
'wsgi.multithread': False,
'wsgi.run_once': False,
'wsgi.url_scheme': 'http',
'wsgi.version': (1, 1)}>

Oh, I finally got the test mail notification

I get feedback emails, but I don't think I'm getting any email notifications.

I periodically get mails like this:

{{{
Traceback (most recent call last):

File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py", line 113, in get_response
response = callback(request, callback_args, *callback_kwargs)

File "/usr/lib/python2.6/site-packages/askbot/views/users.py", line 1123, in user
return user_view_func(request, profile_owner, context)

File "/usr/lib/python2.6/site-packages/askbot/views/users.py", line 690, in user_recent
content_object=award.content_object,

File "/usr/lib/python2.6/site-packages/django/contrib/contenttypes/generic.py", line 80, in get
ct = self.get_content_type(id=ct_id, using=instance._state.db)

File "/usr/lib/python2.6/site-packages/django/contrib/contenttypes/generic.py", line 59, in get_content_type
return ContentType.objects.db_manager(using).get_for_id(id)

File "/usr/lib/python2.6/site-packages/django/contrib/contenttypes/models.py", line 55, in get_for_id
self._add_to_cache(self.db, ct)

File "/usr/lib/python2.6/site-packages/django/contrib/contenttypes/models.py", line 70, in _add_to_cache
key = (model._meta.app_label, model._meta.object_name.lower())

AttributeError: 'NoneType' object has no attribute '_meta'

<WSGIRequest GET:<QueryDict: {u'sort': [u'recent']}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'DOCUMENT_ROOT': '/var/www/html',
'GATEWAY_INTERFACE': 'CGI/1.1',
'HTTP_ACCEPT': '/',
'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
'HTTP_FROM': 'googlebot(at)googlebot.com',
'HTTP_HOST': 'ask01',
'HTTP_USER_AGENT': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)',
'HTTP_X_FORWARDED_FOR': '66.249.73.55',
'HTTP_X_FORWARDED_HOST': 'ask.fedoraproject.org',
'HTTP_X_FORWARDED_SERVER': 'ask.fedoraproject.org',
'PATH_INFO': u'/users/3595/jomoos/',
'PATH_TRANSLATED': '/usr/sbin/askbot.wsgi/users/3595/jomoos/',
'QUERY_STRING': 'sort=recent',
'REMOTE_ADDR': '192.168.1.7',
'REMOTE_PORT': '52906',
'REQUEST_METHOD': 'GET',
'REQUEST_URI': '/users/3595/jomoos/?sort=recent',
'SCRIPT_FILENAME': '/usr/sbin/askbot.wsgi',
'SCRIPT_NAME': u'',
'SERVER_ADDR': '192.168.1.51',
'SERVER_ADMIN': 'admin@fedoraproject.org',
'SERVER_NAME': 'ask01',
'SERVER_PORT': '80',
'SERVER_PROTOCOL': 'HTTP/1.0',
'SERVER_SIGNATURE': '<address>Apache/2.2.15 (Red Hat) Server at ask01 Port 80</address>\n',
'SERVER_SOFTWARE': 'Apache/2.2.15 (Red Hat)',
'mod_wsgi.application_group': '',
'mod_wsgi.callable_object': 'application',
'mod_wsgi.handler_script': '',
'mod_wsgi.input_chunked': '0',
'mod_wsgi.listener_host': '0.0.0.0',
'mod_wsgi.listener_port': '80',
'mod_wsgi.process_group': 'askbot',
'mod_wsgi.request_handler': 'wsgi-script',
'mod_wsgi.script_reloading': '1',
'mod_wsgi.version': (3, 2),
'wsgi.errors': <mod_wsgi.Log object at 0xb24344a8>,
'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0xb2694bf0>,
'wsgi.input': <mod_wsgi.Input object at 0xb2434188>,
'wsgi.multiprocess': True,
'wsgi.multithread': False,
'wsgi.run_once': False,
'wsgi.url_scheme': 'http',
'wsgi.version': (1, 1)}>
}}}

I'm more concerned about how this affects moderators capabilities to send users messages. I periodically send them messages asking them to look at guidelines etc. Are they getting these?

Thanks,
Ankur

Evgeny pointed me to
https://ask.fedoraproject.org/settings/MIN_REP/

The very last entry (Trigger email notifications) was set to '15',
meaning that no user with a reputation less than 15 could trigger an
email. It could be that we haven't noticed any users with reputation
greater than that the past few weeks. I have set that parameter to 0,
which means that any user can trigger an email now.. but also that
spam comments will trigger emails. If this is the case, at least
we'll know that mail works, and we can adjust the value. Evgeny adds that "A remedy could be: set the rep limit to say 10 and make a rule among admins to upvote a new post"

That doesn't seem like the right explanation. I am subscribed to everything as you can see at https://ask.fedoraproject.org/users/2/mether/?sort=email_subscriptions and should be getting emails.

You're right.

It turns out that askbot (if configured to) uses celeryd to send its mail asynchronously. We didn't have a celeryd instance running but we did have askbot configured to look for one.

I've turned off askbot's celeryd expectations in production... let's see if that does the trick.

Don't think it has fixed the problem. Is it working for anyone else?

The problem appears to be fixed for me. Thanks for everyone who worked on this!

I tracked it down to a bug in python-lxml that caused askbot to fail silently when trying to scrub emails before sending[1]. It only occurs on 32bit servers, but not on 64bit ones. Our staging instance is 64bit and our production one is 32bit which explains the discrepancy.

We are patched for now, but we're going to schedule a short downtime in the next few weeks to reinstall our production instance (and take care of a few other issues). Thanks for your patient feedback on this!

[1] - https://bugzilla.redhat.com/show_bug.cgi?id=951233