#5573 [help] Can not import 389 trac tickets to pagure
Closed: Fixed Opened by mreynolds.

For some reason when trying to import tickets from https://fedorahosted.org/389/ it fails with this error from fedorahosted:

$ pgimport fedorahosted https://fedorahosted.org/389/ --tags
Enter your FAS Username: mreynolds
Password:
Repo available

1 - 389.git

Choose the import destination repo [1]:
ERROR: Error in response: {u'message': u'ServiceException details : unsupported type for timedelta microseconds component: unicode', u'code': -32603, u'name': u'JSONRPCError'}

When I try other repos it works fine, only "389" has issues. I would gladly help if you want to do live debugging, etc. I can be found on IRC "mreynolds" (internal #dirsec, freenode #389)


ok. Digging around and adding some debugging, this is the traceback trac outputs:

 2016-11-18 18:13:31,284 Trac[xml_rpc] ERROR: Traceback (most recent call last):
   File "/usr/lib/python2.6/site-packages/tracrpc/web_ui.py", line 158, in _rpc_process
     result = (XMLRPCSystem(self.env).get_method(method_name)(req, args))[0]
   File "/usr/lib/python2.6/site-packages/tracrpc/api.py", line 197, in __call__
     result = self.callable(req, *args)
   File "/usr/lib/python2.6/site-packages/tracrpc/ticket.py", line 140, in get
     t = model.Ticket(self.env, id)
   File "/usr/lib/python2.6/site-packages/trac/ticket/model.py", line 68, in __init__
     self._fetch_ticket(tkt_id, db)
   File "/usr/lib/python2.6/site-packages/trac/ticket/model.py", line 121, in _fetch_ticket
     self.values[field] = from_utimestamp(value)
   File "/usr/lib/python2.6/site-packages/trac/util/datefmt.py", line 98, in from_utimestamp
     return _epoc + timedelta(microseconds=ts or 0)
 ServiceException: unsupported type for timedelta microseconds component: unicode

I see an old trac ticket where someone hit this with wiki pages:
https://trac.edgewall.org/ticket/10287
but the solution there doesn't work for us.

debugging says it's running this sql:

 SELECT t.id AS id,t.summary AS summary,t.owner AS owner,t.type AS type,t.status AS status,t.priority AS priority,t.milestone AS milestone,t.time AS time,t.changetime AS changetime,priority.value AS priority_value
 FROM ticket AS t
   LEFT OUTER JOIN enum AS priority ON (priority.type='priority' AND priority.name=priority)
 ORDER BY COALESCE(t.id,0)=0,t.id

Running a query for just time and updated I see the following tickets that are messed up:

 SELECT t.id AS id,t.time AS time,t.changetime AS changetime FROM ticket AS t

...
47291|query_constraints|[{'component': [u'Security - Access Control (ACL)'], 'milestone': [u'1.2.11.20'], 'status': [u'reopened', u'accepted', u'new', u'assigned']}]
47292|query_tickets|628
47293|query_href|/389/query?component=Security+-+Access+Control+(ACL)&milestone=1.2.11.20&status=reopened&status=accepted&status=new&status=assigned&order=priority&group=status
47295|query_href|/389/query?component=Replication+-+General&status=new&status=assigned&status=accepted&status=reopened&milestone=1.3.2&order=owner&group=status&desc=1
47297|query_constraints|[{'component': [u'Replication - General'], 'status': [u'new', u'assigned', u'accepted', u'reopened'], 'milestone': [u'1.3.2']}]
...

So, we need to delete or fix these tickets... Is there any other record (emails/list, etc) that might tell us what these are and if we can delete them or should try and fix them? They don't seem to contain any useful data. ;(

 sqlite> select * from ticket where id = '47291';
 47291|0|query_constraints|[{'component': [u'Security - Access Control (ACL)'], 'milestone': [u'1.2.11.20'], 'status': [u'reopened', u'accepted', u'new', u'assigned']}]|||||||||||||
 sqlite> select * from ticket where id = '47292';
 47292|0|query_tickets|628|||||||||||||
 sqlite> select * from ticket where id = '47293';
 47293|0|query_href|/389/query?component=Security+-+Access+Control+(ACL)&milestone=1.2.11.20&status=reopened&status=accepted&status=new&status=assigned&order=priority&group=status|||||||||||||
 sqlite> select * from ticket where id = '47294';
 47294|0|query_time|1363880649|||||||||||||
 sqlite> select * from ticket where id = '47295';
 47295|0|query_href|/389/query?component=Replication+-+General&status=new&status=assigned&status=accepted&status=reopened&milestone=1.3.2&order=owner&group=status&desc=1|||||||||||||
 sqlite> select * from ticket where id = '47297';
 47297|0|query_constraints|[{'component': [u'Replication - General'], 'status': [u'new', u'assigned', u'accepted', u'reopened'], 'milestone': [u'1.3.2']}]|||||||||||||

:hotdog:

Kevin,

So a few years ago something changed in the trac database, and our ticket numbers jumped from 400 something to 47298. So all of the tickets listed above (47290 - 47297) never really existed (perhaps they were used or testing?). Anyway they can be deleted, but I don't know how to do that.

Did you see any other tickets that were invalid?

Thanks,
Mark

Nope. Only those ones.

I can manually delete them from command line. (After taking a backup).

Will do so now and then have you try the import again...

ok, done. Can you try the import again?

$ pgimport fedorahosted https://fedorahosted.org/389/ --tags
Enter your FAS Username: mreynolds
Password:

Repo available

1 - clone-nunc-test.git
2 - nunc-test.git
3 - rest389.git
4 - ds.git

Choose the import destination repo [1]: 4
Updated ds.git with issue :1/49046
Updated ds.git with issue :2/49046
Updated ds.git with issue :3/49046
Updated ds.git with issue :4/49046
Updated ds.git with issue :5/49046
Updated ds.git with issue :6/49046
Updated ds.git with issue :7/49046
Updated ds.git with issue :8/49046
Updated ds.git with issue :9/49046
Updated ds.git with issue :10/49046
Updated ds.git with issue :11/49046
Updated ds.git with issue :12/49046
Updated ds.git with issue :13/49046
Updated ds.git with issue :14/49046
Updated ds.git with issue :15/49046
Updated ds.git with issue :16/49046
Updated ds.git with issue :17/49046
Updated ds.git with issue :18/49046
Updated ds.git with issue :19/49046
Updated ds.git with issue :20/49046
Updated ds.git with issue :21/49046
Updated ds.git with issue :22/49046
Updated ds.git with issue :23/49046
Updated ds.git with issue :24/49046
ERROR: Error in response: {u'message': u"Attachment 'dshang_result.tar.gz' not found", u'code': 404, u'name': u'JSONRPCError'}

Two issues:

[1] The ticket number is not incrementing (not sure if that's a pagure issue or not)
[2] Missing attachment is breaking the import

The attachment there on ticket 25 doesn't exist indeed. Any idea what happened to it?

Either we should:

1) create some kind of placeholder attachment there so the import goes on. (Or if there's another copy of it I can just put it in place if you can get it to me)

or

2) open a issue against pagure-importer to ignore missing attachments and go on.

:ping_pong:

I'm currently looking into [2] - playing around with the code now. I'm not sure yet if the error in the response is blocking the import of that ticket

Yeah I fixed the missing attachment issue in the pagure importer. So we are good to go! Thanks for your help Kevin, please close this out.

Great. Please do let us know if you run into any further issues we can assist with.

:japanese_castle:

@kevin changed the status to Closed

Yeah I fixed the missing attachment issue in the pagure importer.

Would it better if that fix gets into importer project?

Metadata