Something in the Fedora mirror logs from Sep 16 caused parse-access-log.py to crash:

[wwoods@log01 ~][PROD-IAD2]$ ./countme-update.sh --checkoutdir ~/mirrors-countme
log 1/20, date=13/Sep/2020:   0%|                                                              | 12.0k/6.44G [00:00<40:52, 2.63Mb/s]                                    
log 2/20, date=14/Sep/2020: 100%|██████████████████████████████████████████████████████████████| 6.56G/6.56G [02:58<00:00, 36.6Mb/s]                                    
log 3/20, date=15/Sep/2020: 100%|██████████████████████████████████████████████████████████████| 6.71G/6.71G [03:00<00:00, 37.1Mb/s]                                    
log 4/20, date=16/Sep/2020:  41%|█████████████████████████▋                                    | 2.80G/6.78G [01:15<01:46, 37.3Mb/s]Traceback (most recent call last):  
  File "/home/fedora/wwoods/mirrors-countme/parse-access-log.py", line 122, in <module>                                                                                 
    main()
  File "/home/fedora/wwoods/mirrors-countme/parse-access-log.py", line 114, in main
    args.writer.write_items(match_iter)
  File "/home/fedora/wwoods/mirrors-countme/countme/__init__.py", line 319, in write_items                                                                              
    self._con.executemany(self._insert_item, items)
sqlite3.IntegrityError: NOT NULL constraint failed: countme_raw.repo_arch

This means we tried to insert an item into the database that had a NULL repo_arch, which is required by the database schema - because it should always be present for well-formed requests from libdnf.

This means we either matched a malformed log entry - a countme=N line that's missing the arch=XXX item - or we mangled/lost the repo_arch item during parsing.

The countme log matcher/parser doesn't require arch=XXX item in the URL's query string, but it does only match log lines where the status is 200 or 302. In theory, those should only be returned if the mirrormanager server accepts the query - and the server is supposed to reject a URL that has no arch. But perhaps this assumption is invalid.

I'm currently trying to find the exact log line that caused the error.

(NOTE: this is a little tricky, because It's unclear from the output which exact file this error corresponds to. This is partly because of how the log files are named - log entries for 16/Sep/2020 are found in /mnt/fedora_stats/combined-http/2020/09/17/mirrors.fedoraproject.org-access.log. It would be helpful for parse-access-log.py to give the exact file / line number corresponding to the error!)

I'm also unsure whether the rawdb is now in an inconsistent state, with part of 16/Sep/2020 logs inserted (and part missing) - which would cause future runs to skip this day, thus dropping a bunch of data. Will investigate that next; might be a separate issue.