#986 Try to guess character set before displaying a file on a web page
Closed: Fixed Opened by bruno.

As far as I can tell, Pagure doesn't try to guess the character set of a file before trying to display it. Instead utf-8 is assumed. This could cause files to be rendered incorrectly. uchardet or python-chardet could be used to guess the charset and then a conversion can be done to utf-8.
There could possibly be performance or security issues with doing this on user supplied files.


Well, the HTML rendered is clearly marked as 'utf-8':

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>

So I do not think that this is or will be a problem.

I don't think you understood the point of the request.
Some project may ship an iso-8859-1 documentation file. If that file is displayed as is on the pagure hosting service marked as a utf-8 file, then it will not be rendered correctly in browsers. There could be a requirement that all text files need to be in utf-8, but projects may have valid reasons for wanting to use other character sets, particularly for documentation.
Now it may be that I missed something and Pagure does do character set conversion.

Hm I see your point, based on the spread of UTF-8 nowadays, I'm going to wait for the issue to arise for real but I would review a patch addressing this is someone wants to work on this.

Also, I must confess that I am not thrilled at the idea to add a new dependency to pagure.

I agree this is not a high priority with the prevelance of utf-8.

One other note, while looking for something else I found that pagure is already importing chardet so adding some other checks won't add any new external requirements.

One other note, while looking for something else I found that pagure is already importing chardet so adding some other checks won't add any new external requirements.

Oh cool, that should make it easier indeed

Implemented in https://pagure.io/pagure/pull-request/1012

Metadata