|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
gstrauss commented 2 years ago | ||
praiskup commented 2 years ago I think that the configuration - as you prpopose - is fine. Lightttpd just doesn't | ||
gstrauss commented 2 years ago
I see that the link is not included. You might strace lighttpd to see if it is trying to open the file after generating the As an aside, I poked around and saw .xml.gz files there. The lua code in this patch handling .log.gz | ||
gstrauss commented 2 years ago doh! Sorry. You're running lighttpd 1.4.61. Latest lighttpd release is lighttpd 1.4.63. There was a bug in lighttpd 1.4.60 and lighttpd 1.4.61 which did not show the readme. https://git.lighttpd.net/lighttpd/lighttpd1.4/commit/cba6a1ab54f063609922ea0085d6ba15ed77fa6f | ||
praiskup commented 2 years ago Answering your questions isn't easy, as it depends on the concrete directory, Also, most of the traffic to lighty goes through CloudFronts so it's cached | ||
praiskup commented 2 years ago I mean load forwarded to lighttpd is pretty low. | ||
gstrauss commented 2 years ago I think I answered my own question. During a build, the logs are being modified, so the caching interval should be brief, or disabled. If not too many people are watching live builds and nothing is polling in a tight loop, then there is little point to enabling the mod_dirlisting cache. That is true as long as once the build completes, an index.html is generated with the directory listing, so that the overhead of generating the directory listing never occurs again for that (completed) build. lighttpd is not returning HTTP Cache-Control in the response headers, so I am unsure how your are controlling the caching at CloudFront. Have you disabled the CloudFront content re-validation? Once an index.html is generated in a directory, then we should allow that to be cached. A short lua script could be substituted for mod_indexfile, and could add appropriate Cache-Control to the response if the file was served from index.html, or if index.html was not present, allow mod_dirlisting to generate the response. | ||
praiskup commented 2 years ago I think we could request an update of Lighttpd in Fedora 35... if going with JS and not Lua. | ||
praiskup commented 2 years ago CloudFronts caches everything, and yes - it really leads to user confusion (people are asking us what happened, that they don't see the build finished, etc.). So I'm sure the directory listing should be controlled at least till we finish the build. | ||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
gstrauss commented 2 years ago roles/copr/backend/files/lighttpd/mime.conf was created in 2014 and the significant change from default /etc/lighttpd/conf.d/mime.conf is the default Content-Type "" => "text/plain", instead of "application/octet-stream". Is this still relevant for downloads? Are there a smaller set of extensions which should have "text/plain" forced or a smaller portion of the site for which this should apply? It is generally preferable to use /etc/lighttpd/conf.d/mime.conf unless there is a specific reason not to do so. | ||
praiskup commented 2 years ago We should go with the default config, yes. | ||
gstrauss commented 2 years ago If the mime.conf is removed from this ansible package, how will the original from lighttpd-filesystem be restored? It is a config file, so it won't be overwritten by lighttpd package upgrades, will it? | ||
praiskup commented 2 years ago This is easier to do manually:, remove the config file and | ||
gstrauss commented 2 years ago It seems like a safer solution is to include /etc/lighttpd/conf.d/mime.conf from lighttpd-filesystem into this repository so that it overwrites existing installations. Then, some time later, after a round of deployment through dev, staging, and prod, then mime.conf can be removed from this package. | ||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
praiskup commented 2 years ago This looks good, but I fail to make it work. Would you mind adding it as a separate patch? 2021-12-10 23:50:03: (response.c.407) -- parsed Request-URI 2021-12-10 23:50:03: (response.c.409) Request-URI : /results/%40copr/copr-dev/epel-7-ppc64le 2021-12-10 23:50:03: (response.c.411) URI-scheme : http 2021-12-10 23:50:03: (response.c.413) URI-authority : copr-be-dev.cloud.fedoraproject.org 2021-12-10 23:50:03: (response.c.415) URI-path (clean): /results/@copr/copr-dev/epel-7-ppc64le 2021-12-10 23:50:03: (response.c.417) URI-query : 2021-12-10 23:50:03: (response.c.495) -- logical -> physical 2021-12-10 23:50:03: (response.c.497) Doc-Root : /var/lib/copr/public_html 2021-12-10 23:50:03: (response.c.499) Basedir : /var/lib/copr/public_html 2021-12-10 23:50:03: (response.c.501) Rel-Path : /results/@copr/copr-dev/epel-7-ppc64le 2021-12-10 23:50:03: (response.c.503) Path : /var/lib/copr/public_html/results/@copr/copr-dev/epel-7-ppc64le 2021-12-10 23:50:04: (response.c.407) -- parsed Request-URI 2021-12-10 23:50:04: (response.c.409) Request-URI : /fedora-rawhide-x86_64/01998568-deepin-system-monitor/ 2021-12-10 23:50:04: (response.c.411) URI-scheme : https 2021-12-10 23:50:04: (response.c.413) URI-authority : copr-be-dev.cloud.fedoraproject.org 2021-12-10 23:50:04: (response.c.415) URI-path (clean): /fedora-rawhide-x86_64/01998568-deepin-system-monitor/ 2021-12-10 23:50:04: (response.c.417) URI-query : 2021-12-10 23:50:04: (response.c.495) -- logical -> physical 2021-12-10 23:50:04: (response.c.497) Doc-Root : /var/lib/copr/public_html 2021-12-10 23:50:04: (response.c.499) Basedir : /var/lib/copr/public_html 2021-12-10 23:50:04: (response.c.501) Rel-Path : /fedora-rawhide-x86_64/01998568-deepin-system-monitor/ 2021-12-10 23:50:04: (response.c.503) Path : /var/lib/copr/public_html/fedora-rawhide-x86_64/01998568-deepin-system-monitor/ 2021-12-10 23:50:04: (response.c.522) -- handling subrequest 2021-12-10 23:50:04: (response.c.524) Path : /var/lib/copr/public_html/fedora-rawhide-x86_64/01998568-deepin-system-monitor/ 2021-12-10 23:50:04: (response.c.526) URI : /fedora-rawhide-x86_64/01998568-deepin-system-monitor/ 2021-12-10 23:50:04: (response.c.528) Pathinfo : 2021-12-10 23:50:04: (mod_dirlisting.c.1245) -- handling the request as Dir-Listing 2021-12-10 23:50:04: (mod_dirlisting.c.1247) URI : /fedora-rawhide-x86_64/01998568-deepin-system-monitor/ And still no readme file ^^^ appended... | ||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
praiskup commented 2 years ago Is this really expected? | ||
gstrauss commented 2 years ago Yes. The default is 4096 since lighttpd 1.4.56. For busy systems and for benchmarking lighttpd, I commented out things that were defaults, or were suboptimal to the defaults. | ||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
praiskup commented 2 years ago This is very interesting. This config did not work with new lighty: f736ed7 | ||
gstrauss commented 2 years ago I am not aware of any such issue with lighttpd 1.4.60 or lighttpd 1.4.61. However, I also do not understand what this mess is trying to accomplish. Then again, I also found that this specific CGI script was wasteful and unnecessary, and could be | ||
praiskup commented 2 years ago I used that condition with the actual version (basically, see the commit) because it worked! | ||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
praiskup commented 2 years ago This would be nice to have as a separate patch, too. I believe the cipher-list is replaced with "default" which should be PROFILE=SYSTEM by default on Fedora. | ||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
gstrauss commented 2 years ago @praiskup, are these obsolete? Did spacewalk archives move to a different server? Did cockpit-preview? Do i586 build logs exist? | ||
praiskup commented 2 years ago Spacewall/cockipt: I don't know personally, perhaps @xsuchy? | ||
gstrauss commented 2 years ago
Yes: bit rot. Since we're looking in lighttpd.conf, it is useful to remove cruft if it is easy to identify. | ||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
praiskup commented 2 years ago This is really interesting, the lua replacement. But we need to have this properly tested. A separate patch would be nice so we can test in izolation. | ||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
praiskup commented 2 years ago After my quick testing, I tend to agree that | ||
gstrauss commented 2 years ago From a secdevops perspective, eliminating dir-generator.php and server-side scripts reduces [Edited for word wrapping. | ||
praiskup commented 2 years ago There was some bug in lighttpd forcing us to "secure" it that way (v1.4.55 on Fedora 33). | ||
gstrauss commented 2 years ago
Is there a ticket somewhere on pagure.io describing this? Was it reported upstream?
lighttpd can do lots of things. Since lighttpd 1.4.60, you could implement dir-generator.php | ||
praiskup commented 2 years ago
No :-( I fell into an assumption the previous behavior was expected.
Ok, I think I can take a look at script how to append the footer? It | ||
|
||
|
||
Draft: lighttpd infra simplification
sketching updates based on recent issues
https://pagure.io/copr/copr/issue/2001
https://redmine.lighttpd.net/issues/3123
and older issues
https://pagure.io/copr/copr/issue/921
A number of items have comments for discussion and validation. @praiskup
One example: roles/copr/backend/files/lighttpd/mime.conf was created in 2014 and the significant change from default /etc/lighttpd/conf.d/mime.conf is the default Content-Type "" => "text/plain",
instead of "application/octet-stream"
. Is this still relevant for downloads? Are there a smaller set of extensions which should have "text/plain"
forced or a smaller portion of the site for which this should apply? It is generally preferable to use /etc/lighttpd/conf.d/mime.conf unless there is a specific reason not to do so.
@praiksup how much of the server load is due to directory listing?
How frequently are the directory listings requested?
How frequently are directories changing?
Is 15 seconds reasonable for caching a directory listing?
Is 60 seconds reasonable?
How large are these directories?
If only a few files, caching might only marginally improve performance.
If directories do not change frequently, or if they change once to compress logs,
then generating a static index.html when the directory contents are compressed,
and then allowing the index.html to be cached, is probably the most performant
solution.