Learn more about these different git repos.
Other Git URLs
Hello All, I am still experiencing issues with s390x builder for Fedora 31+. I can reproduce it 100% - just try rebuilding Erlang and it got stuck eventually.
The latest affected build:
So far the only solution I can think of is to ExcludeArch s390x for a while and will try it again in a few weeks.
See also this thread:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/KEVZTKSHMJJR6EVTRS4ZNC27SPX4Q4DU/
Few years back it was a generic issue in kernel in futex syscall, not specific to s390x. The symptoms were same/very similar. If things work fine in F-30, then one of the suspects might be glibc and some threading changes. On the other hand, a month or so I wasn't able to reproduce that in a local build, but will retry again.
Seems like git-annex is bit by the same issue, and builds fine on F30, but is still doing something on F31 and Rawhide.
Interesting, I've tried F32 very recently and the build was successful:
No luck for F31 yet.
So those above git-annex builds finished. F31 took 2 days 41 minutes and Rawhide took a whopping 8.05 days.
Worth to mention that the git-annex build task on s390x has been restarted 5x, probably after timing out due the 48h limit in mock/koji builder.
One thing I noticed is that the stuck builds are on the KVM based builders only and never on the z/VM ones ... And I wasn't able to reproduce the hang on my z/VM guest so far.
I have noted "stuck" golang build https://koji.fedoraproject.org/koji/taskinfo?taskID=37966413 on s390x. Build usually takes around ~15min, but this has been still pending after 24h so I have canceled the task and resubmitted the build, which succeeded.
Point of the build where it got "stuck" is running tests, log snipped follows.
##### Testing packages. # go tool dist test -run=^go_test:archive/tar$ run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE ok archive/tar 0.034s ok archive/zip 0.083s ok bufio 0.069s ok bytes 0.232s ok compress/bzip2 0.061s ok compress/flate 0.447s ok compress/gzip 0.038s ok compress/lzw 0.010s ok compress/zlib 0.045s ok container/heap 0.016s ok container/list 0.010s ok container/ring 0.022s ok context 0.983s ok crypto 0.002s ok crypto/aes 0.043s ok crypto/cipher 0.026s ok crypto/des 0.007s ok crypto/dsa 0.004s ok crypto/ecdsa 0.691s ok crypto/ed25519 0.121s ok crypto/elliptic 0.034s ok crypto/hmac 0.001s ok crypto/internal/subtle 0.010s ok crypto/md5 0.016s ok crypto/rand 0.036s ok crypto/rc4 0.053s ok crypto/rsa 0.122s ok crypto/sha1 0.003s ok crypto/sha256 0.019s ok crypto/sha512 0.005s ok crypto/subtle 0.007s ok crypto/tls 1.162s ok crypto/x509 0.736s ok database/sql 0.549s ok database/sql/driver 0.013s ok debug/dwarf 0.025s ok debug/elf 0.034s ok debug/gosym 0.034s ok debug/macho 0.014s ok debug/pe 0.004s ok debug/plan9obj 0.016s ok encoding/ascii85 0.025s ok encoding/asn1 0.025s ok encoding/base32 0.062s ok encoding/base64 0.005s ok encoding/binary 0.029s ok encoding/csv 0.036s ok encoding/gob 0.032s ok encoding/hex 0.006s ok encoding/json 0.131s ok encoding/pem 0.017s ok encoding/xml 0.022s ok errors 0.039s ok expvar 0.026s ok flag 0.027s ok fmt 0.086s ok go/ast 0.011s ok go/build 0.176s ok go/constant 0.005s ok go/doc 0.042s ok go/format 0.016s ok go/importer 0.141s ok go/internal/gccgoimporter 0.026s ok go/internal/gcimporter 0.519s ok go/internal/srcimporter 0.897s ok go/parser 0.028s ok go/printer 0.289s ok go/scanner 0.023s ok go/token 0.037s ok go/types 0.828s ok hash 0.037s ok hash/adler32 0.013s ok hash/crc32 0.012s ok hash/crc64 0.022s ok hash/fnv 0.005s ok html 0.002s ok html/template 0.054s ok image 0.109s ok image/color 0.052s ok image/draw 0.094s ok image/gif 0.305s ok image/jpeg 0.343s ok image/png 0.085s ok index/suffixarray 0.381s ok internal/cpu 0.015s ok internal/fmtsort 0.018s ok internal/poll 0.071s ok internal/reflectlite 0.005s ok internal/singleflight 0.024s ok internal/trace 0.059s ok internal/xcoff 0.015s ok io 0.057s ok io/ioutil 0.007s ok log 0.006s ok log/syslog 1.228s ok math 0.008s ok math/big 0.486s ok math/bits 0.020s ok math/cmplx 0.009s ok math/rand 0.145s ok mime 1.060s ok mime/multipart 0.359s ok mime/quotedprintable 0.014s ok net 3.869s ok net/http 3.874s ok net/http/cgi 0.619s ok net/http/cookiejar 0.009s ok net/http/fcgi 0.019s ok net/http/httptest 0.032s ok net/http/httptrace 0.005s ok net/http/httputil 0.058s ok net/http/internal 0.023s ok net/http/pprof 2.021s ok net/internal/socktest 0.002s ok net/mail 0.018s ok net/rpc 0.039s ok net/rpc/jsonrpc 0.017s ok net/smtp 0.028s ok net/textproto 0.006s ok net/url 0.026s ok os 0.713s ok os/exec 0.609s ok os/signal 4.867s ok os/user 0.004s ok path 0.011s ok path/filepath 0.035s ok plugin 0.026s ok reflect 0.162s ok regexp 0.068s ok regexp/syntax 0.499s ok runtime 27.936s ok runtime/debug 0.061s ok runtime/internal/atomic 0.057s ok runtime/internal/math 0.020s ok runtime/internal/sys 0.047s ok runtime/pprof 7.845s ok runtime/pprof/internal/profile 0.004s ok runtime/trace 0.706s ok sort 0.140s ok strconv 0.610s ok strings 0.102s ok sync 0.330s ok sync/atomic 0.048s ok syscall 0.059s ok testing 0.079s ok testing/quick 0.090s ok text/scanner 0.011s ok text/tabwriter 0.010s ok text/template 0.096s ok text/template/parse 0.030s ok time 2.640s ok unicode 0.035s ok unicode/utf16 0.010s ok unicode/utf8 0.039s ok cmd/addr2line 1.727s ok cmd/api 0.009s ok cmd/asm/internal/asm 0.686s ok cmd/asm/internal/lex 0.038s ok cmd/compile 0.021s ok cmd/compile/internal/gc 12.247s ok cmd/compile/internal/ssa 0.742s ok cmd/compile/internal/syntax 0.012s ok cmd/compile/internal/test 0.004s [no tests to run] ok cmd/compile/internal/types 0.012s ok cmd/cover 5.242s ok cmd/doc 0.099s ok cmd/fix 0.044s ok cmd/go 82.524s ok cmd/go/internal/auth 0.030s ok cmd/go/internal/cache 0.351s ok cmd/go/internal/dirhash 0.034s ok cmd/go/internal/generate 0.057s ok cmd/go/internal/get 0.031s ok cmd/go/internal/imports 0.007s ok cmd/go/internal/load 0.025s ok cmd/go/internal/lockedfile 0.021s ok cmd/go/internal/lockedfile/internal/filelock 0.047s ok cmd/go/internal/modconv 0.018s ok cmd/go/internal/modfetch 0.048s ok cmd/go/internal/modfetch/codehost 0.018s ok cmd/go/internal/modfile 0.006s ok cmd/go/internal/modload 0.028s ok cmd/go/internal/module 0.006s ok cmd/go/internal/mvs 0.030s ok cmd/go/internal/note 0.021s ok cmd/go/internal/par 0.083s ok cmd/go/internal/renameio 1.685s ok cmd/go/internal/search 0.008s ok cmd/go/internal/semver 0.002s ok cmd/go/internal/sumweb 0.022s ok cmd/go/internal/tlog 0.619s ok cmd/go/internal/txtar 0.001s ok cmd/go/internal/web 0.007s ok cmd/go/internal/work 0.024s ok cmd/gofmt 0.038s ok cmd/internal/buildid 0.197s ok cmd/internal/dwarf 0.008s ok cmd/internal/edit 0.009s ok cmd/internal/goobj 1.179s ok cmd/internal/obj 0.010s ok cmd/internal/obj/arm64 0.042s ok cmd/internal/obj/x86 2.300s ok cmd/internal/objabi 0.046s ok cmd/internal/src 0.002s ok cmd/internal/test2json 0.140s ok cmd/link 10.497s ok cmd/link/internal/ld 3.411s ok cmd/link/internal/sym 0.002s ok cmd/nm 3.857s ok cmd/objdump 1.308s ok cmd/pack 1.765s ok cmd/trace 0.015s ok cmd/vet 6.368s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE ##### os/user with tag osusergo # go tool dist test -run=^osusergo$ go test -short -count=1 -run= -timeout=10m0s -tags=osusergo os/user ok os/user 0.002s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -timeout=10m0s runtime -cpu=1,2,4 -quick ##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick # go tool dist test -run=^runtime:cpu124$ ok runtime 5.043s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std ##### cmd/go terminal test # go tool dist test -run=^cmd_go_test_terminal$ skipping terminal test; stdout/stderr not terminals run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE ##### Testing without libgcc. # go tool dist test -run=^nolibgcc:crypto/x509$ go test -short -count=1 -tags= -ldflags=-linkmode=internal -libgcc=none crypto/x509 -run= ok crypto/x509 0.395s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std # go tool dist test -run=^nolibgcc:net$ ok net 0.015s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -ldflags=-linkmode=internal -libgcc=none net -run=TestTCPStress run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std # go tool dist test -run=^nolibgcc:os/user$ ok os/user 0.021s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -ldflags=-linkmode=internal -libgcc=none os/user -run= run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std ##### sync -cpu=10 # go tool dist test -run=^sync_cpu$ ok sync 0.385s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= sync -timeout=4m0s -cpu=10 -run= run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std ##### ../misc/cgo/stdio # go tool dist test -run=^cgo_stdio$ run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -c -o test.test . ./test.test -test.short PASS run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std ##### ../misc/cgo/life # go tool dist test -run=^cgo_life$ run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -c -o test.test . ./test.test -test.short PASS gcc -xc -o /dev/null -static - run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE ##### ../misc/cgo/test # go tool dist test -run=^cgo_test$ go test -short -count=1 -tags= -run= PASS ok misc/cgo/test 5.055s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -run= -tags=internal run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std PASS ok misc/cgo/test 4.974s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= PASS ok misc/cgo/test 3.448s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -ldflags -linkmode=auto run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std PASS ok misc/cgo/testtls 0.005s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -ldflags -linkmode=external run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std PASS ok misc/cgo/testtls 0.002s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -ldflags -linkmode=external -extldflags "-static -pthread" run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std PASS ok misc/cgo/testtls 0.002s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std PASS ok misc/cgo/nocgo 0.005s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -ldflags -linkmode=external run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std PASS ok misc/cgo/nocgo 0.015s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -ldflags -linkmode=external -extldflags "-static -pthread" run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std PASS ok misc/cgo/nocgo 0.011s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -run= -tags=static -ldflags -linkmode=external -extldflags "-static -pthread" PASS ok misc/cgo/test 4.341s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -run= -tags=static PASS ok misc/cgo/test 4.519s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std PASS ok misc/cgo/test 3.429s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -buildmode=pie run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std PASS ok misc/cgo/testtls 0.003s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -buildmode=pie run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -buildmode=pie run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std PASS ok misc/cgo/nocgo 0.003s ##### ../misc/cgo/testgodefs # go tool dist test -run=^testgodefs$ run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE ./test.bash run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std ##### ../misc/cgo/testso # go tool dist test -run=^testso$ run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -timeout=20m0s . ok misc/cgo/testso 2.584s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std ##### ../misc/cgo/testsovar # go tool dist test -run=^testsovar$ run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -timeout=20m0s . ok misc/cgo/testsovar 1.842s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std ##### ../misc/cgo/testcarchive # go tool dist test -run=^testcarchive$ run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -c -o test.test . ./test.test -test.short SKIP - short mode and $GO_BUILDER_NAME not set run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std ##### ../misc/cgo/testcshared # go tool dist test -run=^testcshared$ run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -c -o test.test . ./test.test -test.short SKIP - short mode and $GO_BUILDER_NAME not set run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE ##### ../misc/cgo/testshared # go tool dist test -run=^testshared$ go test -short -count=1 -tags= -run= -timeout=20m0s . ok misc/cgo/testshared 43.928s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std ##### ../misc/cgo/testplugin # go tool dist test -run=^testplugin$ ok misc/cgo/testplugin 0.002s run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -timeout=20m0s . run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std ##### ../misc/cgo/errors # go tool dist test -run=^cgo_errors$ run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -c -o test.test . ./test.test -test.short PASS go build -o runtest.exe run.go run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE ##### ../misc/cgo/testsigfwd # go tool dist test -run=^testsigfwd$ go run main.go run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE go test -short -count=1 -tags= -run= -c -o=/dev/null ##### ../test/bench/go1 # go tool dist test -run=^bench_go1$ run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}} STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE /builddir/build/BUILD/go/test/runtest.exe --shard=0 --shards=1 ##### ../test # go tool dist test -run=^test:0_1$
This is happening 100% of the time now on erlang builds for rawhide.
I've created a bz to track down the issue here:
https://bugzilla.redhat.com/show_bug.cgi?id=1770256
So, from the bug the issue was tracked down and worked around... great job!
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.