podman-2:1.4.5-0.78.dev.gite2f38cd.fc31.x86_64 is broken as well.

I wanted to bisect packages in rawhide but there were bunch of failed builds between release -29 and -59 and podman-2:1.4.5-0.59.dev.git680a383.fc31.x86_64  does not work either.

we think this is the registry, not podman or moby.

Then could you explain why does it work with podman on f30 ?

we see it randomly working in ci tests, is it possible this is a fluke on F30?

I've just tried and downgrading podman to 2:1.4.5-0.29.dev.gitd6b41eb.fc31.x86_6
(or moby engine moby-engine-18.09.8-2.ce.git0dd43dd.fc31 -> moby-engine-18.09.8-1.ce.git0dd43dd.fc31)
fixed the probelm.


So maybe it is related to mass rebuild in rawhide and some changes in buildroot.

again, we do see this in our ci tests with PRs though.

(In reply to Brent Baude from comment #5)
> we see it randomly working in ci tests, is it possible this is a fluke on
> F30?

No.
It works like a magic on f30 or after downgrade (comment 6)

Could be the latest version of golang,causing the issue.

(In reply to Lukas Slebodnik from comment #8)
> (In reply to Brent Baude from comment #5)
> > we see it randomly working in ci tests, is it possible this is a fluke on
> > F30?
> 
> No.
> It works like a magic on f30 or after downgrade (comment 6)

BTW  here is my "test"

podman-1.4.4-4.fc30.x86_64
"""
set -e

for i in {1..100}; do
    echo "------$i-----"
    docker pull registry.fedoraproject.org/fedora:30
    docker pull registry.fedoraproject.org/fedora:29
    docker pull registry.fedoraproject.org/fedora:rawhide

    docker rmi registry.fedoraproject.org/fedora:30 registry.fedoraproject.org/fedora:29 registry.fedoraproject.org/fedora:rawhide
done
"""

And it passed all times (300 pulls from fedora registry)

and negative one on rawhide with podman-1.4.5-0.74.dev.git140e08e.fc31.x86_64
"""
set -e

for i in {1..100}; do
    echo "------$i-----"
    podman pull registry.fedoraproject.org/fedora:30 && exit 1 || true
    podman pull registry.fedoraproject.org/fedora:29 && exit 1 || true
    podman pull registry.fedoraproject.org/fedora:rawhide && exit 1 || true

    podman rmi registry.fedoraproject.org/fedora:30 registry.fedoraproject.org/fedora:29 registry.fedoraproject.org/fedora:rawhide || true 
done
"""
and it failed to pull image 300 times.

Maybe it is something different in your CI but it behaves deterministicaly in my env.

(In reply to Lukas Slebodnik from comment #6)
> I've just tried and downgrading podman to
> 2:1.4.5-0.29.dev.gitd6b41eb.fc31.x86_6 … fixed the probelm.

golang 1.12.7-1.fc31 


(failing)
> podman-1.4.5-0.74.dev.git140e08e.fc31.x86_64
golang 1.13-0.beta1.2.fc31.2



(See also historical http://github.com/containers/buildah/issues/1504 , which was, it seems, never tracked down to the real cause.)

This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

We had a couple of issues with the Fedora registry lately that surfaced also in podman's CI.

Podman v1.5.0 includes fixes for those issues. @Lukas, could you try out Podman v1.5.0 in your environment and check if you still see the pull issues?

(In reply to Valentin Rothberg from comment #14)
> We had a couple of issues with the Fedora registry lately that surfaced also
> in podman's CI.
> 
> Podman v1.5.0 includes fixes for those issues. @Lukas, could you try out
> Podman v1.5.0 in your environment and check if you still see the pull issues?

I tried the latest version from rawhide repositories podman-1.5.1-0.6.dev.git2348c28.fc31.x86_64
But it did not help

fwiw --tls-verify=false makes no difference:

   # podman  pull --tls-verify=false  registry.fedoraproject.org/fedora:30
   Trying to pull registry.fedoraproject.org/fedora:30...
     Get http://registry.fedoraproject.org/v2/: local error: tls: unexpected message
   Error: error pulling image "registry.fedoraproject.org/fedora:30": unable to pull registry.fedoraproject.org/fedora:30: unable to pull image: Error initializing source docker://registry.fedoraproject.org/fedora:30: pinging docker registry returned: Get http://registry.fedoraproject.org/v2/: local error: tls: unexpected message

(In reply to Ed Santiago from comment #17)
> fwiw --tls-verify=false makes no difference:
> 
>    # podman  pull --tls-verify=false  registry.fedoraproject.org/fedora:30
>    Trying to pull registry.fedoraproject.org/fedora:30...
>      Get http://registry.fedoraproject.org/v2/: local error: tls:
> unexpected message
>    Error: error pulling image "registry.fedoraproject.org/fedora:30": unable
> to pull registry.fedoraproject.org/fedora:30: unable to pull image: Error
> initializing source docker://registry.fedoraproject.org/fedora:30: pinging
> docker registry returned: Get http://registry.fedoraproject.org/v2/: local
> error: tls: unexpected message

There are more workarounds :-)
But workaround is not the solution of the problem and as I already mentioned few times it works for me on f30 with tls
so there is not any reason why it should not work work on f31/rawhide

[root@host ~]# podman  pull --tls-verify=true  registry.fedoraproject.org/fedora:30
Trying to pull registry.fedoraproject.org/fedora:30...Getting image source signatures
Copying blob e214a48be34a done
Copying config 1cabdcebde done
Writing manifest to image destination
Storing signatures
1cabdcebde84b73e0f536fabf7ea90f304de806319f3fb848ccfa8e7c5ce2ecc

[root@host ~]# rpm -q podman
podman-1.4.4-4.fc30.x86_64

And Valentin is already trying to debug issue in my env.

Back on the issue. To me it looks like a bug in go1.13beta1 which is currently shipped in Rawhide. It works fine (on the same machine) with a skopeo built with go1.12.7.

The debug logs point to `Err:(*net.OpError)(0xc000425bd0)` (i.e., an error in the net package of go's stdlib).

Can we assign the bug to the go maintainers? I consider it a regression in go.

Reassigning to golang to get their input.

Proposed as a Blocker for 31-beta by Fedora user dustymabe using the blocker tracking app because:

 Since a lot of tooling now uses containers we want to be able to pull from our registry using our tools (i.e. podman). If we can't it would be a pretty bad review of the distribution. 

I don't see any thing in the existing criteria that mentions containers. That should probably change.

Well, please go ahead and propose some criteria, then. It's better to do this as soon as possible and let people discuss and reflect on it, than for this to come up in a blocker review meeting without any further prep work and force us to try and invent criteria on the spot or something. Thanks!

(In reply to Valentin Rothberg from comment #19)
> Back on the issue. To me it looks like a bug in go1.13beta1 which is
> currently shipped in Rawhide. It works fine (on the same machine) with a
> skopeo built with go1.12.7.
> 

Could you please re-test with recent version of go 1.12.9 in f30(and rc1 in rawhide), that contains the recent http CVE mitigations?

> The debug logs point to `Err:(*net.OpError)(0xc000425bd0)` (i.e., an error
> in the net package of go's stdlib).
> 
> Can we assign the bug to the go maintainers? I consider it a regression in
> go.

ok it seems like disabling TLS 1.3 seems to workaround this issue:

```
GODEBUG=tls13=0 podman run --rm -it registry.fedoraproject.org/fedora:30
```

Which was recently enabled in golang 1.13 (now in Fedora 31).

Also worth noting that I can pull from quay without the workaround:

```
podman run --rm -it quay.io/fedora/fedora:32-x86_64
```

So it looks like the fedora registry might not support TLS 1.3 ?

Opened http://pagure.io/fedora-infrastructure/issue/8154 to ask for update of the registry to rule issues on this side as it is only registry that triggers this. 

For the record this will also trigger with podman build by older go(1.12) if tls1.3 is explicitly enabled via GODEBUG=tls13=1.

(In reply to Adam Williamson from comment #22)
> Well, please go ahead and propose some criteria, then. It's better to do
> this as soon as possible and let people discuss and reflect on it, than for
> this to come up in a blocker review meeting without any further prep work
> and force us to try and invent criteria on the spot or something. Thanks!

Currently this breaks Toolbox on Silverblue 31, which makes it quite inconvenient to hack on it.

(In reply to Debarshi Ray from comment #27)
> (In reply to Adam Williamson from comment #22)
> > Well, please go ahead and propose some criteria, then. It's better to do
> > this as soon as possible and let people discuss and reflect on it, than for
> > this to come up in a blocker review meeting without any further prep work
> > and force us to try and invent criteria on the spot or something. Thanks!
> 
> Currently this breaks Toolbox on Silverblue 31, which makes it quite
> inconvenient to hack on it.

Check comment#24

(In reply to Dusty Mabe from comment #25) 
> So it looks like the fedora registry might not support TLS 1.3 ?

No, that's easy to disprove, just connect with something other than Go:


$ gnutls-cli registry.fedoraproject.org
Processed 149 CA certificate(s).
Resolving 'registry.fedoraproject.org:443'...
Connecting to '152.19.134.142:443'...
- Successfully sent 0 certificate(s) to server.
- Server has requested a certificate.
- Certificate type: X.509
- Got a certificate list of 2 certificates.
- Certificate[0] info:
 - subject `CN=*.fedoraproject.org,O=Red Hat Inc.,L=Raleigh,ST=North Carolina,C=US', issuer `CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US', serial 0x06001d7080298619406b9702bc155525, RSA key 4096 bits, signed using RSA-SHA256, activated `2017-02-01 00:00:00 UTC', expires `2020-05-01 12:00:00 UTC', pin-sha256="Je3KZzxF/iKAGzb7q7z6QyFf2+LvCoUoBqPx0+8FoiI="
	Public Key ID:
		sha1:de906481f661bb04a3df52240d2a219a806ec92a
		sha256:25edca673c45fe22801b36fbabbcfa43215fdbe2ef0a852806a3f1d3ef05a222
	Public Key PIN:
		pin-sha256:Je3KZzxF/iKAGzb7q7z6QyFf2+LvCoUoBqPx0+8FoiI=

- Certificate[1] info:
 - subject `CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US', issuer `CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US', serial 0x04e1e7a4dc5cf2f36dc02b42b85d159f, RSA key 2048 bits, signed using RSA-SHA256, activated `2013-10-22 12:00:00 UTC', expires `2028-10-22 12:00:00 UTC', pin-sha256="k2v657xBsOVe1PQRwOsHsw3bsGT2VzIqz5K+59sNQws="
- Status: The certificate is trusted. 
- Description: (TLS1.3)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
- Options: OCSP status request,
- Handshake was completed


TLS 1.3 working fine. Anyway the server doesn't need to support TLS 1.3. The connection would just use TLS 1.2 if it didn't.

Could somebody from cryto team help to check TLS 1.3 in the registry and/or in golang?

The TLS 1.3 support in the registry.fedoraproject.org seem to work just fine with openssl and gnutls as Michael writes above. The issue identified here may be an issue in the golang implementation which is not something we cover. Someone from the golang side would have to comment on its TLS1.3 implementation.

My recommendation would be if golang has an in progress TLS1.3 implementation, it will make sense to not enable it by default in fedora.

after duplicating the ClientHello from the podman (as Go TLS implementation does not support SSLKEYLOGFILE environment variable...) the most likely issue is that the go tls implementation is not able to handle a CertificateRequest message from the server in TLS 1.3.

To answer with certainty I'd need a tcpdump from the podman connection with a corresponding key log file (AFAIK, it requires setting tls.Config KeyLogWriter property, see http://github.com/golang/go/issues/13057, and recompiling podman).

An easy way to reproduce http://paste.fedoraproject.org/paste/-AQrAfoCHJCpnLd0qIjIEw

running the above code with TLS 1.3 returns a nil response with the following error Get http://registry.fedoraproject.org/v2/fedora/manifests/latest: local error: tls: unexpected message

(In reply to Hubert Kario from comment #32)
> after duplicating the ClientHello from the podman (as Go TLS implementation
> does not support SSLKEYLOGFILE environment variable...) the most likely
> issue is that the go tls implementation is not able to handle a
> CertificateRequest message from the server in TLS 1.3.

That would make sense since we are using certificates to authenticate pushes to registry.fp.o.

> 
> To answer with certainty I'd need a tcpdump from the podman connection with
> a corresponding key log file (AFAIK, it requires setting tls.Config
> KeyLogWriter property, see http://github.com/golang/go/issues/13057, and
> recompiling podman).

Reported to upstream as http://github.com/golang/go/issues/34040 with all we know so far.

I have just built golang-1.13-0.rc2.1.fc31 and golang-1.13-0.rc2.1.fc32 with revert to the 1.12 behavior regarding tls1.3, i.e. it needs to be explicitly enabled at runtime via GODEBUG=tls13=1.

(In reply to Jakub Čajka from comment #36)
> I have just built golang-1.13-0.rc2.1.fc31 and golang-1.13-0.rc2.1.fc32 with
> revert to the 1.12 behavior regarding tls1.3, i.e. it needs to be explicitly
> enabled at runtime via GODEBUG=tls13=1.

Should we propose the fc31 as bodhi update and attach it to this bug? I assume podman will also need to be rebuilt? If so we'll need to submit a buildroot override?

Discussed during the 2019-09-03 blocker review meeting: [1]

The decision to classify this bug as a "RejectedBlocker" and an "AcceptedFreezeException" was made because, while the new criterion that could cover this is being discussed, we don't think this absolutely needs to block Beta. We will re-consider it for a Beta or Final blocker if it's still a problem when the criterion is decided. Accepted as a freeze exception issue, however, as it is a significant problem we should try to avoid.

[1] http://meetbot.fedoraproject.org/fedora-blocker-review/2019-09-03/f31-blocker-review.2019-09-03-16.01.txt

Discussed during the 2019-09-03 blocker review meeting: [1]

Thank you for reporting this upstream to us. I investigated and found the server to be violating the TLS 1.3 specification. Let me know if you need any more details to track this down server side and/or to report it to that upstream.

http://github.com/golang/go/issues/34040#issuecomment-527986411

I would ask that you please don't disable TLS 1.3 for all Go users just to accommodate one specific broken server. Rather, please disable it only in the client that needs to connect to that server, or even better fix the server. TLS 1.3 is a major security improvement, and having a distribution disable it by default will cause confusion for users, like any divergence from upstream.

I have opened tracking infra issue(when we get fix) http://pagure.io/fedora-infrastructure/issue/8173. 
As this seems to be affecting httpd and based on the comments in the upstream ticket http://github.com/golang/go/issues/34040 reassigning to openssl. I will work on standalone httpd based reproducer as my time allows later this week.

(In reply to Filippo Valsorda from comment #39)
> Thank you for reporting this upstream to us. I investigated and found the
> server to be violating the TLS 1.3 specification. Let me know if you need
> any more details to track this down server side and/or to report it to that
> upstream.
> 
> http://github.com/golang/go/issues/34040#issuecomment-527986411
> 
> I would ask that you please don't disable TLS 1.3 for all Go users just to
> accommodate one specific broken server. Rather, please disable it only in
> the client that needs to connect to that server, or even better fix the
> server. TLS 1.3 is a major security improvement, and having a distribution
> disable it by default will cause confusion for users, like any divergence
> from upstream.

I agree on this with you, as this is not a stdlib issue I will revert the patches. FYI the patched build haven't been yet distributed(as we are in the release freeze period). Thanks for all help.

I see 1.13-1.fc31 and 1.13-1.fc32 were built today, but the TLS 1.3 disabling patch was not removed...

(In reply to Adam Williamson from comment #42)
> I see 1.13-1.fc31 and 1.13-1.fc32 were built today, but the TLS 1.3
> disabling patch was not removed...

My "today" time prior all the development of US pacific TZ "today"... Will fix it my "today".

Please note that fixing the openssl issue will not fix the problem with podman. Only when the fixed openssl is deployed on registry.fedoraproject.org, it will fix the podman problem.

There is no upstream fix yet, so perhaps there should be some workaround done either in podman or in registry.fedoraproject.org.

(In reply to Tomas Mraz from comment #44)
> Please note that fixing the openssl issue will not fix the problem with
> podman. Only when the fixed openssl is deployed on
> registry.fedoraproject.org, it will fix the podman problem.
> 
> There is no upstream fix yet, so perhaps there should be some workaround
> done either in podman or in registry.fedoraproject.org.

Yes that is the reason why I have opened the infra tracking issue #8173.

FYI upstream openssl issue http://github.com/openssl/openssl/issues/9767

FEDORA-2019-b364562f30 has been submitted as an update to Fedora 30. http://bodhi.fedoraproject.org/updates/FEDORA-2019-b364562f30

FEDORA-2019-f4a2ced40c has been submitted as an update to Fedora 31. http://bodhi.fedoraproject.org/updates/FEDORA-2019-f4a2ced40c

openssl-1.1.1c-6.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See http://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: http://bodhi.fedoraproject.org/updates/FEDORA-2019-f4a2ced40c

openssl-1.1.1c-6.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See http://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: http://bodhi.fedoraproject.org/updates/FEDORA-2019-b364562f30

openssl-1.1.1c-6.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See http://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: http://bodhi.fedoraproject.org/updates/FEDORA-2019-f5810ab475

openssl-1.1.1c-6.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Hum. So, I checked this issue against our staging server(s):

podman pull registry.stg.fedoraproject.org/fedora:30       
Trying to pull registry.stg.fedoraproject.org/fedora:30...
  Get http://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message
Error: error pulling image "registry.stg.fedoraproject.org/fedora:30": unable to pull registry.stg.fedoraproject.org/fedora:30: unable to pull image: Error initializing source docker://registry.stg.fedoraproject.org/fedora:30: pinging docker registry returned: Get http://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message

then I upgraded openssl on the proxy server:
[root@proxy01 ~][STG]# rpm -q openssl
openssl-1.1.1c-6.fc29.x86_64

Rebooted the server. 

But the problem persists:
% podman pull registry.stg.fedoraproject.org/fedora:30       
Trying to pull registry.stg.fedoraproject.org/fedora:30...
  Get http://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message
Error: error pulling image "registry.stg.fedoraproject.org/fedora:30": unable to pull registry.stg.fedoraproject.org/fedora:30: unable to pull image: Error initializing source docker://registry.stg.fedoraproject.org/fedora:30: pinging docker registry returned: Get http://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message

What am I missing here? httpd doesn't need to be rebuilt does it?

openssl-1.1.1c-6.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.

FEDORA-2019-d51641f152 has been submitted as an update to Fedora 29. http://bodhi.fedoraproject.org/updates/FEDORA-2019-d51641f152

openssl-1.1.1d-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See http://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: http://bodhi.fedoraproject.org/updates/FEDORA-2019-d51641f152

Re-opening for now, given Kevin's question in #c52.

http://pagure.io/fedora-infrastructure/issue/8173

I am sorry, the patch was not properly applied in the openssl-1.1.1c-6 build. Please try again with the openssl-1.1.1d-1.fc29.

I can confirm that version works. Thanks!

openssl-1.1.1d-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.