Puppet: System Administration Automated

Support

Ticket #899 (new defect)

Opened 1 year ago

Last modified 6 months ago

CRL signature failure when using apache/mongrel

Reported by: bbeveridge Assigned to: community
Priority: normal Milestone:
Component: mongrel Version: 0.24.1
Severity: normal Keywords:
Cc: Triage Stage: Accepted
Attached Patches: None Complexity: Unknown

Description

Following the guide on: http://reductivelabs.com/trac/puppet/wiki/UsingMongrel

running: puppetd --test --server localhost gives me: err: Could not retrieve configuration: Certificates were not trusted: tlsv1 alert decrypt error

in the apache balancer logs i get: [warn] Invalid signature on CRL [error] Certificate Verification: Error (8): CRL signature failure

This is using the current versions: puppet: 0.23.2 apache: 2.2.6 mongrel: 1.1

config is per the urls details.

If i comment out the revocation file all works fine.

Change History

(follow-up: ↓ 2 ) 11/24/07 03:30:17 changed by luke

  • stage changed from Unreviewed to Accepted.

Can anyone else replicate this? Is anyone using a CRL with Apache?

(in reply to: ↑ 1 ) 11/29/07 14:38:27 changed by do

Replying to luke:

Can anyone else replicate this? Is anyone using a CRL with Apache?

Yes, I have exactly the same problem. Setting Apache2 to 'Log Level? DEBUG' reveals the following in /var/log/apache2/error.log

[Thu Nov 29 14:28:40 2007] [warn] Invalid signature on CRL [Thu Nov 29 14:28:40 2007] [error] Certificate Verification: Error (8): CRL signature failure [Thu Nov 29 14:28:40 2007] [debug] ssl_engine_kernel.c(1770): OpenSSL: Write: SSLv3 read client certificate B [Thu Nov 29 14:28:40 2007] [debug] ssl_engine_kernel.c(1789): OpenSSL: Exit: error in SSLv3 read client certificate B [Thu Nov 29 14:28:40 2007] [debug] ssl_engine_kernel.c(1789): OpenSSL: Exit: error in SSLv3 read client certificate B [Thu Nov 29 14:28:40 2007] [info] [client <IP-address>] SSL library error 1 in handshake (server puppetmaster:8140) [Thu Nov 29 14:28:40 2007] [info] SSL Library Error: 67567722 error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01 [Thu Nov 29 14:28:40 2007] [info] SSL Library Error: 67530866 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed [Thu Nov 29 14:28:40 2007] [info] SSL Library Error: 218910726 error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib [Thu Nov 29 14:28:40 2007] [info] SSL Library Error: 336105650 error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned [Thu Nov 29 14:28:40 2007] [info] [client <IP-address>] Connection closed to child 67 with abortive shutdown (server puppetmaster:8140)

The client which tries to connect does not have a vertificate yet. It should get one from the server. No signing request is generated, and also no signed certificate.

01/09/08 00:02:41 changed by dhaveconfig

So I just verified this myself as we were trying to deal with the new ruby ssl library issue on Ubuntu clients.

This was verified on both puppetd 0.23.2 and 0.24.1, both running against our test server, which is using the same Apache/Mongrel setup as above.

The problem only occurs with the newer ruby libraries and when the client does not have a certificate, mine are as shown: 

# dpkg -l |grep ruby
ii  libopenssl-ruby                           1.0.0+ruby1.8.2-1
ii  libopenssl-ruby1.8                        1.8.6.111-2ubuntu1
ii  libruby1.8                                1.8.6.111-2ubuntu1
ii  libshadow-ruby1.8                         1.4.1-8
ii  libxmlrpc-ruby                            1.8.2-1
ii  ruby                                      1.8.2-1 
ii  ruby1.8                                   1.8.6.111-2ubuntu1

and manifests itself on the client side as follows when running puppetd: 

warning: peer certificate won't be verified in this SSL session
/usr/lib/ruby/1.8/net/http.rb:586:in `connect': sslv3 alert handshake failure (OpenSSL::SSL::SSLError)
	from /usr/lib/ruby/1.8/net/http.rb:586:in `connect'
	from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
	from /usr/lib/ruby/1.8/net/http.rb:548:in `start'
	from /usr/lib/ruby/1.8/puppet/network/xmlrpc/client.rb:128:in `start'
	from /usr/lib/ruby/1.8/puppet/network/client.rb:99:in `initialize'
	from /usr/lib/ruby/1.8/puppet/network/client/master.rb:203:in `initialize'
	from /usr/sbin/puppetd:312:in `new'
	from /usr/sbin/puppetd:312

The relevant Apache errors in the logs are: 

[debug] ssl_engine_kernel.c(1770): OpenSSL: Write: SSLv3 read client certificate B
[debug] ssl_engine_kernel.c(1789): OpenSSL: Exit: error in SSLv3 read client certificate B
[debug] ssl_engine_kernel.c(1789): OpenSSL: Exit: error in SSLv3 read client certificate B
[info] [client 172.27.71.100] SSL library error 1 in handshake (server XXX.XXX.XXX.XXX:8140)
[info] SSL Library Error: 336105671 error:140890C7:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:peer did not return a certificate No CAs known to server for verification?
[info] [client 172.27.71.100] Connection closed to child 65 with abortive shutdown (server XXX.XXX.XXX.XXX:8140

So it appears that something has changed in that Ruby SSL library version that changes behavior when talking to Apache with a CRL when you don't have a certificate.

Just to make it obvious, this problem doesn't exist if you talk straight to the webrick server by forcing masterport to be the same as the ca_port, which is to be expected.

I haven't downgraded a client and tested whether this still occurs with 0.22.4, as that doesn't seem to be such a high priority right now.

(follow-up: ↓ 5 ) 03/17/08 02:28:39 changed by emerose

  • version changed from 0.23.2 to 0.24.1.

I'm running into this erro as well, though under slightly different circumstances: the puppetd client in my case is running on the puppetmaster server. Puppetd clients on other servers can connect to apache fine, but every time I run puppetd --test on the puppetmaster, I get the following error: 

/usr/lib/ruby/1.8/net/http.rb:586:in `connect': tlsv1 alert decrypt error (OpenSSL::SSL::SSLError)
	from /usr/lib/ruby/1.8/net/http.rb:586:in `connect'
	from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
	from /usr/lib/ruby/1.8/net/http.rb:548:in `start'
	from /usr/lib/ruby/1.8/puppet/network/xmlrpc/client.rb:128:in `start'
	from /usr/lib/ruby/1.8/puppet/network/client.rb:99:in `initialize'
	from /usr/lib/ruby/1.8/puppet/network/client/master.rb:203:in `initialize'
	from /usr/sbin/puppetd:312:in `new'
	from /usr/sbin/puppetd:312

The apache logs contain: 

[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1143): [client 172.16.0.2] handing out temporary 1024 bit DH key
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: SSLv3 write key exchange A
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: SSLv3 write certificate request A
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: SSLv3 flush data
[2008 Mar 16 10:03:29] [debug] ssl_engine_io.c(1775): OpenSSL: read 5/5 bytes from BIO#86b930 [mem: 87a0d0] (BIO dump follows)
[2008 Mar 16 10:03:29] [debug] ssl_engine_io.c(1722): +-------------------------------------------------------------------------+
<...snip...>
[2008 Mar 16 10:03:29] [debug] ssl_engine_io.c(1753): +-------------------------------------------------------------------------+
[2008 Mar 16 10:03:29] [debug] ssl_engine_io.c(1775): OpenSSL: read 652/652 bytes from BIO#86b930 [mem: 87a0d5] (BIO dump follows)
[2008 Mar 16 10:03:29] [debug] ssl_engine_io.c(1722): +-------------------------------------------------------------------------+
<...snip...>
[2008 Mar 16 10:03:29] [debug] ssl_engine_io.c(1753): +-------------------------------------------------------------------------+
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1190): Certificate Verification: depth: 1, subject: /CN=puppet.mycorp.com, issuer: /CN=puppet.mycorp.com
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1382): CA CRL: Issuer: CN=puppet.mycorp.com, lastUpdate: Feb 24 23:40:40 2008 GMT, nextUpdate: Feb 22 23:40:40 2013 GMT
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1190): Certificate Verification: depth: 0, subject: /CN=puppet.mycorp.com, issuer: /CN=puppet.mycorp.com
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1382): CA CRL: Issuer: CN=puppet.mycorp.com, lastUpdate: Feb 24 23:40:40 2008 GMT, nextUpdate: Feb 22 23:40:40 2013 GMT
[2008 Mar 16 10:03:29] [warn] Invalid signature on CRL
[2008 Mar 16 10:03:29] [error] Certificate Verification: Error (8): CRL signature failure
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1770): OpenSSL: Write: SSLv3 read client certificate B
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1789): OpenSSL: Exit: error in SSLv3 read client certificate B
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1789): OpenSSL: Exit: error in SSLv3 read client certificate B

This is all on ubuntu gutsy, running puppet 0.24.1 from the debian .deb; until now we've been using WEBrick with the same certificates, CA, and CRLs without any SSL problems. The apache config (take mostly from the Using Mongrel wiki page: 

Listen 8140

ProxyRequests Off

<Proxy balancer://puppetmaster>
	<% (minport .. maxport).each do |port| %>
    BalancerMember http://127.0.0.1:<%= port %>
    <% end %>
</Proxy>

<VirtualHost *:8140>
    SSLEngine on
    SSLProtocol -all +SSLv3 +TLSv1
    SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM
    SSLCertificateFile /var/lib/puppet/ssl/certs/<%= fqdn %>.pem
    SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/<%= fqdn %>.pem
    SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
    SSLVerifyClient optional
    SSLVerifyDepth  1
    SSLOptions +StdEnvVars

    RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

    <Location />
       SetHandler balancer-manager
       Order allow,deny
       Allow from all
    </Location>

    ProxyPass / balancer://puppetmaster/
    ProxyPassReverse / balancer://puppetmaster/
    ProxyPreserveHost on
</VirtualHost>

(in reply to: ↑ 4 ; follow-up: ↓ 7 ) 03/17/08 02:41:26 changed by emerose

Replying to emerose:

I guess I should also mention that the problem goes away when the CRL line is commented out of the apache config...

04/24/08 07:08:29 changed by luke

  • component changed from server to mongrel.

(in reply to: ↑ 5 ; follow-up: ↓ 8 ) 05/29/08 21:08:47 changed by greenmoss

Replying to emerose:

Replying to emerose: I guess I should also mention that the problem goes away when the CRL line is commented out of the apache config...

I had the exact same problem and solution. I'm using the apache proxy instead of webrick as well. Fixed by commenting out the "SSLCARevocationFile" line and reloading apache.

(in reply to: ↑ 7 ) 05/29/08 22:07:31 changed by greenmoss

Replying to greenmoss:

I had the exact same problem and solution. I'm using the apache proxy instead of webrick as well. Fixed by commenting out the "SSLCARevocationFile" line and reloading apache.

Strangely enough, after puppetd --test ran fine one time, I re-enabled SSLCARevocationFile and restarted Apache/Puppetmaster. The problem has not yet returned. *Is puzzled*