X509v3Certificate causes handshake failure

[zhengyuzhang]

Hi, I am using the below code to generate a certificate, store it in the keystore and then initialize an SSLServerSocketFactory. However the client always give a handshake failure.

The exceptions and error handling code it omitted to keep it short.

static {
    Security.insertProviderAt(new org.spongycastle.jce.provider.BouncyCastleProvider(), 1);
}
private static X509Certificate generateCertificate(KeyPair keyPair) {

    long current = System.currentTimeMillis();
    long endOffset = 365 * 24 * 60 * 60 * 1000L;
    long endTimeStamp = current + endOffset;

    Date startDate = new Date(current - 365 * 24 * 60 * 60 * 1000L);
    Date endDate = new Date(endTimeStamp);

    X500NameBuilder issuerNameBuilder = new X500NameBuilder(BCStyle.INSTANCE);
    issuerNameBuilder.addRDN(BCStyle.C, "us");
    issuerNameBuilder.addRDN(BCStyle.ST, "az");
    issuerNameBuilder.addRDN(BCStyle.L, "gi");
    issuerNameBuilder.addRDN(BCStyle.O, "sm");
    issuerNameBuilder.addRDN(BCStyle.OU, "sm");
    issuerNameBuilder.addRDN(BCStyle.CN, "zz");
    X500Name issuerX500Name = issuerNameBuilder.build();

    X500Name subjectX500Name = issuerX500Name;

    SecureRandom random = new SecureRandom();

    SubjectPublicKeyInfo subjectPublicKeyInfo =
        SubjectPublicKeyInfo.getInstance(keyPair.getPublic.getEncoded());
    X509v3CertificateBuilder v3CertGen = new X509v3CertificateBuilder(
        issuerX500Name,
        new BigInteger(32, random),
        startDate,
        endDate,
        subjectX500Name,
        subjectPublicKeyInfo);

    ContentSigner signer = new JcaContentSignerBuilder(signAlgo).setProvider("SC").build(keyPair.getPrivate())
    return new JcaX509CertificateConverter().setProvider("SC").
        getCertificate(v3CertGen.build(signer));
}

static KeyStore initializeKeyStore() {
    KeyPairGenerator kpg = KeyPairGenerator.getInstance("RSA");
    kpg.initialize(2048);
    KeyPair keyPair = kpg.genKeyPair();

    X509Certificate cert = generateCertificate(keyPair);
    KeyStore keyStore = KeyStore.getInstance(KEY_STORE_TYPE);
    keyStore.load(null, KEY_STORE_PASSWORD.toCharArray());
    keyStore.setKeyEntry(
                MY_CERT_ALIAS,
                keyPair.getPrivate(),
                MY_CERT_PASSWORD.toCharArray(),
                new Certificate[]{cert});
    return keyStore;
}

// Create SSLServerSocketFactory
static SSLServerSocketFactory createServerSSLSocketFactory(Context context) {
    KeyStore keyStore = initializeKeyStore();

    TrustManagerFactory tmf =
        TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
    tmf.init(keyStore);
    TrustManager[] trustManagers = tmf.getTrustManagers();
    if (trustManagers.length != 1 || !(trustManagers[0] instanceof X509TrustManager)) {
        throw new IllegalStateException("Unexpected default trust managers:" +
            Arrays.toString(trustManagers));
    }

    // Server certificate setup
    KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
    kmf.init(keyStore, KEY_STORE_PASSWORD.toCharArray());

    SSLContext sslContext = SSLContext.getInstance("TLSv1.2");
    sslContext.init(kmf.getKeyManagers(), tmf.getTrustManagers(), null);

    SSLServerSocketFactory factory = sslContext.getServerSocketFactory();

    return factory;
}

Then use the below command, I get a handshake failure. I have tried on Google Nexus 5 devices with Android 5.1.1 and 6.0.1. The errors are the same. From the network trace, the client sends a client hello and later the server returns Handshake Failure (40) and shuts down the connection.

openssl s_client -connect 192.168.0.8:10099

Output:

CONNECTED(00000003)
140091672004248:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 305 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1527744291
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

If I use the following code to generate a keystore and push to the Android device, it works with no problem.

keytool -genkey -keyalg RSA -alias selfsigned -keystore keystore.bks -storepass password -validity 360 -keysize 2048 -validity 9999 -storetype BKS -provider org.bouncycastle.jce.provider.BouncyCastleProvider -providerpath bcprov-jdk15on-159.jar

Did anyone run into the same issue on Android, or have any idea? Thank you!