Curve25519 Derivation doesn't work against RFC test Vector

[jose-gataca]

Hi,

I am having a compatibility issue with another library, similar to Issue #198 . The shared key obtained isn't the expected one.
So, in order to determine which one is correct, I have tested against the Test vector provided by the RFC:
https://www.rfc-editor.org/rfc/rfc7748#section-6.1

It seems your library doesn't provide the expected value.

I attach the code I used for testing in case my test is wrongly implemented:

const EC = require('elliptic').ec;
const x25519 = new EC('curve25519');

export function TestRFCDerivation() {
    try {
        const pairA = {
            publicKey: '8520f0098930a754748b7ddcb43ef75a0dbf3a0d26381af4eba4a98eaa9b4e6a',
            privateKey: '77076d0a7318a57d3c16c17251b26645df4c2f87ebc0992ab177fba51db92c2a'
        };
        const pairB = {
            publicKey: 'de9edb7d7b7dc1b4d35b61c2ece435373f8343c85b78674dadfc7e146f882b4f',
            privateKey: '5dab087e624a8a4b79e17f8b83800ee66f3bb1292618b6fd1c2f8b27ff88e0eb'
        };
        let shared1 = derive(pairA.privateKey, pairB.publicKey);
        let shared2 = derive(pairB.privateKey, pairA.publicKey);
        let expected = '4a5d9d5ba4ce2de1728e3bf480350f25e07e21c947d19e3376f09b3c1e161742';
        console.log('DERIVATION TEST', expected, shared1, shared2, shared1 === expected, shared2 === shared1);
    } catch (error) {
        console.log('Error in RFC Derivation test', error);
    }
}

 derive(privateKey: string, publicKey: string): string {
        let key = x25519.keyFromPrivate(privateKey, 'hex');
        let pkey = x25519.keyFromPublic(publicKey, 'hex');
        return key.derive(pkey.getPublic());
    }

I have tried to debug the issue and it seems that the public keys are not validated (while they are the ones provided by the test vector in the spec), in

elliptic/lib/elliptic/ec/key.js

Line 103 in 43ac7f2

if(!pub.validate()) {

The validation fails on the validation of the montgomery curve:

elliptic/lib/elliptic/curve/mont.js

Line 25 in 43ac7f2

var y = rhs.redSqrt();

If we avoid the validation (commenting it) the test obviously doesn't pass: you get a derived secret which is not the same in both cases and not equal to expected value.