Liptschitz constant estimation above 1 for your tutorial

[ldesmet]

Hello there !

Trying this package in order to have mathematical guarantee of model behavior.
Unfortunately, I simply copy/paste your code of the first tutorial, and the liptschitz constant of the model seems to be above 1. Note that my problem is not 1 Liptschitz, but I except the model to kind of "saturate"... In other words, I except the model to stay 1 Liptchitz even if the problem the model is trying to fit does not respect this property. Here is a piece of my code.

lip_model = lip.model.Sequential([    
        keras.layers.Input(shape=(2,)),
        keras.layers.Flatten(),
        lip.layers.SpectralDense(128),
        lip.layers.GroupSort2(),
        lip.layers.SpectralDense(256),
        lip.layers.GroupSort2(),
        lip.layers.SpectralDense(128),
        lip.layers.GroupSort2(),
        lip.layers.SpectralDense(1)
    ],

)

lip_model.compile(optimizer='adam', loss=keras.losses.mean_absolute_error)
lip_model.fit(x=X_train, y=y_train, batch_size=4096, epochs = 10, validation_data=(X_test, y_test), callbacks=[es])
print('R2 : ', r2_score(y_test, lip_model.predict(X_test, batch_size=4096).ravel()))
print(deel.lip.utils.evaluate_lip_const(lip_model, X_test.values))

Here is the output :

Epoch 1/10
105/105 [==============================] - 5s 37ms/step - loss: 0.1414 - val_loss: 0.0810
Epoch 2/10
105/105 [==============================] - 4s 33ms/step - loss: 0.0519 - val_loss: 0.0351
Epoch 3/10
105/105 [==============================] - 3s 33ms/step - loss: 0.0306 - val_loss: 0.0217
Epoch 4/10
105/105 [==============================] - 3s 32ms/step - loss: 0.0222 - val_loss: 0.0223
Epoch 5/10
105/105 [==============================] - 3s 31ms/step - loss: 0.0212 - val_loss: 0.0225
Epoch 6/10
105/105 [==============================] - 3s 33ms/step - loss: 0.0180 - val_loss: 0.0138
Epoch 7/10
105/105 [==============================] - 3s 33ms/step - loss: 0.0156 - val_loss: 0.0145
Epoch 8/10
105/105 [==============================] - 4s 39ms/step - loss: 0.0149 - val_loss: 0.0124
Epoch 9/10
105/105 [==============================] - 4s 36ms/step - loss: 0.0148 - val_loss: 0.0152
Epoch 10/10
105/105 [==============================] - 4s 33ms/step - loss: 0.0155 - val_loss: 0.0121
52/52 [==============================] - 0s 3ms/step
R2 :  0.9942374229431152
6617/6617 [==============================] - 16s 2ms/step
6617/6617 [==============================] - 19s 3ms/step
lip cst: 2.529

Do you have any idea why this is not 1 Liptschitz ? I tried set up explicity the liptschitz constant at 1, but this did not solve the issue...

Many thanks,

[cofri]

Hi @ldesmet,

Thanks for your feedback!
This is indeed not an expected behaviour: in your code snippet, the Lipschitz constant of the model should be equal or less than 1.

Could you give us more information about your environment ? Python version, TensorFlow and Keras versions, deel-lip version ?
On our side, we will investigate this problem based on the code snippet you provided.

[ldesmet]

Hello !

yes sure, here is my environment :

Python 3.9.2
tensorflow 2.10.1
keras 2.10.0
deel-lip 1.5.0

Here is the full code if you want to generate same data

def compute_func(x, y, constant=200):
    return ((x**2 + y -11)**2 + (x + y**2 -7)**2)/constant

# Generate x and y values
step = 0.01
domain = 4
x_values = np.arange(-domain, domain + step, step)
y_values = np.arange(-domain, domain + step, step)
X, Y = np.meshgrid(x_values, y_values)

# Compute function values
Z = compute_func(X, Y)

# Compute gradients
dZ_dx, dZ_dy = np.gradient(Z, step, step)

# Compute gradient magnitude
gradient_magnitude = np.sqrt(dZ_dx**2 + dZ_dy**2)

# Find maximum gradient
max_gradient = np.max(gradient_magnitude)

print("Maximum gradient:", max_gradient)

# Create a DataFrame for Plotly
df = pd.DataFrame({
    'x': X.ravel(),
    'y': Y.ravel(),
    'z': Z.ravel()
})

Followed by vanilla NN

X_train, X_test, y_train, y_test = train_test_split(df[['x','y']], df[['z']], test_size=0.33)

es = keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True, monitor='val_loss')

model = keras.Sequential([
        layers.Dense(128, input_shape=(2,), activation ='relu'),
        layers.Dense(256, activation ='relu'),
        layers.Dense(128, activation ='relu'),
        layers.Dense(1)
    ])
model.compile(optimizer='adam', loss=keras.losses.mean_absolute_error)
model.fit(x=X_train, y=y_train, batch_size=4096, epochs = 100, validation_data=(X_test, y_test), callbacks=[es])
print('R2 : ', r2_score(y_test, model.predict(X_test, batch_size=4096).ravel()))

And then the Lipschitz version

lip_model = lip.model.Sequential([    
        keras.layers.Input(shape=(2,)),
        keras.layers.Flatten(),
        lip.layers.SpectralDense(128),
        lip.layers.GroupSort2(),
        lip.layers.SpectralDense(256),
        lip.layers.GroupSort2(),
        lip.layers.SpectralDense(128),
        lip.layers.GroupSort2(),
        lip.layers.SpectralDense(1)
    ], k_coef_lip=1,

)

lip_model.compile(optimizer='adam', loss=keras.losses.mean_absolute_error)
lip_model.fit(x=X_train, y=y_train, batch_size=4096, epochs = 10, validation_data=(X_test, y_test))
print('R2 : ', r2_score(y_test, lip_model.predict(X_test, batch_size=4096).ravel()))
print(deel.lip.utils.evaluate_lip_const(lip_model, X_test.values))

Many thanks for your support :)