Changes to control model sparsity and MTSS improvements

[mcloughlin2]

Improvements to MultitaskScaffoldSplitter:

• Notable speedup of GA optimization
• Added early stopping for optimization loop
• Changed fitness function for Tanimoto distances to grade splits based on fraction of valid/test scaffolds with no train scaffold closer than given Tanimoto distance
• Track evolution of fitness function terms during optimization to support split_diagnostics_plot function.

New module split_diagnostic_plots:

• Provides single function to plot multiple aspects of split quality, given split parameters or ModelPipeline as input: Response value distributions by subset, Tanimoto distance histograms, actual vs requested split fractions, and fitness term evolution.
• New individual functions to plot split fractions and fitness term evolution during GA optimization.

Sparsity-related parameters for XGBoost models:

• xgb_alpha: Controls strength of L1 penalty in cost function
• xgb_lambda: Controls strength of L2 penalty in cost function

New search domain parameters for hyperopt optimization of sparsity parameters:

• wdp – for weight_decay_penalty
• wdt – for weight_decay_penalty_type
• xgba – for xgb_alpha
• xgbb – for xgb_lambda (xgbl is already taken)

Feature_importance function to draw line plot of summed NN absolute feature weights vs epoch.

[paulsonak]

Hi, I have tested the following so far mainly by using various tutorial notebooks and running the functions through there. I will update this in a bit when I am finished trying out the rest of the features.

Improvements to MultitaskScaffoldSplitter:

• Notable speedup of GA optimization
• Added early stopping for optimization loop
• Changed fitness function for Tanimoto distances to grade splits based on fraction of valid/test scaffolds with no train scaffold closer than given Tanimoto distance
• Track evolution of fitness function terms during optimization to support split_diagnostics_plot function.

Comments:

• definitely way faster although there's a TensorFlow deprecation warning:

WARNING:tensorflow:From [/Users/apaulson/atomsci-venv/lib/python3.9/site-packages/tensorflow/python/util/deprecation.py:588](http://localhost:8888/lab/tree/repos/AMPL_umbrella/AMPL/atomsci/ddm/examples/tutorials/atomsci-venv/lib/python3.9/site-packages/tensorflow/python/util/deprecation.py#line=587): calling function (from tensorflow.python.eager.polymorphic_function.polymorphic_function) with experimental_relax_shapes is deprecated and will be removed in a future version.
Instructions for updating:
experimental_relax_shapes is deprecated, use reduce_retracing instead

• early stopping works as expected
• updated grading function makes sense and the code looks fine. I had trouble following all the distance matrix calculations but the logic makes sense. (_generate_scaffold_dist_matrix, far_frac_fitness)

New module split_diagnostic_plots:

• Provides single function to plot multiple aspects of split quality, given split parameters or ModelPipeline as input: Response value distributions by subset, Tanimoto distance histograms, actual vs requested split fractions, and fitness term evolution.
• New individual functions to plot split fractions and fitness term evolution during GA optimization.

Comments:

• plot_split_diagnostics() is great. I would recommend moving the tanimoto distance distribution plots to the end with the unweighted fitness scores. That way, all things that are per-task are plotted first and the user can specify num_cols=num_tasks for a neater plot.
• It is confusing that axes can't be None for plot_split_fractions() and plot_fitness_terms(). You can just do a simple if axes is None: fig, ax = plt.subplots() check.

Sparsity-related parameters for XGBoost models:

• xgb_alpha: Controls strength of L1 penalty in cost function
• xgb_lambda: Controls strength of L2 penalty in cost function

New search domain parameters for hyperopt optimization of sparsity parameters:

• wdp – for weight_decay_penalty
• wdt – for weight_decay_penalty_type
• xgba – for xgb_alpha
• xgbb – for xgb_lambda (xgbl is already taken)

Feature_importance

• function to draw line plot of summed NN absolute feature weights vs epoch.