Error in trainModel(): all the ROC metric values are missing

[a-solovyev12]

Hello @joseah ,

Thank you for developing scPred! We have tested the latest version (v1.9.0) and encountered the following issue when training the model:

●  Training models for each cell type...
Something is wrong; all the ROC metric values are missing:
      ROC           Sens          Spec    
 Min.   : NA   Min.   : NA   Min.   : NA  
 1st Qu.: NA   1st Qu.: NA   1st Qu.: NA  
 Median : NA   Median : NA   Median : NA  
 Mean   :NaN   Mean   :NaN   Mean   :NaN  
 3rd Qu.: NA   3rd Qu.: NA   3rd Qu.: NA  
 Max.   : NA   Max.   : NA   Max.   : NA  
 NA's   :9     NA's   :9     NA's   :9    


Error: Stopping
In addition: Warning message:
In nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo,  :
  There were missing values in resampled performance measures.
Execution halted

svmRadialWeights was used as a model. Could you recommend anything to overcome this?

Many thanks,
Andrey

[joseah]

Hi @a-solovyev12,

It seems that there's a cell type with fewer cells than the number of resamples. By default, 5 cross-validations are performed so it's likely that some cross-validations are left without any cells to include for parameter tuning. If that's the case, I'd reevaluate the incorporation of that cell type to be classified given the low number of cells. Otherwise, you could reduce the number cross-validations using the number parameter in the trainModel() function.

Does this happen with other models (besides svmRadialEights)?

Cheers,
Jose

[a-solovyev12]

Hi Jose,

Thanks for a quick reply.
Indeed, taking only cell types with a higher number of cells solves the problem. What minimum number of cells would you suggest for robust performance in the CV step (assuming we keep it 5-fold)?

Best,
Andrey