Please help me interpret grad-cam visualization

[jooseuk]

I trained a binary classification model based on the presence of browning. The model classifies images as True if the flower has brown coloration and False if it doesn't. In images where browning is present, I observed that guided backpropagation or Grad-CAM captures features in the brown areas and visualizes them in red. However, for images without browning, the guided backpropagation images show no features, and Grad-CAM highlights large portions of the image as important. Could you possibly provide an interpretation or opinion on why this is happening?