Best iteration is always 0

[Enferlain]

I've ran around 10 auto merges so far, and for some reason at the end I always end up with it saying "best iteration: 0"

Also the entire time seems to be spent on evaluation.

Thoughts?

Results: 'hyper_score'
   Best score: 0.5331370521490586
   Best parameter set:
      'model_b.BASE'  : 0.26
      'model_b.IN00'  : 0.18
      'model_b.IN01'  : 0.47
      'model_b.IN02'  : 0.01
      'model_b.IN03'  : 0.15
      'model_b.IN04'  : 0.71
      'model_b.IN05'  : 0.11
      'model_b.IN06'  : 0.64
      'model_b.IN07'  : 0.49
      'model_b.IN08'  : 0.91
      'model_b.M00'   : 0.26
      'model_b.OUT00' : 0.7
      'model_b.OUT01' : 0.47
      'model_b.OUT02' : 0.67
      'model_b.OUT03' : 0.13
      'model_b.OUT04' : 0.87
      'model_b.OUT05' : 0.76
      'model_b.OUT06' : 0.56
      'model_b.OUT07' : 0.51
      'model_b.OUT08' : 0.36
      'model_c.BASE'  : 0.66
      'model_c.IN00'  : 0.85
      'model_c.IN01'  : 0.27
      'model_c.IN02'  : 0.26
      'model_c.IN03'  : 0.18
      'model_c.IN04'  : 0.48
      'model_c.IN05'  : 0.82
      'model_c.IN06'  : 0.03
      'model_c.IN07'  : 0.65
      'model_c.IN08'  : 0.36
      'model_c.M00'   : 0.73
      'model_c.OUT00' : 0.59
      'model_c.OUT01' : 0.37
      'model_c.OUT02' : 0.52
      'model_c.OUT03' : 0.05
      'model_c.OUT04' : 0.89
      'model_c.OUT05' : 0.0
      'model_c.OUT06' : 0.16
      'model_c.OUT07' : 0.7
      'model_c.OUT08' : 0.86
   Best iteration: 0

   Random seed: 3908740

   Evaluation time   : 26290.356142520905 sec    [100.0 %]
   Optimization time : 0.13603782653808594 sec    [0.0 %]
   Iteration time    : 26290.492180347443 sec    [120.6 sec/iter]

 - Best weights para =  ['0.26,0.18,0.47,0.01,0.15,0.71,0.11,0.64,0.49,0.91,0.26,0.7,0.47,0.67,0.13,0.87,0.76,0.56,0.51,0.36', '0.66,0.85,0.27,0.26,0.18,0.48,0.82,0.03,0.65,0.36,0.73,0.59,0.37,0.52,0.05,0.89,0,0.16,0.7,0.86'] [True, True, False]
 - Best alpha para =  []
debugs =  ['elemental merge']
use_extra_elements =  True
 - mm_max_models =  3
config hash =  ffccadf55e18e3362e614c3d1a24e39b5abf545d5c8d10d7af27c1ece98694e6
  - mm_use [True, True, False]
  - model_a umbra_mecha.fp16.safetensors [80da973b09]
  - base_model sd_xl_base_1.0.safetensors [31e35c80fc]
  - max_models 3
  - models ['tpn34pdfv10js2ts05tensoradjust.fp16.safetensors [cf4f62151c]', '4thtail3fix.fp16.safetensors [bdc6379d5b]']
  - modes ['DARE', 'Add-Diff']
  - calcmodes ['Normal', 'Normal']
  - usembws [['ALL'], ['ALL']]
  - weights ['0.26,0.18,0.47,0.01,0.15,0.71,0.11,0.64,0.49,0.91,0.26,0.7,0.47,0.67,0.13,0.87,0.76,0.56,0.51,0.36', '0.66,0.85,0.27,0.26,0.18,0.48,0.82,0.03,0.65,0.36,0.73,0.59,0.37,0.52,0.05,0.89,0,0.16,0.7,0.86']
  - alpha [0.5, 0.5]
  - adjust
  - use elemental [False, False]
  - elementals ['', '']
  - Parse elemental merge...
model_a = umbra_mecha.fp16
Loading from file D:\stable-diffusion-webui\models\Stable-diffusion\umbra_mecha.fp16.safetensors...
isxl = True , sd2 = False
compact_mode =  True
 - check possible UNet partial update...
 - partial changed blocks =  ['BASE', 'IN00', 'IN01', 'IN02', 'IN03', 'IN04', 'IN05', 'IN06', 'IN07', 'IN08', 'M00', 'OUT00', 'OUT01', 'OUT02', 'OUT03', 'OUT04', 'OUT05', 'OUT06', 'OUT07', 'OUT08']
 - UNet partial update mode
Open state_dict from file D:\stable-diffusion-webui\models\Stable-diffusion\tpn34pdfv10js2ts05tensoradjust.fp16.safetensors...
mode = DARE, mbw mode, alpha = [0.26, 0.18, 0.47, 0.01, 0.15, 0.71, 0.11, 0.64, 0.49, 0.91, 0.26, 0.7, 0.47, 0.67, 0.13, 0.87, 0.76, 0.56, 0.51, 0.36]
Stage #1/4: 100%|██████████████████████████████████████████████████████████████████| 2263/2263 [00:57<00:00, 39.42it/s]
Check uninitialized #2/4: 100%|████████████████████████████████████████████████| 2263/2263 [00:00<00:00, 215202.24it/s]
Open state_dict from file D:\stable-diffusion-webui\models\Stable-diffusion\4thtail3fix.fp16.safetensors...
mode = Add-Diff, mbw mode, alpha = [0.66, 0.85, 0.27, 0.26, 0.18, 0.48, 0.82, 0.03, 0.65, 0.36, 0.73, 0.59, 0.37, 0.52, 0.05, 0.89, 0.0, 0.16, 0.7, 0.86]
Stage #3/4: 100%|█████████████████████████████████████████████████████████████████| 2263/2263 [00:20<00:00, 109.25it/s]
Save unchanged weights #4/4: 100%|███████████████████████████████████████████████████████████| 253/253 [00:00<?, ?it/s]
 - merge processing in 92.6s (prepare: 14.4s, merging: 78.2s).
 - loading scripts.patches...
 - lora patch
 - Textencoder(BASE) has been successfully updated
 - update UNet block input_blocks.0.
 - update UNet block input_blocks.1.
 - update UNet block input_blocks.2.
 - update UNet block input_blocks.3.
 - update UNet block input_blocks.4.
 - update UNet block input_blocks.5.
 - update UNet block input_blocks.6.
 - update UNet block input_blocks.7.
 - update UNet block input_blocks.8.
 - update UNet block middle_block.
 - update UNet block output_blocks.0.
 - update UNet block output_blocks.1.
 - update UNet block output_blocks.2.
 - update UNet block output_blocks.3.
 - update UNet block output_blocks.4.
 - update UNet block output_blocks.5.
 - update UNet block output_blocks.6.
 - update UNet block output_blocks.7.
 - update UNet block output_blocks.8.
 - update UNet block time_embed.
 - update UNet block out.
 - UNet partial blocks have been successfully updated
 - Reload full state_dict...
 - remove old checkpointinfo
Unloading model 4 over the limit of 3...
 - model 3: umbra_mecha.fp16 + dare_weights(diff tpn34pdfv10js2ts05tensoradjust.fp16) x alpha_0 + (4thtail3fix.fp16 - sd_xl_base_1.0.safetensors [31e35c80fc]) x alpha_1(0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5),(0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5).safetensors [44e1eaaa09]
Creating model from config: D:\stable-diffusion-webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
Loading VAE weights specified in settings: D:\stable-diffusion-webui\models\VAE\sdxl_vae.safetensors
Applying attention optimization: xformers... done.
Model loaded in 3.4s (create model: 0.6s, apply weights to model: 1.8s, apply half(): 0.1s, load VAE: 0.5s, calculate empty prompt: 0.1s).
100%|██████████████████████████████████████████████████████████████████████████████████| 28/28 [00:09<00:00,  3.07it/s]
Total progress: 28it [00:09,  2.92it/s]
 > score_origin = -0.967432594299316/s]
 - Result score = 0.2753925362860547

[wkpark]

what's your optimizing method?
you can attach your particial screenshot for example:

• Normally, I select specific model and blocks to optimize,
• Normally, I select Hill Climbing optimizer. I did not test all optimze method. some optimizer will not work as expected or work with specific condition.

[wkpark]

and this is an example hyperactive app (modified slightly. from https://github.com/SimonBlanke/Hyperactive/blob/master/examples/optimization_applications/hyperpara_optimize.py )

"""
This example shows the original purpose of Hyperactive.
You can search for any number of hyperparameters and Hyperactive
will return the best one after the optimization run.

"""

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import load_wine
from hyperactive import Hyperactive

data = load_wine()
X, y = data.data, data.target


def model(opt):
    gbr = GradientBoostingClassifier(
        n_estimators=opt["n_estimators"],
        max_depth=opt["max_depth"],
        min_samples_split=opt["min_samples_split"],
        min_samples_leaf=opt["min_samples_leaf"],
        criterion=opt["criterion"],
    )
    scores = cross_val_score(gbr, X, y, cv=4)

    return scores.mean()


search_space = {
    "n_estimators": list(range(10, 150, 5)),
    "max_depth": list(range(2, 12)),
    "min_samples_split": list(range(2, 25)),
    "min_samples_leaf": list(range(1, 25)),
    "criterion": ["friedman_mse", "squared_error"],#, "absolute_error"],
    "subsample": list(np.arange(0.1, 3, 0.1)),
}

if __name__ == '__main__':
    hyper = Hyperactive()
    hyper.add_search(model, search_space, n_iter=40)
    hyper.run()

and its result:

[0] model (Random Search): 100%|───────────────────────────────────────────────────────────────────────| 40/40 [01:17<00:00,  1.94s/it, best_iter=0, best_pos=[17  5 12 18  0 21], best_score=0.9498737373737374]


Results: 'model'
   Best score: 0.9776515151515152
   Best parameter set:
      'n_estimators'      : 130
      'max_depth'         : 3
      'min_samples_split' : 15
      'min_samples_leaf'  : 22
      'criterion'         : friedman_mse
      'subsample'         : 0.7000000000000001
   Best iteration: 0

   Random seed: 416581482

   Evaluation time   : 77.45525693893433 sec    [99.99 %]
   Optimization time : 0.005005359649658203 sec    [0.01 %]
   Iteration time    : 77.46026229858398 sec    [1.94 sec/iter]

you can see "Best iteration: 0" in this case
(I guess this is a mitake/bug of the hyperactive)