Adding loggers into TunedModels #193
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev #193 +/- ##
==========================================
+ Coverage 87.53% 87.55% +0.01%
==========================================
Files 13 13
Lines 666 667 +1
==========================================
+ Hits 583 584 +1
Misses 83 83 ☔ View full report in Codecov by Sentry. |
|
Looking good, thanks! Does it all look good on the MLflow service when fitting a |
|
Looking good locally. I've just uploaded the |
|
Played around with this some more. Very cool, thanks! However, there is a problem running in multithread mode. It seem only one thread is logging: using MLJ
using .Threads
using MLFlowClient
nthreads()
# 5
logger = MLFlowLogger("http://127.0.0.1:5000", experiment_name="horse")
X, y = make_moons()
model = (@load RandomForestClassifier pkg=DecisionTree)()
r = range(model, :sampling_fraction, lower=0.4, upper=1.0)
tmodel = TunedModel(
model;
range=r,
logger,
acceleration=CPUThreads(),
n=100,
)
mach = machine(tmodel, X, y) |> fit!;
nruns = length(report(mach).history)
# 100
service = MLJFlow.service(logger)
experiment = MLFlowClient.getexperiment(service, "horse")
id = experiment.experiment_id
runs = MLFlowClient.searchruns(service, id);
length(runs)
# 20
@assert length(runs) == nruns
# ERROR: AssertionError: length(runs) == nruns
# Stacktrace:
# [1] top-level scope
# @ REPL[166]:1 |
|
The problem is we are missing |
|
I think |
|
Thanks for the addition. Sadly, this is still not working for me. I'm getting three experiments, with different id's and same name, "horse" on the server. (I'm only expecting one). One contains 20 evaluations, the other two contains only 1 each, and this complaint is thrown several times: Do you have any idea what is happening? ERROR: TaskFailedException {MLJDecisionTreeInterface.RandomForestClassifier, Vector{LogLoss{Float64}}, Vector{Float64}, Vector{typeof(predict)}, Vector{Vector{Float64}}, Vector{Vector{Vector{Float64}}}, Vector{NamedTuple{(:forest,), Tuple{DecisionTree.Ensemble{Float64, UInt32}}}}, Vector{NamedTuple{(:features,), Tuple{Vector{Symbol}}}}, Holdout}) |
|
Interestingly, I'm getting the same kind of error for using Distributed
addprocs(2)
nprocs()
# 3
using MLJ
using MLFlowClient
logger = MLFlowLogger("http://127.0.0.1:5000", experiment_name="rock")
X, y = make_moons()
model = (@iload RandomForestClassifier pkg=DecisionTree)()
r = range(model, :sampling_fraction, lower=0.4, upper=1.0)
tmodel = TunedModel(
model;
range=r,
logger,
acceleration=CPUProcesses(),
n=100,
)
mach = machine(tmodel, X, y) |> fit!; |
|
Okay, see here for a MWE: JuliaAI/MLFlowClient.jl#40 |
|
Revisiting this issue after a few months. It looks like the multithreading issue is not likely to be addressed soon. Perhaps we can proceed with this PR, after strictly ruling out logging for the parallel modes. For example, if @pebeto What do you think? |
|
The solution to this issue is not part of the |
Details in JuliaAI/MLJ.jl#1029.
Lfor loggers (detailed implementation in MLJBase.jl).