Help Needed with Evaluation Process

[naren200]

Description:

Problem Statement

I am currently working on a project involving training a model, and I have encountered an issue during the evaluation step. I would greatly appreciate your assistance in resolving this problem.

Successful Training Process

As shown in the screenshot above, the training process appears to have completed successfully. The model checkpoints are being saved without any issues.

Error in the Evaluation Step

In the evaluation step, I encounter the following error:

Could not set up the required agent:
> [Errno 2] No such file or directory: 'leaderboard/team_code/interfuser.pth.tar'

Additional Information

• Environment: Conda environment created and training performed on 1 GPU.
• Following Changes made out of README - Batch_size set to 8 (CUDA Out of Memory) && world_size() is set to 1 (1 GPU related error, mentioned in other commits)

Thank you in advance for your assistance!

Updated Information

Run_evaluation.sh (for evaluation)

#!/bin/bash

export CARLA_ROOT=/home/thomfrien/bang/InterFuser/carla
export ROUTES=leaderboard/data/evaluation_routes/routes_town05_long.xml
export TEAM_AGENT=leaderboard/team_code/interfuser_agent.py
export TEAM_CONFIG=leaderboard/team_code/interfuser_config.py
export CHECKPOINT_ENDPOINT=results/interfuser_result.json
export SCENARIOS=leaderboard/data/scenarios/town05_all_scenarios.json

# export CARLA_ROOT=carla
export CARLA_SERVER=${CARLA_ROOT}/CarlaUE4.sh
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla
export PYTHONPATH=$PYTHONPATH:$CARLA_ROOT/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg
export PYTHONPATH=$PYTHONPATH:leaderboard
export PYTHONPATH=$PYTHONPATH:leaderboard/team_code
export PYTHONPATH=$PYTHONPATH:scenario_runner

export LEADERBOARD_ROOT=leaderboard
export CHALLENGE_TRACK_CODENAME=SENSORS
export PORT=2000 # same as the carla server port
export TM_PORT=2500 # port for traffic manager, required when spawning multiple servers/clients
export DEBUG_CHALLENGE=0
export REPETITIONS=1 # multiple evaluation runs
# export ROUTES=leaderboard/data/training_routes/routes_town05_long.xml
# export TEAM_AGENT=leaderboard/team_code/interfuser_agent.py # agent
# export TEAM_CONFIG=leaderboard/team_code/interfuser_config.py # model checkpoint, not required for expert
# export CHECKPOINT_ENDPOINT=results/sample_result.json # results file
# export SCENARIOS=leaderboard/data/scenarios/town05_all_scenarios.json
export SAVE_PATH=data/eval # path for saving episodes while evaluating
export RESUME=True

python3 ${LEADERBOARD_ROOT}/leaderboard/leaderboard_evaluator.py \
--scenarios=${SCENARIOS}  \
--routes=${ROUTES} \
--repetitions=${REPETITIONS} \
--track=${CHALLENGE_TRACK_CODENAME} \
--checkpoint=${CHECKPOINT_ENDPOINT} \
--agent=${TEAM_AGENT} \
--agent-config=${TEAM_CONFIG} \
--debug=${DEBUG_CHALLENGE} \
--record=${RECORD_PATH} \
--resume=${RESUME} \
--port=${PORT} \
--trafficManagerPort=${TM_PORT}

[deepcs233]

Hi!

You need to move the saved checkpoint to the path leaderboard/team_code/interfuser.pth.tar

[naren200]

Thanks,

Pretrain weights

I downloaded the pre-trained weights under 'Pretrain weights' section in ReadME.md which was downloaded here. The model was moved to leaderboard/team_code/. Then, I ran the model and everything seem fine as of now. Now, The issue is sovled.

I have a quick question on this method:

1. I haven't ran 'train.sh' yet. As of now, I have ran only the data generation part, I was wondering why do I need pre-trained weights for generating data for the training process.
2. Is the pre-trained weights created using the training process of the Interfuser or created using the leaderboard under the carla sim!?

Regards,
Naren.