Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

转换规则 torch.hub.load_state_dict_from_url #226

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion paconvert/api_alias_mapping.json
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,6 @@
"torch.utils.data.sampler.BatchSampler": "torch.utils.data.BatchSampler",
"torch.utils.data.sampler.RandomSampler": "torch.utils.data.RandomSampler",
"torch.utils.data.sampler.Sampler": "torch.utils.data.Sampler",
"torch.utils.data.sampler.SequentialSampler": "torch.utils.data.SequentialSampler"
"torch.utils.data.sampler.SequentialSampler": "torch.utils.data.SequentialSampler",
"torch.utils.model_zoo.load_url": "torch.hub.load_state_dict_from_url"
}
19 changes: 19 additions & 0 deletions paconvert/api_mapping.json
Original file line number Diff line number Diff line change
Expand Up @@ -5102,6 +5102,25 @@
"out"
]
},
"torch.hub.load_state_dict_from_url": {
"Matcher": "GenericMatcher",
"paddle_api": "paddle.utils.download.get_weights_path_from_url",
"args_list": [
"url",
"model_dir",
"map_location",
"progress",
"check_hash",
"file_name"
],
"unsupport_args": [
"model_dir",
"map_location",
"progress",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这些不重要的参数可以直接删除,避免影响转换效果,一般一些只关乎性能的、调试的、输入校验的,对计算结果都没有影响,可以采取这种描述方式,代码里写法是对应一个空字符串,文档记得同步更新
infoflow 2023-08-18 12-21-07
这里的progress、check_hash应该不关键

"check_hash",
"file_name"
]
},
"torch.hypot": {
"Matcher": "HypotMatcher",
"args_list": [
Expand Down
66 changes: 66 additions & 0 deletions tests/test_hub_load_state_dict_from_url.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import textwrap

from apibase import APIBase

obj = APIBase("torch.hub.load_state_dict_from_url")


def test_case_1():
pytorch_code = textwrap.dedent(
"""
import torch
state_dict = torch.hub.load_state_dict_from_url('https://s3.amazonaws.com/pytorch/models/resnet18-5c106cde.pth')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

看了下这两个API的返回结果不同,torch返回的是一个Layer的state_dict结果,paddle返回的是一个路径,所以是不是paddle的还需要包装下,比如加一个load之类的

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

加load之后还是不能实现直接转换,原因是:torch 的.pth权重文件,paddle直接load .pth可能会报错

result = True
"""
)
obj.run(pytorch_code, ["result"])


def test_case_2():
pytorch_code = textwrap.dedent(
"""
import torch
result = torch.hub.load_state_dict_from_url('https://s3.amazonaws.com/pytorch/models/resnet18-5c106cde.pth',model_dir="./tests")
"""
)
obj.run(
pytorch_code, ["result"], unsupport=True, reason="paddle not support this arg"
)


def _test_alias_case_1():
pytorch_code = textwrap.dedent(
"""
import torch
state_dict = torch.utils.model_zoo.load_url('https://s3.amazonaws.com/pytorch/models/resnet18-5c106cde.pth')
result = True
"""
)
obj.run(pytorch_code, ["result"])


def test_alias_case_2():
pytorch_code = textwrap.dedent(
"""
import torch
state_dict = torch.utils.model_zoo.load_url('https://s3.amazonaws.com/pytorch/models/resnet18-5c106cde.pth',model_dir="./tests")
result = True
"""
)
obj.run(
pytorch_code, ["result"], unsupport=True, reason="paddle not support this arg"
)