Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

'Selector' object has no attribute '_default_type' #134

Open
winturn opened this issue Jun 7, 2024 · 0 comments
Open

'Selector' object has no attribute '_default_type' #134

winturn opened this issue Jun 7, 2024 · 0 comments
Labels
help wanted Extra attention is needed question Further information is requested 抓取出错

Comments

@winturn
Copy link

winturn commented Jun 7, 2024 •
edited
Loading

目标网站地址

css selector / xpath 内容

from scrapy.selector import Selector

# 示例代码段
html = "<html><body><p>Hello World</p></body></html>"
selector = Selector(text=html)
print(selector.xpath("//p/text()").get())

实际发生了什么？

2024-06-07 22:57:22,720 [ThreadPoolExecutor-0_0:140009119401728] [scheduler:monitor:195] [ERROR]- Traceback (most recent call last):
File "/home/ubuntu/work/WebMonitor/task/utils/scheduler.py", line 138, in monitor
content = get_content(url, is_chrome, selector_type, selector,
File "/home/ubuntu/work/WebMonitor/task/utils/extract_info.py", line 62, in get_content
content_dict = selector_handler.get_by_xpath(url, selector_dict,
File "/home/ubuntu/work/WebMonitor/task/utils/selector/request_selector.py", line 30, in get_by_xpath
result[key] = self.xpath_parse(html, xpath_ext)
File "/home/ubuntu/work/WebMonitor/task/utils/selector/selector.py", line 22, in xpath_parse
res = Selector(
File "/home/ubuntu/work/WebMonitor/lib/python3.8/site-packages/scrapy/selector/unified.py", line 86, in __init__
st = _st(response, type or self._default_type)
AttributeError: 'Selector' object has no attribute '_default_type'

部署相关信息

手动

环境信息

(WebMonitor) ubuntu@ip:~/work/WebMonitor$ pip list | grep parse
parsel               1.9.1

原因

Scrapy 和 Parsel 之间的版本冲突可能会导致这种错误。将 Parsel 降级到 1.7.0 解决了问题

解决办法

修改 requirements.txt 文件，添加下面内容

parsel==1.7.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted Extra attention is needed question Further information is requested 抓取出错
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@winturn