如何利用抓包内容，编写GET中headers配置信息？ #8366

Derekmini · 2021-10-09T11:31:01Z

Derekmini
Oct 9, 2021

您好，想请教一个配置got中header的写法：

描述问题

我遇到一个网页：https://asa.scitation.org/toc/jas/current，想利用如下语句获取每篇文章的信息( 如 const list = $('section.card') )。

const response = await got({
    method: 'get',
    url: 'https://asa.scitation.org/toc/jas/current',
    headers: {
        'Accept-Encoding': 'gzip, deflate, br',
        'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36',
        cookie: 'timezone=480; I2KBRCK=1; MAID=11223344==; _ga=11223344; cookieconsent_status=dismiss; JSESSIONID=11223344; SERVER=11223344==; MACHINE_LAST_SEEN=2021-10-08T22%3A47%3A16.552-07%3A00; osano_consentmanager_uuid=11223344; osano_consentmanager=11223344; osano_consentmanager_expdate=11223344; _gid=11223344; __gads=ID=11223344:RT=11223344; _ga_W1NY9Q2R0V=11223344; _ga=11223344',
});

出现问题的地方在：查看response.data，发现其中的html内容并不完整，经查看不包含每篇文章的内容，导致不能使用const $ = cheerio.load(response.data);获取想要的信息。

我只能暂时认为是headers没有配置好，所以想通过抓包，写出一个有效的headers配置信息。

现有抓包内容以及尝试改写的无效代码

现有的抓包内容

GET https://asa.scitation.org/toc/jas/current HTTP/1.1
Host: asa.scitation.org
Connection: keep-alive
Cache-Control: max-age=0
sec-ch-ua: "Chromium";v="94", "Google Chrome";v="94", ";Not A Brand";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8
Cookie: timezone=480; I2KBRCK=1; MAID=11223344==; _ga=11223344; cookieconsent_status=dismiss; JSESSIONID=11223344; SERVER=11223344==; MACHINE_LAST_SEEN=2021-10-08T22%3A47%3A16.552-07%3A00; osano_consentmanager_uuid=11223344; osano_consentmanager=11223344; osano_consentmanager_expdate=11223344; _gid=11223344; __gads=ID=11223344:RT=11223344; _ga_W1NY9Q2R0V=11223344; _ga=11223344

另附一张图片，我是通过Fillder抓取的：

Fillder抓取的内容

尝试改写的无效代码

const response = await got({
    method: 'get',
    url: 'https://asa.scitation.org/toc/jas/current',
    headers: {
        'Accept-Encoding': 'gzip, deflate, br',
        'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36',
        cookie: 'timezone=480; I2KBRCK=1; MAID=11223344==; _ga=11223344; cookieconsent_status=dismiss; JSESSIONID=11223344; SERVER=11223344==; MACHINE_LAST_SEEN=2021-10-08T22%3A47%3A16.552-07%3A00; osano_consentmanager_uuid=11223344; osano_consentmanager=11223344; osano_consentmanager_expdate=11223344; _gid=11223344; __gads=ID=11223344:RT=11223344; _ga_W1NY9Q2R0V=11223344; _ga=11223344',
});

其他尝试

之前通过puppeteer也完成了想实现的路由功能，但是现在由于资源开销大等原因，路由工作不太正常，所以想换回推荐的获取方式。

虽然接触的时间比较长了，但没有系统地学习过这方面的知识，所以还希望您能给一些提示，能帮我打开一下思路，谢谢您的帮助！

Answered by auto-bot-ty

Oct 9, 2021

const $ = cheerio.load(response.body);
const out = $(".title_item")
  .map((i, e) => {
    $(e).find(".access-text").remove();
    return {
      title: $(e).find(".hlFld-Title").text(),
      link: $(e).find(".ref.nowrap").attr("href"),
      pubDate: $(e).find(".open-access.item-access").text(),
    };
  })
  .get();
console.log(out);

View full answer

auto-bot-ty · 2021-10-09T13:14:13Z

auto-bot-ty
Oct 9, 2021

const $ = cheerio.load(response.body);
const out = $(".title_item")
  .map((i, e) => {
    $(e).find(".access-text").remove();
    return {
      title: $(e).find(".hlFld-Title").text(),
      link: $(e).find(".ref.nowrap").attr("href"),
      pubDate: $(e).find(".open-access.item-access").text(),
    };
  })
  .get();
console.log(out);

7 replies

Derekmini Oct 9, 2021
Author

是的是的，我刚发现后面这个括号问题了，我已经修正了；

cookie的话，我是从抓包软件中获取到的，请问我能在路由中获取cookie吗？

我考虑的是这个cookie是不是有时效性，担心以后提交了路由后别人一段时间后无法使用。

还是要谢谢您，我已经能正常获取到列表了！！！

auto-bot-ty Oct 9, 2021

可以通过环境变量配置 cookie，具体可参考 #7174

Derekmini Oct 10, 2021
Author

简化了 cookie，并获取到数据

按照您提供的思路，我把从 HTML 获取数据的代码改写如下：( 主要简化了 cookie 内容 )

const response = await got({
    method: 'get',
    url: 'https://asa.scitation.org/toc/jas/current',
    headers: {
        'Accept-Encoding': 'gzip, deflate, br',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36',
        cookie: 'JSESSIONID=9e268139-aaaa-bbbb-cccc-ddddc1cc47ca; SERVER=abcdabcdabcdJx41gFJUAw==; MAID=abcdabcdabcdHd6ScOA3jQ==; MACHINE_LAST_SEEN=2021-10-09T19%3A41%3A12.540-07%3A00; I2KBRCK=1',
    },
});

偶然利用Chrome无痕模式访问链接，再用Fillder抓包，发现只需要在 cookie 中填写如上几个内容即可获取到数据。

我对 cookie 时效性的理解以及无效的尝试

我对 cookie 时效性的理解

我觉得网站会每隔一段时间重新派发一个新的时间戳，并且设置一个超期时间。

我尝试把 cookie 中的时间戳和服务器内容去掉，发现还能正常获取到数据，即提供如下cookie：

cookie: 'JSESSIONID=9e268139-aaaa-bbbb-cccc-ddddc1cc47ca; MAID=abcdabcdabcdHd6ScOA3jQ==;',

但考虑到时效性，决定边使用边观察一下这个 cookie 是否能长期有效；

无效的尝试

为了避免时效问题，我查看到抓包软件中有一个内容好像和设置 cookie 有关：

GET https://asa.scitation.org/toc/jas/current?cookieSet=1 HTTP/1.1
Host: asa.scitation.org
Connection: keep-alive
Upgrade-Insecure-Requests: 1

我利用如下代码，想获取与设置 cookie 相关的内容，结果无数据返回。

const response = await got({
    method: 'get',
    url: 'https://asa.scitation.org/toc/jas/current?cookieSet=1',
    headers: {
        'Accept-Encoding': 'gzip, deflate, br',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36',
    },
});

既然上面这个方法失败了，那是否能通过 post 的方式，将 JSESSIONID 和 MAID 提供给网站，重新获取一个时间戳？

请问，在 DIYgod/RSSHub 的路由中，这种做法是常规获取 cookie 的方法吗？赞同这种做法吗？

auto-bot-ty Oct 10, 2021

  const res = await got("https://asa.scitation.org");
  const cookie = res.headers["set-cookie"].join(" ");

  const response = await got({
    method: "get",
    url: "https://asa.scitation.org/toc/jas/current",
    headers: {
      "Accept-Encoding": "gzip, deflate, br",
      "User-Agent":
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
      cookie,
    },
  });

Derekmini Oct 10, 2021
Author

这也太好用了吧！服了！谢谢！现在用起来没什么问题了，我再调整一些内容，完善一下文档，准备提交上去！

另外，作者部分我想写入您的名字——['Derekmini', 'auto-bot-ty']。

再次感谢您的帮助！

刚才又按照您的思路搜索了一下，找到如下内容：

headers: {
  date: "Sun, 10 Oct 2021 06:30:56 GMT",
  "content-type": "text/html; charset=UTF-8",
  "transfer-encoding": "chunked",
  connection: "close",
  "x-xss-protection": "1; mode=block",
  "x-content-type-options": "nosniff",
  "strict-transport-security": "max-age=2592000",
  "x-frame-options": "SAMEORIGIN",
  "cache-control": "no-cache",
  pragma: "no-cache",
  "x-webstats-respid": "12b9d30826f70d895448561d622a6ee3",
  "set-cookie": [
    "JSESSIONID=abcd0f2e-abcd-abcd-abcd-abcda9e1f9c5; path=/; domain=.scitation.org; age=-1; HttpOnly; SameSite=None; Secure",
  ],
  vary: "Accept-Encoding",
  "cf-cache-status": "DYNAMIC",
  "expect-ct": "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"",
  server: "cloudflare",
  "cf-ray": "69bdcd8bc9230c46-DFW",
  "content-encoding": "br",
  "alt-svc": "h3=\":443\"; ma=86400, h3-29=\":443\"; ma=86400, h3-28=\":443\"; ma=86400, h3-27=\":443\"; ma=86400",
},

发现原来您是从这个地方获取的cookie，学习到了，谢谢！

auto-bot-ty · 2021-10-09T13:15:17Z

auto-bot-ty
Oct 9, 2021

[
  {
    title: 'Do I have a hearing loss?',
    link: '/doi/full/10.1121/10.0006522',
    pubDate: 'October 2021'
  },
  {
    title: 'Estimating target strength of estuarine pelagic fish assemblages using fisheries survey data',
    link: '/doi/full/10.1121/10.0006449',
    pubDate: 'October 2021'
  },
  {
    title: 'Assessing room acoustic listening expertise',
    link: '/doi/full/10.1121/10.0006574',
    pubDate: 'October 2021'
  },
...
]

0 replies

auto-bot-ty · 2021-10-09T13:17:53Z

auto-bot-ty
Oct 9, 2021

<section class="card"><article class="card-cont"><div class="toc-graphical-abstract"><a href="/doi/10.1121/10.0006522"><img src="/na101/home/literatum/publisher/aip/journals/content/jas/2021/jas.2021.150.issue-4/10.0006522/20211001/images/medium/10.0006522.figures.online.highlight_f1.jpg" alt=""></a></div><div class="title_item"><div class="open-access item-access"><i class="icon-lock-open"></i><span class="access-text">Free</span>October 2021</div><div class="icon-container"></div><div class="art_title linkable"><a class="ref nowrap" href="/doi/full/10.1121/10.0006522"><h4 class="hlFld-Title">Do I have a hearing loss?</h4></a></div></div><div class="info_item"><label class="tocAuthors author-list afterTitle"><div class="entryAuthor all"><span class="articleEntryAuthorsLinks"><span class="entryAuthor normal hlFld-ContribAuthor"><a href="/author/Yost%2C+William+A">William A. Yost</a></span></span></div><a href="#" class="showMoreAuthors authorsHidden"> more...</a></label><div class="meta-article">The Journal of the Acoustical Society of America <b>150</b>, R7 (2021); <a href="https://doi.org/10.1121/10.0006522">https://doi.org/10.1121/10.0006522</a></div></div><div class="tabs-abstract-cont list-unstyled"><nav class="article-tabs"><ul><li><a href="#" class="js-show-abstract-toc" id="showAbs100006522" data-text-hidden="true" data-abs-id="Abs100006522" data-doi="10.1121/10.0006522" data-visible-state-text="Hide Abstract" data-hidden-state-text="Show Abstract">Show Abstract
                    </a></li><li class="visible-sm visible-xs visible-md pull-right"><span class="icon-small-menu"></span></li><li class="article-options"><ul><li><a class="ref nowrap pdf show-pdf" role="link" aria-label="Open PDF File" target="_blank" title="Opens new window" href="/doi/pdf/10.1121/10.0006522"><span class="icon-download-file"></span>PDF</a></li><li><a class="ref" href="/personalize/addFavoriteArticle?doi=10.1121%2F10.0006522" target="_blank"><span>Add to Favorites</span></a></li><li class="share-button"><div class="share-section-container"><a href="#" class="share-menu-btn social-share"><span>Share</span></a><span class="icon visible-xs visible-md icon-arrow_d_n"></span><div class="share-wrapper"><div><a href="/action/showMailPage?doi=10.1121/10.0006522" class="share-button_email" target="_blank"><span>E-MAIL</span></a></div><div><a href="https://www.facebook.com/sharer/sharer.php?u=https://asa.scitation.org/doi/10.1121/10.0006522%3Fvia%3Dsite" class="share_button_facebook" title="Facebook" target="_blank"><span>FACEBOOK</span></a></div><div><a href="https://www.linkedin.com/sharing/share-offsite/?url=https://asa.scitation.org/doi/abs/10.1121/10.0006522%3Fvia%3Dsite" class="share_button_linkedin" title="LinkedIn" target="_blank"><span>LINKEDIN</span></a></div><div><a class="recommend-to-librarians" href="/action/recommendation?doi=10.1121/10.0006522" target="_blank"><span>RECOMMENDED TO LIBRARIANS</span></a></div><div><a href="https://twitter.com/intent/tweet?text=https://asa.scitation.org/doi/abs/10.1121/10.0006522%3Fvia%3Dsite" class="share_button_twitter" title="Twitter" target="_blank"><span>TWITTER</span></a></div></div></div></li><li><a href="/action/showCitFormats?doi=10.1121%2F10.0006522" target="_blank"><span>Export Citation</span></a></li></ul></li></ul></nav><div id="Abs100006522" class="previewViewSection tocPreview"><p class="previewContent abstract-toc"></p><a class="toc-abs-link Abs100006522" href="/doi/abs/10.1121/10.0006522"><span>GO TO ABSTRACT</span></a></div></div></article></section>

FreeOctober 2021

Do I have a hearing loss?

William A. Yost

more...

The Journal of the Acoustical Society of America 150, R7 (2021); https://doi.org/10.1121/10.0006522

Show Abstract

GO TO ABSTRACT

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

如何利用抓包内容，编写GET中headers配置信息？ #8366

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 7 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Do I have a hearing loss?

Select a reply

如何利用抓包内容，编写GET中headers配置信息？ #8366

Derekmini Oct 9, 2021

描述问题

现有抓包内容以及尝试改写的无效代码

现有的抓包内容

尝试改写的无效代码

其他尝试

Replies: 3 comments · 7 replies

auto-bot-ty Oct 9, 2021

Derekmini Oct 9, 2021 Author

auto-bot-ty Oct 9, 2021

Derekmini Oct 10, 2021 Author

简化了 cookie，并获取到数据

我对 cookie 时效性的理解以及无效的尝试

我对 cookie 时效性的理解

无效的尝试

auto-bot-ty Oct 10, 2021

Derekmini Oct 10, 2021 Author

auto-bot-ty Oct 9, 2021

auto-bot-ty Oct 9, 2021

Do I have a hearing loss?

Derekmini
Oct 9, 2021

Replies: 3 comments 7 replies

auto-bot-ty
Oct 9, 2021

Derekmini Oct 9, 2021
Author

Derekmini Oct 10, 2021
Author

Derekmini Oct 10, 2021
Author

auto-bot-ty
Oct 9, 2021

auto-bot-ty
Oct 9, 2021