如何利用抓包内容，编写GET中headers配置信息？ #8366
Replies: 3 comments 7 replies
-
const $ = cheerio.load(response.body);
const out = $(".title_item")
.map((i, e) => {
$(e).find(".access-text").remove();
return {
title: $(e).find(".hlFld-Title").text(),
link: $(e).find(".ref.nowrap").attr("href"),
pubDate: $(e).find(".open-access.item-access").text(),
};
})
.get();
console.log(out); |
Beta Was this translation helpful? Give feedback.
7 replies
-
|
是的是的，我刚发现后面这个括号问题了，我已经修正了； cookie的话，我是从抓包软件中获取到的，请问我能在路由中获取cookie吗？ 我考虑的是这个cookie是不是有时效性，担心以后提交了路由后别人一段时间后无法使用。 还是要谢谢您，我已经能正常获取到列表了！！！ |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
简化了 cookie，并获取到数据按照您提供的思路，我把从 const response = await got({
method: 'get',
url: 'https://asa.scitation.org/toc/jas/current',
headers: {
'Accept-Encoding': 'gzip, deflate, br',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36',
cookie: 'JSESSIONID=9e268139-aaaa-bbbb-cccc-ddddc1cc47ca; SERVER=abcdabcdabcdJx41gFJUAw==; MAID=abcdabcdabcdHd6ScOA3jQ==; MACHINE_LAST_SEEN=2021-10-09T19%3A41%3A12.540-07%3A00; I2KBRCK=1',
},
});偶然利用Chrome无痕模式访问链接，再用Fillder抓包，发现只需要在 我对 cookie 时效性的理解以及无效的尝试我对 cookie 时效性的理解我觉得网站会每隔一段时间重新派发一个新的时间戳，并且设置一个超期时间。 我尝试把 cookie: 'JSESSIONID=9e268139-aaaa-bbbb-cccc-ddddc1cc47ca; MAID=abcdabcdabcdHd6ScOA3jQ==;',但考虑到时效性，决定边使用边观察一下这个 无效的尝试为了避免时效问题，我查看到抓包软件中有一个内容好像和设置 GET https://asa.scitation.org/toc/jas/current?cookieSet=1 HTTP/1.1
Host: asa.scitation.org
Connection: keep-alive
Upgrade-Insecure-Requests: 1我利用如下代码，想获取与设置 const response = await got({
method: 'get',
url: 'https://asa.scitation.org/toc/jas/current?cookieSet=1',
headers: {
'Accept-Encoding': 'gzip, deflate, br',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36',
},
});既然上面这个方法失败了，那是否能通过 请问，在 |
Beta Was this translation helpful? Give feedback.
-
const res = await got("https://asa.scitation.org");
const cookie = res.headers["set-cookie"].join(" ");
const response = await got({
method: "get",
url: "https://asa.scitation.org/toc/jas/current",
headers: {
"Accept-Encoding": "gzip, deflate, br",
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
cookie,
},
}); |
Beta Was this translation helpful? Give feedback.
-
|
这也太好用了吧！服了！谢谢！现在用起来没什么问题了，我再调整一些内容，完善一下文档，准备提交上去！ 另外，作者部分我想写入您的名字——['Derekmini', 'auto-bot-ty']。 再次感谢您的帮助！ 刚才又按照您的思路搜索了一下，找到如下内容： headers: {
date: "Sun, 10 Oct 2021 06:30:56 GMT",
"content-type": "text/html; charset=UTF-8",
"transfer-encoding": "chunked",
connection: "close",
"x-xss-protection": "1; mode=block",
"x-content-type-options": "nosniff",
"strict-transport-security": "max-age=2592000",
"x-frame-options": "SAMEORIGIN",
"cache-control": "no-cache",
pragma: "no-cache",
"x-webstats-respid": "12b9d30826f70d895448561d622a6ee3",
"set-cookie": [
"JSESSIONID=abcd0f2e-abcd-abcd-abcd-abcda9e1f9c5; path=/; domain=.scitation.org; age=-1; HttpOnly; SameSite=None; Secure",
],
vary: "Accept-Encoding",
"cf-cache-status": "DYNAMIC",
"expect-ct": "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"",
server: "cloudflare",
"cf-ray": "69bdcd8bc9230c46-DFW",
"content-encoding": "br",
"alt-svc": "h3=\":443\"; ma=86400, h3-29=\":443\"; ma=86400, h3-28=\":443\"; ma=86400, h3-27=\":443\"; ma=86400",
},发现原来您是从这个地方获取的cookie，学习到了，谢谢！ |
Beta Was this translation helpful? Give feedback.
Answer selected by
Derekmini
-
|
Beta Was this translation helpful? Give feedback.
0 replies
-
 FreeOctober 2021 more... The Journal of the Acoustical Society of America 150, R7 (2021); https://doi.org/10.1121/10.0006522 |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
您好，想请教一个配置
got中header的写法：描述问题
我遇到一个网页：https://asa.scitation.org/toc/jas/current，想利用如下语句获取每篇文章的信息( 如
const list = $('section.card'))。出现问题的地方在：查看
response.data，发现其中的html内容并不完整，经查看不包含每篇文章的内容，导致不能使用const $ = cheerio.load(response.data);获取想要的信息。我只能暂时认为是
headers没有配置好，所以想通过抓包，写出一个有效的headers配置信息。现有抓包内容以及尝试改写的无效代码
现有的抓包内容
另附一张图片，我是通过Fillder抓取的：
Fillder抓取的内容
尝试改写的无效代码
其他尝试
之前通过
puppeteer也完成了想实现的路由功能，但是现在由于资源开销大等原因，路由工作不太正常，所以想换回推荐的获取方式。虽然接触的时间比较长了，但没有系统地学习过这方面的知识，所以还希望您能给一些提示，能帮我打开一下思路，谢谢您的帮助！
Beta Was this translation helpful? Give feedback.
All reactions