Clash Royale CLAN TAG#URR8PPP

Node.JS: How to scrape a json page for specific data

I would like to scrape this page: calendar events

for specific data, like formattedDate and description. How do I go about that in a module in Node.JS. I am having a hard time understanding the process in Node.JS.

Any help would go a long way, thanks in advance.

2 Answers
2

it's pretty simple, you can import the request module and use it. For example, see code below.

const request = require("request");

request("MY_URL", (error, response, body) =>
console.log('body:', body);
);


Also, you can try this here, on Repl.it

First of all, you need to parse your JSON, this allows you to access fields from received json.

const data = JSON.parse(body);


Now, if you want to access some information about an event you need to loop events and access what you need, something like:

const events = data.bwEventList.events;
events.map((data, index) => console.log(data.calendar))


Final code also on Repl.it

from nodeJS docs here

const http = require('http');

http.get('http://umd.bwcs-hosting.com/feeder/main/eventsFeed.do?f=y&sort=dtstart.utc:asc&fexpr=(categories.href!=%22/public/.bedework/categories/sys/Ongoing%22%20and%20categories.href!=%22/public/.bedework/categories/Campus%20Bulletin%20Board%22)%20and%20(entity_type=%22event%22%7Centity_type=%22todo%22)&skinName=list-json&count=30', (res) =>
const statusCode = res;
const contentType = res.headers['content-type'];

let error;
if (statusCode !== 200)
error = new Error('Request Failed.n' +
`Status Code: $statusCode`);

if (error)
console.error(error.message);
// consume response data to free up memory
res.resume();
return;


res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => rawData += chunk; );
res.on('end', () =>
try
const parsedData = JSON.parse(rawData);
console.log(parsedData["bwEventList"]["resultSize"]);
catch (e)
console.error(e.message);

);
).on('error', (e) =>
console.error(`Got error: $e.message`);
);


see console.log(parsedData["bwEventList"]["resultSize"]);
slice parsedData as an array until you get what you want

console.log(parsedData["bwEventList"]["resultSize"]);

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.