Clash Royale CLAN TAG#URR8PPP

NodeJS - FOREACH function does not run all of elements and stop

I have stored 2775 urls in my mlab database and then I take each URL down to get more information. All of the URL I store in an Array then pass it into a function to process .However, The code only run up to about 1700 urls and process it and then stop. Here is my code (sorry about the code, this is my first time using stackoverflow :

Product.find(, (err, foundProducts) =>
if (err)
console.log("err " + err);
else
foundProducts.forEach(function(foundProduct)
var updateProduct = service.updateTikiProduct(foundProduct.url);
);

);

updateTikiProduct: function(url) {
const options =
url: url,
json: true
;
request(options,
function(err, res, body)
// SOME code to crawl data

Product.findOneAndUpdate(
url: options.url
,
$set:
name: name,
brand: brand,
store: store,
location: location,
base_category: categoryType,
top_description: topDescription,
feature_description: featureDescription

,
upsert: true,
new: true
, (err, createdProduct) =>
if (err)
reject(err);
else
var currentDate = new Date();

if (!createdProduct.hasOwnProperty("price"))
createdProduct.price.push(
current: currentPrice,
origin: originPrice
);
createdProduct.save();
else if (createdProduct.hasOwnProperty("price") &&
createdProduct.price[0].date.getDate() != currentDate.getDate())
createdProduct.price.push(
current: currentPrice,
origin: originPrice
);
createdProduct.save();
console.log("Update price");

counter++;
console.log("url : " + options.url);
console.log("Created product " + counter + " success!");

);



2 Answers
2

I think your code is not processing all the elements is because you are processing all the elements in parallel, which will stop processing at one time when the memory will get full.

foundProducts.forEach(function(foundProduct)
var updateProduct = service.updateTikiProduct(foundProduct.url);


);

what you should do is process them in series. you can use async await for that, do the following changes it will work :-

for(let foundProduct of foundProducts)
var updateProduct = await
service.updateTikiProduct(foundProduct.url);
;


for..of

foreach

try...catch

async

Product.find(, async (err, foundProducts) => {

i guess mongo have limits to get items from db, you should try findAll or https://stackoverflow.com/a/3705615/4187058

foundProducts.length

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.