How to read a large CSV file in Node.js
Today, we will show you how to read a large CSV file in Node.js. In this article, we will create an example to read a large CSV file using csv-parse.
Checkout more articles on Node.js
Here, we will use the previously generated large CSV file. Refer the following article for more details.
How to write a large CSV file in Node.js
Solution
Use the following code to read the large CSV file using Node.js. Here we have used the fs.createReadStream along with the csv-parse npm package to read a CSV file.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | var fs = require('fs'); var { parse } = require('csv-parse'); // ... // ... const fileName = "product_10L.csv"; fs.createReadStream(fileName) .pipe(parse({ delimiter: ",", from_line: 1 })) .on("data", function (row) { console.log('Row: ', row); }).on("end", function () { console.log('Process Completed.'); }).on("error", function (error) { console.error(error.message); }); |
In the above code, we are trying to read product CSV file that contains the product name and product SKU. You can process the data immediately after reading.
API Example
Your server.js file should look like below if you add this logic to the rest API. In this code we read the CSV file from the root level.
Refer this article to setup REST API in node.js.
Create REST API in Node.js
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | var express = require('express'), app = express(), port = process.env.PORT || 4000; var fs = require('fs'); var { parse } = require('csv-parse'); app.get('/read-csv', async (req, res) => { const fileName = "product_10L.csv"; fs.createReadStream(fileName) .pipe(parse({ delimiter: ",", from_line: 1 })) .on("data", function (row) { console.log('Row: ', row); }).on("end", function () { console.log('Process Completed.'); res.send({ success: true }); }).on("error", function (error) { console.error(error.message); res.send({ error: true }); }); }); app.listen(port, () => { console.log('Server started on: ' + port); }); |
Here, we have logged the row data on data listener. Also, we returned success: true message at the end of the process and added the listener to catch the error.
Run the following command to read the CSV.
1 | http://localhost:4000/read-csv |
That’s it for today.
Thank you for reading. Happy Coding..!! 🙂
