Curl is a way you can hit a URL from your code to get an HTML response from it. In NodeJS/ExpressJS we use “request” module for this purpose.

In Simple Words, if we want to access HTML content of any site we pass that URL to request( ) and it returns the HTML source code.

By far out of all the modules ‘Request‘ module is the most popular npm package to make HTTP request. Actually, it is really just a wrapper around Node’s built-in HTTP module, so you can achieve all of the same functionality on your own with HTTP, but request just makes it a whole lot easier.

– In this tutorial, we will scrape data from a webpage using request module and store it in a file(here HTML file).

  1.  Install ‘request’ module in your project folder using following command:- 
    npm install --save request
  2.  We then create a file called server.js and write a basic code for creating a server. 
    //requiring express module
var express = require("express");
//creating express instance
var app = express();

app.get("/", function(req, res) {
 console.log("hello user");
});

app.listen(3000, function() {
 console.log("Node server is running..");
});
  3.  To use Request module we first have to require this module like this:- 
    var request = require("request");
  4.  Request method takes two parameters first the URL to be scrapped and second a callback function. Here I will take “https://www.google.com” as URL. Then in the above code in server.js file, we will call request method.The syntax is as follows:-
    request(url,function(){
    // actions to perform
    });
  5.  Here is the final code,In this, we have made a request call to  “https://www.google.com” and in a callback function, it will return three parameters a ‘response’ variable, an ‘error’ if any error occurs it will get stored in error variable , else the HTML response gets stored in ‘body’ variable. 
    var express = require("express"); //requiring express module
var app = express(); //creating express instance
var fs = require("fs"); //require fs(file system) module
var request = require("request"); //requiring request module

app.get("/", function(req, res) {
 //calling request function
 request("https://www.google.com", function(error, response, body) {
   if (!error && response.statusCode == 200) {
     // writing the response to a file named data.html
     fs.writeFileSync("data.html", body);
   }
 });
});

app.listen(3000, function() {
 console.log("Node server is running..");
});
  6.  Run the above code and in terminal/cmd and in the browser go to http://localhost:3000.As soon as you look in your project directory you will see an HTML file named data open it and your HTML response is there in the file.

Note– In the above code we have to require fs module so that we can create a file and save the HTML response we got from request.

Learn More: