Introducing Scrapingdog API

Scrapingdog is a powerful web scraping API that enables you to extract data from any website on the internet. It supports a wide range of features such as dynamic rendering, rotating proxies, headless browsing, and HTML parsing. In this blog, we will cover some examples of how you can utilize Scrapingdog API in JavaScript.

Getting Started

Before we dive into the examples, we need to get our API key from Scrapingdog. To achieve that, follow these simple steps:

1. Go to https://www.scrapingdog.com/.
2. Click on the "Sign up" button.
3. Fill in the necessary details and proceed to sign up.
4. Once you are logged in, navigate to the dashboard and copy your API key.

Example 1: Scraping a static website

Let's start with the simplest example. In this example, we will extract the title of a static website using Scrapingdog API and display it in the console.

const apiKey = "YOUR_API_KEY_HERE";
const url = "https://example.com";
const headers = {
  "Content-Type": "application/json",
  "API-KEY": apiKey,
};
const body = JSON.stringify({ url });

fetch("https://api.scrapingdog.com/scrape", {
  method: "POST",
  headers,
  body,
})
  .then((response) => response.json())
  .then((data) => console.log(data.title));

Explanation:

1. We define apiKey, url and headers as constants to identify the API key, the URL to be scraped, and the headers for our request, respectively.
2. We create a body object which includes the url and converts to JSON format.
3. We send an HTTP POST request using the fetch method with the API endpoint and headers.
4. Once we receive the response, we extract the title field from the response object and display it in the console.

Example 2: Scraping a dynamic website

In this example, we will scrape data from a dynamic website that loads data dynamically via JavaScript. In order to do this, we need to use the dynamic rendering feature.

const apiKey = "YOUR_API_KEY_HERE";
const url = "https://example.com";
const headers = {
  "Content-Type": "application/json",
  "API-KEY": apiKey,
};
const body = JSON.stringify({ 
  url,
  render_js: true,
  wait: 5000, // wait 5 seconds for page to load
});

fetch("https://api.scrapingdog.com/scrape", {
  method: "POST",
  headers,
  body,
})
  .then((response) => response.json())
  .then((data) => console.log(data));

Explanation:

1. We define apiKey, url and headers as constants to identify the API key, the URL to be scraped, and the headers for our request, respectively.
2. We create a body object which includes the url, render_js set to true and wait for 5 seconds for the page to load.
3. We send an HTTP POST request using the fetch method with the API endpoint and headers.
4. Once we receive the response, we extract the necessary fields from the response object and display them in the console.

Example 3: Scraping data from a search engine

In this example, we will scrape data from a search engine and display the first 10 results.

const apiKey = "YOUR_API_KEY_HERE";
const url = "https://www.google.com/search?q=kittens";
const headers = {
  "Content-Type": "application/json",
  "API-KEY": apiKey,
};
const body = JSON.stringify({ 
  url,
  render_js: true,
});

fetch("https://api.scrapingdog.com/scrape", {
  method: "POST",
  headers,
  body,
})
  .then((response) => response.json())
  .then((data) => {
    const results = data.search_results;
    for (let i = 0; i < 10; i++) {
      console.log(results[i].title);
      console.log(results[i].link);
    }
  });

Explanation:

1. We define apiKey, url and headers as constants to identify the API key, the URL to be scraped, and the headers for our request, respectively.
2. We create a body object which includes the url and render_js set to true.
3. We send an HTTP POST request using the fetch method with the API endpoint and headers.
4. Once we receive the response, we extract the search_results field from the response object and loop through the first 10 elements to display the title and link fields in the console.

Conclusion

Scrapingdog API provides a straightforward and easy-to-use interface for website scraping. With its powerful features and support for JavaScript, you can easily extract data from any website on the internet. We hope these examples give you an idea of how to use Scrapingdog API, and we encourage you to explore its full potential. Happy scraping!