1. Install
  2. Sign in
  3. Scrape!

Join over 150,000 active users

  • Works on most popular sites
  • See results in seconds
  • Automate your data extraction
  • Live customer support

Creating the Automated Job

An Automated Job is a two-step process of scraping layered web pages. It utilizes a List Recipe and a Detail Recipe. Watch the tutorial video or read the step by step written instructions below.

Jobs Tutorial Video:

Part One Video

The List recipe will extract detail page URLs from a search results or list page. These URLs will be uploaded to Data Miner. Once the Job is created, Data Miner will automatically open the URLs one by one in your browser, apply the detail recipe and scrape the detail page data.

This data will be accumulated into a single CSV, which will be saved to the Data Collections folder.

Step By Step Instructions:

1) Starting from a list page, find the data you want to scrape, extract the information using a public recipe as you did in Lesson 1. This recipe will need to extract the list information as well as the URLs of individual detail pages.

Data Miner Pop Up

2) Download the results by clicking “download” in the top right corner. You will need to download it as a CSV to perform a Job.

Download

3) Navigate to your data collections folder. To get to the data collections folder and jobs page, just click the collections icon in the nav bar on the left hand side of the Data Miner window.

Collections

4) Import the CSV containing the URLs. Click, “Import a csv” and select the CSV file from the first scrape.

Import CSV

5) Once the CSV is uploaded, it’s time to create the Job. To create a Job, click on the Jobs tab from the left hand panel. Begin filling out the necessary fields.

  1. Job Name - Name the job what the scrape will accomplish.
  2. Recipe name - will be the Detail Page recipe. The recipe used to scrape detailed information for every individual URL.
  3. Source Collection - Select the CSV containing the URLs that you uploaded to Data Collections.
  4. Column # for URL - The column number where your URL is located in the CSV file. In this example and in most cases it is 1.
  5. Destination Collection - The final output file, name it what you scraped.
  6. Once all the feilds are filled out, press Save
Jobs

6) Once the Job is saved, it will appear at the top of the Jobs page. Press Run.

7) The first URL will be opened up in a new tab. Data Miner will scrape the information and then move onto the next URL.

8) As the recipe runs, you can check the progress by visiting your Data Collections and then clicking on the output file that you named earlier. If you have scraped all the data that you need, click stop/close on the pop up window or wait till the Job reaches the end of the URLs and it will stop automatically.

View Output

9) Once finished, click on Data Collections, select the output file and then download by selecting your file preference Excel(XLS) or CSV.

Download Output
Public Recipe Picture
Updated: 5/2/2018 by Ben
Can't find what you're looking for? Let us know what's missing! Contact Me