1. Install
  2. Sign in
  3. Scrape!

Join over 150,000 active users

  • Works on most popular sites
  • See results in seconds
  • Automate your data extraction
  • Live customer support

How to Write a Recipe

In 8 minutes you'll understand Recipe Creator basics, like hover & shift, selecting rows, selecting columns and Next Page Pagination.

Watch the videos or written step by step instructions below.

Watch The Row Selection Video:

Written Row Selector Instructions:

  1. Navigate to the site you want, launch Data Miner, click New Recipe and pick your page type.
  2. List pages require rows and have multiple pages, while detail pages only have one page and only require columns.
  3. Starting with a list page, navigate to the Rows tab, click the "Find" button and using your mouse hover over the website data until a highlighted box encloses all the information you are looking to scrape.
  4. Once the Row is highlighted, press shift on your keyboard, then over on the tool select one of the suggested classes or html elements to lock in the selection.
  5. Your goal is to get a green outline wrapping around all the individual row containers on your list page. Once this is done you can move onto columns.

Watch The Column Selection Video:

Written Column Selector Instructions:

  1. To begin selecting the individual data. Click on the Column tab and select “column 1”.
  2. Give the column a name, click the "Find" button, and hover & shift over the data you wish to extract into this column.
  3. Pick the class or HTML element that highlights the data the best. (Helpful tip - use the "Select Parent" button for more options if the selector outlines too many other items)
  4. Once the data is highlighted correctly, lock in the selector by clicking "confirm" at the bottom of Recipe Creator and click the Eyeball to check your work.
  5. Continue creating by clicking "Add new Column".
  6. Once you have all your columns done, finish by clicking the Save tab at the top. Give the recipe a name, click save and then run to scrape.

Connecting a List Recipe with a Detail Recipe:

Do you have a link or button on a list page that needs to be clicked or visited to scrape additional information?

This can be done with an Automated Job. Visit the link to learn more or watch the overview video below.

Written Instructions:

  1. Create a List Recipe to extract the URLs from a search results page.
  2. Create a Detail Recipe to extract the detailed info off the sub level detail page.
  3. Upload the list of URLs to Data Miner as a CSV.
  4. Create a Job using the above CSV as source Collection and the Detail Recipe as the Recipe.
  5. Make sure the Column Number for URL field is the same column number in the CSV.
  6. Give the Destination Collection a unique file name. This is where your data will save to.
  7. Click Save.
  8. Click Run.
  9. Visit the full tutorial here.
  10. *Keep the new tab open or drag to a new window for optimized scraping speeds.

Watch The Next Page Video:

  • Next Page Nav is not used to scrape sub-level detail pages. This process requires an Automated Job.

Written Next Page Instructions:

  1. Similar to Rows and Columns, click on the Nav Tab.
  2. Click the "Find" button to activate the find tool.
  3. Locate the next page button on the web page.
  4. Hover your mouse over the next page button and hit shift on your keyboard
  5. Select one of the suggested selectors and hit confirm.
  6. Click "Test Navigation" to see if it works.
  7. If it doesn't work, Recipe Creator needs the next page URL, which is found in the <a> element. Click parent until you find it and choose those selectors.
  8. *An advanced selector that works for most cases is a:contains(Next)

Nav with No Find (Xpath / Auto-generated Recipe):

  • Next Page Nav is not used to scrape sub-level detail pages. This process requires an Automated Job.

**Demonstrated with Data Miner 2017**

Written Next Page Instructions:

  1. Similar to Rows and Columns, click on the Nav Tab.
  2. Right click on page, select Inspect, select Elements in new window.
  3. Select the "arrow in a box" icon on the top left of the new window
  4. Locate the next page button on the web page and click it.
  5. Find the a element and identify the class associated with it.
  6. Now type .//a[@class="XXXX"] into your selector input box replacing the XXXX with your class.
  7. This selector will vary across sites, you may need to replace class with a different attribute. For example .//a[@title="Next""]
  8. *An advanced selector that works for most cases is .//a[contains(text(),"Next")]

Public Recipe Picture
Updated: 2/18/2019 by Ben
Can't find what you're looking for? Let us know what's missing! Contact Me