Scrape Data from a List of URLs by Creating a Simple ScraperThursday, August 10, 2017 10:13 AM
Web scraping can be done by creating a web crawler in Python. Before coding a Python-based crawler, you need to look into the source and get to know the structure of the target website. And of course, you need to learn Python. It will be much easier if you already know how to code. But for a tech noob, it’s very difficult to learn everything from scratch. So we create our app Octoparse to help people who know little to nothing about coding to easily scrape any web data.
In this tutorial, we will learn how to create the simplest and easiest web scraper to scrape a list of URLs, without any coding at all. This method is best suited to beginners like some of you. (We will assume that Octoparse is already installed on your computer. If that’s not the case, download here)
This tutorial will walk you through these steps:
1. Create a “Loop Item” in the workflow
After setting up basic information for your task, drag a “Loop Item” and drop it into the workflow designer.
2. Add a list of URLs into the created “Loop Item”
After creating a “Loop Item” in the workflow designer, add a list of URLs in “Loop Item” to create a pattern for navigating each webpage.
· Select “List of URLs” Loop mode under advanced options of “Loop Item” step
· Copy and paste all the URLs into “List of URLs” text box
· Click “OK” and then save the configuration
1) All the URLs should share a similar layout
2) Add no more than 20,000 URLs
3) You will need to manual copy and paste the URLs into “List of URLs” text box.
4) After entering all the URLs, “Go To Webpage” action will be automatically created in “Loop Item”.
3. Click to extract data points from one webpage
When the webpage is completely loaded, click the data point on the webpage to extract data you need.
· Click the data you need and select “Extract Data”(“Extract Data” action will be automatically created.)
4. Run the scraper set up
The scraper is now created. Run the task with either “Local Extraction” or “Cloud Extraction”.
In this tutorial, we run the scraper with “Cloud Extraction”.
· Click “Next” and then Click “Cloud Extraction” to run the scraper on the cloud platform
1) You are able to close the app or your computer when running the scraper with “Cloud Extraction”.
Just sit back and relax. Then come back for the data. No need to worry about Internet interruption or hardware limitation.
2) You can also run the scraper with “Local Extraction” (on your local machine).
5. Export data extracted
· Click “View Data” to check the data extracted
· Choose “Export Current Page” or “Export All” to export data extracted
1) Octoparse supports exporting data in Excel(2003), Excel(2007), CSV, or having data delivered to your database.
2) You can also create Octoparse APIs and access data. See more at http://advancedapi.octoparse.com/help
Now we've learned how to scrape data from a list of URLs by creating a simple scraper without any coding! Very easy right? Try it for yourself!
Demo data extracted below:
(I also attach the demo task and demo task exported in excel. Find them here)
Now check out similar case studies: