How to Scrape Internet using Laravel Artisan Command

Web scraping is a technique for automatically extracting data from websites. It offers a streamlined way to gather information for analysis, research, or content aggregation. This method eliminates the need for manual data collection, making it invaluable for anyone needing to compile and analyze large amounts of data efficiently. This guide will explore how to use Laravel, a popular PHP framework, and Symfony’s DomCrawler component to scrape data from websites. We’ll provide clear, step-by-step instructions tailored to beginners, helping you easily set up and run web scraping tasks. Following this tutorial, you’ll learn to extract data from any URL and store it in your database, empowering you to harness web data effectively.

Why Use Web Scraping?

Web scraping offers several benefits:

• Automated Data Collection: Saves time by automatically gathering data, eliminating the need for manual entry.
• Real-Time Data: Keeps your information current by retrieving data directly from websites.
• Data Aggregation: Collects and combines data from various sources for a more complete view.
• Market Analysis: Helps monitor competitors and analyze market trends.
• Content Management: Automatically updates your site with fresh content.

Prerequisites

Before you begin, ensure you have the following:

Laravel Framework: A Laravel project is needed. If you don’t have one, create it by following these steps:

Open your terminal or command prompt. Navigate to the directory where you want to create your project. Run the following command:

composer create-project --prefer-dist laravel/laravel your-project-name

Symfony DomCrawler Component: This component helps in parsing HTML data. Install it using Composer:

In your terminal, ensure you are in your Laravel project directory. Run:

composer require symfony/dom-crawler

Basic Understanding of HTML: Familiarity with HTML will help you identify the parts of the webpage you want to scrape.

Database Setup: Make sure you have a database configured. Laravel’s migration system will assist in setting up the necessary tables.

Creating a scraping Artisan Command

Artisan command is a CLI command that you can create with Laravel syntax and logic, within your application. It allows you to perform various background tasks and artisan commands also run very predictably and stable if ran as a queue workers too.

We will use it to create a command for scraping data. To create a command, follow the steps given below.

Step 1: Open the Terminal

Step 2: Execute the following command to create a new Artisan command.

php artisan make:command ScrapeDataCommand

Locate the File: The command will create a new file in app/Console/Commands/ScrapeDataCommand.php.

Now, we will implement the logic for scraping data from a website.

Step 3: Navigate to app/Console/Commands/ScrapeDataCommand.php in your code editor.

Step 4: Replace the content of the file with the following code:

// app/Console/Commands/ScrapeDataCommand.php
namespace App\Console\Commands;
use Illuminate\Console\Command;
use Symfony\Component\DomCrawler\Crawler;
use Illuminate\Support\Facades\Http;
use App\Models\ScrapedData; // Ensure this model exists
class ScrapeDataCommand extends Command
{
    protected $signature = 'scrape:data {url}';
    protected $description = 'Scrape data from a given URL and store it in the database';
    public function __construct()
    {
        parent::__construct();
    }
    public function handle()
    {
        $url = $this->argument('url');
        $response = Http::get($url);
        if ($response->successful()) {
            $html = $response->body();
            $crawler = new Crawler($html);
            $this->scrapeData($crawler);
        } else {
            $this->error('Failed to fetch the URL. Please check the URL and try again.');
        }
    }
    private function scrapeData(Crawler $crawler)
    {
        $scrapedData = [];
        // Adjust these selectors to match the HTML structure of the target website
        $crawler->filter('.item-class')->each(function (Crawler $node) use (&$scrapedData) {
            $title = $node->filter('.title-class')->text();
            $description = $node->filter('.description-class')->text();
            $author = $node->filter('.author-class')->text();
            $scrapedData[] = [
                'title' => trim($title),
                'description' => trim($description),
                'author' => trim($author),
                'created_at' => now(),
                'updated_at' => now(),
            ];
        });
        if (!empty($scrapedData)) {
            ScrapedData::insert($scrapedData);
            $this->info('Data scraping and storage completed successfully.');
        } else {
            $this->error('No data found to scrape.');
        }
    }
}

To store the scraped data, you need a database table. We use Models and Migrations to work with the database in Laravel. So, we will create a Model with migration in this case.

Step 5: In your terminal, run: php artisan make:model ScrapedData -m

This command creates a model and a migration file.

Step 6: Open database/migrations/xxxx_xx_xx_create_scraped_data_table.php in your code editor. Update the file to define the table structure:

// database/migrations/xxxx_xx_xx_create_scraped_data_table.php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
class CreateScrapedDataTable extends Migration
{
    public function up()
    {
        Schema::create('scraped_data', function (Blueprint $table) {
            $table->id();
            $table->string('title');
            $table->text('description')->nullable();
            $table->string('author')->nullable();
            $table->timestamps();
        });
    }
    public function down()
    {
        Schema::dropIfExists('scraped_data');
    }
}

Step 7: Run the Migration

In your terminal, execute: php artisan migrate

Step 8: Run the Scraping Command

Now, you can run the command to scrape data from a website.

• Execute the Command: In your terminal, run: php artisan scrape:data "https://example.com"
  Replace "https://example.com" with the URL of the website you want to scrape.

Further Scaling the Scrapping Logic

Laravel artisan commands are a very powerful tool in Laravel when it comes to scraping automation. In this case, we are scraping a single url. However, you can further scale the same logic with the following ideas.

1. Recursive Scraping: You can create a function in the logic of this command to get all the links from the same/different/specific domain and queue them for further scraping with the same command.
2. Parallel Processing: With the help of Supervisor, Laravel queues, and some logic update in the command, You can also create multiple processes that will pick a task from queue and scrape a specific page. In this case, The above given function to get all the links from the page will also queue the newly found links for scraping.
3. Optimisation: Usually, Websites having rich data to scrape will use multiple/random selectors in different pages. As you start scraping the pages and see the data you are getting, You can further optimise the command to be more efficient.

These are just a few basic improvements that can be made on this scraping application.

Conclusion

This guide covered using Laravel and Symfony’s DomCrawler to scrape website data. Following these steps, you can automate data collection and store it efficiently in your database. Here are some key takeaways from the guide.

• Flexibility: Adjust the scraping logic for various website structures and data formats.
• Automation: Use Laravel commands to automate and streamline data collection.
• Adaptability: Customize the scraping logic and database schema to fit your needs.

With the knowledge gained from this guide, you can start experimenting with different websites and data types. Web scraping is a powerful tool for gathering and utilizing web data.