Scraping the Disney Store for Collectible Pins

The Problem Statement

My partner loves Disney Pins. They're beautiful little enamel pins designed to give one a warn fuzzy feeling when one looks at them.

She would often find herself opening the online store multiple times a day, wondering if a new pin had been listed or if a previosly listed pin had gone on sale. So I thought, what if we could automate the process and send her an email when a new pin was listed or an existing pin was updated? Let's get to work!

The Command

I use Laravel as my preferred backend for personal projects. First, we're going to create a command we can execute to grab all the currently listed pins from the store and notify our user of any updates:

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\Cache;
use Symfony\Component\BrowserKit\HttpBrowser;

class ScrapePins extends Command
{
	protected $signature = 'app:scrape-pins';

	public const PINS_CACHE_KEY = 'app.pins';

	public function handle(HttpBrowser $browser)
	{
		$previousPins = Cache::get(self::PINS_CACHE_KEY);

		$currentPins = $this->currentPins($browser);

		$newPins = collect($currentPins)
			->filter->isNew($previousPins)
			->values()
			->all();

		$updatedPins = collect($currentPins)
			->filter->isUpdated($previousPins)
			->values()
			->all();

		$this->notifyOfPinStoreUpdates(
			newPins: $newPins,
			updatedPins: $updatedPins,
		);
		
		Cache::put(self::PINS_CACHE_KEY, $currentPins);

		return self::SUCCESS;
	}

	...
}

In order to diff the current pin listing to find new and updated pins, we first retieve the previousPins from the Cache. We store this cache entry at the end of the script after we've processed the pin data.

Web Scraping

Next, we'll use Symfony's BrowserKit to traverse the markup in the currentPins method:

<?php

use App\Data\PinData;
use Symfony\Component\BrowserKit\HttpBrowser;
	
/**
 * @return \App\Data\PinData[]
 */
private function currentPins(HttpBrowser $browser): array
{
	$crawler = $browser->request('GET', vsprintf('%s%s', [
		config('app.disney_store_url'),
		config('app.pins_endpoint'),
	]));

	$pins = [];

	$crawler
		->filter('.product')
		->each(function (Crawler $node) use (&$pins) {
			$pins[] = PinData::fromNode($node);
		});

	return $pins;
}

Processing the Data

...And build a DTO from the result. We use a named constructor to contain the logic of converting the markup for each node into data:

<?php

namespace App\Data;
	
use Symfony\Component\DomCrawler\Crawler;

readonly class PinData
{
	public function __construct(
		public string $id,
		public string $title,
		public string $image,
		public string $link,
		public string $price,
		public string $basePrice,
		public bool $isLowInStock,
	) {}

	public static function fromNode(Crawler $node): self
	{
		return new self(
			id: $node
				->filter('.product-grid__tile')
				->attr('data-pid'),
			title: $node
				->filter('.product__tile_name')
				->text(),
			image: $node
				->filter('.product__tile_image')
				->attr('src'),
			link: $node
				->filter('.product__tile_link')
				->attr('href'),
			price: $node
				->filter('.price .sales')
				->text(),
			basePrice: $node
				->filter('.price .strike-through')
				->text(''),
			isLowInStock: filled($node
				->filter('.badge--lowstock')
				->text('')
			),
		);
	}

	...
}

Now we have the pin listings as data, we can work with them! Let's add accessors to our DTO to check if the pin is new or updated:

<?php

/**
 * @param  \App\Data\PinData[]  $previous
 */
public function isNew(array $previous): bool
{
	return collect($previous)
		->doesntContain(fn ($pin) => $this->id === $pin->id);
}

/**
 * @param  \App\Data\PinData[]  $previous
 */
public function isUpdated(array $previous): bool
{
	// A pin has been updated if it has the same 
	// ID, but a different name, price, or has 
	// become low in stock.
	return collect($previous)
		->contains(fn ($pin) => $this->id === $pin->id && (
			$this->title !== $pin->title
			|| $this->price !== $pin->price
			|| $this->isLowInStock !== $pin->isLowInStock
		));
}

...

Sending the Notification

All that's left is to Mail our recipients!

<?php
	
use Illuminate\Support\Facades\Mail;

/**
 * @param  \App\Data\PinData[]  $newPins
 * @param  \App\Data\PinData[]  $updatedPins
 */
public function notifyOfPinStoreUpdates(
	array $newPins,
	array $updatedPins,
): void {
	if (blank($newPins) && blank($updatedPins)) {
		// No pin updates :(
		
		return;
	}

	Mail::to(config('app.to_email'))
		->cc(config('app.cc_emails'))
		->send(new PinStoreUpdated(
			newPins: $newPins,
			updatedPins: $updatedPins,
		));
}

Conclusion

This command can be scheduled as often as we'd like using Laravel's task scheduling. Just like that - we're receiving emails whenever the store's listings change!

Note: I've left out formatting the mail in this article. You can read about Laravel's markdown mail formatting here!