I had a plan. A grand, romantic, and deeply nerdy plan. My wife, Sabrin, is a shopaholic in the best way possible. She has a talent for finding incredible deals, especially from foreign brands on sites like Amazon, AliExpress, Lazada and Shopee etc.. I wanted to build her the ultimate gift: a personal AI agent, the “Sabrin Deal Finder,” that would automatically hunt for the top 10 “girly” deals every day and display them on a beautiful, custom-built website.
I imagined her waking up, checking the site, and seeing a curated list of trendy sunglasses and Korean skincare sets. It was going to be amazing.
After weeks of struggle, debugging, and late nights, I finally launched the agent. I checked the live website, my heart pounding with anticipation. And there it was. The fruit of all my labor. The agent had successfully scraped its first deals.
They were for soyabean oil and toilet cleaner from Chaldal.
This is the story of that glorious failure. It’s a case study in what it *really* takes to build a real-world AI agent, the brutal challenges of web scraping, and the valuable lessons learned when your ambitious project face-plants into reality.
Sabrin Deal Finder PoC link.
The Grand Plan: A Global, Autonomous AI Agent
The idea was solid. I would build a three-part agent:
- The Hunter: A Python script using Selenium to control a real browser, capable of bypassing basic security and reading complex websites.
- The Analyst: A smart system using BeautifulSoup (and sometimes AI) to parse the messy website code and extract clean, structured data: product name, price, and link.
- The Reporter: An automated pipeline to save this data to a Supabase database and display it on a live Next.js website, hosted on Vercel.
The final piece was to set it all on a daily schedule with GitHub Actions. It was a professional stack for a professional-grade gift.
The Reality: Fighting a War on a Thousand Fronts
The moment my agent stepped onto the battlefield of the live internet, it was met with overwhelming force. The fortresses of Amazon, AliExpress, and Shopee were not going to give up their data easily.
Problem #1: The Anti-Bot Fortress
My agent’s first attempts were immediately shut down. These sites are experts at identifying and blocking automated browsers. Even with my most advanced “stealth” settings, the sites would often serve up a different, simplified version of the page, or just a block of confusing JavaScript. My agent was seeing a “welcome mat” but was never invited inside the house.
Problem #2: The Ever-Changing Labyrinth
For the few sites I *could* access, the battle was just beginning. A web scraper works by having a “map” of the page’s HTML structure—it looks for a product title inside a `
` with a specific class name. But these e-commerce giants deliberately change their class names, sometimes daily, to break scrapers. My agent’s map was constantly out of date. It would go to the right street but find the house number had changed.
Problem #3: The Honest but Empty Report
After weeks of tuning, I finally launched the full global mission. The agent ran without crashing—a victory in itself! It dutifully visited every site, tried its best, and reported its findings to the database. The result was this beautiful, heartbreakingly honest table:
As you can see, the agent was a roaring success at scraping Chaldal. For everything else—the sites Sabrin actually cares about—it brought back nothing but honest failure reports. The Hunter was working, but its prey was too elusive.
The Final, Comical Result
My Vercel website, which I had spent so long debugging, worked perfectly. It connected to the database and displayed the results of the agent’s hard work. It proudly presented a list of deals… exclusively for groceries. The “Sabrin Deal Finder” was, for all intents and purposes, a Chaldal bot. It was a beautiful, functional, and utterly useless gift for my wife.
What This “Failure” Taught Me: A Case Study
This project failed to meet its primary objective, but it was the most valuable learning experience I’ve ever had. Here’s what I learned:
- Web Scraping is a Battle of Attrition: Building a scraper is easy. Maintaining it is a full-time job. The web is not a static library; it is a living, hostile environment that actively fights back.
- An Agent is Only as Good as Its Senses: My AI Analyst and Database Reporter were perfect. But if the Hunter at the front lines can’t gather good intelligence, the entire system is useless. The “Perception” part of the loop is the most fragile and most critical.
- The Professional Solution is Different: I was trying to build my own professional hunter from scratch. In reality, large-scale scraping operations use paid services like ScraperAPI’s “Structured Data Endpoints” or residential proxies to get around the blockades. The real pros hire mercenaries.
- Failure is the Best Teacher: Every single error, from the `403 Forbidden` to the `ModuleNotFoundError`, taught me something new. Success teaches you one path; failure teaches you a hundred.
The Road Ahead: I’m Not Giving Up
So, is the “Sabrin Deal Finder” a failed project? For now, yes. But it’s not a dead one. The foundation is built. The pipeline works. Now, the real work begins: iteratively improving the Hunter, one target at a time. I’ll need to dive deeper into the professional tools and maybe even use AI to *understand* a page’s structure instead of just following a map.
For now, my wife knows I tried to build her a high-tech shopping assistant. She also knows that if she ever needs a fantastic deal on soyabean oil, my agent is the best in the world. And in a way, that’s a kind of success, too.