Project

Purpose

The project aimed to develop an AI-powered copilot for e-commerce users to help them find the best products from Alibaba, Amazon, and others. However, it began with an Alibaba scraper. Due to high proxy costs, the client couldn’t afford it, and I had to discontinue the project. After completing the Alibaba scraper, I released it as open-source. This script can scrape the entire Alibaba site, a process that would take approximately 4-6 months.

Features

• Scrape data from Alibaba website
• Multi-threaded
• Save data to Amazon DynamoDB
• Proxy support
• Proper error handling and logging

Lessons Learned

Having worked on numerous scraping projects before, I didn’t encounter any new technical challenges. However, I learned the importance of communicating with clients about additional costs that may arise during a project.

Future Thoughts

I have released this project as open-source, and I welcome anyone to use and contribute to it. I would be delighted if someone finds it useful and decides to build upon it.