Skip to content
Minifetch

Minifetch

monetizing an open-source software module

Created on 31st May 2025

•

Minifetch

Minifetch

monetizing an open-source software module

The problem Minifetch solves

Many developers find the open source NPM module I created & upgraded over the last 9 years useful for fetching metadata from web pages. It has enjoyed increased popularity in the last couple of years. Working with the module has a barrier to entry though: dev skills. Even if the user has dev skills, they may not want to spin up a whole pipeline just for scraping metadata and may find a website, MCP tool, or API client a more approachable form factor.

Some years ago there existed websites that offered this service for free and were frequently used by SEO researchers (for example: comparing product pages). These sites were monetized by banner ads but now that the banner ad revenue has collapsed these websites have largely disappeared, or the functionality has been pared down drastically so it isn't useful for research (tools may only display twitter and open graph (og:) cards now).

These metadata webpage calls are now also useful for AI research because the metadata format helps LLMs parse pages more efficiently. This may be part of the reason for the recent surge in popularity, but I'm not entirely certain because my only source of feedback so far has been Github issues & pull requests.

I do know that over the years I've had to turn many potential users away because they just weren't quite technical enough. There is a small community of users from all around the world that have contributed to this module over the years. Making the module more accessible like this could be the key to expanding the user-base exponentially.

✅ Solution:
We can lower the barrier to entry:

  • Create an API that anyone can use
  • Make the API accessible via a web frontend & MCP tools for AI assistants
  • Make the hosted versions cheap to use, and easy to pay-as-you-go with micropayments thru Coinbase's x402 protocol

👷 MVP Scope (after the Hackathon Prototype)

  • Metadata retrieval by API that respects robots.txt & enforces rate-limits per website
  • Web frontend that queries this API at https://minifetch.com
  • API client for querying the API faster than web frontend
  • Claude AI chat integration via MCP tools: https://github.com/modelcontextprotocol
  • NO account creation or payments w traditional rails
  • Instead we use USDC crypto micropayments via x402 protocol on Coinbase's Base network

The x402 micropayment protocol could be the solution for microservices like this to become sustainable again. The boundary between "free" and "paid" will be simple: if you use this software on your own machine, you get the service for free (ex: local MCP tool with Claude Desktop). If you use this software on our machines, you pay the x402 fee (ex: web frontend, our hosted API or centralized MCP server).

Challenges I ran into

🚀 Wins

  • x402 payment integration was straightforward & easy both on the web frontend side and with x402 example client in Github
  • Obtaining Base Sepolia testnet funds was easy via Coinbase developer portal

🚧 Challenges

  • 🛑 Getting even small funds (<$3) into a production wallet on Base from Coinbase was made difficult by platform delays & took over a week, a sub-optimal UX for the micropayments use-case. Perhaps a "fast-lane" for sub-$5-10 USDC onboarding transactions is in order.
  • It was not initially clear from the code examples or docs that CDP wallets were incompatible with the x402 client example which requires a private key. CDP wallets cannot export a private key by design but i spent a couple frustrating hours (& apparently so did other devs in the Discord) trying to figure out why i couldn't export a private key!
  • It doesn't appear that POST data is available in x402, just GET. Which is fine for this use-case for now.

🚧 Future Challenges

  • This tool could get jailed or banned by websites if users abuse it

    • Mitigations:
      • Micropayments act as a spam deterrent
      • Identify our user agent clearly for website operators
      • Fetch, cache, & obey robots.txt and advertise this fact freely
      • Rate limiting for each website scraped
      • Cache results for a few mins/hours
      • Create a feedback form & FAQ just for website operators
  • One predictable outcome is website/ content/ ip owners in the future may want a cut for having their content scraped for metadata; would be interesting to create a supply chain of micropayments for premium and/or paywalled content!

Tracks Applied (1)

Best Use of x402pay

Discussion

Builders also viewed

See more projects on Devfolio