Why I am Building IndieWeb Search

I recently entertained a question about why I decided to turn what was a search engine for my blog into an IndieWeb Search engine. My motive was not clear until someone shared an article with me written by jpreston.xyz. In this article, jpreston.xyz explains why they had embarked on a similar journey to build a search engine. Quoting from their article:

Other times, we have our preferred sources and we wish to consult those websites first, because we appreciate the style or perspective of the creators who publish there. For example, if I want to find a recipe for quesedillas, I can search ‘the whole Web’ for inspiration, but if I want to make vegan quesedillas, I would prefer to restrict my search to ‘myWeb’, the websites whose vegan recipes I’ve enjoyed in the past...

In this article I'm going to explain a bit more about why I expanded that search engine into an open-source community project that indexes multiple sites owned by members of the IndieWeb community.

Building a search engine for the IndieWeb

Personal websites have given me a lot of value in the last year. I sometimes peruse websites to seek inspiration for what I could do to my site. I sometimes read blog posts to learn about projects people are working. I sometimes "surf the web" in the traditional sense of the term, seeking new content to read, new blogs to which I can subscribe, or pretty websites to look at. Browsing personal websites is fun to me.

But finding personal websites to browse, and content that addresses particular topics on personal websites, is difficult.

I sometimes do Google searches for specific terms but for the most part search results do not meet my intent. I adjusted my behaviour to instead use the IndieWeb wiki when I wanted to learn about something related to personal websites. Failing that, I asked someone in the IndieWeb community chat (i.e. in the case of when I wanted to get thoughts on the history of Facebook's "poking" feature). The wiki and chat were great because I was able to find content or engage with people who knew about topics I was interested in. I wanted to improve content discovery to make it easy to find content that might interest me on topics related to personal websites and web development.

There are so many bloggers out there writing about the IndieWeb or personal sites or web development, among other topics. I wanted to create a programmatic way to aggregate that content so I could browse it at any time and search for a particular term in which I was interested. In other words, I wanted to build a search interface for personal websites.

Thus, IndieWeb Search is a discoverability mechanism. You can use the search engine to find interesting content published by IndieWeb community members and other people who have personal websites.

Surfacing content of personal interest

Going back to the point made by jpreston.xyz about having preferred sources. The IndieWeb community produces so much content that I find interesting but I can only scratch the surface by looking through sites manually. I am bound to miss something. I miss lots. Many sites are too big for me to look at and it is impossible to remember everything someone writes about on their site.

With a search engine, all of the content from IndieWeb sites is aggregated and only a search away. I have found new posts about blogrolls and even coffee roasting that I probably otherwise would not have discovered. Want to know what IndieWeb community members have to say on ARIA roles or h-card microformats or using Rust to build something? You can do so by running a search.

Wrapping up

I expect IndieWeb Search will be rough around the edges for some time because building a search engine for multiple sites is a lot harder than building a search engine for one site. For context, IndieWeb Search was based on the search engine I wrote just for my blog. Indexing content takes time and I still have not quite found an optimal way to rank searches. With that said, I am learning a lot and I have enjoyed the experience of building a search engine.

IndieWeb Search aggregates content from IndieWeb authors. I hope the community uses it as a tool to find interesting content. In any case, I use it to find content, even if that content is not exactly related to web development. I understand that our community has a diverse range of interests, hence why I decided to search for "coffee roasting." And through that search I found articles I otherwise would not have stumbled upon.

If you're ever in the mood to explore some independent websites, I'd love for you to check out the search engine at indieweb-search.jamesg.blog.

Also posted on IndieNews.

Tagged in search engines.