Building my own webmention receiver

A couple of months ago, I started to work on my own webmention receiver. Instead of relying on webmention.io, a service commonly used to receive webmentions, I wanted to challenge myself to create a receiver from scratch. To do so, I realised I would have to read, understand, and interpret a W3C specification, something I had not done up until the point of deciding to work on a webmention receiver. While I knew there would be challenges, I wanted to take control over how I received webmentions and build my own service.

In this post I am going to share why webmentions are important to me and the high-level architecture behind my receiver. If you want to see the code behind the receiver, you can do so by checking out my webmention-receiver GitHub repository.

Why webmentions are important to me

First, a TL;DR on webmentions:

Webmentions are a way of communicating information across websites. A webmention is created on someone's own site. That webmention is then "sent" to another site using a web request. A receiver is necessary to tell a site when a webmention has been sent to a page on that site. The site that received the webmention may choose to display it on their site, usually in a "reactions" or "comments" section.

With this in mind, I can now share why webmentions are important to me. I like being able to receive webmentions because they let people communicate across their own personal websites. You can send a webmentions to me that is hosted on your own site. You can share that webmentions with others on your own site if you want. Or you can keep it private on your own site. I may choose to show your webmentions on my site. I do not support this at the moment but many others do. The nice thing about showing webmentions is you can create your own "reactions" section with supported webmention reactions that you have received (e.g. likes, replies to your article, RSVPs).

The key takeaway from this is webmentions let you communicate in public while owning your data the whole time. I like the principle of webmentions. They let me share my thoughts on cool blog articles without having to send an email or find other contact information. My webmention may even appear as a comment or reaction on someone's site. How cool is that? Something I created on my site could appear as a comment on someone else's site.

Anyway, enough of my talking up webmentions. Now to what I really want to discuss: building my own webmention receiver.

Why I built my own webmention receiver

Above all else, I built my own webmention receiver as a challenge, as I mentioned earlier. I had not worked on a server-side site for a while so I wanted to try to see if I could build one that met all of the W3C webmention specifications. Building my own receiver also made me more aware of how webmentions work. While I would not consider myself an expert on the subject, reading the specification and trying to translate it into code gave me a keen understanding of the inner workings of webmentions that are usually abstracted by services like webmention.io.

The evolution of my receiver

I am not going to repeat W3C specifications. If you want to know what you need to do to build a webmention receiver, you should consult the official specification. What I do want to share is how my webmention receiver evolved.

My first goal was to support receiving webmentions synchronously. While the specification says you should process webmentions asynchronously—an important security measure to prevent denial of service attacks—this was another hurdle that I wanted to revisit later. I chose Python and Flask for building my back-end because I know these two technologies well. To support receiving webmentions synchronously, I worked my way through the webmention specification and implemented all of the necessary checks, from making sure the webmention target and source are not equal to verifying that the source is a valid HTML resource that does not redirect too many times or take too long to load.

This was the easy part. I then had to work out how to process webmentions asynchronously. I played around with a number of options—from thinking about using a cloud service to looking at Python's asyncio library—but the one that I decided was best was cron. cron is built in to UNIX operating systems, is really easy to use, and would require very little overhead. At the time, I didn't realise I could use cron for asynchronous tasks. I thought I had to use a Python library. It wasn't until I spoke with many members of the IndieWeb that I realised cron was indeed asynchronous. I knew how cron worked. I had just never thought it could be used for asynchronous processing.

In line with the webmention specification, I created a new validate_webmentions.py script that I would schedule to be triggered by cron (at the moment I run it every hour on my webmention receiver server). This script contains validation like making sure the source and target is valid and that the source webmention contains a link to the target webmention. Basically everything that involves making a GET request to the source happens after in the validate_webmentions.py script that I trigger with cron.

Once I had the receiver component working, I built a basic JSON API with Flask that let me see webmentions I had received. Then I got to work on building a web interface. While the JSON interface was useful, my main need for this application was to see what webmentions I had received rather than to use them programmatically (which is a more appropriate use for JSON). With that said, I kept my JSON code for later because I still wanted to support getting my webmentions in JSON. Who knows, maybe I will want programmatic access to my webmentions. The first iteration of the web interface listed all of my webmentions using the same stylesheet that I use for this blog and had a simple navigation bar.

Then came authentication. For an application like this to be secure in production, I needed to implement some kind of authentication. I thought about an API key but then realised that a login would be more appropriate. Using Flask-Login and this excellent tutorial from DigitalOcean, I developed a simple login system that let me access all of my webmentions.

In short, the receiver:

1. Is capable of receiving webmentions.
2. Has a dashboard for seeing webmentions I have received.
3. Requires authentication to see the dashboard.

After a lot of iteration, here is how the dashboard turned out:

You may notice in the navigation bar that I also support sending webmentions. This is true. I built a form for sending a webmention and also for seeing webmentions I had sent. This is a replacement for the Telegraph service that I was previously using for sending webmentions. I thought to myself that I may as well support sending webmentions from this receiver application. So technically the receiver is also a webmention sending interface and a receiver. There have been many other evolutions like adding pagination and pages to see individual webmentions sent.

Wrapping up

I have been using my webmention dashboard for a couple of weeks now. In that time, I have not received many webmentions. I'd appreciate it if you could send me a webmention in response to this post if you support them. I love receiving webmentions. But although I haven't received many webmentions, I am happy with the interface I have for exploring them. And I now feel more confident in talking about how webmentions work.

If you are new to webmentions, I'd highly recommend trying out a service like webmention.io. I used it for months. As I found out, building a webmention receiver isn't something you can do in a day. It takes time and careful design. If you do decide to embark on the path of building your own webmention receiver, let me know. I'd be happy to share some of the more technical details around building a webmention receiver that I had to consider for this project. Alternatively, if you want your own receiver but do not want to build your own, you can deploy one based on mine using the code in the project GitHub repository.

Also posted on IndieNews.

Tagged in webmention.