Moving over to my own static site generator

I have been thinking about building a static generator for a while. I enjoy using static site generation to build my site but up until now I have deferred the actual logic behind site generation to Jekyll. This was perfect at the beginning because my site was more about writing with the occasional page change. Now, however, I am interested in expanding my site with a bit more code.

By writing my own generator, I could see and control exactly how all pages on my site are built. I could deepen my understanding of working with the file system and markdown even more. Indeed, more than most factors, this project was a learning opportunity. I could have built on top of Jekyll but I wanted to start from scratch and see how far I could get.

In addition, I have been running into problems with building the retro version of my site. For context, I launched a "retro" version of this website a few weeks ago. My retro version was built with late '90s and early '00s web design in mind. I keep the design simple. The retro site was built as a way to explore how websites were in the past. I learned about the good and not so good features of older websites. I added a guestbook as a reminder of how websites can be communities. I avoided using too many GIFs because we now know they can be a bit overwhelming.

Lunching the retro site made it difficult for me to make consistent changes to my blog. I had to create two themes: my regular blog and my retro blog. These themes were not just different style sheets but different layouts altogether. I used a sidebar instead of header and footer navigation elements. [1] Because these sites were running on different themes, I had them on different branches in my code management system. I had to make every change to my blog twice, once on my main blog and again on the retro blog.

With all of this in mind, I decided to build my own static site generator. The code is open sourced on GitHub at capjamesg/letsjam.

Building letsjam

I did some research and decided to build my static site generator with Python. I like using Python as the syntax is simple and there are so many libraries that extend the language for various tasks. For file loading and parsing work, there were plenty of libraries to help. After choosing a programming language, I got to work on the main feature upon which all other features would be built: page generation. [2]

To reduce the amount of work I would have to do to make my existing blog code work with my new generator, I tried my best to parse my Jekyll files as they were, rather than making changes across dozens or potentially hundreds of files (or writing scripts to do this). I kept the style of front matter at the top of each page that stated the layout of the page (what template it should use), the title of the page, what image to show at the top of the page (if any), and other pieces of information that affect how a page is created.

My generator reads this front matter and then looks for a template, if one is specified. Then, the contents of the file that appear after the front matter—the blog post or the HTML or the plain text—will be added to the template. Here's how the process looks for creating a file:

1. I create a file in a folder (i.e. _posts or _likes).
2. I add some front matter that indicates the template name, page title, and any other pieces of "meta" information I need to track.
3. The generator reads the file and looks for the template associated with a page.
4. The generator retrieves the template and adds the content from a page.
5. The generator creates a file name and path for the new web page.
6. The generator saves the entire file.

Figuring out where to save files took a bit of work. I had to experiment a bit on what file paths worked for me. I decided that all files that should not end in .html (i.e. /blogroll/ or /contact/)should be in folders, followed by index.html. When my generator creates a file path, it turns the name of the file into a directory, creates that directory, and then saves the page contents in an index.html file in that directory.

All of my posts are saved using the date in their file name, just like they were in Jekyll. Here is an example:

/2021/01/10/post-name/index.html

Using this path means that visiting /2021/01/10/post-name/ will show you the post you want to see, without having to add .html onto the end. This structure is key because I wanted to preserve my existing URLs to the extent I could. Good URLs don't change. (Not to mention the fact that changing URLs would involve setting up a lot of redirects from the old page URL to the new one).

Expanding the site generator

As I mentioned, I wanted letsjam to be as close to my old site as I could. But rather than just match features, I thought about what I had on my old site that I could not do without writing custom extensions onto Jekyll. This got me thinking about what itches I have had but left alone.

To match the style of my old site, I created a function that generates category pages. If a category has less than 10 posts, one page will be created. If there are more than 10 posts in a category, more pages will be created for each 10 posts in the category. These category pages list all pages that are tagged under a particular category, such as IndieWeb or Coffee. This approach creates URLs like:

/category/indieweb/1/ /category/coffee/2/

The category name comes first followed by the page name.

I extended this same logic towards date archive pages, too. This was something I hadn't figured out on Jekyll but wanted to include on my site. Instead of just having a massive list of posts published in a month or a year, I wanted to have them paginated. I wrote a function that retrieves all posts written in each month and year the blog has been running. Then, these posts are saved to files. These archives are paginated so no more than 10 posts will show up on each page.

Here is an example of the URL structure of an archive page:

/2021/ /2020/12/

These slugs are already part of the folder structure for posts. I took this approach so that I did not add another namespace like /archives/ which would have made the URL structure for the project more complicated. With that said, I did create a single /archives/ page to list all of the archive by month and year pages that are present on the blog. This page is now linked in the footer. Using this page, you should easily be able to navigate around my site by month and year.

I made a few other minor changes to my site but there's one big one I want to end with: supporting retro themes. I spoke about this concept at the beginning of the article. Creating a retro theme on my site with Jekyll meant maintaining two separate versions of my site. With my own generator, I could build my retro site and my old site at the same time, in a way I could easily customise to my needs.

Now my site build process creates the main version of my site, the one at jamesg.blog, and then goes on to build the retro version. The retro version works on similar templates as the main site but there are some retro styles applied and HTML changes that support the unique structure for the pages.

In total, building my main site takes under 30 seconds on my local machine, usually about 20-24 seconds. Building the retro site takes pretty much the same amount of time. This means my build process takes less than 50 seconds. I have over 900 pages on the site in total, many of which are archive pages, so I think the build time is reasonable. I have not spent much time thinking about performance optimisation although I might as I add more pages to my site.

Wrapping up

Working on my own static site generator has been an exciting project. I understand there are many site generators out there. Peter Molnar's site generator is called "not another static generator" (nasg) which I believe articulates the extent to which the static generator idea has been explored. For me, however, this was a foray into the technical details behind building static pages. I ran into a lot of bugs. I practised writing clean code. I gained more experience working with yaml in Python, something I had only explored with my Micropub client up until now.

There may be bugs on this site. If you find any, please do let me know. You can email me at jamesg@jamesg.blog. I have done my best to make my site look like it was before but some changes have broken features that would otherwise have worked. I believe I have caught most of the bugs but a few may remain. A few people have already emailed me about issues. I intend to write a blog post thanking everyone for reaching out.

[1]: I did try to use exclusively CSS to style the page—and indeed it appears I could have created the theme with pure CSS—but I worried about the semantics and the cleanliness of my code. I did not want to use JavaScript either because one issue could impact the layout of my site. [2]: This is the first time I have named an IndieWeb project anything other than its purpose. My webmention receiver was called "webmention receiver," my Micropub client and server was called "Micropub." I am proud of coming up with a unique name for the project that is not predictable.

Also posted on IndieNews.

Tagged in static site generator.