PeterSmith.Org

Third steps to IndieWeb


My new theme, is working well enough that I have now republished my website using it. It also means that I have webmentions working as well. I have some rough edges to sort out but its definitely a MVP (minimum viable product).

It feels as if a cast of thousands have been involved in the this new theme. I have so many people to thank.

First there is Aaron Parecki. I had been watching his videos to help me get into doing my video production (for the BusDev@Auckland seminar series) but then I stumbled into his website, and was introduced to the world of IndieWeb and webmentions. Aaron's site webmention.io also provided the glue to get webmentions working here.

Soon after my first foray into webmentions, I got a mention by Robbi Nespu. That was important not just because of the excellent advice Robbi gave me, but it also gave me the impetus to charge ahead with trying to fully implement webmentions. Looking at Robbi's site, I still need to do more work to get things like 'bookmarks', 'rsvps', and other IndieWeb things working here.

That all prompted me to start (re)theme-ing this site to better accommodate webmentions. As I say, it works, but there is still more to be done.

The basic idea for the current style of this site came from The Ministry of Type. In many ways it is reminiscent of the style I was using about seven or eight themes (site versions) ago. In other ways, I am paying a big homage to that site. I was rather surprised that it hasn't been updated since 2015.

My basic process for 'dealing to' webmentions is use the API at webmention.io to download them all as a single JSON file. First I did it using curl. Then I wrote a Python script based on the work of Brian Wisti. I added in some code to process the JSON with a view to splitting it up into separate JSON files for go into the director of each post (My site is structured using Hugo's page bundles, with each post (and associated material) living in its own directory).

I then moved on to using webmention.io-backup to fetch the JSON file. This cunning tool from Evgeny Kuznetsov works pretty well. I would have liked to use the option to place the 'mentions' in the same directory as the posts, but I couldn't because his directory structure (the one the tool supports) and mine aren't compatible. Whilst Evgeny also seems to be using page bundles, his structure is different. Que sera. Having said that, I need to get a bit techy …

… so, I use Evgeny's tool to download my webmentions. I then access them using .Site.data (data templates) as shown a bit later on.. E.g., with the following, it I want to access the author's name, then I would use .Site.data.author.name in my Hugo templates.

  {
    "author": {
      "name": "Robbi Nespu",
      "photo": "https://webmention.io/avatar/robbinespu.gitlab.io/6e26274b5791d929b75d7450e7a5c1f9e0d19134bb0c641705f0a1e41742e4d2.png",
      "type": "card",
      "url": "https://robbinespu.gitlab.io/"
    },

Easy eh! Well, it actually took me a long time, and posts such as this and this from The New Dynamic, to get it working. On the way I ran in to a couple of problems. First, in testing (i.e., running the site locally using hugo server), the URLs on the local test site didn't match those in the mentions. E.g., my local URLs began http://localhost:1313/ where as the live posts/mentions had URLs that started with https://petersmith.org. So I bodged that up with a little string replacement in my templates (see, a bit further on).

The second problem had to do the names of the keys used in JSON file. In the above JSON example, there are keys such as "author" and "name". That's great. But later on, following the webmention docs, there are keys such as "wm-target" and "mention-of" … all beginning with "wm-". And the - doesn't place nice with with hugo. A key such as wm-target when accessed through .Site.data.wm-target falls over because hugo treats it as it it were .Site.data.wm - target; it tries to subtract something called target from wm. Doh!

So, I run a teeny tiny sed script on the JSON file from webmention.io to change all the hyphens in key names into underscores. It's not a pretty script of sophisticated script, but it seems to do the job. In total, getMentions.sh looks like this:

#!/bin/sh

FILENAME="data/mentions.json"
rm -fr ${FILENAME}

webmention.io-backup -t -y2QezN2pCs2CNVshkMHaQ -d petersmith.org -f ${FILENAME}  -jf2 -tlo=false -p 

#
# Now fix the pesky hypens in the poperty names
#
sed -i  '
s/\(^ *\)"like-of":/\1"like_of":/
s/\(^ *\)"mention-of":/\1"mention_of":/
s/\(^ *\)"in-reply-to":/\1"in_reply_to":/
s/\(^ *\)"wm-/\1"wm_/' ${FILENAME}

So, instead of trying to access .Site.data.wm-target, I go for .Site.data.wm_target, and that works great. As an example, here is the hugo partical template I use to check to see how many mentions or replies a post has.


{{- $permalink := replace .Permalink "http://localhost:1313/" "https://petersmith.org/" -}}
{{- $webmentions := .Site.Data.mentions -}}
{{- $likes := 0 -}}
{{- range $webmentions -}}
{{- if (and (or .in_reply_to .mention_of ) (eq .wm_target $permalink)) -}}
{{- $likes = add $likes  1 -}}
{{- end -}}
{{- end -}}
{{- $likes -}}

The 'partial' returns the count (0 or more) of the mentions as page has. I have similar 'partials' that produce a count of the number of 'likes' a page has.

So far, so good. I don't quite have the displaying of mentions perfect, but it works.

That said, there are some design/architecture choices I have made that I need to revisit, but I don't think any changes to that will change how the site looks and functions from a user's perspective. In practice, I use one large JSON file from webmention.io. I still wonder if I should use individual JSON files for each post. I think that would make processing time/site build time/scalability better. But that would me either writing a 'post processor' to run after Evgeny's tool, or to write my own. There's some thinking I need to do about that.

I also wonder about finding a way to fetch and save the author images/avatars to improve page load times. As others have found this can be a bit of a pig. I do already use 'loading="lazy"' for the avatars, but is that enough. It would be nice to download load them and save them when I build the site.

Oh, well. That's enough for now. Time to go back and tweak the stylesheet to get the 'mentions' looking nice.

If you webmention this page, please let me know the URL of your page.

BTW: Your webmention won't show up until I next "build" my site.

  • Wow! I’m glad to know my little tools are of use to someone else.

    I wonder just how different your directory structure is from mine. Perhaps, we could add an option or two to webmention.io-backup to make your life easier? ;-) Drop me a line if you’re interested, we’ll try to figure something out.

    Also, there’s a workaround I use to avoid seding the JSON for dashes: a parsed JSON is (obviously) a map in Hugo, so you can get the values of its keys by indexing them. And since index takes a string for the key name, dashes can be used freely, too. So, while .Site.data.wm-target fails, index .Site.data "wm-target" can be used to achieve the same goal successfully.

  • I see.

    The reason webmention.io-backup is failing to correctly work your directory structure is that it expects the directory structure to mirror the URL path, that is, it expects petersmith.org/blog/2022/01/10/third-steps-to-indieweb/ to reside in content/blog/2022/01/10/third-steps-to-indieweb/. In your setup, that directory doesn’t exist; instead, content/blog/20220110-third-steps-to-indieweb is used. I suppose, you are using permalinks config option to achieve that? If so, that’s quite a problem, because in this case webmention.io-backup can not really infer the directory name, since the URL is formed based on the date parameter and not based on the first 8 digits of the directory name, right?

    I mean, I can make a patch for webmention.io-backup to work on your directory structure, it’s fairly trivial. But I’d be reluctant to incorporate it in the main branch, because it wouldn’t be flexible at all.

  • To accomodate a complex layout like this, webmention.io-backup would have to parse the config file and look up the date field in all the content files' front matter to figure out which directory corresponds to which website page (basically, the way Hugo does it when it creates the pages in the first place). Programming all this is not impossible, of course, but it would be quite an endeavor that I’m not sure I want to embark on, especially considering the fact that one’s config.toml may contain something along the lines of

    [frontmatter]
      date = [':git', 'lastmod']
    

    I don’t want to open that can of worms!

    These ‘mentions’ on my side aren’t being threaded back to the original post.

    Beware, this is Salmention territory you’re stepping into. There be dragons.

Show all the shares aka https://keithjgrant.com/posts/2019/01/low-friction-workflow-for-notes/