Expanding Wikipedia Seismograph Into a Related Concept Explorer

I’ve been spending the weekend expanding Wikipedia Seismograph. Before it used Wikipedia page view spikes to create date-bounded Google News searches, but I added in a bunch of different APIs and functionality.

Now it’s a concept-exploring machine and I’ve barely gotten started adding stuff to it.

I’ve been using Kendrick Lamar as one of my test searches because I’m fascinated by the beef between him and Drake (in my day it was Kool Moe Dee running over LL Cool J’s Kangol hat.)

Screenshot of Wikipedia Seismograph at work. It's analyzed Kendrick Lamar's Wikipedia page views between late August 2024 and today. A chart shows the spikes in his page views over that time period. Two red bars bracket a small spike in early September. A button underneath invites the user to click for more information.

Instead of just opening a Google News search link, here’s how this version of Wikipedia Seismograph works: after the news results are fetched, they’re displayed locally and sent to an OpenAI call. The call extracts the primary relevant concepts from the news items. This OpenAI call is *supposed* to only supply those concepts which are represented by Wikipedia pages, but OpenAI can’t handle that perfectly no matter how much prompt-tweaking I do. (I didn’t figure this to be a tough ask for AI, but apparently I was wrong.)

A GNews search for "Kendrick Lamar" over the period of September 4 - September 11 2024 has been run. At the bottom of the page are the news results, while above them in the grid are key concepts as extracted by an OpenAI call. They include Joe Budden, JAY-Z, Drake, and Lil Wayne.

I put in a little Wikipedia-search backup to the OpenAI concept call — it keyword-searches Wikipedia for the concept provided and accepts the first result, getting around slightly-misstated or formatted page titles. Happily much of the time this method does work, and when it does you get these lovely detail / search divs for each extracted concept! The left gives you a picture, relationship overview, and official links. The right, a Wikipedia article summary but also a bunch of *very* useful tools.

A detail page for Pharrell Williams. On the left is an image of him, a brief paragraph explaining his relationship in this time period with Kendrick Lamar (they're doing/did an animated feature together), and a number of external official links (social media, Library of Congress, official website, etc.)

On the right are several sets of information that I'll continue to talk about in this thread. The top item is a summary of Pharrell's Wikipedia article.

There’s an “Individual Search” section that opens up into two search sections: web and news. The Web search uses a Mojeek API call to offer searches over a variety of time spans, starting with the original dates bracketed on the Seismograph.

The Pharrell Page, only now on the right it's showing the results for a Mojeek Web search between September 4-11, 2024. A dropdown menu gives additional date span choices and a search form lets the user edit the search.

The News portion, meanwhile, uses the GNews API to get its results. I’m using a free tier so what I’m getting is pretty limited but on the other hand it is free so I am not complaining at all!

Pharrell again, this time with news sourced from the GNews API for September 4-11 2024. Apparently he's going to be a Lego or something?

A combined search lets me run queries for both the explored topic and with the original query. Only one result in this case but imagine this with a more-expensive, better apportioned news API.

I know it looks weird that I have separate sections for individual and combined searches instead of putting them all into one place. I wanted to have two separate searches so I could run comparative queries.

GNews search for Pharrell AND Kendrick Lamar over the same September time period. Only 1 result (because I'm using a very very very free news API) but it shows the overlap between the two musicians.

Finally, if — after you explore this related concept — you decide this is a topic you want to follow in the future, you have your choice of keyword-based RSS feeds from Bing, Bing News, WordPress, and Google News.

The last section in the Pharrell detail div, this time with a set of keyword-based RSS feeds for monitoring the topic of Pharrell in the future. The fees are for Bing, Bing News, Google News, and WordPress.

This version of Wikipedia Seismograph has given me a strong tool to explore related concepts when I’m exploring a topic or person from Wikipedia. I’ve still got plenty of things to add, though — Crony Corral or some of the other MegaGladys tools will go PERFECT with this.

Expanding Wikipedia Seismograph Into a Related Concept Explorer

Tara Calishain

Related Posts