Brainstorming a daily scavenger hunt game

I have spent a lot of time thinking about creative ways to work with embeddings. Embeddings are numeric representations of data, such as text or an image. Embeddings are "smart" in the sense that they encode semantic information.

You can run a cosine similarity operation to compare the similarity between two embeddings. You could compare an image of a cat and a sofa and the embeddings would not be similar; you could compare two images of cats and the embeddings would be similar. You could compare an image of a cat and the text "cat" to see if the image is close to the text prompt.

I wrote down an idea on my TODO list for a game that generates a random object each day and tasks you to go and take a photo of that object. This is like a mini-scavenger hunt. You have to find one object a day. You could use a tool like GPT to generate the object each day, which means that you will not know what objects you may have to find.

This game would be an interesting way to seek out new things in one's day. Imagine if you were asked to take a photo of a banana and you have no bananas in the house. You might have to go to a supermarket or a fruit stand to take a photo and participate for the day. Or if the prompt was a dress and you don't own a dress, you might go for a walk to see if there are any dresses in the window.

CLIP, an image vision model developed by OpenAI, would be a good candidate for generating image and text embeddings for use in this application. Then either GPT or a random selection from a pre-defined list will run. Then, there would be a web page that lets you take a photo, uploads it, and compares an embedding for the image with an embedding for the day's text prompt. CLIP probably wouldn't work as well with more abstract concepts like "pink dress" or "speed bump" (NB: I haven't tested on those exact prompts), but for more general concepts CLIP would work.

I can imagine that there may be some prompts, even with random selection, that one can't find in a day. For instance, if a tool like GPT generates "zebra" and you aren't near a zoo, it would be difficult to participate (you could take a photo of a photo of a zebra from the internet, but that wouldn't be as fun!). A "skip" tool that generates a new prompt for the day would be useful. Perhaps use of this feature is limited to three turns each day to encourage players to seek out more difficult objects to find.

It could even be a competition: who can take a photo of the day's prompt fastest?

If you want to build this and need a bit of guidance, email me at readers [at] jamesg [dot] blog and I would be happy to help.