BioCLIP: A vision model for the Tree of Life

Last year, I learned about BioCLIP, a vision model for the Tree of Life. Using BioCLIP, you can provide a list of categories and an image. BioCLIP will then tell you how related each category is to the image. BioCLIP is able to classify plants, animals, and fungi. The model was developed by researchers at The Ohio State University, Microsoft Research, the University of California, Irvine, and Rensselaer Polytechnic Institute.

For example, you can upload a photo of an Arabica coffee plant and provide three categories like "Arabica", "Robusta", and "Liberica". The model will tell you which category best represents the image.

You can use the model in two ways: "zero-shot", which lets you provide a list of categories (i.e. specific plants, animals, or fungi that you think could be in the image) and identify the one most related to the image, or you can use the pre-set categories. Using the pre-set categories is ideal if you want to identify an object without having to write your own list of categories to use for classification.

There is a demo you can use to experiment with the model on the official BioCLIP website. To use the demo with your own categories, click "Zero-shot". Add the categories you want to use to classify an image, then upload an image, Click "Submit" to start the classification. This process will take a few seconds.

There are so many potential applications of this technology. For example, you could build a nature surveying app that lets people index different plants in their local area. You could build a plant scavenger hunt game to help young people learn about the plants around them (this probably exists, but this technology opens up the opportunity to easily expand the number of plants an app could identify). Machine learning experts could use BioCLIP to train a narrower model to identify species of plants, animals, or fungi that the BioCLIP model cannot identify.

I encourage you to play around with the demo. If you try it out, blog about your results.

Tagged in IndieWeb.