Know your model outputs while writing logic

Earlier today, I was working on a computer vision project. I trained a model to identify and distinguish between different chocolates in a box. With this model ready, I wanted to be able to verify that every chocolate was in the right place. I tested the model to make sure it could accurately identify chocolates in an image. But, my testing was limited; something that would come back later. I wrote logic that worked on one input but not another. So I changed my inputs. In the process, I forgot an important rule: check your model outputs.

While I was writing logic with a colleague, we noticed that the output of our logic was not expected. We were working under the assumption that the model identified all 22 chocolates ([insert Taylor Swift reference here]). We were surprised when our logic was missing a few chocolates. Then we realised that we were testing on an image that wasn't representative of the data on which our model was trained: that image I used to test how well my logic performs on different images.

I thought I knew what data I was working with, but after verifying the model output I realised the model doesn't perform as well on the second image on which I was testing. In the process, I learned:

1. The model needed more data to improve performance, and;
2. When writing logic, make sure you know your model outputs.

If you assume your model outputs without directly checking, there is a risk that you end up finding a "bug" in your code, being stumped, and then realising it was not your code but the model.