📸 Image Classification

[davanstrien]

This is a discussion for the image classification task:

Image classification is the task of assigning a label or class to an entire image. Images are expected to have only one class for each image. Image classification models take an image as input and return a prediction about which class the image belongs to. Source

• Do you know any teaching resources that use image classification in a GLAM setting?
• Do you have an example of using image classification in a GLAM setting?
• Have you found particular tools helpful in creating datasets for image classification tasks or training models?
• If you know of GLAM-related image classification datasets, feel free to suggest them below.

[davanstrien]

Here is a quick list of training resources that spring to mind for this:

• Computer Vision for the Humanities: An Introduction to Deep Learning for Image Classification(Part 1). A Programming Historian Lesson focused on using computer vision in a humanities setting. While the focus is on using computer vision for research questions, the approaches shown can be adapted for more practical tasks too.
• How we built a Pragmatic Machine Learning Pipeline to Identify Fake Flysheets as part of a Library Workflow: a 'book' outlining the process of creating a machine learning project at the British Library using image classification approaches to help identify mislabeled digitized book images. The book focuses on the high level, so it is less about coding and more about the overall steps taken and why some of the different approaches were taken.

[davanstrien]

This might also be useful if you want to train computer vision models without having to write any code yourself. Models I've trained using this approach often do a good job without much time investment required compared to managing training yourself 🙂

[altomator]

At the BnF, we used trained models and more recently CLIP to address classification tasks:

• gallicapix.bnf.fr PoC: https://github.com/altomator/Image_Retrieval
• CLIP: https://github.com/altomator/CLIP_test

[davanstrien]

This is very cool! There is a lot of interest in CLIP models for GLAM applications. The zero/few-shot capabilities are especially appealing!

Have you/are you planning to move any of these experiments into more Business and Usual applications? There still seem to be some ethical/practical challenges to using CLIP in a GLAM context which to me make a full deployment a bit more tricky.

[altomator]

Yes, we 're planning to use text2image models in a near future, but we will probably leverage OpenCLIP, due to the concerns you are quoting.

[MikeTrizna]

The crazy thing, @altomator, is that even OpenCLIP claims in its model card https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K#out-of-scope-use) that "Any deployed use case of the model - whether commercial or not - is currently out of scope".

I saw that the recently-released Align multimodal models list intended use (https://huggingface.co/kakaobrain/align-base#model-use) for AI researchers, which might be more permissive for folks from the GLAM community. But I haven't tested out that family of models yet.

[altomator]

This is not cool :-) What's the point of creating open source models if they can't be deployed?
It seems that the statement is mainly intended as a self-protection against the uses that would be made of the model...

btw, Giles Bergel is about to deploy a public image search app using all Wikimedia content and OpenCLIP.

[davanstrien]

It seems that the statement is mainly intended as a self-protection against the uses that would be made of the model...

This is probably more based on the potential biases of the model. See the bias section of the model card for some discussion of this.

Model cards don't represent a legal barrier to using the model, i.e. it's not a licence, and many people/organisations have put these models into production. This is probably either because they have evaluated the risks in deploying in their situation or just YOLO'd the model into production and not given much thought to potential risks. I think, at the moment, the legal consequences of this are quite low, but I suspect in a year or two, taking this approach will be much less wise.

My own 2c's is that there are many GLAM applications of CLIP models where the bias risks are fairly low, but there are also a whole bunch where the risk is pretty high. I think some of this can be overcome by making sure that the potential biases of models are communicated to end-users of the model and thinking very carefully about what types of collections CLIP style models will be used with and also whether the system will be internally facing or also be exposed to public users.

[MikeTrizna]

Our group at the Smithsonian just published a paper on an image classifier model that our fellow built to classify freshwater fish from the Amazon river: https://doi.org/10.1002/ece3.9987. It was built in collaboration with local Peruvian communities to enable them to measure fish populations before and after a major pipeline being built across the Amazon. We used a combination of images of fish in the field, with images of preserved museum specimens.

We actually ended up with better accuracy by building and using a separate image masking model (we can also add this to the Image Segmentation category here when that's built), to remove the background from the fish images before classifying.

Here's a demo we set up on Hugging Face Spaces: https://huggingface.co/spaces/Smithsonian/amazonian_fish_classifier. The images and models are on FigShare, but we'll get them mirrored to Hugging Face soon, along with a model card and dataset card.

[davanstrien]

This is a really interesting project. It's also really interesting that creating a separate image masking model improved performance. It's also a nice example that deep learning does still sometimes require some 'feature engineering'.

It was built in collaboration with local Peruvian communities to enable them to measure fish populations before and after a major pipeline being built across the Amazon.

Are there plans to also use these models 'on the edge', i.e. setup camera traps (or something similar) in local communities along the Amazon river (sorry if this is discussed in the paper, didn't have a chance to read it yet!).

[sighmon]

We've been experimenting with video classification and captioning on our ACMI collection:

• Audio transcriptions
• Video image frame captions

Try the resulting video search here: https://www.acmi.net.au/videos/

[sighmon]

@davanstrien The confidence code runs, but I'm not sure how to get a percentage out of it - do you have any ideas?

[davanstrien]

IUC x 100 + rounding should be sufficient. I suspect because of how image captioning works, the confidence scores will all be low and possibly a little bit noisy so I'd be interested to see how much the confidence scores translate to 'quality'.

[sighmon]

@davanstrien Is there a better way to infer quality? I added a couple more examples and you're right about the confidence scores being very low!

[sighmon]

And also some output from one of our videos:

Prediction at 0 (0.0 secs): [{'prediction': 'a vintage car driving down a country road', 'confidence': 0.24626527447253466}]
Prediction at 100 (4.2 secs): [{'prediction': 'a sign pointing to a road sign', 'confidence': 0.001155625159299234}]
Prediction at 200 (8.3 secs): [{'prediction': 'a man walking down a dirt road with a sign pointing to the left', 'confidence': 0.00046116647354210727}]
Prediction at 300 (12.5 secs): [{'prediction': 'a sign that says whitefield please slow down', 'confidence': 0.5517419427633286}]
Prediction at 400 (16.7 secs): [{'prediction': 'a street sign that says fern-tree gulley', 'confidence': 0.1732164528220892}]
Prediction at 500 (20.8 secs): [{'prediction': 'a sign that says ovens river on a bridge', 'confidence': 0.06027290946803987}]
Prediction at 600 (25.0 secs): [{'prediction': 'a colorful bird sits on a branch in a tree', 'confidence': 0.0556458136998117}]
Prediction at 700 (29.2 secs): [{'prediction': 'porepunkah, nsw, australia', 'confidence': 0.003496685530990362}]
Prediction at 800 (33.3 secs): [{'prediction': 'a sign that says great seal road at the entrance to a rural area', 'confidence': 0.00014232408602765645}]
Prediction at 900 (37.5 secs): [{'prediction': 'an old truck and camper trailer parked on the side of the road', 'confidence': 0.0014286674741015304}]
Prediction at 1000 (41.7 secs): [{'prediction': 'a view of a waterfall in the woods', 'confidence': 0.015348590386565775}]
Prediction at 1100 (45.8 secs): [{'prediction': 'a man is looking at a waterfall from a car window', 'confidence': 0.00012174664334452245}]
Prediction at 1200 (50.0 secs): [{'prediction': 'from acmi collection', 'confidence': 5.89255653321743}]

[davanstrien]

I'm sorry I missed this follow-up. I'm unsure of other ways that don't rely on having some ground truth data, but I imagine there must be some other methods.

Outputs look super cool already!