Attribute based eval

[rom1504]

https://arxiv.org/abs/2209.06103

[JeniaJitsev]

Rationale: introduce a "true" zero-shot eval into the benchmark, based on attribute-based zero-shot task.

The task is harder, as in contrast to standard zero-shot transfer used currently for CLIP eval, it requires to deal with new classes which were never presented during pre-training by describing any entity to be recognized by a set of attributes (eg sandy yellowish, furry, four legged, fast, strong) instead of using a class label (eg lion). For more details, see https://arxiv.org/abs/2209.06103

[mehdidc]

That would be great, is the code available somewhere ? @hildekuehne

[hildekuehne]

Will check and get back soon.

[mehdidc]

I think it would still be cool to re-evaluate the newest models on this, will see if it is easy to incorporate from the repo https://github.com/felixVogel02/VL-Taboo