OpenAI's GPT-2 + DALL-E PyTorch

[jp-krow]

GPT-2 Extra Large model (1775M Parameters) + DALL-E PyTorch implantation.
(Or a Fine-Tuned 355M Model)

Do you think that would be a feasible idea?

It will be a dumbed down version of DALL-E for sure, but it will open up much more possibilities.

[sorrge]

I think that's worthy of implementation. Instead of text tokens, we can feed GPT2 embeddings from the last layer. It might be even smarter with text than original DALL-E, because GPT2 was trained on a large amount of text. DALL-E has to rediscover everything about language from just the captions, which are small and fragmented.

[afiaka87]

Surely it doesn't have to rediscover everything considering CLIP was trained on GPT-3, right?

[sorrge]

CLIP has nothing to do with GPT. Also, although CLIP is trained on large amount of text, its knowledge is not used for generation in DALL-E. It's only used to rank the generated pictures. If DALL-E doesn't know the meaning of some text, it will not generate the correct picture in the first place.
So yes, everything about the language has to be learned from scratch.

[afiaka87]

CLIP has nothing to do with GPT. Also, although CLIP is trained on large amount of text, its knowledge is not used for generation in DALL-E. It's only used to rank the generated pictures. If DALL-E doesn't know the meaning of some text, it will not generate the correct picture in the first place.
So yes, everything about the language has to be learned from scratch.

Cool thanks for the clarification