Skip to content

Create a Gemma compatible transformer implementation #30

@iislucas

Description

@iislucas

Different transformer implementations have variations (e.g. in positional encoding, where skip connections are, use of MQA, etc). Lets provide a Gemma standard implementation of transformers. This could be verified by being able to load and evaluate with a Gemma weights file.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is neededproject:TTPFor the Tiny Transformer Playground

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions