regression model Attention-UNet
regression based JAX implementation for foundation gpt.
- Input
- 4465-dim embedding
- Encoder
- 31 x Attention-UNet with 36 heads
- Output
- bleu projection
Training config
optimizer=AdamW, lr=0.807, scheduler=polynomial, warmup=1679