Skip to content

Add TIPSv2 Scenic demo#12

Open
bingyic wants to merge 2 commits intomainfrom
scenic_demo
Open

Add TIPSv2 Scenic demo#12
bingyic wants to merge 2 commits intomainfrom
scenic_demo

Conversation

@bingyic
Copy link
Copy Markdown
Collaborator

@bingyic bingyic commented Apr 10, 2026

Summary

Add a unified Scenic (JAX/Flax) demo notebook supporting both TIPSv1 and TIPSv2 models.

Supersedes #10 with a cleaner implementation that leverages the configurable posembs parameter from #11.

Key Changes

Notebook (scenic/notebooks/TIPS_Demo.ipynb)

  • Unified demo supporting TIPS and TIPS_V2 model types
  • Model config (tips_model_config.py) handles V1 vs V2 positional embedding sizes automatically
  • No monkey-patching: TIPSv2's (32, 32) positional embeddings are passed cleanly via:
    model_vision = tips_model.VisionEncoder(
        variant=model_config.variant,
        posembs=model_config.positional_embedding.shape,  # (16,16) for V1, (32,32) for V2
    )

Checkpoint Downloads

  • Added TIPSv2 checkpoint URLs for both PyTorch and Scenic formats
  • Updated download_checkpoints.sh scripts

Model Config (tips_model_config.py)

  • Added TIPSv2 variant mappings (B/14, L/14, So400m/14, g/14)
  • Config automatically sets correct positional_embedding.shape per model version

Co-authored-by: washingtonsk8

washingtonsk8 and others added 2 commits April 11, 2026 01:26
…-patch

Now that posembs is a configurable parameter in VisionEncoder (merged in PR #11),
we can simply pass posembs=model_config.positional_embedding.shape instead of
monkey-patching vit.py source code at runtime.

Changes:
- Remove the entire ViT.setup() monkey-patch and source file string-replace hack
- Pass posembs directly via VisionEncoder constructor
@bingyic
Copy link
Copy Markdown
Collaborator Author

bingyic commented Apr 11, 2026

I've force-pushed the correct commit (e2c7264) to the scenic_demo branch, replacing the accidental merge commit. The PR is now ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants