Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions docs/source/content/migrating_to_v3.md
Original file line number Diff line number Diff line change
Expand Up @@ -335,3 +335,54 @@ def cache_activations(model: TransformerLensModel, text: str):

Use `TransformerLensModelWithWeights` instead when the helper also needs the
weight-processing surface used by advanced `ActivationCache` operations.


### Build a TL-native model from scratch

For toy models and interpretability-research training loops, the bridge
replaces `HookedTransformerConfig + HookedTransformer(cfg)` with
`TransformerBridgeConfig + TransformerBridge.boot_native(cfg)`:

```python
# Before
from transformer_lens import HookedTransformer, HookedTransformerConfig

cfg = HookedTransformerConfig(
d_model=64,
d_head=8,
n_heads=8,
n_layers=2,
n_ctx=256,
d_vocab=50257,
act_fn="gelu_new",
seed=42,
)
model = HookedTransformer(cfg)

# After
from transformer_lens.config import TransformerBridgeConfig
from transformer_lens.model_bridge import TransformerBridge

cfg = TransformerBridgeConfig(
d_model=64,
d_head=8,
n_heads=8,
n_layers=2,
n_ctx=256,
d_vocab=50257,
act_fn="gelu_new",
seed=42, # optional: makes initialisation reproducible
)
bridge = TransformerBridge.boot_native(cfg, device="cpu")
```

`boot_native` makes no HuggingFace Hub call and requires no `transformers`
import. `cfg.seed` seeds the weight initialiser; omitting it lets the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HT's init_weights calls torch.manual_seed(cfg.seed) globally, while boot_native forks the RNG, so a training loop migrated verbatim silently loses reproducible data shuffling. Could this note state that difference explicitly?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also fixed in 870ccb9 — added an explicit note that boot_native forks the RNG (unlike HookedTransformer.init_weights() which calls torch.manual_seed(cfg.seed) globally), so training loops ported verbatim will silently lose reproducible data shuffling.

global RNG advance normally. Passing a `HookedTransformerConfig` (or any
other legacy config object) to `boot_native` raises `TypeError` — construct
a `TransformerBridgeConfig` directly. To reinitialize weights in place, call `bridge.init_weights()` — if `cfg.seed` is set,
`init_weights()` rebuilds from that same seed and produces identical weights; clear or
change `cfg.seed` first for a genuinely fresh draw. Unlike `HookedTransformer.init_weights()`,
which calls `torch.manual_seed(cfg.seed)` globally, `boot_native` forks the RNG — training
loops ported verbatim that rely on global seed state for data shuffling will silently
lose reproducibility.
Loading