OUTTA Basic — Organizing BERT, Hugging Face, and Gemini Safely

Three parts of a BERT input

I went beyond the phrase “bidirectional Transformer encoder” and wrote the BERT input as the sum of token, position, and segment embeddings. The 40-page BERT module also helped me locate [CLS], [SEP], and padding in the sequence.

Eᵢ = E(token, i) + E(position, i) + E(segment, i)

Sequence classification uses the [CLS] representation, while token classification uses every timestep. Checking the output shape first prevents connecting the wrong task head.

Attention masks and shapes

Self-attention mixes values according to query-key similarity.

Attention(Q, K, V) = softmax(QKᵀ / √dₖ + M)V

The mask (M) prevents padding positions from contributing. I checked that input_ids.shape == attention_mask.shape, both with (batch, length), and expected classification logits to have (batch, classes).

Hugging Face workflow

I matched the tokenizer and model checkpoint names, passed the tokenizer dictionary directly into the model, and selected the required field such as logits or last_hidden_state from the output object.

encoded = tokenizer(
    texts,
    padding=True,
    truncation=True,
    return_tensors="pt",
)

assert encoded["input_ids"].shape == encoded["attention_mask"].shape
result = model(**encoded)
assert result.logits.shape[0] == len(texts)

Outputs retained in the notebooks

The Hugging Face copy retained 39 cells, 25 code cells, and 125 text or table output objects. Its code source matched the base notebook; the outputs were the difference. The BERT copy had 72 cells and 31 output objects, with 17 of 32 code cells differing from the base. The Gemini copy retained 33 cells, 24 code cells, and 10 output objects.

These notebooks contained text and table outputs rather than saved image outputs, so I centered this note on tokenization, masks, shapes, and code flow. I did not describe the stored material as a fresh execution.

Separating Gemini credentials

The old Gemini notebook contained a plaintext API key. I excluded that raw notebook and rewrote the public example to use an environment variable after the owner revokes and replaces the old key. No value or original credential cell is reproduced here.

import os

api_key = os.environ["GEMINI_API_KEY"]
if not api_key:
    raise RuntimeError("GEMINI_API_KEY is empty")

# Pass api_key to the SDK client without printing or saving it.

Only the environment-variable name remains in source. The value belongs in a shell, Colab secret, or CI secret. Before sharing a notebook, I also check outputs and metadata for accidental credential retention.

Comparing BERT and GPT

The 28-page GPT module uses a causal mask that hides future tokens. BERT reads context on both sides for masked-token learning and downstream representations; GPT-style models predict the next token from left context. The 22-page CV and NLP trends module tied both to the shared workflow of selecting a pretrained backbone, tokenizer, task head, and loss.

Previous and next

• Previous: Following the time axis in RNNs and Seq2Seq
• Next: 2024 OUTTA AI Bootcamp Basic and its projects