/10_text_feature_engineering/03_sentiment_evolution.py

# ---
# jupyter:
#   jupytext:
#     cell_metadata_filter: tags,-all
#     text_representation:
#       extension: .py
#       format_name: percent
#       format_version: '1.3'
#       jupytext_version: 1.19.3
#   kernelspec:
#     display_name: Python 3 (ipykernel)
#     language: python
#     name: python3
# ---

# %% [markdown]
# # Sentiment Analysis Evolution: TF-IDF → Word2Vec → Transformers
#
# **Chapter 10: From Text to Features - The Transformer Breakthrough**
# **Section Reference**: See Sections 10.2, 10.3, 10.5 for conceptual discussion
#
# **Docker image**: `ml4t-py312`
#
# > **Docker required**: This notebook uses `gensim`, which has no Python 3.14 support.
# > Run with:
# > ```bash
# > docker compose --profile py312 run --rm py312 python 10_text_feature_engineering/03_sentiment_evolution.py
# > ```
#
# ## Purpose
# This notebook demonstrates the evolution of text representation for sentiment
# classification, comparing three approaches on the Financial PhraseBank dataset.
# We show how each generation of NLP techniques addresses limitations of its
# predecessors, culminating in Transformer-based models.
#
# ## Learning Objectives
# After completing this notebook, you will be able to:
# - Implement TF-IDF vectorization for document classification
# - Use pre-trained static embeddings (GloVe) for document-level features
# - Apply a pre-trained Transformer (FinBERT) without task-specific fine-tuning
# - Compare accuracy and F1 scores across NLP paradigms
# - Interpret confusion matrices to identify class-specific errors
# - Understand why pre-trained models need fine-tuning for new tasks
#
# ## Prerequisites
# - Sections 10.1–10.4 of the chapter (TF-IDF, static embeddings, Transformers).
# - Financial PhraseBank `sentences_allagree` subset on disk (loaded via
#   `data.load_financial_phrasebank`).
#
# ## Related Notebooks
# - `01_word2vec_training.py` — Skip-gram mechanics on the same corpus.
# - `04_bert_finetuning.py` — fine-tunes FinBERT on PhraseBank (this notebook
#   shows the pre-fine-tuning baseline).

# %%
"""Sentiment Analysis Evolution — compare TF-IDF, Word2Vec, and Transformer approaches on Financial PhraseBank."""

import json
import warnings

import gensim.downloader as api
import matplotlib.pyplot as plt
import numpy as np
import polars as pl
import seaborn as sns
import torch
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, f1_score
from sklearn.model_selection import train_test_split
from transformers import pipeline
from transformers import set_seed as set_transformers_seed

from data import load_financial_phrasebank as load_financial_phrasebank_canonical
from utils.paths import get_chapter_dir
from utils.reproducibility import set_global_seeds

warnings.filterwarnings("ignore", category=UserWarning)

# %% tags=["parameters"]
# Production defaults — Papermill can override for fast CI runs.
SEED = 42
MAX_SAMPLES = 0  # 0 = use the full sentences_allagree subset
FINBERT_TEST_SAMPLES = 0  # 0 = run FinBERT on the entire stratified test split

# %%
# Reproducibility — set_global_seeds covers Python random / NumPy / Torch.
# transformers has its own RNG (used by Trainer + pipelines) that needs explicit seeding.
set_global_seeds(SEED)
set_transformers_seed(SEED)

CONFIG = {
    "random_seed": SEED,
    "test_size": 0.2,
    "dataset": {
        "name": "takala/financial_phrasebank",
        "subset": "sentences_allagree",
        "description": "Financial PhraseBank — 100% annotator agreement subset (2,264 sentences)",
    },
    "tfidf": {
        "max_features": 5000,
        "ngram_range": (1, 2),
        "min_df": 2,
    },
    "glove": {
        "model": "glove-wiki-gigaword-100",
        "dim": 100,
    },
    "finbert": {
        "model_id": "yiyanghkust/finbert-tone",
        "description": "FinBERT fine-tuned on analyst reports for sentiment (NOT PhraseBank)",
        "tokenizer_id": "yiyanghkust/finbert-tone",
        "max_length": 512,
        "labels": {"Negative": 0, "Neutral": 1, "Positive": 2},
    },
}

print("=" * 70)
print("EXPERIMENT CONFIGURATION")
print("=" * 70)
print(json.dumps(CONFIG, indent=2))
print("=" * 70)

# %% [markdown]
# ## 2. Load Financial PhraseBank Dataset
#
# The Financial PhraseBank (Malo et al., 2014) consists of English-language
# sentences from financial news, each labelled positive / negative / neutral
# by 5–8 annotators. We use the `sentences_allagree` subset — the 2,264
# sentences where every annotator picked the same label, i.e., the
# highest-precision portion of the corpus.

# %%


def load_financial_phrasebank() -> pl.DataFrame:
    """Load Financial PhraseBank from canonical local storage (sentences_allagree)."""
    return load_financial_phrasebank_canonical()


df = load_financial_phrasebank()
print(f"Loaded {len(df):,} sentences")
print("\nLabel distribution:")
print(df.group_by("label").len().sort("label"))

if MAX_SAMPLES > 0 and len(df) > MAX_SAMPLES:
    per_label = max(MAX_SAMPLES // df["label"].n_unique(), 1)
    df = (
        df.sort(["label", "sentence"])
        .group_by("label", maintain_order=True)
        .head(per_label)
        .sort("sentence")
    )
    print(f"Reduced dataset for test run: {len(df):,} sentences")

# Map numeric labels to text
label_map = {0: "negative", 1: "neutral", 2: "positive"}
df = df.with_columns(pl.col("label").replace_strict(label_map).alias("sentiment"))

# %%
# ============================================================================
# SANITY CHECKS
# ============================================================================
# These checks ensure the data and label mapping are correct before training.

print("\n" + "=" * 70)
print("DATASET SANITY CHECKS")
print("=" * 70)

# 1. Class distribution in full dataset
print("\n1. CLASS DISTRIBUTION (Full Dataset)")
class_counts = df.group_by("label").len().sort("label")
print(class_counts)

# Majority class baseline
total = len(df)
majority_row = class_counts.sort("len", descending=True).row(0)
majority_label = majority_row[0]  # label column
majority_count = majority_row[1]  # len column
majority_baseline = majority_count / total
print(f"\nMajority class: {label_map[majority_label]} (label={majority_label})")
print(f"Majority baseline accuracy: {majority_baseline:.1%}")

# 2. Label mapping verification
print("\n2. LABEL MAPPING VERIFICATION")
print("   Dataset labels → Our labels:")
for numeric, text in label_map.items():
    print(f"   {numeric} → {text}")

# Assert label mapping matches FinBERT expectations
finbert_label_map = {"Negative": 0, "Neutral": 1, "Positive": 2}
assert label_map == {0: "negative", 1: "neutral", 2: "positive"}, "Label mapping mismatch!"
print("   [OK] Label mapping verified")

# 3. Sample sentences by class
print("\n3. SAMPLE SENTENCES BY CLASS")
for label_id, label_name in label_map.items():
    sample = df.filter(pl.col("label") == label_id).head(1)["sentence"][0]
    print(f"   {label_name}: '{sample[:80]}...'")

print("\n" + "=" * 70)

# %%
# Train/test split (convert Polars columns to numpy for sklearn)
X_train, X_test, y_train, y_test = train_test_split(
    df["sentence"].to_numpy(),
    df["label"].to_numpy(),
    test_size=CONFIG["test_size"],
    random_state=SEED,
    stratify=df["label"].to_numpy(),
)

# Print split details
print("\n" + "=" * 70)
print("TRAIN/TEST SPLIT DETAILS")
print("=" * 70)
print("Split protocol: Stratified random split (preserves class proportions)")
print(f"Test size: {CONFIG['test_size']} ({CONFIG['test_size'] * 100:.0f}%)")
print(f"Random seed: {SEED}")
print(f"\nTrain samples: {len(X_train):,}")
print(f"Test samples: {len(X_test):,}")

if FINBERT_TEST_SAMPLES > 0 and len(X_test) > FINBERT_TEST_SAMPLES:
    sample_idx = np.random.choice(len(X_test), FINBERT_TEST_SAMPLES, replace=False)
    X_test_finbert = X_test[sample_idx]
    y_test_finbert = y_test[sample_idx]
    print(f"FinBERT evaluation sample: {len(X_test_finbert):,}")
else:
    X_test_finbert = X_test
    y_test_finbert = y_test

# %%
# Class distribution in each split.
train_counts = dict(zip(*np.unique(y_train, return_counts=True), strict=False))
test_counts = dict(zip(*np.unique(y_test, return_counts=True), strict=False))
pl.DataFrame(
    {
        "class": [label_map[i] for i in sorted(train_counts.keys())],
        "train": [train_counts[i] for i in sorted(train_counts.keys())],
        "test": [test_counts[i] for i in sorted(train_counts.keys())],
    }
)

# %% [markdown]
# ## 3. TF-IDF + Logistic Regression (Lexical Baseline)
#
# The simplest baseline: represent documents as weighted term frequencies,
# then train a linear classifier. TF-IDF captures word importance but cannot
# understand semantic similarity or context.

# %%
# TF-IDF vectorization
tfidf = TfidfVectorizer(
    max_features=5000,
    ngram_range=(1, 2),
    min_df=2,
    stop_words="english",
)

X_train_tfidf = tfidf.fit_transform(X_train)
X_test_tfidf = tfidf.transform(X_test)

print(f"TF-IDF feature dimension: {X_train_tfidf.shape[1]}")

# Train logistic regression
lr_tfidf = LogisticRegression(max_iter=1000, random_state=SEED)
lr_tfidf.fit(X_train_tfidf, y_train)

# Evaluate
y_pred_tfidf = lr_tfidf.predict(X_test_tfidf)
acc_tfidf = accuracy_score(y_test, y_pred_tfidf)
f1_tfidf = f1_score(y_test, y_pred_tfidf, average="macro")

print("\nTF-IDF + Logistic Regression:")
print(f"  Accuracy: {acc_tfidf:.1%}")
print(f"  F1 (macro): {f1_tfidf:.3f}")

# %% [markdown]
# ## 4. Static Embeddings (GloVe) + Logistic Regression
#
# Static embeddings map words to dense vectors that capture semantic similarity.
# We use GloVe (Global Vectors for Word Representation) pre-trained on Wikipedia/Gigaword.
# We average word vectors to create document representations, then train a classifier.
# Limitation: each word has ONE vector regardless of context.

# %% [markdown]
# ### Load GloVe Embeddings
# Load pre-trained GloVe vectors for document representation.

# %%
from gensim.utils import simple_preprocess


def get_embedding_model():
    """Load pre-trained GloVe embedding model."""
    # Use 100-dim GloVe vectors trained on Wikipedia + Gigaword
    model_name = "glove-wiki-gigaword-100"
    print(f"Loading {model_name}...")
    return api.load(model_name)


# %% [markdown]
# ### Document Vector Computation
# Average word vectors to create a fixed-size document representation.


# %%
def document_vector(doc: str, model, dim: int = 100) -> np.ndarray:
    """Compute document vector as average of word vectors.

    Uses gensim's simple_preprocess for consistent tokenization:
    - Lowercases text
    - Removes punctuation and special characters
    - Filters very short/long tokens
    """
    # Use gensim's tokenizer for cleaner preprocessing
    words = simple_preprocess(doc, deacc=True, min_len=2, max_len=15)
    vectors = []
    for word in words:
        if word in model:
            vectors.append(model[word])
    if vectors:
        return np.mean(vectors, axis=0)
    return np.zeros(dim)


# %%
# Load model
glove_model = get_embedding_model()
embed_dim = glove_model.vector_size

# Compute document vectors
print("Computing document vectors...")
X_train_emb = np.array([document_vector(doc, glove_model, embed_dim) for doc in X_train])
X_test_emb = np.array([document_vector(doc, glove_model, embed_dim) for doc in X_test])

print(f"GloVe document dimension: {X_train_emb.shape[1]}")

# Train logistic regression
lr_glove = LogisticRegression(max_iter=1000, random_state=SEED)
lr_glove.fit(X_train_emb, y_train)

# Evaluate
y_pred_glove = lr_glove.predict(X_test_emb)
acc_glove = accuracy_score(y_test, y_pred_glove)
f1_glove = f1_score(y_test, y_pred_glove, average="macro")

print("\nGloVe + Logistic Regression:")
print(f"  Accuracy: {acc_glove:.1%}")
print(f"  F1 (macro): {f1_glove:.3f}")

# %% [markdown]
# ## 5. FinBERT Pre-trained (No Task Fine-tuning)
#
# Transformers learn contextual representations that vary with surrounding words.
# FinBERT (yiyanghkust/finbert-tone) is pre-trained on financial text and already
# has a sentiment classification head. However, it was trained on analyst reports,
# which have different phrasing patterns than Financial PhraseBank's news sentences.
# Same labels (positive/negative/neutral), different text distribution.
#
# **Critical Note**: This is NOT "zero-shot" since FinBERT-tone already has a
# sentiment classification head trained on analyst reports. We're testing
# **cross-dataset transfer** without task-specific fine-tuning on PhraseBank.
# The poor performance demonstrates **distribution shift**, not model quality.

# %%
# ============================================================================
# FINBERT CHECKPOINT IDENTITY
# ============================================================================
# Explicit documentation for reproducibility and reviewer scrutiny.

print("\n" + "=" * 70)
print("FINBERT CHECKPOINT DETAILS")
print("=" * 70)
print(f"Model ID:     {CONFIG['finbert']['model_id']}")
print(f"Tokenizer ID: {CONFIG['finbert']['tokenizer_id']}")
print(f"Max Length:   {CONFIG['finbert']['max_length']}")
print(f"Description:  {CONFIG['finbert']['description']}")
print("\nLabel mapping (FinBERT → our numeric labels):")
for label, idx in CONFIG["finbert"]["labels"].items():
    print(f"  {label} → {idx}")
print("\nThis checkpoint was fine-tuned on analyst reports, not PhraseBank;")
print("differences vs the TF-IDF / GloVe baselines below reflect distribution shift,")
print("not a like-for-like comparison of model quality.")
print("=" * 70)


def get_finbert_predictions(texts: list[str], batch_size: int = 32) -> np.ndarray:
    """Get sentiment predictions from FinBERT.

    Uses the yiyanghkust/finbert-tone model which is already fine-tuned
    for financial sentiment classification on analyst reports.

    Returns:
        Array of predictions mapped to our label scheme (0=neg, 1=neu, 2=pos)
    """
    model_id = CONFIG["finbert"]["model_id"]
    tokenizer_id = CONFIG["finbert"]["tokenizer_id"]
    max_length = CONFIG["finbert"]["max_length"]

    # Use GPU if available
    device = 0 if torch.cuda.is_available() else -1

    # Create pipeline with explicit parameters
    classifier = pipeline(
        "sentiment-analysis",
        model=model_id,
        tokenizer=tokenizer_id,
        device=device,
        truncation=True,
        max_length=max_length,
    )

    # Map FinBERT labels to our numeric labels
    label_to_id = CONFIG["finbert"]["labels"]

    predictions = []
    for i in range(0, len(texts), batch_size):
        batch = texts[i : i + batch_size]
        results = classifier(batch)
        for r in results:
            predictions.append(label_to_id[r["label"]])

    return np.array(predictions)


print("\nRunning FinBERT inference...")
y_pred_finbert = get_finbert_predictions(X_test_finbert.tolist())

acc_finbert = accuracy_score(y_test_finbert, y_pred_finbert)
f1_finbert = f1_score(y_test_finbert, y_pred_finbert, average="macro")

print("\nFinBERT (Pre-trained on analyst reports, no PhraseBank fine-tuning):")
print(f"  Accuracy: {acc_finbert:.1%}")
print(f"  F1 (macro): {f1_finbert:.3f}")

# Per-class diagnostic to understand failure patterns
from sklearn.metrics import classification_report

print("\nPer-class breakdown (explains why cross-dataset transfer struggles):")
print(
    classification_report(
        y_test_finbert, y_pred_finbert, target_names=["negative", "neutral", "positive"]
    )
)

# %% [markdown]
# ## 6. Results Comparison

# %%
# Summary table
results = pl.DataFrame(
    {
        "Method": ["TF-IDF + LR", "GloVe + LR", "FinBERT (pre-trained)"],
        "Accuracy": [acc_tfidf, acc_glove, acc_finbert],
        "F1 (macro)": [f1_tfidf, f1_glove, f1_finbert],
    }
).with_columns(
    pl.col("Accuracy").map_elements(lambda x: f"{x:.1%}", return_dtype=pl.String),
    pl.col("F1 (macro)").map_elements(lambda x: f"{x:.3f}", return_dtype=pl.String),
)

print("\n" + "=" * 50)
print("COMPARISON SUMMARY")
print("=" * 50)
print(results)
print("=" * 50)

# Relative change calculation (negative = regression vs the lexical baseline).
relative_change = (acc_finbert - acc_tfidf) / acc_tfidf * 100
direction = "above" if relative_change >= 0 else "below"
print(
    f"\nFinBERT (pre-trained) accuracy is {abs(relative_change):.1f}% {direction} the TF-IDF baseline."
)

# %%
# Confusion matrices
fig, axes = plt.subplots(1, 3, figsize=(14, 4))
fig.suptitle("Sentiment Classification Confusion Matrices", fontsize=12, y=1.02)
labels = ["negative", "neutral", "positive"]

for ax, (name, y_pred) in zip(
    axes,
    [("TF-IDF", y_pred_tfidf), ("GloVe", y_pred_glove), ("FinBERT (pre-trained)", y_pred_finbert)],
    strict=False,
):
    y_true = y_test_finbert if "FinBERT" in name else y_test
    cm = confusion_matrix(y_true, y_pred)
    sns.heatmap(
        cm,
        annot=True,
        fmt="d",
        cmap="Blues",
        xticklabels=labels,
        yticklabels=labels,
        ax=ax,
    )
    ax.set_title(name)
    ax.set_xlabel("Predicted")
    ax.set_ylabel("Actual")

plt.show()

# %%
# Bar chart comparison
fig, ax = plt.subplots(figsize=(8, 5))

methods = ["TF-IDF + LR", "GloVe + LR", "FinBERT (pre-trained)"]
accuracies = [acc_tfidf, acc_glove, acc_finbert]
f1_scores = [f1_tfidf, f1_glove, f1_finbert]

x = np.arange(len(methods))
width = 0.35

bars1 = ax.bar(x - width / 2, accuracies, width, label="Accuracy", color="#0a1628")
bars2 = ax.bar(x + width / 2, f1_scores, width, label="F1 (macro)", color="#D4A84B")

ax.set_ylabel("Score")
ax.set_title("Sentiment Classification Method Comparison")
ax.set_xticks(x)
ax.set_xticklabels(methods)
ax.legend()
ax.set_ylim(0, 1)

# Add value labels
for bar in bars1:
    height = bar.get_height()
    ax.annotate(
        f"{height:.1%}",
        xy=(bar.get_x() + bar.get_width() / 2, height),
        xytext=(0, 3),
        textcoords="offset points",
        ha="center",
        va="bottom",
    )

for bar in bars2:
    height = bar.get_height()
    ax.annotate(
        f"{height:.3f}",
        xy=(bar.get_x() + bar.get_width() / 2, height),
        xytext=(0, 3),
        textcoords="offset points",
        ha="center",
        va="bottom",
    )

plt.show()

# %% [markdown]
# ## What the Comparison Shows
#
# The comparison table above is the chapter's headline result for §10.1, §10.2,
# and the lead-in to §10.4. Three patterns to read off it:
#
# - **TF-IDF + Logistic Regression** is a hard baseline. With bigram features
#   and stop-word removal, lexical signal alone classifies most sentences
#   correctly because financial news vocabulary (`profit`, `loss`, `revenue`,
#   `narrowed`, `tumbled`) is highly polarised.
# - **GloVe averages add semantic similarity** — synonyms cluster, so a held-
#   out phrase like "earnings retreated" benefits from proximity to training
#   examples about "profits falling". The gain over TF-IDF is modest because
#   the document-vector mean throws away word order and negation.
# - **FinBERT-tone evaluated without task-specific fine-tuning** depends
#   strongly on which PhraseBank subset is in scope. On `sentences_allagree`
#   (this notebook's current subset — only sentences where every annotator
#   agreed), FinBERT outperforms both lexical baselines because the high-
#   agreement labels are the cleanest signal and the pre-trained head can
#   transfer well. On the larger mixed-agreement subset that the chapter
#   §10.4 table reports, FinBERT underperforms TF-IDF — the same checkpoint
#   degrades on noisier labels because its training distribution is analyst-
#   report tone, not journalistic news. This is "pre-trained, no task-
#   specific fine-tuning" — not "zero-shot" in the prompted-LLM sense,
#   because the classification head already exists.
#
# The next code cells quantify each contrast with the actual run; the
# follow-up notebook `04_bert_finetuning.py` shows what fine-tuning recovers
# on the harder mixed-agreement subset.

# %%
# Structured output for automated extraction
print("=" * 70)
print("KEY STATISTICS FOR CHAPTER PROSE")
print("=" * 70)
print("\nDataset: Financial PhraseBank")
print(f"Train/Test split: {len(X_train)}/{len(X_test)}")
print(f"\nTF-IDF + LR: Accuracy={acc_tfidf:.1%}, F1={f1_tfidf:.3f}")
print(f"GloVe + LR: Accuracy={acc_glove:.1%}, F1={f1_glove:.3f}")
print(f"FinBERT (pre-trained): Accuracy={acc_finbert:.1%}, F1={f1_finbert:.3f}")
print(f"\nRelative change (FinBERT vs TF-IDF): {relative_change:.1f}%")

# %%
# Save results for chapter integration - both markdown and JSON artifacts
output_dir = get_chapter_dir(10) / "output" / "sentiment_evolution"
output_dir.mkdir(parents=True, exist_ok=True)

# Save structured JSON artifact (for reproducibility verification)
results_artifact = {
    "config": CONFIG,
    "dataset": {
        "name": CONFIG["dataset"]["name"],
        "subset": CONFIG["dataset"]["subset"],
        "total_samples": len(df),
        "train_samples": len(X_train),
        "test_samples": len(X_test),
        "finbert_test_samples": len(X_test_finbert),
        "majority_baseline_accuracy": float(majority_baseline),
    },
    "results": {
        "tfidf_lr": {"accuracy": float(acc_tfidf), "f1_macro": float(f1_tfidf)},
        "glove_lr": {"accuracy": float(acc_glove), "f1_macro": float(f1_glove)},
        "finbert_pretrained": {
            "accuracy": float(acc_finbert),
            "f1_macro": float(f1_finbert),
            "note": "Cross-dataset transfer (trained on analyst reports, tested on news)",
        },
    },
}

json_file = output_dir / "results.json"
with open(json_file, "w") as f:
    json.dump(results_artifact, f, indent=2)

# %%
# Save markdown summary
results_file = output_dir / "results.md"
with open(results_file, "w") as f:
    f.write("# Sentiment Evolution Results\n\n")
    f.write("## Experiment Configuration\n\n")
    f.write(f"- Dataset: {CONFIG['dataset']['name']} ({CONFIG['dataset']['subset']})\n")
    f.write(f"- Train/Test split: {len(X_train)}/{len(X_test)} (stratified, seed={SEED})\n")
    f.write(f"- Majority baseline: {majority_baseline:.1%}\n\n")
    f.write("## Performance Comparison\n\n")
    f.write("| Method | Accuracy | F1 (macro) |\n")
    f.write("|--------|----------|------------|\n")
    f.write(f"| Majority Baseline | {majority_baseline:.1%} | - |\n")
    f.write(f"| TF-IDF + LR | {acc_tfidf:.1%} | {f1_tfidf:.3f} |\n")
    f.write(f"| GloVe + LR | {acc_glove:.1%} | {f1_glove:.3f} |\n")
    f.write(f"| FinBERT (pre-trained*) | {acc_finbert:.1%} | {f1_finbert:.3f} |\n")
    f.write("\n*FinBERT-tone: trained on analyst reports, NOT Financial PhraseBank\n")
    f.write("\n## Key Finding\n\n")
    f.write(f"FinBERT vs TF-IDF change: {relative_change:+.1f}%\n")
    f.write("\n## Critical Insight\n\n")
    if acc_finbert < acc_tfidf:
        f.write("Pre-trained FinBERT (without PhraseBank fine-tuning) **underperforms** TF-IDF.\n")
        f.write(
            "This demonstrates **distribution shift**: same sentiment labels, "
            "different text domains.\n"
        )
        f.write("FinBERT-tone was trained on analyst reports (formal, technical language),\n")
        f.write("while PhraseBank contains financial news sentences (journalistic style).\n")
    else:
        f.write(
            f"Pre-trained FinBERT (without PhraseBank fine-tuning) **outperforms** TF-IDF "
            f"by {relative_change:+.1f}% on this subset.\n"
        )
        f.write(
            "On the high-agreement subset the labels are clean enough that the pre-trained "
            "FinBERT head transfers well, and the analyst-report-trained classifier picks "
            "up the polarised sentence-level financial vocabulary directly.\n"
        )
        f.write(
            "The §10.4 chapter table reports the larger mixed-agreement subset, where the "
            "same checkpoint degrades because labels are noisier and analyst-report tone "
            "and journalistic style diverge more visibly.\n"
        )
    f.write("\n## Model Details\n\n")
    f.write(f"- FinBERT checkpoint: `{CONFIG['finbert']['model_id']}`\n")
    f.write(f"- Max length: {CONFIG['finbert']['max_length']} tokens\n")

print("\nResults saved to:")
print(f"  - {results_file}")
print(f"  - {json_file}")

# %% [markdown]
# ## Key Takeaways
#
# 1. **TF-IDF + logistic regression** reaches 83.2% accuracy / 0.742 macro F1 on
#    this `sentences_allagree` split, setting a hard baseline that the
#    embedding-based methods must clear.
#
# 2. **GloVe averages do not beat TF-IDF on this split** (80.6% accuracy /
#    0.709 macro F1). Averaging static word vectors discards word order and
#    negation, and the resulting document representation loses information that
#    the n-gram features preserve.
#
# 3. **Pre-trained transformer transfer is subset-dependent**: on the high-
#    agreement PhraseBank subset used here, FinBERT outperforms the lexical
#    baselines because clean labels expose the value of pre-training. On the
#    larger mixed-agreement subset (the §10.4 table), the same checkpoint
#    underperforms TF-IDF — distribution shift between analyst-report training
#    text and journalistic test text matters more when label noise grows.
#
# 4. **Distribution shift is the key lesson**: same labels (positive/neutral/
#    negative) do not mean same text distribution, and the gap between subsets
#    here illustrates that label-quality and text-domain gaps interact. Always
#    validate on your target domain *and* your target label-quality regime.
#
# **Next**: See `04_bert_finetuning` for how fine-tuning transforms performance.