A multi-method ML study combining supervised classifiers, unsupervised clustering, and novel contrastive learning approaches to detect ideological framing in news articles — achieving 90% weighted F1 with SetFit.
01 — Project Overview
News media shape public understanding of politics, but bias often operates through subtle mechanisms — framing, selective coverage, and source attribution — rather than overt partisanship. This project asked: can we detect ideological bias beyond surface-level vocabulary?
Working as a 4-person team in Georgia Tech's CS 7641 Machine Learning course, we applied supervised classification, unsupervised clustering, and two novel contrastive learning methods (SimCSE and SetFit) to a corpus of 6,000+ politically-labelled news articles across five bias categories: Left, Lean Left, Center, Lean Right, and Right.
Managed team, coordinated deliverables across supervised and unsupervised tracks, and led Agile task planning and milestone reviews throughout the semester.
Led comparative analysis across all methods including bias distribution analysis, confusion matrix interpretation, ROC curve generation, and the clustering evaluation heatmap — translating model outputs into clear research narrative.
Authored key sections of the project report including the comparative model analysis, next steps, and the SetFit vs. SimCSE comparison — ensuring technical findings were communicated accessibly.
02 — Development Process
Select any step to expand details — including decisions made, my specific contributions, and key tradeoffs.
As project lead, I structured the semester-long project into parallel workstreams and facilitated early alignment on what we were trying to build and measure.
We combined four datasets to ensure robustness and coverage: AllSides (Kaggle), BASIL, GDELT, and Media Bias Fact Check (MBFC). My contribution was designing the preprocessing quality checklist and leading the analysis visualisations at each stage.
Before applying supervision, we asked whether ideological structure exists naturally in the embedding space. I led the multi-algorithm evaluation and produced the comparative metrics heatmap.
Three supervised methods were implemented across increasing representational complexity. I designed the comparative framework and led the error analysis.
The novelty contribution of the project was applying two contrastive learning frameworks to political bias detection. I co-led the SetFit implementation and authored the comparative analysis between the two approaches.
The final phase synthesised results across all methods into a unified narrative and identified concrete improvement directions. I led this analysis and produced the comparative model table.
03 — Key Results
Select a method to explore the findings. Performance improved consistently as representational power and supervision increased.
PCA reduced 384-dim embeddings to 50 components (91% variance) for clustering and 2 components for 2D visualisation. All nine methods showed overlapping clusters, indicating that the political bias categories are not naturally separable in the embedding geometry.
Key Insight: The clustering experiments confirmed a critical finding: ideological bias does not organise itself into clearly separable geometric clusters. This directly motivated the supervised approach — bias detection fundamentally requires learning discriminative patterns from labelled data, not just finding natural groupings.
The MLP with TF-IDF outperformed fine-tuned RoBERTa despite being architecturally simpler. TF-IDF's 5,000-feature vocabulary successfully captured overt partisan vocabulary differences, while RoBERTa's contextual power was constrained by the limited fine-tuning dataset size.
Key Insight: Extreme categories (Left and Right) were easiest to classify — F1 of 0.93 each. Centrist and "lean" categories caused the most confusion, reflecting genuine semantic proximity between ideologically adjacent positions. This pattern appeared across all supervised methods and directly motivated the contrastive approaches.
SimCSE used dropout noise as augmentation — each sentence produced two slightly different representations via different dropout masks, forming positive contrastive pairs without labels. Trained on 300k+ political sentences, this unsupervised approach created a semantically rich foundation for bias classification.
Key Insight: SimCSE demonstrated that unsupervised contrastive pre-training on domain-specific political text produces useful representations for downstream classification. However, it learns broad semantics rather than class-discriminative signals — task-specific fine-tuning adds meaningful lift (+5%), but supervised contrastive pairing (as in SetFit) yields even greater gains.
SetFit (all-mpnet-base-v2 backbone) amplified supervision by generating N² contrastive pairs from N labelled examples. Stage 1 shaped the embedding space around bias labels using pair-based contrastive loss; Stage 2 froze the encoder and trained a lightweight classifier on the learned representations.
Key Insight: SetFit's performance advantage over SimCSE comes from four compounding factors: supervised contrastive signal directly aligned to bias labels, task-specific representation sculpting, a stronger sentence encoder backbone, and dense supervision with few labels. The two-stage design also prevents the catastrophic forgetting that plagues standard fine-tuning on small datasets.
Placing all methods side-by-side shows a clear performance progression from simple statistical models to contrastive transformers. The pattern reflects increasing representational power — but also the importance of matching model architecture to dataset characteristics.
Overall Conclusion: Supervised models consistently outperformed unsupervised clustering by learning discriminative patterns from labelled data. The highest macro F1 came from SetFit, confirming that label-efficient contrastive learning outperforms both simple lexical features and standard transformer fine-tuning when labelled data is limited. MLP + TF-IDF remains a strong, computationally cheap alternative when computational resources are constrained.
04 — Skills Demonstrated
This project required end-to-end ownership across leadership, technical ML, data analysis, and communication — across a semester-long team environment.
Key Takeaways
No unsupervised method produced clean ideological clusters. Bias categories rely on linguistic cues that only become discriminable under supervision. This is a practically important finding for any system trying to automate media analysis.
MLP + TF-IDF matched SetFit on weighted F1 despite being architecturally far simpler. The right feature representation — lexical in this case — can close the gap to transformer-scale models at a fraction of the compute cost.
SetFit's N² pair amplification achieved the best macro F1 without needing large labelled datasets. For real-world political bias detection, where labelling is expensive and subjective, label-efficient methods are not just practical — they are essential.
The clustering heatmap, confusion matrices, and AUROC curves weren't generated after decisions were made — they drove them. Producing clear, comparable visualisations at each stage was central to how the team navigated method selection and identified failure modes.