FIG-Loneliness

Automated detection of loneliness self-disclosure in Reddit posts — and the production system that makes a research pipeline usable.

The problem#

Loneliness is a significant public-health concern, associated with heightened risk of depression, anxiety, and premature mortality. On platforms like Reddit, people disclose feelings of social isolation frequently without ever using the word "lonely" — the language is indirect, contextually grounded, and overlaps with adjacent constructs like depression and social anxiety. That makes automated detection genuinely hard.

The team worked with the FIG-Loneliness dataset (Jiang et al., 2022): 5,633 Reddit posts from four communities (r/loneliness, r/lonely, r/youngadults, r/college), annotated for a binary task — does the post disclose loneliness (2,633 posts) or not (3,000 posts)? The research question was which text-representation strategy, from handcrafted psycholinguistic features through to fine-tuned transformers, best captures the subtle markers of self-disclosed loneliness, while weighing predictive performance against interpretability.

The team's modelling approach#

The pipeline a teammate led compared six text representations across sixteen experimental configurations, so I'll summarise it briefly for context before getting to the part I built.

Six representations were evaluated: a 13-dimensional psycholinguistic feature vector (pronoun ratio, negation ratio, social-word ratio, and similar); TF-IDF over unigrams and bigrams; a concatenation of TF-IDF and the linguistic features; mean-pooled Word2Vec embeddings; Sentence-BERT embeddings via all-MiniLM-L6-v2; and end-to-end fine-tuning of distilbert-base-uncased. The five non-neural representations were each paired with Logistic Regression, Linear SVM, and Random Forest; DistilBERT was fine-tuned directly. Models were selected on validation F1, and the classification threshold was derived from the validation ROC curve using the Youden J statistic rather than defaulting to 0.5.

The headline results, evaluated once on the held-out test split:

DistilBERT topped the table, but the strong showing of the combined TF-IDF + Linguistic logistic-regression model (F1 = 0.9570) was the more interesting finding for interpretability — near-transformer performance from a fully transparent model.

What I built: from pipeline to product#

A research pipeline that produces JSON and PNG artefacts in a results/ folder is not usable by anyone who isn't running the code. My role was to turn it into a system: a deployed API that serves every artefact, an inference engine that runs the trained models on new text, a diagnostic module that interrogates model errors, and a dashboard that makes all of it explorable.

Error-analysis module (100% mine)#

Before the API, I built the qualitative error-analysis module (src/error_analysis.py). For each trained model it collects every false positive and false negative on the test set with full text, true label, predicted probability, and all 13 feature values, then generates four diagnostics: an FP/FN count chart, text-length distributions by error type, a linguistic-feature comparison between the FP and FN groups, and a cross-model confusion-overlap heatmap showing which posts the most models get wrong.

It auto-generates qualitative observations, but conditionally — only when the feature-mean difference between error groups exceeds one standard deviation, so the system never narrates noise.

FastAPI backend and inference engine (90% mine)#

I implemented the complete read-only REST backend (api/main.py, routers for EDA, models, and prediction, plus the Pydantic schemas). The endpoints serve pipeline status, EDA statistics, model results, evaluation data (confusion matrices, ROC curves, classification reports), interpretability summaries, error-analysis data, PNG plot serving, and both single and batch inference.

The inference engine (src/inference.py) is the part I'm most pleased with as a piece of engineering:

• Input validation rejects inputs under three tokens with a typed InvalidInputError, so the model never runs on noise.
• Threshold loading reads the ROC-derived optimal threshold from the persisted validation JSON, so inference matches the evaluation methodology exactly rather than assuming 0.5.
• A thread-safe singleton model loader uses double-checked locking to prevent redundant model loads under concurrent requests — important on a CPU-bound Hugging Face Space where loading a transformer more than once would be costly.
• Per-item failure handling in the batch endpoint means one malformed post doesn't fail the whole request.

Next.js dashboard (90% mine)#

I designed and built the full Next.js 14 App Router dashboard that consumes the API — a typed API client, shared TypeScript interfaces matching every API schema, and a shared component library (metric cards, plot images, section cards, tags, callouts, representation badges). Navigation uses the shadcn/ui collapsible sidebar on desktop and a Sheet drawer on mobile.

Eight pages make the whole pipeline explorable:

1. Overview — pipeline status, headline metrics, dataset-split table, research-question cards
2. Preprocessing — the 8-step pipeline, a 13-feature reference table, and a paginated sample-post explorer with per-feature sparklines
3. EDA — a seven-tab interface with interactive Recharts for class distribution, text length, linguistic features, n-grams, word clouds, POS distribution, and a correlation heatmap
4. Models — a sortable results table with a validation/test toggle
5. Model detail — per-model confusion matrix, ROC curve, and classification report
6. Interpretability — a performance-vs-transparency trade-off scatter, Logistic-Regression coefficient bars, and DistilBERT attention
7. Error analysis — an FP/FN browser backed by the module I built, with a model dropdown and paginated example cards
8. Live demo — single-post inference with examples, plus a batch mode with an individual textarea per post, add/remove controls, and an expandable results table

What I'd do differently#

The dashboard does a lot, and in hindsight some of its eight pages could have been consolidated — the model and model-detail pages especially. I optimised for completeness because it doubled as the assessment artefact, but a sharper product would lead with the live demo and the error analysis and treat the rest as drill-down.

I'd also add request-level caching to the API. Right now every dashboard page hits the backend fresh; since the underlying results are static artefacts, an HTTP cache layer (or even pre-generating the JSON into the frontend build) would remove the cold-start wait that the free HF Spaces tier introduces.

Finally, the high F1 scores deserve adversarial probing. I'd want to feed the deployed model deliberately ambiguous posts — self-narration without loneliness, loneliness expressed through neutral vocabulary — and use the error-analysis tooling I built to map where the decision boundary actually sits.

Code and links#