AI systems can identify changes in mental-health signals across social-media timelines, but accuracy alone is not enough for clinical applications. At the Leiden AI Community, Leiden University researcher Maryia Zhyrko presented an approach that combines local language models, rule-based methods and traditional machine learning to make such analysis more transparent, reproducible and privacy-conscious.
Mental-health analysis is increasingly becoming an application area for natural language processing, but using AI with highly personal data creates challenges around privacy, explainability and reliability. During the Leiden AI Community, MSc Computer Science student Maryia Zhyrko presented research into a hybrid approach designed specifically around those constraints.
Zhyrko’s research focuses on Natural Language Processing, Clinical NLP, Computational Social Science and Generative AI. Her presentation centred on DreamerNLplus, a framework for analysing social-media timelines and identifying changes in mental-health signals over time while keeping sensitive information under local control.
The work was developed for a shared research task and presented at ACL. Rather than simply determining whether an individual post expresses a positive or negative sentiment, the system attempts to recognise deeper psychological states and, crucially, how these develop across a sequence of posts.
Moving beyond the black box
A central objective of the project was interpretability. According to Zhyrko, mental-health NLP research faces problems with reproducibility, while opaque models make it difficult for clinicians and researchers to understand how a prediction was reached.
The team therefore deliberately avoided relying on a single black-box model. Instead, it combined several techniques, including rule-based matching, a random forest model, classifiers and locally operated open-source language models. The intention was to make individual predictions inspectable and, where possible, accompany them with a human-readable explanation.
Privacy was another reason for choosing this architecture. Because social-media and clinical narratives can contain highly sensitive personal information, the language models were run locally rather than sending data to external AI services.
The system divided the problem into three stages: identifying a person’s current psychological state, detecting changes between states over time and producing summaries of those developments.
Finding signals in individual posts
For the first stage, the researchers classified psychological elements in individual messages. Because some states occurred rarely and the available test data contained only 30 dialogues, a local language model was also used to generate synthetic training data.
The team then compared machine-learning classification with a more transparent rule-based method. In the latter approach, characteristic words and phrases were extracted from the augmented dataset and associated with particular states. New posts could subsequently be matched against these lexical signatures.
That provided an advantage for explainability: researchers could see directly which phrases contributed to a classification. Zhyrko gave examples such as expressions indicating tiredness or withdrawal that could act as signals of a maladaptive state.
The research also highlighted an important distinction between classification and intensity. A system might successfully recognise the type of psychological state expressed in a text while still struggling to determine how strongly that state is present.
According to Zhyrko, models that performed well at identifying specific subcategories often had more difficulty assessing overall intensity. The two problems therefore cannot automatically be treated as the same modelling task.
Detecting when something changes
The second part of the system looked at transitions over time. For each new post, a local language model received the current message together with the five previous posts. It was also provided with labelled examples representing different types of change. The model then generated both a classification and a short explanation of its reasoning.
The researchers compared this with a conventional model built around features including text embeddings, changes relative to earlier posts and sentiment signals. The two approaches showed different weaknesses.
The language model detected more potential changes, resulting in relatively high recall but lower precision. The non-LLM approach showed the reverse pattern: it produced fewer false positives but had greater difficulty recognising rare psychological states.
One difficulty was distinguishing gradual deterioration from a sudden change. A series of posts showing increasingly negative behaviour creates a visible trajectory. A sudden improvement or deterioration without a clear textual lead-up is harder to identify reliably. The small dataset also limits the conclusions that can be drawn. With only 30 timelines, Zhyrko noted that rare transition events are particularly difficult to model.
Rules versus generative summaries
The third task involved summarising developments across an entire timeline. Here the researchers compared two substantially different approaches. The first was a deterministic rule-based summariser. It evaluated sequences, selected predefined templates and assembled these into a fixed narrative.
The advantage was predictability. Because the system used templates rather than generating text freely, its output was reproducible and avoided generative hallucinations. The second approach used a language model to generate the summary directly from the underlying posts. Relevant examples were retrieved dynamically from the training data to guide generation.
This created a familiar AI trade-off. The rule-based system offered greater control and consistency, while the language model could produce more flexible summaries. In the shared-task evaluation, the LLM-based approach performed strongly on similarity metrics, while the rule-based summaries scored particularly well for internal coherence.
Zhyrko argued that this exposes a broader evaluation problem. Metrics measuring textual similarity to human-written reference answers do not necessarily reward the same properties as metrics designed to measure logical consistency.
Looking for patterns across users
The researchers also attempted to identify patterns that recurred across multiple timelines.
To keep the amount of information presented to the language model manageable, timelines were divided into batches. A local model first identified recurring patterns inside each batch. A second stage then combined these findings into a more general pattern.
This approach performed particularly well at identifying improvement. It ranked first in the improvement category of the evaluation and third in deterioration. According to Zhyrko, one explanation could be that deteriorating psychological states are often more ambiguous and therefore harder to classify consistently.
Across the different experiments, however, there was no single method that performed best on every criterion. Improving one metric frequently came at the expense of another. Detecting local changes, maintaining temporal consistency and generating coherent summaries represented partly conflicting objectives.
Potential role in clinical research
Zhyrko emphasised several limitations. Synthetic data may not capture the nuance of real-world mental-health narratives, while the pipeline structure means that errors at an early stage can propagate into later analyses.
The system is therefore better understood as research into interpretable decision support than as an automated diagnostic tool.
Possible applications mentioned during the presentation include helping researchers or clinicians analyse session notes, identifying changing patterns between clinical check-ins and providing an open system whose conclusions can be inspected and evaluated.
The broader issue, according to Zhyrko, is that mental-health NLP is moving closer to practical clinical workflows. In that environment, a model output alone is insufficient: researchers and clinicians need to understand why a system reached its conclusion, while sensitive narratives require strong privacy protections.
One remaining challenge is evaluation. Current metrics for coherence, temporal consistency and similarity can reward conflicting characteristics. Developing a more unified evaluation method, particularly one that reflects clinical usefulness rather than only statistical performance, is therefore an important next step.
The presentation illustrated a broader direction in applied AI: for sensitive domains, the most useful system may not be the largest or most complex model. Combining language models with simpler, inspectable techniques can provide a way to balance flexibility with privacy, reproducibility and explainability.
