Architecture & Concepts

This page explains how PsychiatryNLPKit is structured and how the components work together. The package is developed in the CEYMH / Douglas Research Group at McGill University.

Data Flow

The analysis pipeline follows a unidirectional data flow:

Raw Text → Section → TextData → compute() / BatchAnalyzer → Results

1. Sections — Named text segments (paragraphs, utterances, interview responses). Each section has a name and text field.

2. TextData — The central container that holds sections, language setting, and optional model references. It lazily computes and caches linguistic features on first access:

POS tags and lemmas (via Stanza)
Constituency parses (via Benepar)
Word and sentence embeddings (via HuggingFace models)
Dependency parses

In practice, a TextData object often represents one participant, one transcript, or one document split into named sections. Descriptive names help when exporting results or joining them back to metadata tables.

from PsychiatryNLPKit.data import Section, TextData

sections = [
  Section(
      text="I work in a factory that produces humanoid robots. The assembly line runs all day.",
      name="Paragraph 1",
  ),
  Section(
      text="I just came back from a vacation in the mountains. The weather was calm and cool.",
      name="Paragraph 2",
  ),
  Section(
      text="The interview discussed school, work, and family stress. The participant described each in detail.",
      name="Paragraph 3",
  ),
]
data = TextData(sections=sections, lang="en")

If your raw data arrives as a CSV or a folder of text files, convert each row or file into a Section before creating TextData.

3. Analysis Registry — A public registry (PsychiatryNLPKit.analysis.get_analysis_spec) maps every analysis name to the TextData property it needs, the language and models it requires, and its keyword arguments. Both TextData.compute and BatchAnalyzer resolve analyses through this registry, so callers never look up function signatures.

4. Analysis Functions — Pure functions that accept pre-computed features from TextData and return dictionaries of metrics keyed by section name. They don’t modify input data and have no side effects.

5. BatchAnalyzer — Orchestrates multiple analyses, keeps any required models loaded for the whole run, collects results, and handles errors gracefully (one failed analysis doesn’t stop the others).

Design Principles

Scientific Grounding

Each analysis function implements metrics derived from peer-reviewed research on language markers of psychosis risk, formal thought disorder, disorganization, and cognitive impairment. Metrics have been shown to correlate with clinical rating scales such as PANSS, TLC, and TLI.

Batch Efficiency

Expensive computations (tokenization, embedding generation, constituency parsing) are:

Lazy-loaded: Models load only when a feature is first accessed
Cached: Results are stored on the TextData object; subsequent accesses return cached values
Shared: A single TextData instance serves all analyses — no redundant computation

Hardware Acceleration

All deep learning pipelines automatically detect and use available accelerators:

CUDA (NVIDIA GPU / AMD HIP)
MPS (Apple Silicon Metal)
Intel XPU (Integrated graphics)
CPU (fallback)

Device selection is automatic via pnlp.device. You don’t need to configure it manually.

Composable Architecture

The package supports two usage patterns:

Single analyses: Run one analysis by name with data.compute(name). The registry supplies the right property, language, and models automatically.
Batch API: Run all or a subset of analyses with one call via BatchAnalyzer, which delegates to compute while keeping required models loaded for the run.

Both patterns share the same TextData container, and both resolve analyses through the registry.

Language Support

PsychiatryNLPKit supports English ("en") and French ("fr"). The language is set when creating a TextData object:

data = TextData(sections=sections, lang="en")  # or "fr"

The package automatically selects the appropriate NLP models for each language:

Stanza pipelines (POS tagging, lemmatization) — separate models per language
Benepar constituency parsers (benepar_en3, benepar_fr2)

Analysis Categories

Category	Description
Syntax	POS tag ratios, clause structure, syntax tree depth, sentence complexity metrics
Similarity	Semantic coherence measured as cosine similarity between adjacent word or sentence embeddings
Perplexity	Language model perplexity at paragraph and sentence levels (both generative LM and masked LM)
Graph	Network metrics from structural word-transition graphs (node count, edge count, diameter, z-scores) — requires `networkx`
Density	Semantic space dimensionality via PCA explained variance, intrinsic dimension estimation, vector-unpacking semantic density, and propositional idea density
Lexicon	Disfluency markers and filler word frequency analysis
ImageSimilarity	Cross-modal cosine similarity between images and text sections using multimodal embeddings — requires `pillow`

Module Organization

Module	Purpose
`PsychiatryNLPKit.data`	Data containers (Section, TextData, ImageData) and text utilities
`PsychiatryNLPKit.analysis`	Analysis functions organized by category, plus BatchAnalyzer orchestration
`PsychiatryNLPKit.model`	Model wrappers for HuggingFace LLMs and multimodal embedding models with device management
`PsychiatryNLPKit.config`	Logging configuration, device detection, and HuggingFace token handling