PsychiatryNLPKit.analysis.image_text_similarity

PsychiatryNLPKit.analysis.image_text_similarity(image, text, vit_model, sections=None, manage_lifecycle=True)[source]

Compute token-patch similarity between an image and text sections.

Text is split into chunks that fit the model’s context window, encoded at the token level alongside patch-level visual embeddings, and pairwise cosine similarity is aggregated with mean pooling.

Notes:

Theoretical basis - Lower CLIP scores predict higher conceptual disorganization and more disorganization in thoughts (He et al., 2024).

Args:

image: PIL Image object or path to image file. text: Dict mapping section names to text strings (from TextData.data). vit_model: HuggingFaceViTModel instance. Must be loaded unless

manage_lifecycle is True.

sections: Sections to process. None processes all sections in text. manage_lifecycle: If True (default), load and unload the model

internally. Set to False when the caller manages the lifecycle.

Returns:

Dict mapping section names to a metric dict with key "image_text_similarity" (mean-pooled token-patch cosine similarity).

Raises:

RuntimeError: If patch or token embedding extraction fails for the model.

References:

He, R., Palominos, C., Zhang, H., Alonso-Sánchez, M. F., Palaniyappan, L., & Hinzen, W. (2024). Navigating the semantic space: Unraveling the structure of meaning in psychosis using different computational language models. Psychiatry Research, 333, 115752. https://doi.org/10.1016/j.psychres.2024.115752