PsychiatryNLPKit.analysis.intrinsic_dimensionality_density

PsychiatryNLPKit.analysis.intrinsic_dimensionality_density(token_embedding_vectors, sections=None, k=None)[source]

Estimate intrinsic dimensionality using MLE (Levina & Bickel, 2004).

Intrinsic dimensionality quantifies the local geometric complexity of the semantic space formed by token embeddings. Lower values indicate more redundant speech.

Notes:

Theoretical basis - Lower intrinsic dimensionality indicates more redundant speech (Palominos et al., 2025).

Args:

token_embedding_vectors: Dict mapping section names to token-level: embedding tensors (from TextData.token_embedding_vectors).

sections: Sections to process. None processes all sections in the dict. k: Number of neighbors for MLE estimator. Defaults to

min(10, n_samples - 1).

Returns:

Dict mapping section names to a metric dict with key "ID_MLE". Empty sections receive float("nan").

References:

Palominos, C., Stein, F., Kircher, T., Ayesa-Arriola, R., Palaniyappan, L., Homan, P., Sommer, I. E., & Hinzen, W. (2025). Lexical meaning is lower dimensional in psychosis. Scientific Reports, 16(1), 859. https://doi.org/10.1038/s41598-025-30443-1