PsychiatryNLPKit.analysis.zipf_frequency

PsychiatryNLPKit.analysis.zipf_frequency(pos_tags, sections=None, lang='en')[source]

Mean and standard deviation of Zipf word frequency per section.

Word frequency is computed via wordfreq.zipf_frequency on the word lemma. Each word occurrence contributes once; lemmas not recognized by the frequency database (Zipf frequency 0.0) are excluded from both the mean and the standard deviation.

Notes:

Theoretical basis - Word frequency patterns are altered in schizophrenia, with patients using less frequent words that relate to verbal productivity deficits and can distinguish healthy controls from first-episode and chronic schizophrenia patients (Figueroa-Barra et al., 2022).

Args:

pos_tags: Dict mapping section names to lists of POS-tagged sentences: (from TextData.pos_tags).
sections: Sections to process. None processes all sections in: pos_tags.

lang: The language of the text. Must be one of “en” or “fr”.

Returns:

Dict mapping section names to a metric dict with keys "zipf_frequency_mean" and "zipf_frequency_std".

References:

Figueroa-Barra, A., Del Aguila, D., Cerda, M., Gaspar, P. A., Terissi, L. D., Durán, M., & Valderrama, C. (2022). Automatic language analysis identifies and predicts schizophrenia in first-episode of psychosis. Schizophrenia, 8(1), 53. https://doi.org/10.1038/s41537-022-00259-3