PsychiatryNLPKit.data.TextData

class PsychiatryNLPKit.data.TextData(sections, lang='en', embedding_model=None, generative_model=None)

Main text data container with lazy-computed linguistic properties.

Args:

sections: List of Section objects. lang: Language code (“en” or “fr”). Determines tokenizer and stopwords. embedding_model: Optional embedding model for vector computations. generative_model: Optional generative model for perplexity computations.

Properties compute on first access and cache results. The _computed set tracks which properties have been materialized.

__init__(sections, lang='en', embedding_model=None, generative_model=None)

Methods

__init__(sections[, lang, embedding_model, ...])

Attributes

attention_scores	Attention scores per section, shape (n_tokens, n_tokens).
content_word_embedding_vectors	Word2Vec embeddings for content words per section, shape (n_words, dim).
content_words	Content words (nouns, verbs, adjectives, adverbs) per section.
data	Raw section text keyed by section name.
paragraph_generative_tokens	Tokenized paragraphs for generative perplexity.
pos_tags	list of sentences, each a list of (word, lemma, tag) tuples.
section_names
sentence_embedding_vectors	Sentence-level embeddings per section, shape (n_sentences, dim).
sentence_generative_tokens	Tokenized sentences for generative perplexity.
sentences	Sentences per section (list of strings).
syntax_trees	Benepar constituency trees per section (one per sentence).
token_embedding_vectors	Token-level embeddings per section, shape (n_tokens, dim).

__init__(sections, lang='en', embedding_model=None, generative_model=None)

property data: dict[str, str]

Raw section text keyed by section name.

property section_names: list[str]

property pos_tags: dict[str, list[list[tuple[str, str, str]]]]

list of sentences, each a list of (word, lemma, tag) tuples.

Type:

POS tags per section

property syntax_trees: dict[str, list[Any]]

Benepar constituency trees per section (one per sentence).

property sentences: dict[str, list[str]]

Sentences per section (list of strings).

property token_embedding_vectors: dict[str, Tensor]

Token-level embeddings per section, shape (n_tokens, dim).

Computed via the embedding model’s last_hidden_state with attention-mask aware mean pooling. Requires output_attentions=True on the model.

property attention_scores: dict[str, Tensor]

Attention scores per section, shape (n_tokens, n_tokens).

Aggregated over layers and heads. Computed alongside token embeddings.

property sentence_embedding_vectors: dict[str, Tensor]

Sentence-level embeddings per section, shape (n_sentences, dim).

property content_words: dict[str, list[str]]

Content words (nouns, verbs, adjectives, adverbs) per section.

property content_word_embedding_vectors: dict[str, Tensor]

Word2Vec embeddings for content words per section, shape (n_words, dim).

property paragraph_generative_tokens: dict[str, dict[str, Tensor]]

Tokenized paragraphs for generative perplexity.

property sentence_generative_tokens: dict[str, dict[str, Tensor]]

Tokenized sentences for generative perplexity.