Skip to content

API reference

Tokenizer

Bases: Model

__call__(text)

Calling a tokenize config instance like a function just calls the tokenize method.

MecabTagger

Bases: Model

userdic_path: backend: ekonlpy verbose: false

NLTKTagger

Bases: Model

lemmatize: false stem: true lemmatizer: target: nltk.stem.WordNetLemmatizer stemmer: target: nltk.stem.PorterStemmer

Normalizer

Bases: Model

Main Normalizer class for generic text. Normalize unicode, hyphens, quotes, whitespace. By default, the normal form NFKC is used for unicode normalization. This applies a compatibility decomposition, under which equivalent characters are unified, followed by a canonical composition. See Python docs for information on normal forms: http://docs.python.org/2/library/unicodedata.html#unicodedata.normalize

__call__(text)

Calling a normalizer instance like a function just calls the normalize method.

normalize(text)

Run the Normalizer on a string. :param text: The string to normalize.

Stopwords

Bases: Model

A word is a stopword when stopwords_fn returns True for it or it is in the loaded list.

The set operators combine that rule, not just the lists: a | b and a + b match a word either operand matches, a & b one both match, a - b one a matches and b does not, and a ^ b one exactly one operand matches. The result keeps the operands' lowercase; operands that differ in lowercase raise ValueError, and so does & when both have a stopwords_fn (the result would have no finite list).

name: stopwords lowercase: true stopwords_fn: stopwords_list: stopwords_path: nltk_stopwords_lang: verbose: False

load_stopwords()

Load stopwords from file or function

__call__(word)

Calling a stopwords instance like a function just calls the is_stopword method.

is_stopword(word)

:type word: str :returns: bool

__le__(other)

Subset: every word self matches, other matches too.

Tokenize texts and datasets.Dataset columns; each function works as a HyFI workflow step.

In a workflow, pass the tokenizer as a config with _target_ (or ${tokenizer} from the tokenizer group); HyFI builds it before calling the step.

tokenize_texts(texts, tokenizer=None)

Tokenize each text (a single string counts as one text), line by line.

plot_dendrogram(model, **kwargs)

Plots the hierarchical clustering dendrogram.

plot_similarity_distribution(similarity_matrix, percentile=80, distance_threshold=None, title_name='', title_fontsize=10, output_dir='.', show_fig=False, save_fig=True)

Plots the distribution of cosine similarities between pairs of documents.

find_similar_docs_by_clustering(data, num_workers=2, min_num_docs=5, percentile=80, distance_threshold=None, linkage='average', grouping_freq='W', grouping_name='Week', timestamp_col='timestamp', token_col='tokens', id_col='id', cluster_col='cluster', duplicate_col='duplicate', dist_fig_col='dist_fig', output_dir='.', show_fig=False, save_fig=False, verbose=False)

Finds similar documents in the given data using cosine similarity and Agglomerative Clustering.