TOKENIZATION METHODS FOR TAJIK TEXT USING PYTHON

Istamuqlov Hasanjon, Muzafarov Dilshod

This scientific article examines tokenization methods for Tajik text using the Python programming language. The authors analyze the characteristics of the Tajik alphabet and grammar, as well as typical tokenization problems related to its specificity. The article provides an overview of the main libraries and packages for text processing in Python and describes approaches to tokenization based on examples from other languages. The work presents the results of experiments using morphological, statistical, and neural network approaches to tokenization, and suggests directions for future research in this field.

Keywords:tokenization, Tajik language, Python programming language, morphological approach, statistical approach, neural networks, deep learning, natural language processing, alphabet, grammar