Natural Language Processing Using Neighbour Entropy-based Segmentation

Jianfeng Qiao, Xingzhi Yan, Shuran Lv


In natural language processing (NLP) of Chinese hazard text collected in the process of hazard identification, Chinese word segmentation (CWS) is the first step to extracting meaningful information from such semi-structured Chinese texts. This paper proposes a new neighbor entropy-based segmentation (NES) model for CWS. The model considers the segmentation benefits of neighbor entropies, adopting the concept of "neighbor" in optimization research. It is defined by the benefit ratio of text segmentation, including benefits and losses of combining the segmentation unit with more information than other popular statistical models. In the experiments performed, together with the maximum-based segmentation algorithm, the NES model achieves a 99.3% precision, 98.7% recall, and 99.0% f-measure for text segmentation; these performances are higher than those of existing tools based on other seven popular statistical models. Results show that the NES model is a valid CWS, especially for text segmentation requirements necessitating longer-sized characters. The text corpus used comes from the Beijing Municipal Administration of Work Safety, which was recorded in the
fourth quarter of 2018.

To cite this article: J. Qiao, X. Yan, and S. Lv, “Natural Language Processing Using Neighbour Entropy-based Segmentation” in CIT. Journal of Computing and Information Technology, vol. 29, no. 2, pp. 113–131, 2021, doi: 10.20532/cit.2021.1005393


Text Mining; Text Segmentation; Chinese Word Segmentation; Safety Management; Hazard Analysis

Full Text:


Creative Commons License
This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.

Crossref Similarity Check logo

Crossref logologo_doaj

 Hrvatski arhiv weba logo