Extraction of Transaction Information on Handwritten Notes Using Optical Character Recognition (OCR) and Named Entity Recognition (NER)

Meyry Ardianti; Puguh Hiskiawan

doi:10.30871/jaic.v10i5.13985

Authors

Meyry Ardianti Universitas Bunda Mulia
Puguh Hiskiawan Universitas Bunda Mulia

DOI:

https://doi.org/10.30871/jaic.v10i5.13985

Keywords:

Handwritten Notes, IndoBERT, Information Extraction, NER, OCR

Abstract

Handwritten notes remain widely used as transaction proofs, yet extracting information from them automatically is challenging due to variations in handwriting styles, layouts, and image qualities. This research aims to develop a transaction information extraction method for handwritten notes by integrating Optical Character Recognition (OCR) and Named Entity Recognition (NER). The research stages include image preprocessing using grayscale conversion, denoising, and Contrast Limited Adaptive Histogram Equalization (CLAHE), text recognition using EasyOCR across multiple image orientations, and transaction entity identification using IndoBERT with the BIO scheme. OCR performance is evaluated using Character Error Rate (CER) and Word Error Rate (WER), while NER is evaluated using Accuracy, Precision, Recall, and F1-score. Test results show that the NER model achieved an Accuracy of 0.8889, Precision of 0.7857, Recall of 0.8462, and F1-score of 0.8148 on the test data. However, the OCR-NER integration still exhibits character recognition errors and fails to completely extract all transaction attributes, particularly PRICE and AMOUNT. These findings indicate that the OCR-NER integration holds potential for supporting transaction data digitization in handwritten notes, though improving OCR quality and using a larger dataset are necessary to enhance system accuracy and generalizability.

Downloads

Download data is not yet available.

References

[1] E. L. Afriana and D. F. Suyatno, “Development of a Point of Sale (POS) System for the Digitalization of Sales Transaction Recording Based on AI (Case Study: PT XYZ),” J. Emerg. Inf. Syst. Bus. Intell., vol. 7, no. 3, pp. 351–363, 2026, [Online]. Available: https://ejournal.unesa.ac.id/index.php/JEISBI/

[2] E. Dethier, D. R. Kern, G. Stevens, and A. Boden, Making Order in Household Accounting - Digital Invoices as Domestic Work Artifacts, vol. 33, no. 4. Springer Netherlands, 2024. doi: 10.1007/s10606-024-09495-w.

[3] J. Sutrisno, Y. Viencent, S. D. Wijaya, N. Setiawan, D. Novytasari, and R. Nattassha, “Perancangan Aplikasi Inventaris Web dengan Rekomendasi Stok AI untuk Toko Kelontong Mandiri,” J. Digismantech (Jurnal Progr. Stud. Bisnis Digit., vol. 5, no. 1, 2025, doi: 10.30813/digismantech.v5i1.8396.

[4] P. Hiskiawan, C.-C. Chen, and Z.-K. Ye, “Processing of electrical resistivity tomography data using convolutional neural network in ERT-NET architectures,” Arab. J. Geosci., vol. 16, no. 10, pp. 1–14, 2023, doi: 10.1007/s12517-023-11690-w.

[5] P. Hiskiawan, M. K. Sari, R. E. Siregar, N. A. Valencia, and T. P. Wijayanti, “A Deep Learning Data Fusion Approach for Prostate MRI Zonal Segmentation Using Dual-Channel U-Net with T2 and ADC Images,” in 2026 International Conference on Current Research in Artificial Intelligence and Data Science (ICCRAIDS), Banyumas, Indonesia: IEEE, 2026, pp. 1–7. doi: 10.1109/ICCRAIDS67816.2026.11519671.

[6] S. V. Mahadevkar, S. Patil, K. Kotecha, L. W. Soong, and T. Choudhury, “Exploring AI-driven approaches for unstructured document analysis and future horizons,” J. Big Data, vol. 11, no. 1, 2024, doi: 10.1186/s40537-024-00948-z.

[7] Q. Zhang, F. Liu, and W. Song, “IMTLM-Net: improved multi-task transformer based on localization mechanism network for handwritten English text recognition,” Complex Intell. Syst., vol. 11, no. 1, pp. 1–18, 2025, doi: 10.1007/s40747-024-01713-8.

[8] Y. Zhang and G. Xiao, “Named Entity Recognition Datasets: A Classification Framework,” Int. J. Comput. Intell. Syst., vol. 17, no. 1, 2024, doi: 10.1007/s44196-024-00456-1.

[9] L. Cooper and F. Richardson, “Automatic Receipt Segmentation and Entity Recognition using OCR and Named Entity Recognition (NER),” pp. 1–13, 2023.

[10] O. Tüselmann and G. A. Fink, “Neural models for semantic analysis of handwritten document images,” Int. J. Doc. Anal. Recognit., vol. 27, no. 3, pp. 245–263, 2024, doi: 10.1007/s10032-024-00477-8.

[11] J. Andrés, C. Wall, S. Tarride, M. Coustaty, A. H. Toselli, and E. Vidal, “The PARES Database: Information Extraction over Historical Parish Records,” Int. J. Doc. Anal. Recognit., vol. 28, no. 3, pp. 377–389, 2025, doi: 10.1007/s10032-025-00531-z.

[12] V. Choudhary, P. Guha, G. Pau, and S. Mishra, “Comparative Analysis of Open-Source OCR Frameworks for Receipt Data Extraction and Post-Processing,” Environ. Sustain. Indic., vol. 26, pp. 1–10, 2025.

[13] P. Hiskiawan, Y. M. Geasela, H. Heryanto, E. Stephanie, M. Ardianti, and F. A. Sukarno, “Trustworthy Data Science Framework for Non-Invasive Nutritional Screening Using Computer Vision,” in 2025 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS), Jakarta, Indonesia: IEEE, 2025, pp. 1743–1748. doi: 10.1109/ICIMCIS68501.2025.11327268.

[14] F. Wolf and G. A. Fink, “Self-training for handwritten word recognition and retrieval,” Int. J. Doc. Anal. Recognit., vol. 27, no. 3, pp. 225–244, 2024, doi: 10.1007/s10032-024-00484-9.

[15] Z. Yan et al., “DocExtractNet: A novel framework for enhanced information extraction from business documents,” Inf. Process. Manag., vol. 62, no. 3, p. 104046, 2025, doi: 10.1016/j.ipm.2024.104046.

[16] V. Choudhary, P. Guha, G. Pau, and S. Mishra, “An overview of smart agriculture using internet of things (IoT) and web services,” Environ. Sustain. Indic., vol. 26, no. January, p. 100607, 2025, doi: 10.1016/j.indic.2025.100607.

[17] J. R. Ilagan and J. B. Ilagan, “Graph-partitioning entity resolution for resolving noisy product names in OCR scans of retail receipts,” Procedia Comput. Sci., vol. 239, no. 2023, pp. 338–345, 2024, doi: 10.1016/j.procs.2024.06.180.

[18] A. Takano, T. C. H. Cole, and H. Konagai, “A novel automated label data extraction and data base generation system from herbarium specimen images using OCR and NER,” Sci. Rep., vol. 14, no. 1, pp. 1–8, 2024, doi: 10.1038/s41598-023-50179-0.

[19] A. Rombach and N. Mehdiyev, “Beyond Accuracy: Understanding Model Confidence in Key Information Extraction with Conformal Prediction,” Int. J. Doc. Anal. Recognit., vol. 29, no. 2, pp. 551–563, 2026, doi: 10.1007/s10032-026-00572-y.

[20] A. Sarhan et al., “Egyptian car plate recognition based on YOLOv8, Easy-OCR, and CNN,” J. Electr. Syst. Inf. Technol., vol. 11, no. 1, pp. 1–27, 2024, doi: 10.1186/s43067-024-00156-y.

[21] N. U. Muchtar, D. Arifianto, and R. Umilasari, “Analisis Kinerja Transformer Untuk Named Entity Recognition (NER) Menggunakan IndoBERT Pada Transkrip Video Politik Berbahasa Indonesia,” Discov. J. Ilmu Pengetah., vol. 10, no. 2, pp. 180–199, 2025, doi: 10.33752/discovery.v10i2.10133.

[22] A. De Nardin, S. Zottin, C. Piciarelli, G. L. Foresti, and E. Colombi, “In-domain versus out-of-domain transfer learning for document layout analysis,” Int. J. Doc. Anal. Recognit., vol. 28, no. 2, pp. 161–175, 2025, doi: 10.1007/s10032-024-00497-4.

[23] C. A. Romein, A. Rabus, G. Leifert, and P. B. Ströbel, “Assessing advanced handwritten text recognition engines for digitizing historical documents,” Int. J. Digit. Humanit., vol. 7, no. 1, pp. 115–134, 2025, doi: 10.1007/s42803-025-00100-0.

[24] S. Chakraverti, P. Agarwal, H. S. Pattanayak, S. P. S. Chauhan, A. K. Chakraverti, and M. Kumar, “De-noising the image using DBST-LCM-CLAHE: A deep learning approach,” Multimed. Tools Appl., vol. 83, no. 4, pp. 11017–11042, 2024, doi: 10.1007/s11042-023-16016-2.

[25] P. Härtinger and C. Steger, “Adaptive histogram equalization in constant time,” J. Real-Time Image Process., vol. 21, no. 3, pp. 1–9, 2024, doi: 10.1007/s11554-024-01465-1.

[26] M. Mayr et al., “Data-efficient handwritten text recognition of diplomatic historical text,” Multimed. Tools Appl., vol. 84, no. 32, pp. 39107–39122, 2025, doi: 10.1007/s11042-024-20545-9.

[27] X. Chen, W. Zhang, and S. Pan, “CLESR: Context-Based Label Knowledge Enhanced Span Recognition for Named Entity Recognition,” Int. J. Comput. Intell. Syst., vol. 17, no. 1, pp. 1–13, 2024, doi: 10.1007/s44196-024-00595-5.

[28] S. C. Sreenivas, S. Chowdhury, and M. Masum, “Enhancing Clinical Named Entity Recognition via Fine-Tuned BERT and Dictionary-Infused Retrieval-Augmented Generation,” Electron., vol. 14, no. 18, pp. 1–19, 2025, doi: 10.3390/electronics14183676.

[29] D. Chicco and G. Jurman, “The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification,” BioData Min., vol. 16, no. 1, pp. 1–23, 2023, doi: 10.1186/s13040-023-00322-4.

[30] P. Hiskiawan, T. P. Wijayanti, H. Heryanto, S. Everlin, S. A. Yasodhara, and D. Alexander, “Mel-Frequency Cepstral Coefficients and Neural Networks for Indonesian Traditional Music Recognition,” in 2025 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS), Jakarta, Indonesia: IEEE, 2025. doi: 10.1109/ICIMCIS68501.2025.11327436.