Prediksi Performa Large Language Model Berdasarkan Benchmark 2020-2026 dengan Algoritma Random Forest dan LSTM
Keywords:
LLM, benchmark, performance prediction, Random Forest, LSTM, model evaluationAbstract
This study aims to predict the performance of Large Language Models (LLM) based on benchmark scores for the 2020-2026 period by comparing the Random Forest and Long Short-Term Memory (LSTM) algorithms. The data used comes from uploaded dataset archives, consisting of benchmark scores, model catalogs, computational estimates, capability milestones, and price history. After data merging, 1,276 score observations were obtained from 113 models and 15 types of benchmarks. The experiment was conducted with a chronological division: 2020-2024 data as training data, 2025 as validation, and 2026 as test data. The results show an increase in the average benchmark score from 46.26% in 2020 to 88.13% in 2026. On the 2026 test data, LSTM obtained an MAE of 11.77 and an RMSE of 14.62, better than Random Forest with an MAE of 23.67 and an RMSE of 30.11. These findings suggest that temporal patterns and interannual capability shifts are important considerations when predicting LLM performance. However, the negative R-squared value in the test data indicates that the frontier performance spike in the 2026 model is still difficult to reliably extrapolate from historical data. This study recommends the use of more recent benchmark data, architectural feature enhancement, and cross-source validation to enhance the predictive model's robustness.
Downloads
References
[1] P. Liang et al., “Holistic evaluation of language models,” Transactions on Machine Learning Research, 2023.
[2] Y. Chang et al., “A survey on evaluation of large language models,” ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 3, Article 39, 2024, doi: 10.1145/3641289.
[3] M. T. R. Laskar et al., “A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations,” Proceedings of EMNLP 2024, pp. 13785-13816, 2024, doi: 10.18653/v1/2024.emnlp-main.764.
[4] D. Hendrycks et al., “Measuring massive multitask language understanding,” in International Conference on Learning Representations (ICLR), 2021.
[5] A. Srivastava et al., “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,” Transactions on Machine Learning Research, 2023.
[6] J. Hoffmann et al., “Training compute-optimal large language models,” Advances in Neural Information Processing Systems, vol. 35, 2022.
[7] A. Chowdhery et al., “PaLM: Scaling language modeling with Pathways,” Journal of Machine Learning Research, vol. 24, no. 240, pp. 1-113, 2023.
[8] J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, 2022.
[9] S. Yao et al., “Tree of thoughts: Deliberate problem solving with large language models,” Advances in Neural Information Processing Systems, vol. 36, 2023.
[10] R. Iranzad and X. Liu, “A review of random forest-based feature selection methods for data science education and applications,” International Journal of Data Science and Analytics, vol. 20, pp. 197-211, 2025, doi: 10.1007/s41060-024-00509-w.
[11] A. Casolaro, V. Capone, G. Iannuzzo, and F. Camastra, “Deep learning for time series forecasting: Advances and open problems,” Information, vol. 14, no. 11, p. 598, 2023, doi: 10.3390/info14110598.
[12] K. Liao, X. Xuan, and K.-L. Ma, “Deep learning for time series forecasting: A survey of recent advances,” Frontiers of Computer Science, vol. 20, p. 2011359, 2026, doi: 10.1007/s11704-025-50947-3.
[13] K. Cobbe et al., “Training verifiers to solve math word problems,” arXiv:2110.14168, 2021.
[14] S. Minaee et al., “Large language models: A survey,” arXiv:2402.06196, 2024.
[15] OpenAI, “GPT-4 technical report,” arXiv:2303.08774, 2024.
[16] Y. Saleh, M. Abu Talib, Q. Nasir, and F. Dakalbab, “Evaluating large language models: A systematic review of efficiency, applications, and future directions,” Frontiers in Computer Science, vol. 7, p. 1523699, 2025, doi: 10.3389/fcomp.2025.1523699.
[17] H. Touvron et al., “LLaMA: Open and efficient foundation language models,” arXiv:2302.13971, 2023.
[18] H. Touvron et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv:2307.09288, 2023.
[19] L. Wang et al., “Enterprise large language model evaluation benchmark,” arXiv:2506.20274, 2025.
[20] Y. Yang and H. Wang, “Random forest-based machine failure prediction: A performance comparison,” Applied Sciences, vol. 15, no. 16, p. 8841, 2025, doi: 10.3390/app15168841.
