Recent advances in artificial intelligence (AI) have introduced powerful tools for machine translation (MT) and automated language proofreading (ALP) into academic publishing. However, their evaluation in highly specialized fields such as medicine and pharmacy remains methodologically underexplored. This study presents a multidimensional evaluation framework that integrates human expert judgment with AI-based replication to assess the quality of AI-generated translations and proofreading outputs. The framework covers three quality dimensions applicable to both MT and ALP: semantic fidelity, terminological accuracy and consistency, and grammatical correctness and fluency, as well as a fourth dimension specific to ALP, the appropriateness of edits.In the pilot phase of the research, five machine translation systems were compared using the multidimensional evaluation framework, with DeepL serving as a strong baseline; under advanced idiomatic prompting, the Large Language Model (LLM) systems achieved performance levels comparable to this baseline. The semantic fidelity dimension was further evaluated through an AI simulation of human judgment using ChatGPT-5. The agreement between human and AI evaluators reached κ = 0.81 (95% CI = 0.73–0.88), indicating high consistency and no observed hallucinations within this pilot sample.These preliminary results demonstrate that large language models have promising potential to reproduce human expert quality reasoning when guided by structured prompts. Beyond its technical contribution, our ongoing research represents a step toward transparent and reproducible evaluation for AI-generated academic writing and its automated quality assessment.

Keywords: AI-assisted writing; machine translation; automated proofreading; evaluation Framework; semantic fidelity; terminological accuracy; COMET QE; QuickUMLS; biomedical language; scientific communication; domain-specific NLP; language quality assessment