Some updates:
For an identical system, I successfully ran the entire decoupling process in the 2018.8 version of gmx, also using totally the same mdp settings. But I met the same problem with this: How to solve Lincs warning in free energy calculation in 2023.1 version. (particularly from init-lamda-state 11 to init-lambda 18, simulation can run well in the rest of lambda states.)
I successfully ran the example of gmx tutorial (the methane-water Free Energy Calculations) in gromacs version 2023.1. So I guess it’s the topology of the ligand. I tried to use LigParGen (OPLS/AA) and charmm-gui (CHARMM36) to build the topology, but they both encountered the above problem (How to solve Lincs warning in free energy calculation) in such calculation.