The lack of transparency and those inflated administrative costs are actually the stronger part of your case here. An executor has a legal duty to act in the beneficiaries' interest and account for their spending, so when someone refuses to provide those details or charges seem unreasonable, that's textbook breach of fiduciary duty. You've got grounds.
What matters is having proof. Start by documenting every time you asked for accounting and got stonewalled - dates, what you asked for, how he responded. If you can get anything in writing showing those "administrative costs," grab it. Then get a probate attorney in your state (they usually charge less than you'd think for an initial consultation) and bring everything. Most courts will remove an executor if you can show he's either incompetent or acting in bad faith, and two years with no distribution plus refusal to explain himself hits both marks.
One thing people sometimes miss: you might not even need to get him removed entirely. Some states let you petition for a conservator or co-executor to oversee him, which forces transparency without the nuclear option. Your attorney can tell you if that's an option where you are and which route makes sense for your situation.