Conference is a self-sterile pear variety, and that's almost certainly your problem. It needs a pollinator of a different, compatible variety. Apple trees won't help here - pears are only pollinated by pears. Bees can fly from far away, but if there are no other pear trees nearby, you'll get minimal fruit set or none at all, even with abundant flowering.
The solution is simple: plant another pear variety as a pollinator next to it. For Conference, varieties like Williams, Beurré Bosc, Forest Beauty, or other summer-autumn pears work well. You don't even need a mature tree - a young sapling will pollinate in just a year or two. If space is tight, you can graft a cutting of a different variety directly onto your pear tree, though this method is more complicated.
Soil and fertilizer are a secondary issue. Of course, depleted soil won't help, but if the tree is growing well, nutrition is more or less fine. Check the drainage (pears don't like waterlogging), but first and foremost solve the pollinator problem - that's what will give you results.