The main thing is to distinguish between two types of subordinate clauses with "which". If you can remove the part with "which" and the main clause still makes complete sense, you need a comma. If you can't remove it because that part clarifies which specific object you're talking about, there won't be a comma.
In your example "I bought a book, which was lying on the shelf" - the sentence makes sense even without the second part. You bought a book, period. And the information about the shelf is extra explanation. You need a comma here. But if you say "I bought a book which you recommended to me" - there's no comma here, because without the clarification "which you recommended to me" it's unclear which book you mean.
It's simple to check: try mentally removing the subordinate clause. If the main clause is still logical and complete, you need a comma. If the meaning falls apart or becomes unclear without the subordinate clause, there's no comma.