WHY DOES IT TAKE SOME COUPLES AGES TO CONCEIVE?

I was just wondering why it takes some people so long to have a baby.. you'd think it's as simple as having sex and the sperm fertilizes the egg. But what I don't understand is why it takes some couples years to have a baby? What stops it?