Buying a home may no longer be the American dream (now people just want to get out of debt someday), but it’s still a huge goal for many. But if you don’t have a good credit score, that dream might go up in flames.

A lot of things factor into getting a mortgage, but it starts with your credit score. If your credit score isn’t good enough, nothing else matters. Depending on the loan program, lender, and applicant’s specific credit history, the minimum credit score necessary to buy a home varies. Heather McRae, a senior loan officer for Chicago Financial Services, says 580 is the lowest score she’s seen in recent loan approvals, and she hasn’t seen anything lower than that in a long time.

A 580 is a bad credit score (on the often-used scale of 300 to 850). A 600 credit score isn’t the best credit score, either, but that’s generally considered the minimum for an FHA-backed loan, says Scott Sheldon, a senior loan officer at Sonoma County Mortgages. FHA loans are popular among first-time home buyers and have a minimumdown payment of 3.5%. If you’re going for a conventional loan (one not backed by a government agency), the minimum credit score is 620, McRae and Sheldon say.

Those minimum scores are just that—they get your foot in the door. But mortgages are complicated and require a lot of paperwork, so if you’re barely squeaking in, you can expect a rough ride. Take that 600 minimum on a 30-year fixed-rate FHA loan.

“You’re going to have a lot—I mean, a lot—of hoops to jump through,” Sheldon says. “It’s going to be very unpleasant.”

For people with lower credit scores, a down payment plays a huge factor in approval. McRae says a 20% down payment and a 620 credit score should get you a loan, but a smaller down payment will tighten credit standards.

McRae adds, “680 is kind of like the unspoken minimum, but you can definitely get something done below that. If you’re under a 680 credit score, the mortgage insurance premiums get pretty steep.”

Loans with less than a 20% down payment require mortgage insurance, which you can sometimes get rid of later.

Sheldon also says 680 is the unofficial minimum: “680 would be the goal, the credit score to strive for if you want to have choice and flexibility.”

If you’re already close to a 680, he says to aim higher for even better loan terms. “You really want your score in the 740 range to get optimal pricing on any given day.”

There are home loans for people with a variety of credit scores, but with a financial decision of this size, it helps to set yourself up for the most affordable terms possible. If you’re thinking of buying a home at any point in the future, start preparing for the mortgage process now by building a good credit score. It can take a while, especially if you had credit problems in the past, but there are many ways to rebuild your credit and position yourself to buy a home someday.

As you work on improving your credit, you can see how you’re doing by looking at your two free credit scores every month on Credit.com. Make sure you’re looking at your free credit reports each year, too, at AnnualCreditReport.com, because sometimes errors can be dragging your score down—and you need to find them to fix them.

This article was originally posted on Realtor.com