Rob Werner, president and CEO of Ardent Credit Union, estimates that about 10 percent of mortgage applications coming through his institution include some retirement funds to cover closing costs. In many cases, money from an individual retirement account or 401(k) is only for a portion of the down payment instead of the whole amount.

Experts are divided over whether this is a smart financial move. Some financial advisors say workers should keep their hands off retirement funds, no matter what. Others argue there are times when it makes sense to dip into a retirement account for a home purchase. As with so many financial questions, there is no one answer that will satisfy everyone’s situation.

[See: 10 Ways to Reduce Your Housing Costs in Retirement.]

The price of retirement fund withdrawals. Before pulling money from a retirement fund, it’s important to understand the rules. Withdrawals made from traditional IRAs or 401(k)s prior to age 59 ½ will incur a 10 percent penalty and are also subject to income tax. Up to $10,000 may be exempt from the penalty if taken from an IRA for a qualified first-time home purchase.

“If you have a Roth IRA, that’s probably the best option,” says Kathy Cummings, senior vice president of homeownership solutions and affordable housing programs at Bank of America. Since contributions to Roth accounts have already been taxed, early withdrawals of the principal amount are not subject to income tax.

Rather than taking a distribution from a retirement account, another option is to take a 401(k) loan. Many plans allow loans worth up to half the vested value of the account or $50,000, whichever is less. This money may be paid back over five or more years when it’s used to purchase a primary residence, depending on the plan’s rules. Should a worker leave his or her job while the loan is out, the balance must be repaid immediately or be subject to tax and penalties.

[See: 10 Ways to Avoid the IRA Early Withdrawal Penalty.]

Missing out on investment gains. The cost of taking money from a retirement fund is more than simply the taxes and penalties you’ll pay, says Ethan Vickery, an agent with Triplemint. He works with clients buying condos and co-ops in New York City and urges caution before taking money from retirement funds. “Understand that these actions will change your financial picture in such a way that may make you look less desirable to a board, which would make the whole exercise moot,” he says.

Vickery isn’t the only one not sold on the idea of a retirement fund withdrawal or loan. “I strongly think it’s the wrong thing to do,” says Brent Wilsey, president of Wilsey Asset Management in San Diego. According to Wilsey, taking $30,000 out of a 401(k) account for 20 years could mean a worker will have $600 less a month in retirement. That’s assuming the money would have gained 7 percent per year while invested.

While property often gains value, Wilsey cautions buyers against assuming that will be the case. “[People] are buying a depreciating asset on an appreciating piece of land,” he says. Another downturn like 2008 could leave you without a home and without adequate money for retirement if you deplete your savings for a down payment.

Although not as easy as taking out a 401(k) loan, Wilsey says, “Our parents bought houses and scrimped and scraped and put away money.” He recommends today’s borrowers do the same.

Becoming a homeowner. Not everyone thinks it’s a bad idea to borrow from a 401(k) to buy a house. Without 20 percent down, lenders require private mortgage insurance, which could add $100 or more to your monthly payments. “If you can borrow from your 401(k) to get that last $5,000 to avoid PMI, … it’s really not a terrible strategy for the purchase of a home,” Werner says.

Cummings says borrowing from a 401(k) may also make sense in a low-interest, high-rent environment. She notes her hometown of Charlotte, North Carolina, has seen rental rates increase an average of 6 percent per year. Someone who wants to avoid rising rental costs and lock in a low interest rate may find borrowing from a retirement fund helps keep long-term housing costs affordable.

Taking loans from a 401(k) becomes a problem when people start feeling comfortable dipping into the money for any purpose. What’s more, it’s risky for anyone whose employment is not stable. Therefore, Werner recommends only taking out what is absolutely necessary and only doing so if you feel confident you will remain at the job until the loan is paid.

[See: 10 Retirement Planning Moves to Make in Your 20s.]

Options available to lower costs. Before borrowing from a retirement fund, home buyers should understand there are a number of federal, state and local programs that can help reduce or even eliminate closing costs. Some of these programs are available only to first-time home buyers — often defined as those who haven’t owned a house in three years — or those who meet certain income requirements.

Vince Liuzzi, executive vice president and chief banking officer for DNB First, says his institution participates in programs through the First Front Door initiative and Pennsylvania Housing Finance Agency. Both offer down payment assistance to qualified buyers. “So long as you stay within your house for five years, it turns into a grant that doesn’t have to be repaid,” Liuzzi says.

To find out which programs are available, Cummings recommends looking for a housing counseling agency approved by the U.S. Department of Housing and Urban Development, or HUD. While there may be a fee for using a counselor’s services, it could be money well-spent. “They are going to be familiar with every option and all the qualifications,” Cummings says.

Regardless of how you get the money for your down payment, you shouldn’t deplete all your cash on closing costs. “Make sure you have some money in savings post-closing,” Liuzzi says. “You’ve got to put up those new drapes.”

More from U.S. News

How to Max Out Your 401(k) in 2017

10 Tips for Finding a Great Place to Retire

How to Reduce Your Tax Bill by Saving for Retirement

Should You Use Your Retirement Savings to Buy a House? originally appeared on usnews.com