WEST PALM BEACH (CBSMiami/AP) – A Palm Beach Church and its pastors caught up in a Ponzi scheme are refusing to turn over near $2 million that federal regulators say belongs to people caught up in the scam.

A federal judge temporarily froze the accounts of Winners Church and pastors Fred and Whitney Shipman. The church leaders are fighting efforts to allow $1.7 million to be returned to hundreds of people who were lured into a diamond and bitcoin investment fraud.

Bishop Fred Shipman told the judge Thursday that returning the money would have a serious “ripple effect.”

Securities and Exchange Commission lawyer Amie Berlin isn’t claiming the church or pastors were involved in the scheme orchestrated by a former church director.

The judge gave both sides until June 3rd to file papers explaining their positions.

(© Copyright 2019 CBS Broadcasting Inc. All Rights Reserved. The Associated Press contributed to this report.)