WEST PALM BEACH (CBSMiami/AP) – A Palm Beach Church and its pastors caught up in a Ponzi scheme are refusing to turn over near $2 million that federal regulators say belongs to people caught up in the scam.

A federal judge temporarily froze the accounts of Winners Church and pastors Fred and Whitney Shipman. The church leaders are fighting efforts to allow $1.7 million to be returned to hundreds of people who were lured into a diamond and bitcoin investment fraud.

READ MORE: Jury Continues Deliberations Dayonte Resiles Murder Trial After Juror Disagreed With Manslaughter Verdict

Bishop Fred Shipman told the judge Thursday that returning the money would have a serious “ripple effect.”

READ MORE: Have You Seen This Woman? Andreae Lloyd Missing After Being Abducted From Miami-Dade Home

Securities and Exchange Commission lawyer Amie Berlin isn’t claiming the church or pastors were involved in the scheme orchestrated by a former church director.

The judge gave both sides until June 3rd to file papers explaining their positions.

MORE NEWS: Davie Police Give Missing Woman's Daughter Some Closure After 46 Years

(© Copyright 2019 CBS Broadcasting Inc. All Rights Reserved. The Associated Press contributed to this report.)