WEST PALM BEACH (CBSMiami/AP) – A Palm Beach Church and its pastors caught up in a Ponzi scheme are refusing to turn over near $2 million that federal regulators say belongs to people caught up in the scam.

A federal judge temporarily froze the accounts of Winners Church and pastors Fred and Whitney Shipman. The church leaders are fighting efforts to allow $1.7 million to be returned to hundreds of people who were lured into a diamond and bitcoin investment fraud.

READ MORE: Self Proclaimed "Witch" Shannon Ryan Charged With Second Degree Murder In Leila Cavett's Disappearance

Bishop Fred Shipman told the judge Thursday that returning the money would have a serious “ripple effect.”

READ MORE: Search For Missing Florida Teen Tristan Bailey Ends After Body Found, Teen Charged With Murder

Securities and Exchange Commission lawyer Amie Berlin isn’t claiming the church or pastors were involved in the scheme orchestrated by a former church director.

The judge gave both sides until June 3rd to file papers explaining their positions.

MORE NEWS: ME: Florida Teen Tristyn Bailey Was Stabbed To Death, 13-Year-Old Suspect Is Classmate

(© Copyright 2019 CBS Broadcasting Inc. All Rights Reserved. The Associated Press contributed to this report.)