One family source in Florida told MailOnline: 'People will think that Andrew is the totally innocent party in all this.
'But if this divorce gets ugly, then it won't be one-sided, Lauren knows everything about Andrew - and believe me, he's no angel.'
Andrew is currently in the process of divorcing wife Lauren, who recently revealed she is pregnant with X-factor judge Simon Cowell.
He's no angel: Friends close to the couple say that Andrew isn't the innocent party in his divorce
Meanwhile, sources close to Andrew say the wealthy real estate developer is 'P****' by rumors spread by Lauren that he knew about her affair all along.
In a report on TMZ, the friend said rumors that Andrew knew about the affair are 'ridiculous' and that Lauren 'knows this isn't true'.
The comments were made in response to an earlier TMZ article that said Lauren was 'outraged' by her husband's feigned shock over pictures released of her kissing Cowell on a yacht last year.
The website reported that Lauren has alleged to close friends that Andrew 'knew what was going on the whole time and he was doing the same thing.'
The also reported that Lauren accused her husband of messing around, partying with girls all over the world.
But the friend says that these rumors sound like a 'desperate attempt to smear him.'
Unattentive husband: Silverman's stepfather says Lauren only slept with Cowell because she was lonely
Wealthy property developer Andrew, who also owns a casino in Panama, and Lauren are hoping to have their divorce settled within weeks.
However, a friend of Andrew said on Friday that if it leads to a legal battle, Andrew knows 'plenty' about Cowell and could expose some embarrassing secrets.
But the Silverman source said: 'It goes both ways.'
Friend say Silverman cheated on her husband with Cowell because he was never there. 'Cuddling up to five Gucci bags' at night did not satisfy her longing for a man.
Lauren Silverman, who has enjoyed holidays on Cowell's yacht - sometimes with her husband and sometimes just with the X-Factor creator, lived a life of luxury that was empty without her husband Andrew around because he was often away on business, the man who raised her has claimed.
Speaking exclusively to MailOnline, Stewart Eisenberg said that Lauren slept with Cowell because she was lonely and you ‘can’t put a credit card under your arm when you go out to dinner’.