Fisher Investments, the financial firm founded by the now-scorned Ken Fisher, took another hit Thursday after the Los Angeles Fire and Police Pension System voted to withdraw funds managed by the company because of lewd comments made by the CIO at a private industry conference.

The $522 million withdrawn, which is only a small chunk of the $24 billion in the LA pension, was the second highest removed so far, only behind the Michigan Investment Board’s $600 million taken away earlier this month when the original story broke.

Goldman Sachs, the massive investment bank, also announced it was cutting ties with money manager, making it the second major financial institution to drop Fisher after Fidelity cut ties Monday. Fisher’s firm managed $234 million for Goldman.

Fisher was invited to speak at the Los Angeles meeting where a nine-person board of commissioners voted to approve the withdrawal, but he declined, according to LA Fire and Police Pension System General Manager Ray Ciranna.

“The only explanation is that Mr. Fisher was unable to attend and had business in the office,” Ciranna told CNBC.

All told, eight institutional clients have dropped Fisher Investments, and the combined losses exceed $2.7 billion. The six government-run pensions totaled just over $2 billion, while Fidelity Investments and Goldman Sachs, make up the other $734 million.

On Tuesday, Ray Higgins, executive director of the Public Employees Retirement System of Mississippi, announced his pension program was also placing the troubled firm on a watch list. If it withdrew its $558 million, the total losses would blow past $3 billion.

All these withdrawals are only a drop in the bucket for Fisher’s firm, though, which managed $112 billion of assets as of Sept. 30, 2019, according to its SEC filing. Individuals investors make up more than half of the total managed by the firm at $59.5 billion.

Fisher came under fire after making sexually inappropriate comments at the Tiburon conference on Oct. 8.

“Money, sex, those are the two most private things for most people,” Fishers said, according to leaked audio from the conference. “It’s like going up to a girl in a bar … [inaudible] … going up to a woman in a bar and saying, hey, I want to talk about what’s in your pants.”