According to Black's Law Dictionary:
A frivolous lawsuit is a lawsuit lacking a legal basis or legal merit; not serious; not reasonably purposeful.
Abuse (misuse) of process is the improper and tortious use of a legitimately issued court process to obtain a result that is either unlawful or beyond the process's scope.
Malicious is something that is sure to cause injury; without just cause or excuse.
Malicious use and abuse of process is using the institution of a criminal or civil proceeding for an improper purpose and without probable cause (malicious prosecution).
They are closely related and can overlap. Frivolous lawsuits, however, are generally dismissed as they have no basis in law. They can, however, be filed "with malice" - or with a reckless disregard of the law or a person's legal rights.
Wait for attorneys to post with more.