Gerund is used after certain verbs/phrases that are followed by an appropriate preposition. The gerund form (-ing) is used after a verb + preposition: insist on, look forward to, etc. So, 'insisted on paying' is the correct phrase grammatically.
Hence, option (a) is correct.
The correct sentence is: He insisted on paying a lump-sum money to his friend because his friend was in dire need.
NOTE:
Gerund: Gerund is a verb form that functions as a noun, ending in -ing. A gerund can be either the subject or object of a sentence in English. Example: "Everybody enjoyed her dancing and singing at the party." The words 'dancing' and 'singing' are gerunds.