Hellman & Friedman has been in talks to acquire business payments firm Bill Holdings Inc., people familiar with the matter said, in what would be another sizable buyout of a financial software company.