The Huddersfield Town chairman also acknowledged that designing and producing his own cards after resisting pressure from giant US supplier Hallmark was "the making of Card Factory".
Hoyle was the special guest at a Q&A for Insider's Growth 100 Awards and fielded questions about his successes at Card Factory and his hopes for Huddersfield Town, with the football club currently lying just outside the play-off spots in the Championship. The full interview is in the November issue of Insider.
Asked what advice he would give a growing business, Hoyle said Card Factory's early profitability meant he had the funds to recruit the best people into the business.
"Going on from there – and with apologies to anyone in marketing – we kept our head very much below the radar. People knew we were a national chain but they didn't know we were based in Wakefield. I always have this fear that those who shout the loudest fall the hardest.
"We never wanted to alert our competitors to how profitable we were. We used to slice and dice our accounts, quite legally, but this was part of the game."
Hoyle went on to highlight the importance of taking on the competition: "We opened our first store in 1997 in Wakefield, which cost us about £20,000 to open. In the first year it made about £100,000 profit.
"Card Factory is now the market leader within 15 years of starting from nothing, and Clinton Cards has gone into administration. You have got to face your competitors head-on. If you are going to be number one, you have got to overtake the dominant market players."
At the outset, Card Factory aimed to double in size each year and stuck to that until the input of a supplier changed the company's fortunes.
"The dynamics of the business changed in 2005 when we had big supply pressures and we were told 'unless you put your prices up, we're not going to supply you anymore'," said Hoyle.
"My world fell apart. We had the foresight to link up with a design company and started producing our own – that was the making of Card Factory."
Hoyle sold Card Factory to Charterhouse for £350m in 2010 after securing backing from LDC in 2006.
He still has a board role at the Wakefield-based business but is now investing millions in building a squad of players at Huddersfield Town capable of challenging for a place in the super-rich Premier League after a penalty shoot-out success against Sheffield United earned promotion to the Championship last term.
"In any business, you focus on your main strengths. Card Factory was all about cashflow, takings, sales. Football is all about what happens on a Saturday afternoon when your team steps over the white line, and keeping your fingers crossed.
"In that respect, it's not a proper business, it's just a big game."