Michy Batshuayi scored two goals either side of a Victor Moses strike at Stamford Bridge to send the home side into the third round, but they were given a fright when the visitors twice halved a two-goal deficit through Peter Hartley and an Ellis Harrison penalty.
The victory is Conte's third in competitive matches as Chelsea boss, following 2-1 wins over West Ham and Watford in the Premier League - both of which came about thanks to late goals from Diego Costa.
And the former Italy boss wants his players to become more ruthless in their approach in order to prevent similarly nervy encounters in future.
"These games are very tough," he told BBC Radio London Sport. "When you dominate the game, you must kill the game. In this case, we didn't do this.
"It's a pity because when you're playing very well, creating a lot of chances to score, I don't like to concede a soft goal from a free-kick and a penalty.
"We can improve this aspect, and stay concentrated until the end.
"We started [the second half] very well, created the situation to score another goal and then we conceded the penalty on a counter-attack. This created a bit of imbalance in the players.
"It's important to understand that we must stay concentrated, always. And when you have the opportunity to kill the game, you must kill it, not keep it open."
Captain John Terry came off the bench for the final 13 minutes of Tuesday's match, on the day that England manager Sam Allardyce claimed he could consider offering the 35-year-old a recall to international football.
Conte, however, refused to become involved in the debate over whether the veteran centre-back should represent his country again.
"I'm a former coach of a national team and I can tell in this situation that it's better to sort the situation between Sam Allardyce and John Terry," he said.