Celtic were level by the half-hour mark when Joe Ledley took advantage of poor defending to nod in an equaliser.
But Gers’ patience, drive and will to win paid off as extra time followed and Jelavic hit his first derby goal to edge a thoroughly entertaining game.
It’s a fantastic result for Smith, with this particular triumph being his 12th in the League Cup with the Ibrox side and Dundee United.
And he will hope now his men can kick on when their league campaign resumes on April 2 at home to the Arabs.
Before the game, there had been much speculation over whether Smith would go with a more attacking 4-4-2 formation than he has in the most recent derbies.
Sure enough, the manager chose to take Celtic on at their own game as he listed Jelavic and Steven Naismith in attack.
Gregg Wylde kept his place after an impressive show against PSV Eindhoven in the Europa League, while Kyle Lafferty was fit enough to start on the right of midfield.
As expected, Neil Alexander started ahead of Allan McGregor in goals with Steven Whittaker, Madjid Bougherra, David Weir and Sasa Papac in front of them.
Davis and Maurice Edu were the central pairing in the engine room and they were tasked with dictating play when possible.
There was a fantastic atmosphere as the game kicked off and the Parkhead side settled quicker.
But it was Gers who had the first real chance and if truth be told, they should have done much better with it.
Papac’s hanging ball found Jelavic in the box and he had more time than he appreciated to send an angled volley towards goal but he didn’t connect well and was off target.
Lafferty then had a go when his own header was only half cleared by Thomas Rogne but his shot was deflected away by the big Norwegian.
Naismith’s presence and the freshness he brought after a month’s rest was making a real difference up front for the SPL champions.
While he wasn’t always creating opportunities, his added energy was forcing Celtic to play the ball quicker than they would have liked and it made them more disjointed.
Neil Lennon’s team was denied a penalty when Mark Wilson went down in the box under a challenge from Papac.
But it didn’t seem there was any contact made and after taking a good look at it, referee Craig Thomson chose not to award anything and it was the correct decision.
By and large, Rangers were making most of the running in the early stages of this one and after 23 minutes their industry paid off as the took the lead.
Naismith did well to give Rogne enough of a nudge when a high ball was lofted upfield that the stopper could only head as far as Davis.
And when a glorious first touch took him beyond Ledley, he drove towards the area with space opening up all around him.
Jelavic peeled off and it seemed he was destined to receive the pass but Davis took it on himself and although his shot wasn’t the cleanest, it crept in from 20 yards.
Predictably, Celtic responded well and it needed Lafferty to hook clear after Gary Hooper’s header was blocked.
And within seven minutes the scores were level as Ledley made up for his failure to keep up with Davis by hitting the equaliser.
It was a poor goal to lose as Emilio Izaguirre’s cross deflected off Lafferty and looped up for Georgios Samaras to flick on.
Ledley then stole in ahead of Weir to make it 1-1 but within moments there was a huge moment of controversy as it seemed Rangers were denied a penalty.
Jelavic took Rogne on then fell, with ref Thomson pointing to the spot – only to change his mind on linesman Graham Chambers’ advice and book the Croatian for simulation.
TV replays showed there was minimal contact made but it was enough that the kick should have been given and it was a massive let off.
The Ibrox outfit lost a bit of their momentum after that as play evened out but Bougherra still sent a header wide from a Papac cross before Lafferty nodded wide too.
And at the break, as the teams remained level, there was little doubt Smith’s men deserved more from a pulsating game at that point.
Rangers picked up from where they left after the pause in play, with Lafferty seeing another header flash away before Fraser Forster saved well from Whittaker.
Back came Celtic though, with Gers lucky that Papac’s fresh air swipe didn’t allow a low cross to sneak in at the back post.
In the main, however, it was the Light Blues who looked more dangerous in attack and Jelavic should have done more when he met a Wylde corner.
Incredibly, Rangers were denied another penalty when Edu tried to play past Wilson, the ball struck his arm and Thomson shook his head.
And for all that trophy holders were controlling things, they weren’t getting the breaks you sometimes need to shade these games.
Wylde went close with a free kick which was probably intended as a cross but almost dipped in from 35 yards and had to be pushed over by Forster.
And Weir nodded wide from the corner which followed as another chance went begging with a quarter of the regulation 90 minutes left.
Wylde provided a disappointing end to a good move involving Edu and Lafferty when his drilled attempt bent away from goal.
But with time ticking down, Rangers were starting to pick up knocks and Bougherra had to go off for Kyle Hutton with a hamstring injury.
With Smith’s men tailing off, Celtic made a late push for a winner and Alexander had to save on the line.
Substitute Ki also had a free kick on target after a foul by fellow replacement Kyle Hutton but it was held as well.
And when Vladimir Weiss was thrown on seconds before the whistle, he won a free kick then took it but it hammered off the wall so extra time beckoned.
The Slovakian was stretching Celtic and won another set piece when he was taken out by Wilson with a cynical foul.
Jelavic took the kick and it took a big nick off the wall from 25 yards then changed direction before Forster got a crucial hand to it.
To their credit, Rangers simply didn’t let their heads drop and their grit and perseverance finally paid off eight minutes into the first additional period.
Thomson pulled play back for a foul on Weiss just inside the Celtic half and the winger got up to take it quickly.
His weighted ball set Jelavic scampering between Charlie Mulgrew and the recently-introduced Glenn Loovens and he was in on Forster.
With just the keeper to beat, his right-footed shot bounced off one post and seemed to be trundling back out.
But it took a curved path along the line before nestling in the side netting on the other side of the goal and sparking bedlam at the Mount Florida end of the ground.
Jelavic was having a good game and his strike – his first against Celtic – was merited for the shift he had put in.
And after another interval, the former Rapid Vienna star could very nearly have added another to his tally.
He showed common sense to stay onside with Celtic pushed upfield before being released and taking on Mulgrew.
Jelavic eventually got the better of him but was hauled down on the edge of the box and given it was a scoring opportunity, why the defender wasn’t sent off is unclear.
Regardless, there was a free kick to be taken but Papac hammered it into the wall and the chance came to nothing.
The Parkhead side wasn’t registering many shots on goal and the less that’s said about Ki’s long-range drive, the better.
With four minutes all that were still on the clock, El-Hadji Diouf came on for Jelavic as Rangers looked to firm up and see out the win.
But he very nearly set up a clincher too for Weiss, breaking clear and playing in the youngster only for Forster to save well.
Diouf then collected a laughable yellow card for diving when he was brought down by Mulgrew who, sitting on a booking himself, should have gone.
With just seconds to go, Hooper – who had been so quiet for so long – nearly forced penalties but Whittaker, excellent as an auxiliary centre half, blocked magnificently.
There was still time for more incident though as Forster joined the attack and got caught out, leaving his goal exposed.
Weiss just had to outpace Izaguirre but the Honduran took him out instead and rightly saw red.
With that, the match was won and as Thomson’s final whistle went, a huge roar heralded the arrival of Smith’s 20th honour as boss.