Mixu Paatelainen’s men went a goal up though Conor Sammon but two penalties from Miller gave Rangers the lead against a team down to ten men after the dismissal of Mohamadou Sissoko.
Remarkably, Sammon equalised with his 13th goal of the campaign before Miller scored his third to clinch a memorable victory.
Rangers can play better than they did this afternoon but credit must go to Killie for the style of play they have adopted this season.
At times they were superb with Finnish star Alexei Eremenko pulling the strings, especially in the opening 45 minutes.
But Rangers again showed their fantastic team spirit in a match that exploded in to life from referee Euan Norris’ first whistle.
With not even a minute played Manuel Pascali brought down Lafferty before he reached the Killie 18-yard box but Davis’ free-kick was cleared.
The home side then pushed forward and Eremenko’s diagonal pass over the Gers defence found Sammon.
The big striker controlled the ball on his chest but McGregor was there to make a superb stop at his feet.
It was clear early on that Killie’s confidence was high after their recent four-game unbeaten run in the league.
Rangers, however, were also showing plenty of quality in the middle of the park but on 14 minutes it was a route one ball that gave the Ibrox club their first goalscoring opportunity of the game.
Kirk Broadfoot punted the ball up the park and Lafferty’s flick-on found Miller unmarked 20 yards from goal.
He took one touch but his deft lob went over the bar with Cameron Bell nervously watching the ball as it clipped the top of the net.
It was end to end stuff and on 16 minutes a low, driven effort from Eremenko dipped and swerved in front of McGregor who somehow palmed the ball over the bar and on to the roof of the net.
It was a heart stopping moment for the visiting support but the opening goal soon arrived and it was not a surprise that Eremenko was the architect.
He received the ball in the 18 yard box and, after turning his marker, delivered a clever cross to the back post to find Liam Kelly who shot across goal.
McGregor got a touch but the ball fell to Sammon who had the easiest of tasks to find the net from four yards.
Rangers immediately looked to respond and a good cross from Whittaker on the left was inches away from the head of Miller who again had found space.
Killie, however, were proving to be a real threat going forward and when Mehdi Taouil jinked his way in to the box all that was missing was a bit of power on his left foot shot which was easily gathered by McGregor.
Before the half hour mark there was a bizarre moment that left everyone in the stadium scratching their heads.
Sissoko tried to put the ball out of play under pressure from Miller and the ball hit the Main Stand linesman, who was actually on the pitch, stopping it from going out of play.
Miller then kicked it out and the referee awarded a Killie throw-in!
In the very next attack Kyle Lafferty then made another good dart in to the box from his wide position and the Northern Ireland forward felt his shirt was pulled by Jamie Hamill but the ref waved away his pleas.
The Killie fans were clearly enjoying what they were seeing in the first-half and little wonder; with Eremenko, Taouil and Sammon on this form they will be a handful for most clubs in the SPL.
Nevertheless a moment of madness in defence allowed Rangers to get back in to this contest on 41 minutes.
Keeper Bell played a short goal kick to Hamill who tried to pass it to centre-half Mohamadou Sissoko.
Miller nicked in and the centre-half made a clumsy challenge to bring the Gers no. 9 down in the box.
The ref immediately pointed to the spot and Miller stepped up to send Bell the wrong way with a composed finish.
The goal seemed to galvanise Walter Smith’s men and before the break they had a great opportunity to take the lead.
Foster used his pace to get down the right and his cut back found McCulloch who sent a soaring shot past the post.
The goal was just what Rangers needed but the scoreline at half-time flattered the SPL leaders who were second best to a resurgent Killie.
The home side continued to look dangerous at the start of the second-half and Foster had to be sharp to make an important goalline clearance after another defence splitting move.
From there the action was relentless with both teams playing their part in making this a terrific spectacle.
On 54 minutes Rangers had their noses in front for the first time when Lafferty forced his way past Hamill yet again but Sissoko tugged at his top and the official had no option but to award another penalty.
The Frenchman was subsequently shown his second yellow card and Miller gave Bell no chance with an unstoppable strike which almost ripped through the net.
Kenny was on fire but he should have had his hat-trick before the hour mark.
Davis fed a perfect pass through to Scotland ace but he swerved his effort past the post when it looked certain he would score.
This looked to be costly for the Gers as Killie went up the other end of the pitch and grabbed an equaliser.
Hamill delivered a teasing low cross to the back post and Sammon was there to force the ball over the goal line from close range.
It was proving to be a remarkable game and on 65 minutes Miller clinched his hat-trick with a little bit of help from the Killie defence.
Whittaker did brilliantly to skip over an Eremenko challenge on the right and his cross looked to be heading in the direction of Italian defender Pascali.
He managed to miss the ball entirely and it fell at the feet of Miller who smashed it home with style before standing, chest puffed out, in front of the jubilant Rangers support.
But even with ten men Killie kept pushing men forward and McGregor had to be at his best to stop a 25 yard curler from Taouil before Hamill came close with a long-range free kick.
At the other end Miller sprinted clear of the Killie defence but his shot was well saved by Bell although he should have played in Lafferty who had made a lung-bursting run to provide support.
In the end Rangers held on to record a precious win from this thrilling 90 minutes of football.