Boyd thought he had won it for Rangers with six minutes to go but a handball by Alan Hutton in the 91st minute gave Steven Naismith a chance to level from the spot and he made no mistake, sending Alan McGregor the wrong way.
After making four changes last Saturday, injuries and illness forced Le Guen to alter things again this weekend as Lionel Letizi and Chris Burke missed out with calf and ankle problems respectively and Phil Bardsley called off with a stomach bug.
It meant a return to the starting line-up for Hutton and Libor Sionko, while McGregor was in goal for his first competitive game of the season.
Having convincingly brushed the Jambos aside at Ibrox, confidence was rightly high within the side ahead of this match.
Rather than bursting into life, however, the game started slowly with neither side creating any chances in the opening spell.
The first sniff in front of goal came after 14th minute and what a opportunity it was for ex-Killie hitman Kris Boyd.
The Scotland international made a scoring return on his last visit to Rugby Park and after collecting Brahim Hemdani’s through ball, burst into the box to fire a shot on goal.
In fairness Gordon Greer did well to usher him onto his left foot but Boyd was still angry at screwing his effort wide of the post.
Two minutes later the Kilmarnock defender wasn’t so effective as he made a poor clearance, which fell directly to Dado Prso.
He fed Boyd, but Graeme Smith was alert and rushed off his line to smother the at his former team-mates’ feet.
Le Guen’s side were really starting to take hold of the game at this point and after 17 minutes Jeremy Clement found Libor Sionko on the edge of the area.
He cut inside and fizzed a low shot on goal, but again Smith collected. The pressure was building on Kilmarnock and on 25 minutes Steven Naismith came to the rescue when he cleared Julien Rodriguez’s header off the line, following Lee Martin’s corner.
The on-loan winger looked eager to impress again and just after the half hour mark he cut in from the right to latch on to Clement’s pass and drive a fine effort on goal.
Smith saved well and a minute later produced another fine block to deny Boyd, after Hemdani played a quick free kick into the box catching the defenders cold.
It took 37 minutes before the home side registered their first effort on goal and even then Hutton did well to block Danny Invincible’s shot, before it reached McGregor.
Having withstood such immense pressure throughout the first half, the Killie defence eventually gave way in the 42nd minute when Boyd finally broke the deadlock.
Once again Prso was the key man, picking up the ball in midfield and driving at the opposition, he beat two men before passing to Boyd, who was positioned around the penalty spot.
The 22-year-old shielded the ball from his marker before showing quick feet to skip past and fire a shot low into the net.
Clearly desperate for his side to get back into the match, Jim Jefferies changed things at the break bringing on Gary Wales and David Fernandez to bolster the attack.
But it was ex-Ibrox star Alan Johnston who tested McGregor’s resolve after 52 minutes when he collected Hutton’s soft clearance to curl an effort on goal. The young Scot saved well though to keep Rangers’ lead intact.
Four minutes later Prso should have extended his side’s advantage when he blazed a shot over the bar, after being picked out by Sionko unmarked inside the area.
On the hour mark, Martin once again created another fantastic chance when he collected the ball on the right and cut inside before unleashing a fierce shot on goal. It was destined for the top left corner but Smith flew across his area to palm away for a corner.
As has been the tale in so many of Rangers’ games this season, the opposition were allowed back into the match, after being totally outplayed and Kilmarnock equalised after 63 minuutes.
Frazer Wright managed to evade his marker to latch on to Gary Wales’ corner and head home from close range.
With 16 minutes remaining Invincible had a great chance to put the home side ahead when he was perfectly placed to head Naismith’s cross, but put his effort over the bar.
Immediately afterwards Le Guen threw on Thomas Buffel and Filip Sebo and the latter broke down the right after 76 minutes to cut the ball back for Boyd.
The Scotland international was unable to get a shot away but laid the ball back to Prso, who should have burst the net, but hit a weak shot allowing Smith to save.
Just when it looked like missed chances would cost maximum points again, that man Kris Boyd stepped up again with another crucial strike.
With six minutes remaining he ran in behind his marker to take Prso’s pass and coolly slid the ball under the out-rushing Smith.
Buffel could have made it 3-1 after 88 minutes, following great work by Sebo, but again the Killie keeper produced an excellent save to paw the Belgian’s shot wide.
But the home side never gave up and in the first minute of injury time Hutton was judged to have handled Fernadez’s cross and the spot kick was awarded.
Naismith stepped up and fired home, leaving Rangers cursing their luck in front of goal again.
It was another promising performance from Le Guen’s side but for the umpteenth time this season missed chances proved so costly.
KILMARNOCK: 13 Smith, 2 Fowler, 3, Hay, 5 Greer, 7 Naismith, 9 Nish (10 Wales 45), 11 Invincible, 12 Johnston (16 Dodds 82), 14 Wright, 15 Murray, 22 Leven (20 Fernandez 45)
SUBS: 26 Bell, 18 Di Giacomo, 21 O’Leary, 25 Adams
RANGERS: 22 McGregor, 20 Hutton, 19 Svensson, 27 Rodriguez, 26 Smith; 14 Sionko (23 Sebo 74), 7 Hemdani, 8 Clement, 32 Martin (4 Buffel 74), 9 Prso (11 Rae 87), 15 Boyd
SUBS: 34 Robinson, 28 Ponroy, 25 Adam, 30 N’Diaye
Referee: Alan Freeland