a dilemma

I have not yet mentioned this, but in a fit of cardigan lust I cast on Sweaterbabe's Cables and Lace Cardigan with some Queensland Collection Kathmandu DK. Great yarn, decent pattern (it has no charts, so I made some). Anyway, I've been working away on it since some point in January. Yesterday and today I had jury duty, and I brought along the pattern, a ball of yarn and my needles. I cast on and knit the second front panel of the cardigan. It was really great jury duty knitting, and by this afternoon the yarn was used up with only an inch or two left to finish. Which is interesting because the first front panel took less than a ball of yarn.

When I got home I steamed both sides and lined them up, and they don't line up. I was pretty careless when I did the decreases on the first panel (I think I was being overly social and not paying very close attention) whereas on jury duty I gave it my full attention. The question now is: how important is this? The neckline will get a knitted on ribbed border, which might make them seem more equal. And it's sort of a wrappy thing, not so precise. I'm feeling lazy about it, but I'd also like them to match more or less. Or look like they match.

Ideas? What do you all think?