Hi C.J.
We went to Quebec city with our daughter when she was 3.5 m.
I can say it was a pretty easy trip :) now she's 18 months.
Since she was sleeping most of the time and it was easy to enjoy ourselves and walk for hours or go to a restaurant, because she was very portable.
Also it's a very small place (relatively). The only thing is that you need big wheels on your stroller, since lots of streets are cobble stone.
For examble there is one street that brings you down from Le Chateau Frontenac Hotel which is on the hill to lower part of the town near harbour.
There I recommend you to take cable-car all the way down, since there are many stairs and you need to lift the stroller couple times.
However the boardwalk by Le Chateau Frontenac Hotel is really nice and even soothing for the baby.
Here is the link to some attractions in Quebec city and reviews:
http://www.tripadvisor.com/Attractions-g155033-Activities-Quebec_City_Quebec.html
I'm sure you'll enjoy it a lot!!!
Take care,
Marina