I am in a similar boat! I usually wear a light blouse (or T) under a sweater blazer (JCREW Sophie) and top it with a very light super soft puffer that has a hood -- that way I'm covered if it rains and it's not all that bulky if I need to shove it in a bag or carry it around. For office-appropriate bottoms, I live in the Delancy Moto, Headlands Hybrid Trek jogger (which actually looks great with heels and a sweater blazer), and the Headlands Hybrid Cargo II Tight. It's a combination that works year-round for me because, in the warmer months, the AC is cranked up so high that sitting at a desk requires bundling up. In the winter months since I'm in and out of the car just adding gloves and a scarf is typically good enough but the standard sweater blazer works as an essential year-round as do the pants -- even the thicker ones are fine in the summer when you're inside.