Expectations?

Why does fifty degrees during daylight hours feel bone chilling freezing while fifty degrees when you get off work at 11:00pm feel  downright warm? I mean, I'm assuming that it has something to do with what your eyes and mind are expecting when encountering daylight and nighttime hours.