Think of deep sea. The deeper you go, there is more water stacked on top of you pressing down. So, higher pressure at down deep sea, less pressure at top of water.
Although air is not heavy like water, but air has also weight. The higher you go the fewer air is above you.
So you get the same effect --> high pressure at sea level, low pressure at high altitude

Ref: Yahoo! Answers - How come at high altitude ther is low pressure?