old people

Why is it that when you use the term 'old man' you are reffering to your father, but someone's 'old lady' is their wife or girlfriend?