come down in the world

[come down in the world] {v. phr.} To lose a place of respect or honor, become lower (as in rank or fortune).

The stranger plainly had come down a long way in the world.

Compare: DOWN ON ONE'S LUCK.