have an affair with

[have an affair with] {v. phr.} To have a sexual relationship withsomeone, either before marriage or outside of one's marriage.

Towand Jane had a long and complex affair but they never got married.