Why not talking in different way?

I had taken an appointment with my house owner to discuss about continuing on rent for the extended period. I was on my way to his house, but I stopped at ATM to take money. While I was in the queue at ATM, house owner called me and asked weather I was having plan to visit him. I said yes I was planning. But I could have said that I was on my way to you, but if you had any other plans go head with them. I could meet him later.