Some years ago I had an obviously Muslim student in my evening class. He came up to me one day and told me that due to his schedule he had to switch to another section of the class. I was genuinely disappointed - he was a good student. I said "lets go to lunch one day" his remark was "You would have lunch with me?".
We were never able to put it together, but this is an example of why we cannot get together. We need to start on the individual level.
It's up to each one of us.