A few weeks ago, my wife Stella was invited to an annual event for the LGBT community at No 10 Downing Street. The emphasis this year was on tackling homophobia and transphobia in sport. As someone who has quite a high public profile, and yet who has always been, and still is, very much old Labour, Stella was unsure about attending an event hosted by a Conservative government, not least because of the negative response by some people – how could a Labour supporter be part of a Tory event? Stella, while pondering whether or not she should go – asked if I wanted to join her. As with most things, I immediately said no, I couldn’t possibly join a Tory party, it would be wrong. As the days passed, Stella decided to go, one reason was because she had never been invited to a LGBT event by any Labour government. As someone who gives a great deal of her time to LGBT issues, I agreed that she should go. I raised the question on Face Book and Twitter, asking others what they thought I should do. Except for one person, everyone said GO. And so yesterday, with passports in hand and excitement rising, we arrived promptly at four o’clock along with everyone else, inching closer to No 10. We passed the security check (men with guns have always scared me) and walked across the street (which resembles a film set) and through those famous, highly polished doors. I then realised this was an event I had given very little thought to, giving more thought to what I would wear instead.

I was as unashamedly inquisitive as the next person. I wanted to have a good nose around the house, after all I’d seen it on television since I was a child. My parents have always been Tory supporters, but are wavering now. They are unhappy with David Cameron and his team, not that they would ever vote Labour. They told me to have a word with him and I explained it was unlikely we would meet. My parents like to call me a cappuccino drinking (extra sprinkles), Guardian reader, who, in their minds, is very radical. I am not. I like to think I am very tolerant and mostly, I am. Like Stella, I am old Labour, my values have changed as I have grown older, but I am all for looking out for my society, am happy to pay higher taxes so that the less well off can afford to survive, and do what I can do contribute. But what was I doing here, at No 10?

After we left our mobiles at reception (I loved the absence of mobiles for two whole hours), we made our way down the hallway and I noticed an L.S. Lowri painting. I had to look twice at the signature. I don’t know why I was so surprised and excited, perhaps because I am not used to seeing great works of art in people’s homes, but this didn’t feel like someone’s home, more a temporary accommodation-come-workplace, which is filled with people like me and Stella and dignitaries and celebrities all day every day. I thought about all those people traipsing through our house, and then reminded myself it was never going to happen. On we walked, fresh flowers accompanying us everywhere we went, up the famous staircase, with photographs of all the PMs, and I did tingle and it was special and I even managed a quick glance at Margaret Thatcher’s photo. Upstairs to three large reception rooms, I likened it to our small terrace being broken through from one end to the other, just bigger and deeper and higher, and full of priceless objects. I ignored the first room of people and walked instead through to the furthest room, where a couple of men were being nosy too! The first man I spoke to was the MD of the Huddersfield Giants (did you know that Huddersfield was the birthplace of rugby league?). We were both amazed at the view through the large windows, looking out onto Horse Guards Parade. I didn’t expect that view, not sure why, I know London’s layout pretty well, I just didn’t put the two locations together! So that’s how the Queen travels so quickly from Buck House to No 10.

The afternoon progressed, along with more famous paintings, a few famous faces, but actually not that many. I was more excited to see Ben Cohen than David Cameron, even more thrilled that I managed to introduce myself to Billie Jean King, someone I have respected since I was a teenager. She was a reminder to why we were all there – addressing the problem of homophobia in sport. Billie Jean King lost all her endorsements within 24 hours of being outed. And still there are so few sports people who have come out. I was told there are twenty premiership players who are gay. Imagine what a difference it would make if they came out, all of them. That is why I was at No 10. To try and make a difference, to try and make it okay for those who are not okay.

And now to the food – canapés and alcohol were plentiful, the sausages particularly delicious, as were the gigantic strawberries dipped in chocolate. Yes, I admit to enjoying the food. I met so many interesting people, some gay, some lesbian, some heterosexual, some transgender, some bisexual. Who cared? I didn’t. Because in those rooms, for me at least, something important happened. I thought about the protests by people who refused to attend the event, or thought that the likes of me shouldn’t attend, or that we were just playing up to the Tories, or that we were there to gape (what’s wrong with gaping?), and I questioned myself, and Stella and the rest of us. David Cameron spoke very briefly, and then seemed to disappear. A few of his colleagues (Theresa May, Margot James, Anne Milton) were on hand to talk to the more important guests, the ones who make a real difference, Lisa Power, Policy Director from the Terence Higgins Trust, Claire Harvey, a brilliant woman and paralympian. We are both patrons of Diversity Role Models, a new foundation set up by Suran Dickson. Ben Cohen, now retired, was there as Chairman of the Ben Cohen StandUp Foundation, the world’s first anti-bullying organisation. Gareth Thomas and plenty of other well-known, and many more not so well known people, in and out of the sports world, all came together because of a cause, not a Tory cause, but a world wide cause that was important to me. Homophobia and transphobia in sport. And yes, David Cameron is not my best friend and I still disagree with pretty much everything he stands for, but this is a cause that crosses parties, and I wanted to be party to this cause. And that is why I went and that is why I will never apologise for going because the cause is what mattered, irrespective of whether it was hosted by Labour, Tory or a government from outer space. The food didn’t matter, and the décor didn’t matter, and ultimately what I wore didn’t matter, what did matter was that I found myself speaking to so many different people, and not once did I question whether they were gay or lesbian, heterosexual, transgender or bisexual or Tory or Labour or Lib Dem. For two hours, the people in those spacious and bright reception rooms, through my eyes at least, provided a microcosm of the possibilities for the society we live in. We still have a long way to go with sexuality and racism and all the other isms you can add to the list, the event showed a glimmer of hope. And if I haven’t said it enough, the emphasis was on the cause, not which political party you align yourself with. It allowed me a chance to show my support to my colleagues who make the groundbreaking work happen, and to those I respect. If we don’t talk to each other, and break down barriers, what hope is there for a better world? If I remain in the room across the hall, do I have to wait for the person opposite to make the first move, or can I be braver than that? I like to think I am always braver. If only Israel and Hamas would speak, the world might change overnight. Yes, I’m an optimist, but it’s better than being a pessimist because I might as well give up. I honestly believe that it is far better to talk to your enemies, to stand beside them with your differences in one hand and your hope in the other, than rage at them from across the street and wait for a miracle to happen. We create the miracles. It is better to share the same space, note our differences face to face, rather than ghettoise ourselves and push ourselves so deeply into corners, that the only way out is if someone else prises us out, and that rarely happens. There are quite a few brave people in the world, but more who would like to be brave but never make that step, because they are afraid of losing jobs, being hated, being bullied.

I also realised this yesterday; that our wonderful and accepting heterosexual friends, who have no problems with our sexuality, might actually think it’s fixed, so they don’t speak out or consider that perhaps all is not so great. See above, Ben Cohen, ‘the first straight sports star to donate his philanthropic efforts for the benefit of LGBT people.’ We need heterosexual allies, and those we have are plentiful, but when one of their kids comes out and they realise it’s not fixed, I don’t want to be the one to say I told you so.

At the end of the afternoon, after signing a flip chart, (I’m still unsure what I signed), the man who I had first talked to, in the room with the view of Horse Guards Parade, gave me a rugby shirt. Here, he said, this is for you, because you were the first person I spoke to and you were so lovely. Before he went, he invited us both to a match when we’re in Huddersfield. I’m going to take him up on his offer. I left No 10, no better or worse as a person, but full of hope. Surely that’s what it was all about?