FRANKFORT, Ky. (CBSDC/AP) — U.S. Sen. Rand Paul said Wednesday that “the ball is moving forward” on immigration reform despite the surprise defeat of House Majority Leader Eric Cantor in the Virginia primary.

Cantor lost to Dave Brat, a little-known economics professor whose campaign focused largely on his opposition to immigration reform. Cantor’s defeat had some wondering if it would be difficult for the Republican-controlled U.S. House of Representatives to pass immigration reform.

But Paul said a number of issues contributed to Cantor’s defeat, including his past votes to raise the debt ceiling and the controversy surrounding the National Security Administration’s domestic spying program. He noted the easy primary victory of U.S. Sen. Lindsey Graham, an author of an immigration bill that failed to pass the House last year.

“It’s a mistake to … decide that one issue decided this,” Paul told reporters on a Wednesday morning conference call organized by the Partnership for a New American Economy, a pro-immigration reform group. “Some people, myself included, think that you can go too far negative. And apparently millions of dollars in negative ads were run and it may well have increased the name identification of a lesser known candidate – a lesser known candidate who also had a lot of popular things to say on other issues.”

Paul, a potential 2016 Republican presidential candidate, has been actively trying to expand the Republican Party by making it attractive to minorities — and immigration reform is part of that strategy. But Paul acknowledged Wednesday that immigration remains a tricky area for Republicans, including himself.

For example, Paul voted against the Senate’s bipartisan immigration bill last year because he said it did not do enough to secure the border and it did not provide enough work visas. That bill passed the Senate but has stalled in the House.

“I’ve been someone who is for the concept of immigration reform. I’m very, very careful that I want to vote for something that works,” he said.

Paul said Republicans have become “somewhat trapped by rhetoric and words” – including amnesty.

“We’re trapped in a word that means different things to different people,” he said. “We have to get beyond that. But I think the ball is moving forward, and it is complicated.”

One such complication, Paul said, is what to do with immigrants who served in the United States military.

“Do I have sympathy if you served in our military, that we ought to find a place for you in our country? Absolutely,” he said. “But do I want to send a signal to everybody in Mexico that if you come join our military you get to be a citizen? That’s a bad signal.”

Sen. Lindsey Graham also pushed for continued immigration reform following Cantor’s loss, saying it would be a mistake to scuttle it.

“It’s not a Democratic issue. It’s an American issue. It’s a national security issue,” Graham said. “For God’s sake, let’s fix immigration before it destroys our economy, our culture and leads to another 9/11-type attack.”

(TM and © Copyright 2014 CBS Radio Inc. and its relevant subsidiaries. CBS RADIO and EYE Logo TM and Copyright 2014 CBS Broadcasting Inc. Used under license. All Rights Reserved. This material may not be published, broadcast, rewritten, or redistributed. The Associated Press contributed to this report.)