There can be no final decisions on the future of the Irish border until the UK and the EU have reached a trade agreement, Liam Fox has said.
The UK's international trade secretary also blamed the EU for Brexit delays.
The comments came after the Irish Republic's EU commissioner said Dublin could veto Brexit trade talks.
The EU has said "sufficient progress" has to be made on the Irish border before negotiations on a future relationship can begin.
Downing Street has said the whole of the UK will leave both the customs union and the single market when it leaves the EU in 2019.
"We don't want there to be a hard border but the UK is going to be leaving the customs union and the single market," Mr Fox told Sky News.
He added: "We can't come to a final answer to the Irish question until we get an idea of the end state. And until we get into discussions with the EU on the end state that will be very difficult – so the quicker we can do that the better, and we are still in a position where the EU doesn't want to do that."
Mr Fox accused the European Commission of having an "obsession" with ever-closer union between EU member states, which was delaying progress in Brexit talks.
Phil Hogan, the EU's agriculture commissioner, told the Observer that staying in the customs union would negate the need for a hard border – with customs posts and possible passport checks – on the island.
He said Dublin would "play tough to the end" over its threat to veto trade talks until it had guarantees over the border.
Shadow chancellor John McDonnell said he was "worried" by Mr Fox's comments, adding that Labour would not take continued membership of the single market and the customs union off the table.
"I think the one thing that we don't want to do is jeopardise any movement quickly, because we need movement to enable us to get into the proper trade negotiations," Mr McDonnell told ITV's Peston on Sunday.
"So I'm hoping that isn't a Downing Street-sanctioned statement that's he's made."
The EU has given Prime Minister Theresa May until 4 December to come up with further proposals on issues including the border, the Brexit divorce bill and citizens' rights, if European leaders are to agree to moving on to trade talks.
But Mr Hogan accused some in the British government of having what he called "blind faith" about securing a comprehensive free-trade deal after Brexit.
He said it was a "very simple fact" that "if the UK or Northern Ireland remained in the EU customs union, or better still the single market, there would be no border issue".
In these circumstances regulations on either side of the border would remain the same, and so a near-invisible border would be possible.
The Irish government has always insisted there must not be a hard border between the Republic and Northern Ireland, with Taoiseach Leo Varadkar saying he must have written assurance from the UK before Brexit talks can move on.
Irish Foreign Minister Simon Coveney has said the UK's desire for no hard border on the island of Ireland was "aspirational".
But in her speech in Florence, this September, Mrs May restated that both the UK and EU would not accept any physical infrastructure at the border.
The Democratic Unionist Party said Northern Ireland and the rest of the UK must not be different.
Arlene Foster, the leader of the DUP, which is in a confidence-and-supply arrangement with the Conservative government, said she would not support "any suggestion that Northern Ireland, unlike the rest of the UK, will have to mirror European regulations".
Irish MEP Mairead McGuinness said she was "troubled" by Mr Fox's remarks and urged the UK to stay in the single market and customs union.
"I hope that the United Kingdom is not holding the Irish situation to ransom in these negotiations – it is far too serious and far too critical," she told the Sunday Politics on BBC One.
But Conservative MP Owen Paterson told the same programme that there should be no question of Northern Ireland leaving the "single market of the UK".
Mr Paterson, a former Northern Ireland secretary, accused some politicians of trying to "force and blackmail the UK into giving a special status to Northern Ireland outside the rest of the UK".
Suggestions for alternate arrangements have included a new partnership that would "align" customs approaches between the UK and the EU, resulting in "no customs border at all between the UK and Ireland".