It is not possible to see how the Irish border issue can be resolved after Brexit, the influential group of MPs scrutinising the process has said.
The government wants no hard border between the Irish Republic and Northern Ireland and no customs border between the latter and the rest of the UK.
Ministers have suggested technology could enable a "frictionless border".
But the Committee for Exiting the EU said the proposals were "untested" and "to some extent speculative".
Northern Ireland will be the only part of the UK to share a land border with an EU member state after the UK leaves.
There is currently no physical infrastructure on the border but there is concern that this will have to change after Brexit.
If the UK leaves the EU's single market and customs union, as the government intends, the Irish border will become the external border for the EU's single market and customs union.
The Irish Republic wants Northern Ireland to keep following EU rules, so that goods can continue moving across the border – in effect, staying within the customs union and single market.
But this would effectively push the customs border out into the Irish Sea, becoming an internal customs border between Northern Ireland and the rest of the UK – which the UK government rejects.
In its report, the Exiting the European Union Committee says it does not see how it will be possible to reconcile these positions.
The government has put forward two proposals, one using "technology-based solutions", such as pre-screening of goods and trusted trader schemes, to reduce the need for customs checks at the Irish border.
The other would involve a "customs partnership", with the UK leaving the single market without introducing an EU-UK border – something the UK has admitted would be "challenging".
The committee is urging the government set these proposals out in more detail.
UK Prime Minister Theresa May has said the UK and Irish governments "have the same desire" on the border – to ensure that the movement of trade and people continues "as now" and that no new barriers are created.
The government added that it remained "absolutely committed to finding a solution that works for the people of Northern Ireland and Ireland".
The Irish government has always insisted there must not be a hard border between the Republic and Northern Ireland, with Taoiseach Leo Varadkar saying he must have written assurance from the UK before Brexit talks can move on.
But Northern Ireland's Democratic Unionist Party has warned that any attempt to "placate Dublin and the EU" could lead to the end of its confidence-and-supply agreement with the UK's Conservative government.
The DUP struck a deal with the minority Conservative government in June, agreeing to support Tory policies at Westminster, in return for an extra £1bn in government spending for Northern Ireland.
The Committee for Exiting the EU itself was split over the report, with five of its 21 members – four Conservatives and a DUP MP – voting to reject it.
The report also includes a call for the government to publish the likely terms of any transition period governing what will happen immediately after Brexit in March 2019.
It says it is "essential" that the details of the arrangements be published by the end of March so as to give businesses enough time to prepare.
It adds that any deal reached on the rights of EU citizens in the UK and vice versa should not be dependent on getting the so-called Brexit divorce bill or the Irish border issue agreed.