Can a robot plan a holiday? Consumer group Which? asked bots to arrange a trip to Greece – but they proposed dodgy hotels, flights from the wrong COUNTRY and spouted ‘nonsense’

• The watchdog put five AI systems, including Bard and ChatGPT, to the test
• Which? asked the bots 10 questions, addressing almost every aspect of the trip 
• READ MORE: I use AI bots to attend meetings for me and manage my diary

You need to be very careful in how you use AI bots to help with holiday planning, Which? reveals.

The consumer group put their vacation-booking skills to the test, and found them wanting when asked to help organise a trip to Greece.

The watchdog put five AI systems to the test – OpenAI’s free ChatGPT and the premium version with a Kayak plug-in; Google’s Bard; Microsoft’s Bing Chat; and Expedia, which has integrated the ChatGPT software.

Which? asked the bots 10 questions to eke out advice for almost every aspect of the trip, from Greek islands that are best for peace and quiet to the best travel insurance options, car rental firms and hotels. 

There were some positives – such as the bots simplifying travel insurance jargon – but, from recommending hotels with terrible reviews to sourcing flight prices for airports in the wrong country, on the whole, AI wasn’t particularly useful. Here’s what the investigation uncovered…

THE GOOD

You need to be very careful in how you use AI bots to help with holiday planning, Which? reveals 

Which? was ‘impressed’ when Bing Chat suggested Kefalonia as an island to visit, as the holiday hotspot was voted Which? members’ favourite Greek island and was awarded five stars for peace and quiet.

The watchdog notes that Bing Chat was also the only bot to cite its sources, even citing a report by Which? in its advice.

In another positive, Which? found the AI bots were handy for jargon-busting – for instance, it notes that ChatGPT quickly translated a ‘wordy and confusing’ extract from a travel insurance policy into simpler, easier-to-understand terms.

THE BAD

One major negative, Which? found, was how AI bots’ responses could sound like a ‘backhanded sales pitch’. It explains that when it asked Bing for the best car hire companies at Kefalonia Airport, it recommended the ‘obscure’ CBR Car Hire Kefalonia, using language that appeared to be ‘lifted almost word for word from the company’s own website’ as justification for picking them. ‘It was a similar story for its recommendation of local rental Flydrive,’ Which? says.

As mentioned, ‘downright dodgy’ hotel recommendations were another red flag. Which? says that Erietta Studios was top of the list for ChatGPT Premium’s ‘best-rated hotels for under £150 a night’, even though the property has a mediocre 6.7 rating on Kayak and 2.5 on Tripadvisor, with more than a third of reviewers rating it ‘terrible’.

Which? was ‘impressed’ when Bing Chat suggested Kefalonia as a Greek island to visit, as the holiday hotspot was voted Which? members’ favourite Greek island. Above is the isle’s Fteri Beach

On top of that, Bard gave Which? several hotel recommendations with links that take users to various holiday providers’ websites. One of the links led users to ‘On the Beach’– a company that was awarded just three stars for customer service by Which? members, the watchdog notes.

Which? faced more problems when they sought advice for buying travel insurance. Bard’ lost the plot’ when the watchdog asked about holiday cover for a 70-year-old with pre-existing medical conditions and recommended a ‘global travel insurance company’ called Interpol – a firm that Which? couldn’t find any trace of online. The watchdog says that following the link provided led to an Interpol (the International Criminal Police Organisation) themed notebook for sale on Amazon.

The AI bots were similarly wayward when it came to sourcing flights. ChatGPT wrongly told Which? there were no direct flights from Birmingham to Kefalonia, as ChatGPT is only ‘educated’ up until September 2021 – the same year the Birmingham to Kefalonia route was launched by Jet2.com.

The watchdog then put the same question to the premium version of ChatGPT, which costs $20 (£16) per month, but the bot confused Birmingham in England for Birmingham, Alabama.

Financial advice from each of the AI bots varied wildly, Which? found. When it asked how much spending money would be needed for a two-week all-inclusive holiday to Kefalonia, answers ranged from £430 to a whopping £2,920 per person, the watchdog reveals.

Another warning sign flashed when Which? asked if the AI bots could book the trip. All said no, except Bard, which found the watchdog a Ryanair flight and asked for its credit card details, saying: ‘I will book the flight and send you a confirmation email.’ However, Google told the watchdog that Bard was getting ahead of itself and doesn’t have the ability to book flights yet.

Google also told Which? that it continues to make improvements ‘to ensure that accurate information is provided in response to queries’.

Meanwhile, touching on the negative aspects of the responses offered by Bing, Microsoft told Which? that Bing includes its sources so that users can ‘fact check’ and research its responses.

‘We are constantly looking to improve the authority and credibility of our web results,’ Microsoft said.

When Which? asked how much spending money would be needed for a two-week all-inclusive holiday to Kefalonia, answers ranged from £430 to a whopping £2,920 per person. Above is the pretty village of Assos on Kefalonia 

And OpenAI acknowledged that ChatGPT sometimes gives ‘plausible-sounding but incorrect or nonsensical answers’ – and admits that fixing this issue is ‘challenging’, Which? reveals.

In a statement to the watchdog, Kayak added that it was ‘early days’ for the tech, vowing to make changes to help it ‘prioritise nearby locations’ in future.

The upshot from Which? is that you should ‘make sure to do some extra research before turning your fictional trip into a reality’.

Which? senior researcher/writer Laura Sanders says: ‘The chatbots were able to converse with us in a very natural way, which gives them an air of expertise – but don’t be fooled… several [responses we received] were biased, some were out of date and others were utter nonsense. Worse still, AI often doesn’t reveal its sources, meaning you’re none the wiser if that glowing room recommendation has come directly from the hotel manager.’

And Rory Boland, Editor of Which? Travel, said: ‘More and more consumers are likely to encounter artificial intelligence when booking holidays, but worryingly we’ve found that these services can often give travellers information and recommendations that are inaccurate, biased or even out of date.

‘Because AI chatbots have learned to communicate in a natural, conversational style, it can be easy to be lulled into a false sense of security, and accept their answers at face value – even though the information AI uses can often be directly lifted from marketing materials, or unreliable sources. Those that do decide to experiment with AI to get ideas for their next trip should always make sure to take the time to do their own research and check reviews to make sure they’re getting reliable recommendations.’

Source: Read Full Article