The global travel industry is currently navigating a significant technological paradox: while major brands publicly emphasize their commitment to artificial intelligence and digital transformation, their underlying website infrastructures are often failing to communicate with the very AI tools they hope will drive future revenue. A recent investigative review conducted by Skift, which examined the robots.txt files of 50 major travel websites, reveals a critical disconnect between corporate strategy and technical execution. These findings suggest that a majority of the industry’s leading players have yet to optimize their digital presence for the burgeoning era of AI-driven search and discovery.
At the heart of this issue is the robots.txt file, a fundamental component of web architecture that acts as a gatekeeper for automated programs, commonly known as crawlers or bots. These files provide instructions to search engine spiders and other automated agents regarding which parts of a website they are permitted to access and index. As generative artificial intelligence (AI) and large language models (LLMs) like ChatGPT, Claude, and Gemini become primary tools for consumer travel planning, the ability of these models to accurately "read" brand content has become a vital competitive advantage. However, the Skift audit found that only 20% of the surveyed travel sites—just 10 out of 50—explicitly addressed AI-specific crawlers in their protocols. The remaining 40 sites either grouped AI bots with generic web crawlers or relied on outdated default settings that may inadvertently restrict or confuse the machines attempting to parse their data.
The Adobe Analysis: A Surge in AI-Driven Travel Traffic
The urgency of addressing this technical gap is underscored by recent consumer behavior data. In July, Adobe released a report highlighting a massive shift in how travelers interact with the internet. According to Adobe’s findings, AI-driven visits to travel websites surged by 119% year-over-year. This exponential growth indicates that travelers are increasingly bypassing traditional search engines in favor of conversational AI interfaces to research destinations, compare prices, and seek personalized recommendations.
Despite this clear shift in consumer preference, Adobe’s lead analyst for digital insights, Vivek Pandya, has identified a "major bottleneck" in the digital ecosystem. Pandya noted that significant portions of U.S. travel and retail sites remain functionally unreadable to AI machines. This lack of readability restricts a brand’s presence across AI search results, effectively rendering their latest offers, amenities, and pricing updates invisible to the latest generation of digital shoppers.
"When an LLM cannot easily read brand content—such as specific amenities, real-time pricing, or room availability—potential revenue is being left on the table," Pandya wrote in a recent industry analysis. The implication is clear: if an AI agent cannot confirm that a hotel offers a pet-friendly policy or a specific discount for a summer booking, it will exclude that property from its recommendations, directing the consumer to a competitor whose site is more "machine-friendly."
Chronology of the Shift: From SEO to AEO
The current friction between travel brands and AI crawlers is the latest chapter in the evolving relationship between the travel industry and the wider internet. To understand the gravity of the current situation, it is necessary to look at the timeline of digital discovery in the travel sector.
For the past two decades, the industry focused almost exclusively on Search Engine Optimization (SEO). The goal was to rank highly on Google’s traditional search results pages (SERPs) by optimizing for keywords and backlinks. However, the introduction of OpenAI’s ChatGPT in late 2022 and the subsequent launch of Google’s Search Generative Experience (SGE) in 2023 fundamentally altered the landscape.
In early 2023, several major travel brands, including Expedia and Kayak, were among the first to launch plugins for ChatGPT, signaling an early adoption of generative AI. Yet, as the technology moved from experimental plugins to integrated search features, the focus shifted toward "AI Engine Optimization" (AEO). By mid-2024, it became evident that many brands had focused on the "front-end" of AI—such as chatbots on their own homepages—while neglecting the "back-end" infrastructure that allows external AI models to index their data accurately.
The Technical Barrier: Understanding Robots.txt and AI Agents
The Skift review highlights a fundamental misunderstanding of how AI agents interact with the web. In the traditional web era, a single set of instructions for "Googlebot" was often sufficient. Today, however, there are dozens of distinct AI-specific crawlers, such as OpenAI’s GPTBot, Anthropic’s user-agent, and Common Crawl’s CCBot.
When a travel brand fails to specify instructions for these bots, it creates a vacuum. Some AI crawlers are programmed to be aggressive, scraping as much data as possible unless explicitly told otherwise. Conversely, many AI models are designed to respect the "Disallow" directives intended for older, less sophisticated bots. If a travel site has a legacy "Disallow: /" command aimed at preventing old-school price-scrapers, it may inadvertently be blocking the very AI models that could drive high-value traffic to their site.
Furthermore, the quality of data ingestion is at stake. LLMs do not just index keywords; they attempt to understand context and relationships between data points. A website that uses non-standard formatting or lacks structured data (Schema markup) makes it difficult for an AI to distinguish between a "starting at" price and a "final" price, or between a "seasonal" amenity and a year-round service.
Industry Responses and the Strategic Dilemma
While many travel executives acknowledge the importance of AI, the decision to open their digital doors to AI crawlers is fraught with strategic tension. This tension likely explains why 80% of the sites in the Skift study have not yet updated their robots.txt files to welcome AI specifically.
There are three primary concerns currently circulating among travel industry stakeholders:
-
The Risk of Data Scraping and Competition: Many travel brands, particularly Online Travel Agencies (OTAs) and airlines, are protective of their proprietary pricing data. There is a fear that by making their sites easily readable for AI, they are essentially handing over their "secret sauce" to competitors who could use LLMs to monitor price fluctuations and undercut them in real-time.
-
Copyright and Intellectual Property: The debate over whether AI companies should be allowed to train their models on proprietary brand content remains unresolved. Some travel brands may be keeping AI bots at arm’s length as a defensive measure until clearer legal frameworks regarding AI training data are established.
-
Traffic Cannibalization: There is a growing concern that if an AI can provide a traveler with all the information they need (pricing, photos, and reviews) directly within the search interface, the user may never actually click through to the brand’s website. This "zero-click" search phenomenon threatens the traditional direct-booking model that many hotels and airlines have spent years cultivating.
Despite these concerns, the data from Adobe suggests that the risk of being invisible to AI search results may outweigh the risks of data scraping. As AI agents become more capable of performing complex tasks—such as booking an entire multi-city itinerary—the brands that are most accessible to these agents will likely capture the lion’s share of the market.
Broader Implications for the Travel Ecosystem
The failure of major travel brands to optimize for AI has implications that extend beyond individual revenue losses. It reflects a broader "technical debt" within the industry. Many major hotel chains and legacy airlines operate on aging tech stacks that were not designed for the high-frequency, high-context demands of modern AI.
Moreover, this gap creates an opportunity for smaller, more agile players. Boutique hotel groups or niche travel startups that prioritize machine readability from the outset may find themselves punching above their weight in AI-driven search results, simply because their data is easier for LLMs to digest and present to consumers.
The situation also places a spotlight on the role of intermediaries. If direct brand sites remain unreadable, AI models will likely turn to third-party sources—such as travel blogs, review sites, and social media—to gather information. This could lead to the dissemination of outdated or inaccurate information, as the AI relies on secondary sources rather than the brand’s official, real-time data.
Conclusion: The Path Toward Machine-Readability
To bridge the gap between AI aspiration and technical reality, travel brands must move toward a more proactive digital strategy. This includes not only updating robots.txt files to explicitly manage AI crawlers but also adopting advanced structured data formats that provide the context AI models require.
As Vivek Pandya’s analysis suggests, the transition to an AI-first search environment is not a distant future prospect; it is a current reality. With AI-driven visits increasing by triple digits, the "bottleneck" of machine-unfriendly websites represents a significant barrier to growth in an increasingly competitive global market. For travel brands, the message is clear: being found by human travelers now requires first being understood by the machines they use. The brands that fail to update their digital gatekeepers risk being left out of the conversation entirely, as the next generation of travelers increasingly relies on AI to navigate the world.
