xAI: The game of catching up

Elon Musk has had dreams and concerns about AI for a long time. This July, his AI lab xAI celebrates its first year, and it’s already among the few most important generative AI unicorns with a recent valuation of over $24 billion. Despite controversies, the company’s rapid progress with Grok-1 has sparked curiosity and skepticism. What are Musk’s plans, how does he X-utilize his assets to create Muskonomy, and what lies ahead for xAI and its main creation, Grok? Let’s investigate!

In today’s episode:

In 2015, Elon Musk supported Sam Altman’s idea of spinning up an AI lab through YC with the goal of building safe human-level AI. Elon even suggested the name: Open AI. Knowing his obsession with inserting X in all his companies, one might argue he didn’t want this lab to be his initially, but he was really a keen supporter. According to Greg Brockman’s memories: “I was struck by how inquisitive Elon was, and how much he sought others' opinions and really listened to them.”

As we wrote in our OpenAI profile, everything changed in 2018: Elon Musk resigned from the board, officially due to the potential conflict of interest between Tesla and OpenAI, but most likely because he couldn't take over the company and run it his way. Tesla was indeed using machine learning and AI quite extensively, but Elon didn’t demonstrate any signs that he wanted to start his own AI lab to pursue human-level AI. His goal seemed to trash Sam Altman and OpenAI.

Especially when OpenAI split the world into before and after with their ChatGPT in November 2022. A month before that, in October 2022 – at that point unrelated completely – another important event happened. As the New York Times put it: “After months of waffling, lawsuits, verbal mudslinging, and the near miss of a full-blown trial, Elon Musk now owns Twitter.” A gold mine for data. But at first, he kept his course trashing OpenAI everywhere and even signed the famous Pause in AI training letter, in March 2023, advocating a halt for six months.

But how can you resist when you have enormous financial resources, billions of data points in your hands, a wounded ego, and you are Elon Musk? You simply can’t. So, no matter the call for a halt in AI research, in July 2023, Musk, of course, launched his own company. Now we know for sure he is serious about it because he calls it xAI.

Grok-0, a 33-billion-parameter dense transformer model, was completed on 18 August 2023, and by 3 November 2023, Grok-1 was released for early access, with hundreds of users starting to use the conversational AI.

The community was a bit skeptical about these results published in the Grok-1 announcement because xAI wrote: “These benchmarks can be found on the web and we can’t rule out that our models were inadvertently trained on them.” Users on the OpenAI subreddit had a lively debate about “How did xAI create Grok so quickly?” People suggested that xAI quickly created Grok by leveraging existing architectures such as GPT and LLaMA and using publicly available datasets. They utilized over a decade of Twitter data – a wealth of high-quality conversational material. By employing Tesla's powerful GPU data centers, they accelerated the training process. Additionally, xAI incorporated open-source models and data from platforms like HuggingFace. The combination of high-volume, quality data, efficient training techniques like fine-tuning and transfer learning, and advanced neural network processors significantly reduced training time. There is also speculation that Grok may be a customized version of existing models like GPT, further simplifying the development process.

The Grok-1 model card, a succinct one-page table, reveals minimal about its architecture other than being an autoregressive transformer-based model pre-trained for next-token prediction, fine-tuned with extensive feedback from both humans and its predecessor model, Grok-0, and featuring a context length of 8,192 tokens.

To develop Grok, xAI constructed a robust deep learning infrastructure utilizing Kubernetes, Rust, and JAX. The team implemented custom distributed systems designed to automatically detect and manage GPU failures, ensuring high Model FLOP Utilization (MFU)* and minimal downtime.

*Model FLOP Utilization (MFU) measures how efficiently a model uses floating point operations (FLOPs). It indicates the proportion of FLOPs that contribute to actual model performance.

Rust was chosen for its performance, robust ecosystem, and ability to prevent common bugs typical in distributed systems. This choice is particularly critical given the small size of the team, as it enables the infrastructure to operate reliably with minimal oversight. Regarding training, despite common doubts, the documentation states: “Like most LLMs today, Grok-1 was pre-trained by xAI on a variety of text data from publicly available sources from the Internet up to Q3 2023 and datasets reviewed and curated by AI Tutors who are human reviewers. Grok-1 has not been pre-trained on X data (including public X posts).” However, Grok does have “real-time knowledge of the world,” including posts on X.

When xAI released Grok-1 in November 2023, the waiting list for the model was available only through a subscription to the X Premium plan or higher. Concerns arose that xAI was using data from X, formerly Twitter, to train Grok; however, the documentation indicated otherwise. By December 2023, Grok became accessible to premium users on the X website and its applications.

In March, Musk sued OpenAI and CEO Sam Altman, claiming they prioritized profits over public good. As he was also continually accusing OpenAI of being very closed-source, there was nothing left for him but to open-source xAI's chatbot code. Of course, it wasn’t the model you would pay for on X. It was a raw base model checkpoint from its pre-training phase, which concluded in October 2023. It was not until this release that it became known that the base Grok-1 model is a 314 billion parameter Mixture-of-Experts model, with 25% of the weights active for any given token, developed entirely from scratch. Musk’s move fuels the debate on whether open-source AI is safer or riskier.

The latest model updates

The latest model updates: The same month that xAI open-sourced Grok-1, the company also released Grok-1.5, which features improved reasoning and problem-solving capabilities, along with an extended context window of 128K tokens. However, specific details about the model’s architecture were not disclosed.

 In April 2024, Grok-1.5 was upgraded to Grok-1.5V. This version of Grok is capable of processing both textual and visual data. Along with the model, the team introduced a new benchmark called RealWorldQA, designed to evaluate the basic real-world spatial understanding capabilities of multimodal models. This benchmark includes over 700 images, each accompanied by a question and an easily verifiable answer. Technical specifications for Grok-1.5V, however, have not been made public.

He also mentioned that currently, Grok 2.0 is being trained on 20,000 GPUs. Later, on Twitter Spaces, Elon Musk said that they would need about 100,000 GPUs to train Grok 3.0.

Elon Musk needs compute not only for his spanky, anti-Woke xAI. A while ago, for Tesla and Tesla Bot, he started to accumulate resources. For its ML and AI needs, Tesla developed a supercomputer called Dojo, which utilizes Tesla's custom D1 chips. In addition to Dojo, Tesla leverages 35,000 Nvidia H100 GPUs to train its self-driving AI, with plans to double that capacity by year-end.

According to The Information, Elon plans to "string all these chips into a single, massive computer" – or what he's calling a "gigafactory of compute." This supercomputer, using Nvidia's H100 GPUs, is expected to be operational by fall 2025 and will be significantly larger than the current biggest GPU clusters, such as those by Meta Platforms. Musk promised he would be "personally responsible for delivering it on time." But everyone who follows Tesla cars knows that Musk’s timeline is not correlated with an ordinary human's timeline.

Despite this ambitious plan, xAI will still lag behind competitors like OpenAI and Microsoft, who are working on even larger clusters, potentially worth $100 billion and containing millions of Nvidia GPUs. To catch up, xAI aims to raise $6 billion to invest in Nvidia’s advanced chips to develop AI with human-like capabilities. Of course, xAI is among the first companies to receive Nvidia’s upcoming Blackwell chip, but again along with OpenAI, Amazon, and Google.

SemiAnalysis calculated that renting 100,000 GPUs costs over $4 billion in server capital expenditures. Additionally, the power consumption for such a cluster is about 1.59 terawatt hours annually, costing approximately $123.9 million at a rate of $0.078 per kWh.

Musk’s plan to position the venture as a key player in the AI landscape by leveraging unique data assets, significant infrastructure investments, and strategic collaborations. The integration with Musk’s broader business ecosystem, referred to as the "Muskonomy," is central to the startup’s monetization strategy.

Integration with Existing Companies: xAI plans to monetize its products by integrating them with Musk's other companies, such as Tesla. For example, xAI's chatbot Grok will be sold through these companies, enabling direct user engagement and reducing intermediary costs​.

Data Utilization: Musk intends to leverage the vast amounts of proprietary data from his companies, including Tesla, SpaceX, and Neuralink, to enhance xAI's models. This data includes visual, sensory, and navigation information, which will help improve the real-world performance of AI models​.

Optimus Robot: Tesla’s Optimus robot will serve as a platform for xAI’s models to learn and improve through real-world interactions. This not only aids in the development of more advanced AI but also creates another potential revenue stream through sales and subscriptions​. The push toward autonomous AI agents accelerated across the industry — Cognition AI's Devin redefined what AI agents can do for software teams, raising $400M at a $10B valuation and intensifying the race Musk is part of.

Premium Services on X: xAI's products, such as the Grok chatbot, will be available to subscribers of X’s Premium+ service. This model aims to monetize the vast user base of the social media platform formerly known as Twitter by offering advanced AI capabilities as part of a premium package​.

In June 2024, Tesla shareholders filed a lawsuit against Elon Musk, accusing him of "brazen disloyalty" by diverting AI talent and resources to his new venture, xAI, as reported by Business Insider. The plaintiffs claim that Musk poached employees, redirected Nvidia GPUs initially intended for Tesla, and demanded more control over the company, all contributing to a significant drop in Tesla's stock value.

Musk will argue but internal emails from Nvidia reveal that Musk indeed directed the company to prioritize shipments of AI chips for X and xAI, delaying Tesla's receipt of over $500 million worth of processors. This move has raised concerns among Tesla shareholders about Musk's commitment to the automaker, especially as Tesla's stock has declined significantly.

The conflict highlights Musk's management of multiple ventures and the potential conflicts of interest that arise. Critics argue that his actions demonstrate a clear disregard for Tesla's interests, impacting its technological advancements and market position in the development of autonomous vehicles and robotics.

Elon Musk is at the helm.

Musk's researchers emerged just a year ago, yet its team is already competing with leading generative AI model providers like OpenAI, Google, and Cohere, drawing on similar foundational expertise. has recruited talented researchers and practitioners who have significantly contributed to the development of widely recognized algorithms, methods, and models in the machine learning industry.

Here are some key contributions along with brief descriptions:

Members of xAI's team also participated in the creation of Transformer-XL, SimCLR, AlphaStar and AlphaCode, Minerva, GPT-3.5, and GPT-4. Leading the team at xAI is Dan Hendrycks, a young and prominent figure known for co-authoring the GELU activation function, which is integral to state-of-the-art models such as BERT, GPT, and Vision Transformers.

According to their website, xAI is primarily focused on the development of advanced AI systems that are truthful, competent, and maximally beneficial for all of humanity. The company’s mission is to understand the true nature of the universe.

Elon Musk's AI startup, xAI, has registered in Nevada as a "benefit corporation," a type of for-profit company that prioritizes benefiting society over maximizing shareholder profits. This structure, mirroring AI rivals OpenAI and Anthropic, emphasizes the company's commitment to a positive societal impact. Becoming increasingly popular, benefit corporations often appeal to employees, customers, and investors seeking to align themselves with socially responsible endeavors.

The company's recent funding round, totaling $6 billion, brought together prominent investors like Valor Equity Partners, Andreessen Horowitz, and Sequoia Capital, amongst others. This investment elevates the company's valuation to $24 billion. While this figure trails OpenAI's substantial $86 billion valuation, it surpasses Anthropic's $18 billion valuation.

“Maybe the real AGI was the friends we made along the way. @xAI”

Thank you for reading, please feel free to share with your friends and colleagues. 🤍