Claude Operates Computers: Auto-Farming Upgrade Materials in Honkai: Star Rail

Discover how Claude 3.5 Sonnet automates GUI tasks, plays games, and revolutionizes human-computer interaction with advanced visual capabilities.

Where Does the Execution Power of Large Models Come From?

Last month, Anthropic released the upgraded Claude 3.5 Sonnet. This latest version can mimic human-computer interactions by moving cursors, clicking specific locations, and typing through a virtual keyboard based on user instructions.

In simple terms, Claude 3.5 Sonnet can now use a computer like a human!

Enhanced with the Computer Use feature, Claude 3.5 Sonnet becomes the first AI model capable of controlling a computer as a graphical user interface (GUI) agent during public testing.

As an early beta version, Anthropic noted during its release that the model's abilities in real-world complex environments remain uncertain.

Recently, a research team from the Show Lab at the National University of Singapore (NUS) designed and conducted a series of carefully curated tasks to explore the depths of the Computer Use feature.

The results? Beyond the "AI can now work independently" showcased in the demo, it turns out:

AI can also play games independently! Don’t believe it? Check out the VCR where Claude takes over Honkai: Star Rail gameplay:

Tasks such as finding dungeons and clicking teleport are now effortlessly handled by Claude’s "invisible hands."