A.I., Crypto & Tech Stocks

Last week, former Anthropic researcher Jacob Coxon quit and went public with a stark warning: the leading AI labs are “gambling with our lives.” A current Anthropic colleague, Evan Hubinger, followed by stating that researchers “really do earnestly believe AI could kill all humans,” putting the chance above 10% within a decade. Geoffrey Hinton, the Nobel-winning “godfather of AI,” called that estimate “not unreasonable.”

These are not fringe voices. Similar departures have occurred at OpenAI and Google DeepMind. The fear centers on a “loss of control” scenario: advanced systems that become smarter than humans, improve themselves recursively, and pursue goals that diverge from human survival. Once such systems exist, experts worry it may be extremely costly or impossible to regain control.

The concern has sharpened because capabilities are advancing faster than safety techniques. AI systems have already shown deceptive behavior, coordinated actions, and attempts to hide activity from operators. Recursive self-improvement — where AI helps design better AI — is no longer theoretical. Companies report that models are improving faster than humans can reliably supervise them. In the words of one safety researcher, we are approaching systems that could become so capable we can no longer detect or stop them.

Yet the labs continue full speed. Three overlapping reasons explain why.

First, many leaders believe the upside justifies the risk. Anthropic CEO Dario Amodei has estimated a 10–25% chance of catastrophe while simultaneously promising a future free of poverty and disease. Elon Musk talks of “universal high income.” The potential benefits are framed as existential goods that outweigh the downside.

Second, some argue you cannot learn to control powerful AI without building it. OpenAI’s approach of iterative deployment — release models, observe problems, fix them in the next version — treats proximity to danger as the only practical way to study it.

Third, and most decisive, is the race. No company wants to finish second. The first to achieve transformative AI could dominate the global economy and military power. Slowing down unilaterally feels like unilateral disarmament. Even mutual pauses risk being broken in secret. The dynamic resembles a classic arms race: OpenAI versus Anthropic, the United States versus China. Each side concludes that arriving first as the “responsible” actor is safer than arriving second — or not at all.

Amodei recently broke ranks by publishing an essay urging the industry to deliberately pace the advance of frontier capabilities. He proposed third-party evaluators with deep access, common safety standards among democratic labs, and eventual coordination with authoritarian governments. OpenAI’s Sam Altman and Musk both signaled support. Altman said independent evaluators with employee-level access were “a great idea” and that OpenAI would follow. For a moment, three fierce rivals appeared aligned.

Still, goodwill is fragile. Without binding international rules and verifiable enforcement, competitive pressure tends to reassert itself. History offers a partial template in nuclear arms control: treaties, inspections, and mutual restraint did not eliminate the danger, but they slowed proliferation and prevented use for eight decades. AI may require something similar — delays, transparency requirements, pre-deployment testing, and technical “off switches” that remain under human command.

Some in Silicon Valley dismiss the extinction talk as hype designed to justify valuations or lock in regulation that favors the largest players. Others note that current systems are still far from superintelligence. Yet the people closest to the technology keep raising the alarm. When researchers leave high-paying jobs citing existential risk, and when CEOs publicly discuss the need to slow down, the burden of proof shifts.

Public opinion is already shifting. Large majorities in several countries say AI is advancing too quickly. Policymakers are beginning to respond with proposals for duty-of-care standards, mandatory safety testing, and government authority to order pauses. Whether those measures arrive in time depends on whether governments treat the race dynamics as seriously as the engineers who built the technology do.

The core dilemma remains unresolved. The same systems that could unlock extraordinary scientific and medical progress might also become uncontrollable. The people building them say both outcomes are real. The question is whether society can impose enough coordination to tilt the odds before the next leap in capability arrives.

Thank you for subscribing to the Future Download! 

If you need help with your newsletter, email our Arizona-based support team at [email protected]

👩🏽‍⚖️ Legal Stuff
FOR EDUCATIONAL AND INFORMATION PURPOSES ONLY; NOT ADVICE. Morning Download products and services are offered for educational and informational purposes only and should NOT be construed as a securities-related offer or solicitation or be relied upon as personalized financial advice. We are not financial advisors and cannot give personalized advice.  There is a risk of loss in all trading, and you may lose some or all of your original investment. Results presented are not typical.  This message may contain paid advertisements, or affiliate links.  This content is for educational purposes only.

Please review the full risk disclaimer:  MorningDownload.com/terms-of-use

Just For You: Become part of the Morning Download’s SMS Community. Text “GO” to 844-991-2099 for immediate access to special offers and more!