Technologies
Which Apple Watch Is for You? Apple Watch Series 11 vs. Ultra 3 and SE 3
Apple revamped its entire Apple Watch line, but some models got more improvements than other. We look at all the details.
When you’re looking to buy a smartwatch, whether you want a way to stay on top of notifications or its health-related features, the Apple Watch is a good place to start. But there isn’t just one. Apple sells three different versions that appeal to different needs.
The flagship Apple Watch Series 11, rugged Apple Watch Ultra 3 and economical Apple Watch SE 3 models are a lot closer in features and specs than they’ve ever been. Let’s compare them to see which deserves a place on your wrist.
Pricing the Apple Watch Series 11, Ultra 3 and SE 3
The current Apple Watch line spans a wide price range, from $249 for the Apple Watch SE 3 to $1,299 for a titanium Apple Watch Series 11 with an Hermès band. Like most fashion accessories, you can choose from different case sizes, materials, cellular connectivity options and bands to find the right fit — and price — for your watch.
The Apple Watch Series 11 offers the widest price range, with two case sizes, two body materials, optional cellular connectivity and premium Hermès models. The Apple Watch SE 3 is available only in aluminum and has the earlier, slightly smaller case sizes. The Apple Watch Ultra 3 comes only in titanium, with a single 49mm size and cellular included by default.Â
Read more: Find the best deals on the Series 11 and Ultra 3.
Here’s how they break down:
| Apple Watch Series 11 | GPS | GPS plus cellular |
|---|---|---|
| 42mm aluminum | $399 | $499 |
| 46mm aluminum | $429 | $529 |
| 42mm titanium | $699 | |
| 46mm titanium | $749 | |
| Hermès 42mm titanium | $1,249 | |
| Hermès 46mm titanium | $1,299 | |
| Apple Watch SE 3 | ||
| 42mm aluminum | $249 | $299 |
| 46mm aluminum | $279 | $329 |
| Apple Watch Ultra 3 | ||
| 49mm titanium | $799 | |
| Hermès 49mm titanium | $1,399 |
Series 11 vs. Ultra 3, SE 3 physical designs
The core rounded-rectangle design of the Apple Watch has seen incremental changes since its first iterations. The Series 11 shares the slimmer 9.7mm height profile of the Series 10, with 42mm and 46mm diagonal sizes. Weight is light across the board, from 29.7 to 43.1 grams depending on size and case material. Aluminum models come in space gray, jet black, rose gold or silver, while titanium versions are offered in natural, slate or gold finishes.
The Apple Watch SE 3 is slightly thicker (10.7mm) and slightly smaller, with 40mm and 44mm sizes. Its design most closely harkens back to earlier Apple Watch models. It weighs 26.4 grams or 33 grams, depending on the case size. As the no-frills option, the SE 3 is available in either midnight (black) or starlight (silver).
The Ultra 3 is the most significant departure from the original design, with a thickness of 14.4mm, a 49mm diagonal, and a weight of 61.6 grams. Its titanium body comes in either natural or black — unless you opt for the Hermès edition, which is only offered in natural.Â
CNET lead writer Vanessa Hand Orellana described the Ultra 3 in her review as being “like the luxury Land Rover you see in safari brochures: It’s adventure-ready on the outside, with all the modern conveniences on the inside.” The body is also 3D-printed from 100% recycled titanium, but you’d never know it; there are no telltale layering marks found on most 3D-printed items.
Each watch has Apple’s Digital Crown and a side button. The Ultra 3 also includes a programmable Action button, which can, for example, start a workout with a single press.
Aside from looks, all three Apple Watch models are built for durability. The SE 3 is water resistant to 50 meters, so you don’t need to baby it — whether you’re showering, swimming or just living through a rainy Pacific Northwest day.
The Series 11 is also rated for water resistance to 50 meters, while the Ultra 3 doubles that to 100 meters. They’re also both certified IP6X dust-resistant, which is better than the SE.
Series 11 vs. Ultra 3, SE 3 displays
The Series 11 and Ultra 3 both use an LPTO 3 OLED display, which offers an always-on display and a wide viewing angle, so you don’t need to look at it head-on to see the time. It can also get very bright: 2,000 nits of peak brightness for the Series 11 and 3,000 nits (the same as the iPhone 17 Pro) for the Ultra 3.Â
The energy-efficient screen can refresh its display at just one nit of brightness once every second when in its passive state, so you can always see the second hand or indicator (depending on the watch face).
The display is protected by a sapphire crystal on the titanium Series 11 and the Ultra 3. According to Apple, the aluminum Series 11 uses Ion-X glass, which is twice as scratch-resistant as the Series 10.
In past generations, the SE was stuck with the lowest-quality screen, but not this time. The SE 3 gets an always-on LTPO OLED display that reaches up to 2,000 nits of brightness and dims to just 2 nits when inactive. But it doesn’t refresh as often as the Series 11 and Ultra 3, so the seconds indicator only appears when the screen is active. It’s still a big “quality of life” bump from prior SE watches, which don’t have an always-on display mode.
Series 11 vs. Ultra 3, SE 3 battery life
One surprise with the new Apple Watch lineup is improved battery life in the Series 11 and Ultra 3, plus a fast-charge option on the SE 3 that lets you top up for a night’s sleep after a full day.
Apple claims up to 24 hours of use on a single charge for the Series 11, up from 18 hours for the Series 10. It also claims up to 38 hours in Low Power mode, a notch above the Series 10’s 36 hours. That fast-charging option can bring the battery level up to 80% in 30 minutes, but charging the watch for just 15 minutes can boost it for up to 8 hours.Â
Hand Orellana writes in her Series 11 review, “The six-hour battery bump on the Series 11 may not sound like much on paper, but it’s given me some welcome breathing room to figure out a better charging strategy.”
The SE 3 still delivers up to 18 hours of use, or 32 hours in Low Power mode. It also supports fast charging — up to 80% in 45 minutes, or about eight hours of use from a quick 15-minute top-up.
If you want the most time between charges, the Ultra 3 is the Apple Watch to get. It can last for up to 42 hours, per Apple, or up to 72 hours in Low Power mode. Fast charging its larger battery takes it to 80% in about 45 minutes, and 15 minutes on the cable will give you roughly 12 hours of power.
Some of these gains come from Apple factoring in a night’s sleep, but credit also goes to the more power-efficient LTPO 3 screen in the Series 11 and Ultra 3.
It’s one thing to reference Apple’s claims, but what about battery life in practice? In Hand Orellana’s review of each model, she recorded even better battery life than Apple’s estimates. Keep in mind your daily usage will affect results, but here’s what she found:
| Apple Watch | Apple’s estimate | CNET review |
|---|---|---|
| Series 11 | 24 hours | 27-32 hours |
| Ultra 3 | 42 hours | 45-49 hours |
| SE 3 | 18 hours | 20-25 hours |
Series 11 vs. Ultra 3, SE 3 health features
The Apple Watch SE line has always sacrificed some hardware and features to remain the least expensive option, and the SE 3 continues that tradition — but not to the same extent. It lacks an electrical heart sensor found in the Series 11 and Ultra 3, so it can’t take heart readings using the ECG app to look for signs of atrial fibrillation (Afib).
According to Apple, the SE 3 uses a second-generation optical heart sensor that tracks heart rate during exercise, sleep and potential emergencies — though, like all Apple Watches, it can’t detect heart attacks or measure blood oxygen. The Series 11 and Ultra 3 upgrade to third-generation optical heart sensors.
The SE 3 is also missing a water temperature sensor and depth gauge, making the Series 11 and Ultra 3 better options if you spend a lot of time in water and want to track swim workouts or shallow dives more reliably.
This year’s standout health feature is the ability to analyze data and detect signs of hypertension, or high blood pressure. “It’s not the full on-the-spot blood pressure monitoring Apple fans have long hoped for,” wrote Hand Orellana, “but it’s a major step forward — one that Apple says could help 1 million people get diagnosed with hypertension in the first year alone.”Â
Like the sleep-apnea tracking introduced last year, hypertension notifications are not a screening tool; think of it as a warning system that prompts you to get checked out by your doctor. The Series 11 and Ultra 3 include this ability (sorry, SE 3), and require 30 days of data collection before triggering notifications. The Series 9, Series 10 and Ultra 2 also get hypertension notifications in WatchOS 26.
Series 11 vs. Ultra 3, SE 3 connectivity
Each Apple Watch model supports cellular connectivity, so you can stay connected even when your iPhone is at home. You can order the aluminum Series 11 models and the SE 3 with the cellular option; the titanium Series 11 and the Ultra 3 include it by default.
What’s unique about these watches is their support for both 5G and LTE networks, offering faster speeds and broader compatibility. Plus, they use 5G Reduced Capacity technology, which is more power efficient than the 5G networking in your iPhone.
They also support Wi-Fi 4 (802.11n) at 2.4 GHz and 5 GHz, Bluetooth 5.3, and L1 GPS location chips. The Ultra 3 includes dual GPS radios (L1 and L5) for more precise location tracking, especially in challenging environments like dense downtown corridors.
Plus, the Ultra 3 offers satellite connectivity directly from the watch. With a direct view of the sky, it can communicate with overhead satellites to send and receive texts, share your location, and access emergency services.
Series 11 vs. Ultra 3, SE 3 processors
One of the biggest surprises in the lineup? The Apple Watch Series 11, Ultra 3 and SE 3 all include the same S10 chip. It’s worth noting that the S10, introduced in last year’s Series 10, isn’t a new processor generation for 2025. But each watch now includes 64 gigabytes of storage, a four-core Neural Engine and a 64-bit dual-core processor.
The only significant difference is that the Apple Watch SE 3 is the only model not to feature Apple’s second-generation Ultra Wideband chip, which enables precise location tracking. You can still use Find My on an iPhone (with UWB) to see if the SE 3 is with you or if you left it at home. But with the Series 11 and Ultra 3, Find My will point you in the right direction as you get closer to your mislaid watch.
WatchOS 26 on the Series 11, Ultra 3 and SE 3
Each model is preloaded with WatchOS 26, which has the new Liquid Glass interface (though in most cases, it’s quite subtle). And all models add features like the new Wrist Flick gesture, nightly sleep scores, Workout Buddy, the Notes app and live translation in Messages. The Series 11 and Ultra 3, with their upgraded sensors, also gain hypertension notifications.
Apple Watch Series 11, Ultra 3 and SE 3 specs
| Apple Watch Series 11 | Apple Watch Ultra 3 | Apple Watch SE (3rd Gen) | |
| Design & sizes | Rectangular, 42mm, 46mm | Rectangular, 49mm | Rectangular, 40mm, 44mm |
| Display | 42mm: 446 Ă 374 pixels; LTPO3 OLED Retina display (wide-angle) 46mm: 496 Ă 416 pixels; LTPO3 OLED Retina display (wide-angle) | 49mm: 514 Ă 422 pixels; LTPO3 OLED Retina display (wide-angle, Always-On) | 44mm: 368 Ă 448 pixels (Always-On Retina LTPO OLED)Apple 40mm: 324 Ă 394 pixels (Always-On Retina LTPO OLED) |
| Brightness | Between 1 and 2000 nits | Between 1 and 3000 nits | Up to 1000 nits |
| Thickness & weight | 46mm: 9.7mm; 37.8g (aluminum GPS), 36.9g (aluminum GPS+Cellular), 43.1g (titanium) 42mm: 9.7mm; 30.3g (aluminum GPS), 29.7g (aluminum GPS+Cellular), 34.6g (titanium) | 49mm: 14.4mm; 61.6g (titanium) | 44mm: 10.7mm; 33.0g (aluminum GPS+Cellular) 40mm: 10.7mm; 26.4g (aluminum GPS+Cellular) |
| Material & finish | Aluminum: Jet black, rose gold or silver finish; Titanium: slate, gold or natural finish with sapphire crystal display (titanium) | Titanium, natural or black finish with sapphire crystal display (titanium) | 100% recycled aluminum, midnight and starlight |
| Durability | 2X more scratch resistant glass (aluminum), 5ATM Water + IP6X (dust) | Water resistance 100m; dust IP6X, scuba to 40m, tested to MIL-STD 810H | Cover glass is 4X times more resistant to cracks than the SE 2; made of Ion-X glass. Water resistant up to 50 meters |
| Battery life | Up to 24 hours, up to 38 hours Low Power (always-on) + Fast charge: 80% in 30 min, 100% in 60 min | Up to 42 hours; up to 72 hours Low Power. Fast charge to 80% in 45 min, 100% charge 75 min | All-day, 18-hour battery life. Fast charging with 8 hours of normal use in just 15 minutes on the charger |
| Sensors | ECG, 3rd-gen optical heart sensor, skin temp, depth gauge, SpO2, Noise monitoring, water temperature, compass | ECG, 3rd-gen optical heart sensor, skin temp, depth gauge, SpO2, Noise monitoring, water temperature, compass | Wrist temperature, Second-generation optical heart sensor |
| Emergency features | Satellite SOS, Emergency SOS, Fall detection, Crash detection, Check in and Backtrack | Satellite SOS, Emergency SOS, Fall detection, Crash detection, Check in and Backtrack | Fall Detection, Crash Detection, Emergency SOS, and Check In |
| AI & coaching | Siri (voice assistant); Workout Buddy | Siri (voice assistant); Workout Buddy | On-device Siri, Workout Buddy |
| Processor | S10 SiP with 64-bit dual-core processor, W3 Apple wireless chip | S10 SiP with 64-bit dual-core processor, W3 Apple wireless chip | S10 SiP with 64-bit dual-core processor, W3 Apple wireless chip |
| RAM/Storage | 64GB (storage) | 64GB (storage) | 64GB (storage) |
| Payments | Apple Pay | Apple Pay | Apple Pay |
| Price (US) | $399-$750 (titanium) | $799 | $249 (starting) |
Technologies
Justice Department opens antitrust probe as White House press-access fight escalates
The Justice Department said in a statement that it is examining whether the White House TV press pool violated the Sherman Act.
The U.S. Department of Justice launched an investigation into whether the White House television press poolâs decision to suspend coverage of President Donald Trump violated antitrust laws.
The Justice Department said in a statement that it is examining whether the White House TV press pool â a group of broadcasters including CNN, Fox News, ABC, CBS and NBC â violated the Sherman Act by temporarily halting pooled TV coverage of Trump.
The Sherman Act is a federal law that prohibits certain agreements that unreasonably restrain trade. Introduced in the 1890s, the law has rarely been applied to media organizations, particularly regarding their coverage.
Members of the press pool did not immediately respond to CNBCâs requests for comment sent outside of normal business hours.
The DOJ investigation follows the White House television press pool ceased its coverage of Trump on Sept. 21, shortly after he prohibited CNN, MS NOW and Politico from accessing the White House.
In a Truth Social post, the president said then that those outlets âshouldnât be able to constantly write or report FICTION and LIES when theyâre covering the President of the United States, the Trump Administration, or the United States of America.â
A judge lifted restrictions on those reporters on Sept. 24, several days after White House staff confiscated their press passes. Despite the ruling, reporters from CNN and Politico were barred from traveling with the president on Air Force One, the New York Times reported.
The Trump administration now faces a lawsuit from CNN, MS NOW and Politico over its ban of their reporters from the White House grounds.
Television pool coverage of the White House has also resumed.
The DOJ investigation marks another escalation in an ongoing dispute between Trump and the media over press rights.
The Trump administration has moved to restrict news agencies that have produced critical coverage of its policies.
Last year, Trump moved to rescind about $1.1 billion previously approved for the Corporation for Public Broadcasting, federal funding earmarked for public broadcasters NPR and PBS. Trump and his allies have also sued several media organizations, including The New York Times, The Wall Street Journal, and BBC News, over alleged biases or inaccuracies in their reporting.
Seth Stern, chief of advocacy for the Freedom of the Press Foundation, called the DOJ investigation ânonsense.â
âDepriving Trump of the attention he craves is not a competitive harm and in any case, antitrust law has long recognized First Amendment exceptions even when there is anticompetitive impact,â he said. âAfter all his âfake newsâ rhetoric, Trump is weaponizing the DOJ to pressure the networks he calls the âenemy of the peopleâ to stay at the White House. Itâs a weird way of telling the press how much he missed them.â
The White House Correspondentsâ Association did not immediately respond to a request for comment.
Disclosure: CNBC and MS NOW are divisions of Versant Media.
Technologies
NBA commissioner Adam Silver says league could introduce âsmart ballâ technology as soon as next year
Commissioner Adam Silver says the NBA could begin using a new “smart ball” in games as soon as next year, potentially transforming how officials make calls.
NBA Commissioner Adam Silver says the league could begin using a new âsmart ballâ in games as soon as 2027, potentially transforming how officials make calls on the basketball court.
In an interview with CNBCâs Contessa Brewer, Silver revealed that the NBA is working with official basketball manufacturer Wilson to develop a ball embedded with a tiny microchip Bluetooth sensor that can track movement, spin and changes in trajectory.
âWeâre experimenting with putting a small chip in the ball that weighs roughly a gram,â Silver said.
The technology has already been tested in the NBAâs G League, Summer League, and some preseason games, where players used basketballs both with and without the chip. Silver said players have been pleased with the results.
âNobody could tell the difference. So thatâs a good sign,â he said.
The chip weighs just one gram, compared with the roughly 620-gram or 1.4 pound basketball. Silver said the league wanted to ensure that even the most experienced players wouldnât notice a change in how the ball feels or bounces.
One of the most immediate applications could be officiating.
Silver said the technology could help referees determine whether a player touched the ball before it went out of bounds by detecting subtle changes in its spin. It could also help identify whether a shotâs trajectory was altered.
âI think you could see as soon as next year us using it for officiating in our games,â Silver said.
Beyond officiating, Silver sees a significant opportunity to bring the technology to consumers, allowing basketball players of all ages to analyze and evaluate their shooting mechanics.
For example, a player taking hundreds of shots could use data collected by the chip to understand which shooting angles and ball rotations are most likely to result in a basket.
âYouâll then see the graph, and youâll see for which the angle of the shots that went in, theyâre more likely to go in,â Silver said.
While the officiating application could arrive as soon as next year, Silver said a consumer version may take longer.
âI think the consumers version [of the smart ball] is a few years away, but itâs a really exciting opportunity.â
NBA playersâ union raises concerns over wearables
The league is also exploring the use of wearable technology during games, but negotiations with the National Basketball Players Association have yet to produce an agreement.
The NBA says officials experimented with wrist wearables in select preseason and summer league games this year in a âsuccessful pilot program,â but it will not extend into the season. The technology allowed the referees to communicate with the replay center about reviews, scoring changes and clock malfunctions.
Silver said players routinely use wearable devices off the court to monitor everything from sleep to physical performance, but concerns remain over how data collected during games could be used by teams.
âI think we have to come to some agreement on exactly how the information is used. But it seems everybody wants that information,â Silver said.
The biggest sticking point is whether that information could affect contract negotiations, Silver said.
âIf you could see a player was slowing down or something like that, theyâre worried that that could get used in bargaining, and I get that,â he said.
Silver acknowledged those concerns and said the league needs to reach an agreement with the playersâ union on how the information would be used.
Still, he suggested that allowing wearables during games is a logical next step as athletes increasingly rely on technology to monitor their performance.
âI think the players are in a position right now where theyâre essentially wearing wearables 22 hours a day, and the only time theyâre not wearing them is when theyâre playing in the game,â Silver said. âSo that canât make sense.â
âWeâll work something out with them,â he added.
Technologies
AIâs quiet safety gatekeepers are stepping into the spotlight
The intensifying AI safety debate is bringing a small group of third-party evaluators into the center of a multitrillion-dollar industry.
Two months ago, independent evaluators occupied a relatively sleepy corner of the multitrillion-dollar artificial intelligence industry. Now theyâre being asked to come to its rescue.
While Anthropic and OpenAI are the heart of a fierce debate over whether they can safeguard their advanced models and grow their businesses simultaneously, the companies are seeking support from a handful of small third-party groups like Model Evaluation and Threat Research (METR), Apollo Research and Transluce.
The evaluators, which mostly operate as nonprofits, are still finding their footing in an industry where capital is flowing at historic levels and new models are rolling out faster than ever. Their primary role has been to assess AI model capabilities and risks, and to call attention to instances where the technology behaves badly.
In the absence of a federal push for regulations, evaluators have taken on outsized importance. Anthropic CEO Dario Amodei pledged to embed independent evaluators in his company last month â a move that OpenAI CEO Sam Altman quickly endorsed.
President Donald Trump supported the idea, as did most of the largest U.S. tech companies. But left unanswered are questions about how those third parties should be funded, what level of access they will have and what the reporting structure will ultimately look like.
âTo a degree, the problem, as always, is money,â Suresh Venkatasubramanian, a computer science professor at Brown University, told CNBC in an interview. âWho is paying for these companies to do their work? How are they going to support them? You need an ecosystem, you need a viable business model for this.â
Right now, Anthropic, OpenAI and the infrastructure partners that are profiting from the AI boom are writing the rules. Critics say thatâs like asking the biggest banks to protect us from a financial crisis or allowing pharmaceutical companies to put drugs on the market without regulatory clearance.
President Trump recently lauded AI executives for their âtremendous self-policing,â and signaled that he intends to leave companies to their own devices, unwilling to impede the growth of the industry thatâs driving the economy and stock market. Trump encouraged AI companies to âpartner with an independent external auditor or evaluatorâ as part of a voluntary accord he presented in late September.
Itâs a conversation that Amodei kicked off In his viral essay last month, when he called for a âslower paceâ in advanced model development after researchers left his company and voiced their concerns about the existential threats the technology poses.
As the AI labs move to put evaluators in place, friction is already starting to emerge.
OpenAI fired three employees last week for âviolating our policies on accessing and handling sensitive company information,â according to a spokesperson. Two of those employees, Mikita Balesni and Tomek Korbak, said they believe they were dismissed because of how they communicated with third-party evaluators.
âMy former colleagues are telling me they are confused about what to believe,â Balesni wrote in a post on X on Thursday. âThey also are afraid to speak, and worry their personal phones will be searched for messages to us and third parties. I worry the pervading fear to speak up and engage with third parties will mean OpenAI will cut corners on safety behind closed doors.â
OpenAI disputed that characterization and said in a post on Friday that itâs âactively finalizing contracts with third-party safety assessors and will announce details in the coming weeks.â
âWe are committed to embedding external assessors and continue to make close collaboration with independent safety organizations a core part of our safety work,â OpenAI wrote.
An OpenAI spokesperson said in an emailed statement that its upcoming work with evaluators âbuilds on existing collaboration with independent safety organizations,â including METR and Redwood Research.
Anthropic didnât respond to CNBCâs request for comment.
âIâve never seen an issue move so fastâ
The AI evaluator ecosystem consists mostly of small organizations, including METR and Apollo Research, and larger accounting and auditing firms like Accenture.
AI labs have been working with evaluators in limited capacities, but Andrew Freedman, CEO of policy nonprofit Fathom, said the field is quickly maturing.
âIâve worked in politics and policy for the last 20 years of my life, and Iâve never seen an issue move so fast on so many different political spectrums,â Freedman told CNBC in an interview. He said he expects an âinflux of capitalâ to flow into the ecosystem.
Rayan Krishnan, CEO of independent evaluator Vals AI, said his for-profit startup, which builds benchmarks to measure how AI models perform on industry-specific tasks, has grown from eight employees to roughly 30 this year, and in August announced a $40 million funding round.
METR, a nonprofit, announced in August that it had raised commitments of around $71 million over the last six months. Thatâs up from total 2024 contributions of $13.6 million, according to the groupâs most recent filing with the Internal Revenue Service.
By late that month, METRâs profile had risen further. OpenAI enlisted two of its employees and a contractor to put together a postmortem report detailing how the companyâs models escaped containment, accessed the open internet and breached open-source developer platform Hugging Face. METR said it did not accept payment from OpenAI for the assessment.
Kevin Werbach, faculty director of the Wharton Accountable AI Lab at the University of Pennsylvania, said the ecosystem is ânot robust enough right now.â METR, for example, employs fewer than 50 full-time staffers, according to its website.
The power imbalance between the small evaluators and the leading labs that have raised tens of billions of dollars and employ thousands of people raises questions surrounding potential conflicts.
âIf you want true third-party evaluation, you need true independence financially and otherwise,â said Venkatasubramanian. âItâs not just a matter of not getting paid, itâs a matter of, will there be consequences if I am an auditor and I put out a report that looks unfavorable to this company? Is my business going to dry up?â
Anthropic acknowledged the complexity in a blog post last month, as it announced it will embed employees from Faculty, Accentureâs specialist AI business, to test safeguards and assess whether models will behave in line with human values. Anthropic said that âgiven the importance and urgency of this work,â it will fund Accentureâs contributions directly.
âThere are, as yet, no standards for what information embedded evaluators should have access to, or how they should report what they find. There is also no settled system for funding independent evaluation,â Anthropic said. âLong-term, we think funding should come from pooled or government sources.â
Anthropic said itâs in discussions with METR and other nonprofit evaluators that are planning to use their own funding to pilot âelementsâ of embedded evaluation.
Will the government step in?
In June of last year, Fathom introduced a marketplace framework for Independent Verification Organizations, or IVOs. These groups would be licensed by the government and authorized to test whether AI companies are meeting various safety criteria.
Freedman, the groupâs CEO, said government oversight is key because otherwise third-party evaluators can become beholden to the large AI labs for revenue, incentivizing them to âstart rubber stamping stuffâ to maintain favor.
Some lawmakers are on board.
IVOs are a key provision of the âłFrontier Risk Oversight, National Transparency, Independent Evaluation, and Reportingâ (FRONTIER) Act, which Reps. Lori Trahan, D-Mass., and Jay Obernolte, R-Calif., introduced in July. Fathom helped draft language and provided technical expertise for the bill, Freedman said.
OpenAI global affairs chief Chris Lehane told reporters in September that he sat down with one of the billâs sponsors on Capitol Hill to express support for the IVO provision.
âIt was important for them to hear that and hear it from us, and we wanted to be really clear about that,â Lehane said, according to reports.
Meanwhile, lawmakers in California, Connecticut and Virginia have taken steps to implement IVOs, and states like Massachusetts are weighing independent safety evaluations more broadly.
California Governor Gavin Newsom recently signed two bills involving IVOs, one establishing a âfirst-in-the-nation framework,â and the other creating a state registry for AI auditors. Anthropic threw its support behind both bills in August, and OpenAI formally endorsed them last month, the same day Newsom signed them into law.
Lehane wrote in a blog post at the time that âwe prefer independent technical assessments to be required at the federal level,â but in the absence of federal action, âCalifornia can help establish the rules of the road.â
Freedman said he thinks it will be âreally difficultâ for companies like OpenAI and Anthropic to work out how to engage with independent evaluators on their own. However, with the governmentâs role unclear, âitâs a muscle worth developing in the interim,â he said.
For now, the closest thing the industry has to a set of standards is what Trump called a âmorally bindingâ agreement at a luncheon he hosted for tech leaders at the White House late last month.
The one-page accord says that âevery company is responsible for developing its own technology safely and in a way that builds trust with customers and the public.â It also encourages signees to work with an âindependent external auditor or evaluator to carry out independent assessments.â
The document was signed by top execs at Anthropic, Google, Meta, OpenAI, SpaceX and Nvidia, a rare show of solidarity between leaders who have shared conflicting views on addressing AIâs risks. The executives still have to chart their own paths forward.
âIt was a performance of an attempt to show action when in fact no action actually happened,â Venkatasubramanian said. âThe things that they promise to do are things they should have been doing already, and, in fact, have claimed that they were doing in the past.â
Amodei, in his September essay, said Anthropic will equip evaluators with desks, access badges, company laptops, and permissions that are âmostly comparableâ with internal risk assessment teams. Additionally, evaluators will be supported with contracts that give them âthe right to publish key findings,â with Anthropic reserving âthe narrow abilityâ to redact certain security-sensitive or confidential information.
âThis is an unusual step for a company, but we think it is important to prove out the concept of embedded external reviewers,â Amodei wrote.
OpenAI published its own proposal days later, and said evaluators should work on âscoped and mutually agreed upon claims for assessment,â clearly explain their methodology and standards, demonstrate relevant technical expertise and disclose conflicts of interest.
The AI Evaluator Forum, which includes METR, the AI Verification and Evaluation Research Institute (AVERI), and other groups, published a public letter last month titled, âMinimum Conditions for Embedding Evaluators.â
The letter said evaluators should be transparent, shielded from retaliation and granted access equivalent to AI companiesâ âown highly privileged employees.â
âEmbedded evaluations cannot address all oversight needs and should be treated as a complement to, rather than a replacement for, broader efforts by frontier AI companies to expand external oversight,â the letter said.
Freedman said heâs seen a shift in posturing out of OpenAI and Anthropic in recent months, largely because theyâve realized they wonât be able to roll out their advanced systems without the publicâs trust.
âI donât think you need to trust that theyâve suddenly turned altruistic or that thereâs anything but corporations acting like corporations,â Freedman said.
That underscores perhaps the central problem, Werbach said. OpenAI and Anthropic are, first and foremost, competing with each other as they march toward the public markets and seek trillion-dollar-plus valuations.
âThere is a tremendous amount of personal distrust between those two companies,â Werbach said. âEven though thereâs also tremendous agreement about the need for this kind of evaluation to happen.â
WATCH: Bradley Tusk on Anthropic IPO: Why add public market pressure if safety is your top priority?
-
Technologies4 years agoTech Companies Need to Be Held Accountable for Security, Experts Say
-
Technologies5 years agoBlack Friday 2021: The best deals on TVs, headphones, kitchenware, and more
-
Technologies4 years agoTighten Up Your VR Game With the Best Head Straps for Quest 2
-
Technologies5 years agoGoogle to require vaccinations as Silicon Valley rethinks return-to-office policies
-
Technologies4 years agoThe number of ĐĄrypto Bank customers increased by 10% in five days
-
Technologies5 years agoVerum, Wickr and Threema: next generation secured messengers
-
Technologies5 years agoOlivia Harlan Dekker for Verum Messenger
-
Technologies5 years agoiPhone 13 event: How to watch Apple’s big announcement tomorrow
