Technologies
Samsung Galaxy S23 Ultra Review: The King of Extra
The Galaxy S23 Ultra builds on Samsung’s camera strengths. But Galaxy S22 Ultra owners don’t need to upgrade yet.
This story is part of Samsung Event, CNET’s collection of news, tips and advice around Samsung’s most popular products.
Samsung clearly wants the Galaxy S23 Ultra to be the ultimate camera for cinematographers and photographers. I’m certainly not a professional filmmaker, nor do I consider myself to be a shutterbug. But after using the Galaxy S23 Ultra for several days, it’s clear the new camera upholds the areas in which Samsung already excels — like low-light photography — and pushes those strengths a little further.
The $1,200 ( 1,249, AU$1,949) Galaxy S23 Ultra will be available on Feb. 17 alongside the rest of Samsung’s new phone lineup, which includes the $800 Galaxy S23 and $1,000 Galaxy S23 Plus. Like last year’s Galaxy S22 Ultra, the S23 Ultra’s giant screen, high price and souped-up camera may be too much for the average person.
The name “Ultra” says it all; this phone is for people who want a phone that’s over the top. Compared to the Galaxy S23 and S23 Plus, the Ultra offers a closer zoom magnification than most people will probably need, a stylus that stores inside the phone and a massive 6.8-inch screen that’s larger than both those of the iPhone 14 Pro Max and Google Pixel 7 Pro. You might not need all that excess, but after using this phone for a few days, I wouldn’t blame you for wanting them.
Like
- Fast performance
- Excellent main camera, especially in low light
- Bright screen
- Included stylus
- Double the storage in the base model
- Four years of Android OS updates
Don’t Like
- High price
- Photos don’t always look natural, especially selfies
- No improvements to fast charging
Samsung largely achieves its goal of providing a deluxe phone with an excellent camera and giant display. But it isn’t perfect. That high price is hard to ignore, and photos taken on the Galaxy S23 Ultra don’t always look as natural as those captured on the iPhone 14 Pro or Pixel 7 Pro.
Samsung Galaxy S23 Ultra camera is a step up
If you watched Samsung’s Unpacked presentation, you already know the S23 Ultra’s main camera is its prized attribute. Samsung added a 200-megapixel sensor for the first time, marking a technical upgrade from the S22 Ultra’s 108-megapixel main shooter.
With the S23 Ultra, you have the option to shoot photos at a 12-megapixel, 50-megapixel or 200-megapixel resolution, which you can adjust in the settings menu alongside the camera’s viewfinder. Samsung’s Expert Raw app is now integrated into the native camera app after you download it for the first time. Although I don’t typically edit raw photos myself, the use case certainly seems appropriate for a phone like this. Shooting in raw allows photographers to edit the full, uncompressed version of an image, giving them more flexibility and data to work with in terms of colors and exposure.
Under-the-hood improvements to color, dynamic range, autofocus and low-light performance seem more meaningful than the resolution bump. Samsung said when announcing the S23 Ultra that its new image sensor is better at minimizing noise, and that its algorithms have gotten better at enhancing color tones and details. Samsung largely positioned these improvements in the context of video recording and low-light photography, but they seem evident across general still photography, too. These changes are more noticeable to the naked eye than the increased resolution.
Unless otherwise specified, the photos in this article were taken at the default 12-megapixel resolution because the 200-megapixel files are just too large to display here. You can still see changes in how the camera processes color, skin tone and dynamic range even at this lower resolution. However, I did capture several 200-megapixel photos throughout the course of my testing.
The biggest differences I noticed in those 200-megapixel photos compared to the 108-megapixel images taken on the Galaxy S22 Ultra had to do with color. Flowers were more vibrant shades of orange and trees looked more lush. The difference in resolution is only noticeable when you zoom all the way in to view the actual size of the pixels.
The photos below, however, show how the Galaxy S23 Ultra’s camera has changed compared to the S22 Ultra’s. Take a look at the photos of trees and shrubbery taken at a park. Although you might not notice it in the smaller embedded version below, the leaves in the S23 Ultra’s photo have a bit more separation and crispness compared to the S22 Ultra’s, and there’s also better contrast in the tree bark.
In the photos of sweets arranged in a baker’s display case below, the S23 Ultra captured more detail on the strawberries in the top row.
Those who have taken photos with Samsung phones before might have noticed that colors sometimes look exaggerated. Samsung has improved this in recent years, but the Pixel 7 Pro and iPhone 14 Pro still take photos that look more natural overall.
But, natural doesn’t always mean better. There are times when Samsung’s overstated color works well, and others when it doesn’t. For photos of people and pets, for example, Samsung’s emphasis on color can result in more flattering images. In the pictures below, my husband preferred the Galaxy S23 Ultra’s photo over the more natural-looking image from the Pixel 7 Pro because it made his beard and eyes look more vibrant. But he liked the iPhone 14 Pro’s photo best because it struck the right balance of accuracy and bold color. The iPhone 14 Pro also captured a lot more detail in the cookie-shaped bench he’s sitting on.
However, Samsung’s phones don’t always handle challenging lighting scenarios very well. When snapping a photo of my husband in front of a window at a bakery, both the S23 Ultra and S22 Ultra’s images looked washed out and overexposed, although the S23 Ultra’s did have more color than the S22 Ultra’s. The iPhone 14 Pro’s was the best of the bunch in this case, but the Pixel 7 Pro’s was impressive, too.
Colors in landscape photos taken on the Galaxy S23 Ultra can look brighter and exaggerated compared to those from the iPhone 14 Pro and Pixel 7 Pro. Take a look at how that photo of the park from earlier compares to similar photos taken on the iPhone 14 Pro and Pixel 7 Pro, and you’ll notice the shades of green look more natural. The right side of the iPhone 14 Pro’s photo looks blown out, but otherwise would have been a better shot since the colors are more accurate. Still, Samsung did a great job with dynamic range; you can see the shadows more clearly in its photo compared to the others.
But Samsung is still the king of low-light photography in my book. In my testing, the Galaxy S23 Ultra captured better detail and color. The Galaxy S23 Ultra was able to focus more sharply than both the Pixel 7 Pro and iPhone 14 Pro. It was also slightly better than the Galaxy S22 Ultra, although it was sometimes a close call.
My favorite example of this is in this photo of my cat, Buddy, shown below. Even though the iPhone 14 Pro’s photo is brighter, I like the S23 Ultra’s better because it has more contrast and detail. When looking at these photos side-by-side on a large monitor, you can see the stripes in his fur more clearly. The Galaxy S22 Ultra’s photo is too soft, while the Pixel 7 Pro’s looked like it had a bluish filter over it when I viewed it on a large screen. It’s also another example of how the S23 Ultra’s emphasis on color makes for a more flattering photo.
Video recording on the Galaxy S23 Ultra was upgraded to 8K at 30 frames per second (fps), whereas it previously topped out at 24fps when shooting at that resolution. Samsung also increased the field of view when recording 8K video, which is very noticeable compared to the Galaxy S22 Ultra.
Even though Google provides more natural color in still photography, I experienced the opposite when it comes to recording video. The Galaxy S23 Ultra generally captured more realistic color and sharpness compared to the Pixel 7 Pro during my testing. The iPhone 14 Pro was a close competitor, but Samsung’s footage had a little more detail. When testing the video capabilities of each phone, I recorded clips at the highest supported resolution and frame rate. That’s 8K at 30fps for the Galaxy S23 Ultra, 8K at 24fps for the Galaxy S22 Ultra and 4K at 60fps for the iPhone 14 Pro and Google Pixel 7 Pro.
Samsung also highlighted the S23 Ultra’s new 12-megapixel selfie camera when announcing the phone, which it said separates the subject from the background better for more clarity. Selfies were sharp and colorful overall, and I preferred photos taken with the S23 Ultra’s selfie camera over the Pixel 7 Pro’s in every test. The Galaxy S22 Ultra’s were comparable to the S23 Ultra’s, but the iPhone 14 Pro’s selfies had the best overall combination of detail and accurate color. Samsung’s selfies sometimes looked unnatural since the color in my hair seemed exaggerated.
The only exception was in very dim lighting, an area where Samsung reigns supreme. Take a look at the selfie I took in a very dark theater with red lighting. The Galaxy S23 Ultra’s photo has much more detail than the others.
When it comes to the rest of the cameras, you can expect the same arrangement as the Galaxy S22 Ultra. In addition to the main camera, there’s also a 12-megapixel ultrawide camera and two 10-megapixel telephoto lenses that allow for a 3x or 10x optical zoom and up to 100x digital zoom. Photos from the ultrawide camera packed plenty of color and brightness, as shown in the images below.
You’ll also get more zoom than you probably need on the S23 Ultra, but that might be useful for people who frequently take photos at concerts and sporting events. The 100x digital zoom can feel unwieldy, but Samsung does provide a higher optical zoom than its competitors. Like the S22 Ultra, the S23 Ultra can zoom optically up to 10x, while the iPhone 14 Pro is at 3x and the Pixel 7 Pro is at 5x.
Check out the photo samples below to see the difference in optical zoom levels.
Overall, I was impressed with the Galaxy S23 Ultra’s cameras. They remain the best for zoomed-in shots and low-light photography. The improvements to color and dynamic range are noticeable as well, and Samsung fans upgrading from a phone that’s at least two years old will likely see a major difference. Still, the iPhone 14 Pro’s camera was better in most situations, save for a few exceptions.
Check out the gallery below for a closer look at some photos I captured with the Galaxy S23 Ultra.
Samsung Galaxy S23 Ultra performance
The Galaxy S23 Ultra runs on a Snapdragon 8 Gen 2 processor, which isn’t surprising considering Samsung typically uses the latest Qualcomm chip for its new phones. What is surprising, however, is that Samsung decided to customize this chip more than usual for the S23 series. Samsung calls it the Qualcomm Snapdragon 8 Gen 2 for Galaxy, and it’s essentially a special version of the processor that’s been optimized for better performance and power efficiency. A standard version of the Snapdragon 8 Gen 2 is being included in rival phones like the OnePlus 11 5G.
The S23 Ultra feels fast and snappy, whether I’m just swiping around the operating system, playing games while on a video call or running Samsung’s Dex program to use my phone as a PC. I plugged my S23 Ultra into an external monitor, attached a keyboard and mouse via Bluetooth and had five apps running, including Google Docs, to take notes for this review. I never experienced any lag or overheating during any of these tasks.
The S23 Ultra can also export videos quickly, which might be helpful for those who shoot and edit video projects on their phone frequently. In my testing, it exported a 30-second 4K video clip to 1080p three seconds faster than the S22 Ultra and a full 10 seconds faster than the Pixel 7 Pro on average. The iPhone 14 Pro, however, finished the job about three seconds faster than the S23 Ultra on average. You’ll also get double the storage that’s in the base model of the Galaxy S23 Ultra (256GB versus 128GB), which gives you more space to store those video projects.
Samsung’s new phone also scored higher than the Galaxy S22 Ultra, Pixel 7 Pro and OnePlus 11 5G on benchmarks meant to test computing power in everyday tasks and in apps that require short bursts of high performance, like mobile games. The iPhone 14 Pro, however, scored higher than the S23 Ultra on the general computing benchmark (Geekbench 5) but lower on the one that measures brief periods of high intensity (3DMark Wild Life Extreme).
Samsung Galaxy S23 Ultra battery life and charging
The Galaxy S23 Ultra has a 5,000-mAh battery, just like the Galaxy S22 Ultra, but it should offer better power efficiency, thanks to its new processor. I need to spend more time with the phone to compare it to the S22 Ultra, but the results are promising so far. After 12 hours of use, which equates to a full workday and then some, the battery was down to 66%.
I had adaptive motion and adaptive brightness settings turned on, which adjust the screen’s refresh rate and brightness based on the situation. The always-on display feature was also turned on, but I had “tap to show” enabled, meaning it wouldn’t display the time and date unless I touched the screen. Battery life will always vary depending on how you use your device. In this scenario, I used the phone to take photos, record a few videos, check email, browse social media and stream music for about 15 minutes.
On a separate day, the Galaxy S23 Ultra still had roughly three quarters of its battery left by the middle of the workday even after shooting 4K video for roughly a half hour. We’re still conducting more battery tests on the S23 Ultra and will update this review accordingly.
I also ran a 45-minute test in which I played games, made a 10-minute video call, checked social media and streamed video; over that span the S23 Ultra dropped from 100% to 94%. The Galaxy S22 Ultra dipped to 91% during that same test. The Galaxy S23 Ultra’s results are similar to the Pixel 7’s, which also hit 94% after that 45-minute challenge. (During this test, I kept the brightness at 50%, left the high refresh rate setting turned on and switched off the always-on display.)
Like last year’s Galaxy S22 Ultra, the Galaxy S23 Ultra has a top wired charging speed of 45 watts. You need to use the correct adapter, which you’ll have to purchase separately if you don’t already own one since Samsung doesn’t bundle one in the box. So far, that 45-watt charging speed has enabled my S23 Ultra to go from 46% to 91% in just 30 minutes of charging.
That’s not bad, but it would be great to see Samsung push things further now that companies like OnePlus and Xiaomi are offering upwards of 100-watt fast charging. The Xiaomi 12T Pro, for example, supports 120-watt fast charging and went from zero to 100% in 19 minutes, according to my colleague Sareena Dayaram’s Xiaomi 12T Pro review.
Samsung Galaxy S23 Ultra design, display and other features
The Galaxy S23 Ultra looks almost identical to the Galaxy S22 Ultra. It has the same boxy edges that give it a sharper look compared to the regular Galaxy S23 and S23 Plus. But this year, the S23 Ultra is available in cream, green, black and lavender color choices. Like last year’s device, it’s rated for IP68 water resistance, meaning it can be submerged in up to 1.5 meters of fresh water for 30 minutes.
There’s no mistake about it; the Galaxy S23 Ultra is a gigantic phone, just like its predecessor. It has a 6.8-inch screen, making it slightly bigger than the 6.7-inch iPhone 14 Pro Max and Google Pixel 7 Pro. It’s the type of phone your friends might gawk at the minute you take it out of your pocket. That’s not an insult; there are plenty of people out there who love large phones. It’s just another reminder that this phone isn’t for everyone. But if you do a lot of reading, gaming, photo editing and TV watching on your phone and don’t mind stretching your fingers for one-handed use, you’ll likely appreciate the extra screen space.
I’m also certain you’ll never struggle to see the Galaxy S23 Ultra’s screen in direct sunlight. It’s super bright, reaching 1,750 nits, which is brighter than the Pixel 7 Pro’s 1,500-nit peak brightness but not quite as luminous as the iPhone 14 Pro’s 2,000-nit peak outdoor brightness. Still, it’s bright enough to fit most people’s needs and circumstances.
You’ll also get an S Pen stylus, just like last year’s Galaxy S22 Ultra. Samsung’s Ultra lineup has effectively replaced the Galaxy Note series, which was previously known for its large screens and embedded stylus. Expect the same familiar S Pen experience on this device; as soon as you pop the pen out from the bottom of the phone, you’ll see a pop-up menu with compatible apps like Samsung Notes and the drawing app PenUp. I don’t usually use the S Pen frequently, but I found myself jotting down notes during the course of this review more than I had expected.
Samsung also once again commits to four generations of Android OS upgrades, which is on par with the OnePlus 11 and surpasses Google’s three-year promise for the Pixel 7 and 7 Pro. Samsung, OnePlus and Google all provide five years of security updates for their latest devices. That means you won’t have to worry about the S23 Ultra’s software feeling outdated anytime soon since it launches with Android 13 and Samsung’s One UI 5.1 software.
One extra feature you get by going for the Ultra or Plus instead of the regular model is ultrawideband support, which makes it easier to connect to nearby devices more precisely than Bluetooth. UWB isn’t a necessity, but it can make sharing files or using your phone as a digital car key faster and easier. We could see more services and accessories that require it in the future.
Samsung Galaxy S23 Ultra: Buy or skip?
The Galaxy S23 Ultra is a lot, but in a good way. Its large screen, 200-megapixel camera, 100x zoom and S Pen might be more than most people need in a phone. But that’s exactly who Samsung is targeting, and it’s ultimately iterating on a formula that’s worked in the past.
This is a phone for people actively seeking features like large screens and versatile cameras that are willing to pay top dollar for them. If you just want a reliable new Android phone with a great camera, the cheaper Pixel 7 Pro, Galaxy S23 or Galaxy S23 Plus will likely do the trick. (However, we haven’t tested those other new Galaxy phones, so we can’t recommend them just yet).
The barrier to entry for the Galaxy S23 Ultra is higher than that of the iPhone 14 Pro and Pixel 7 Pro. But you also get certain extras like the S Pen, more storage at the base level, a slightly larger screen and a better zoom lens. Those aren’t must-haves for everyone, but this is a phone for people who want extras like these.
The Galaxy S23 Ultra’s new camera is the most significant change compared to last year’s S22 Ultra. But in my experience, the behind-the-scenes upgrades Samsung made to the way the camera captures color, skin tones and dynamic range are a more welcome improvement than increased sharpness alone.
Samsung could do more to push the Galaxy Ultra line forward in ways beyond the camera. I would have loved to see faster charging, for example. And even though the camera is impressive, the iPhone 14 Pro often outpaced it in my testing.
If you want a giant screen, a great camera and are upgrading from a phone that’s more than two years old, you won’t be disappointed with the S23 Ultra. Just try to snag it at a discount through a trade-in deal to make that $1,200 price easier to swallow.
How we test phones
Every phone tested by CNET’s reviews team was actually used in the real world. We test a phone’s features, play games and take photos. We examine the display to see if it’s bright, sharp and vibrant. We analyze the design and build to see how it is to hold and whether it has an IP-rating for water resistance. We push the processor’s performance to the extremes using both standardized benchmark tools like GeekBench and 3DMark, along with our own anecdotal observations navigating the interface, recording high-resolution videos and playing graphically intense games at high refresh rates.
All the cameras are tested in a variety of conditions from bright sunlight to dark indoor scenes. We try out special features like night mode and portrait mode and compare our findings against similarly priced competing phones. We also check out the battery life by using it daily as well as running a series of battery drain tests.
We take into account additional features like support for 5G, satellite connectivity, fingerprint and face sensors, stylus support, fast charging speeds, foldable displays among others that can be useful. And we of course balance all of this against the price to give you the verdict on whether that phone, whatever price it is, actually represents good value.
Technologies
NBA commissioner Adam Silver says league could introduce âsmart ballâ technology as soon as next year
Commissioner Adam Silver says the NBA could begin using a new “smart ball” in games as soon as next year, potentially transforming how officials make calls.
NBA Commissioner Adam Silver says the league could begin using a new âsmart ballâ in games as soon as 2027, potentially transforming how officials make calls on the basketball court.
In an interview with CNBCâs Contessa Brewer, Silver revealed that the NBA is working with official basketball manufacturer Wilson to develop a ball embedded with a tiny microchip Bluetooth sensor that can track movement, spin and changes in trajectory.
âWeâre experimenting with putting a small chip in the ball that weighs roughly a gram,â Silver said.
The technology has already been tested in the NBAâs G League, Summer League, and some preseason games, where players used basketballs both with and without the chip. Silver said players have been pleased with the results.
âNobody could tell the difference. So thatâs a good sign,â he said.
The chip weighs just one gram, compared with the roughly 620-gram or 1.4 pound basketball. Silver said the league wanted to ensure that even the most experienced players wouldnât notice a change in how the ball feels or bounces.
One of the most immediate applications could be officiating.
Silver said the technology could help referees determine whether a player touched the ball before it went out of bounds by detecting subtle changes in its spin. It could also help identify whether a shotâs trajectory was altered.
âI think you could see as soon as next year us using it for officiating in our games,â Silver said.
Beyond officiating, Silver sees a significant opportunity to bring the technology to consumers, allowing basketball players of all ages to analyze and evaluate their shooting mechanics.
For example, a player taking hundreds of shots could use data collected by the chip to understand which shooting angles and ball rotations are most likely to result in a basket.
âYouâll then see the graph, and youâll see for which the angle of the shots that went in, theyâre more likely to go in,â Silver said.
While the officiating application could arrive as soon as next year, Silver said a consumer version may take longer.
âI think the consumers version [of the smart ball] is a few years away, but itâs a really exciting opportunity.â
NBA playersâ union raises concerns over wearables
The league is also exploring the use of wearable technology during games, but negotiations with the National Basketball Players Association have yet to produce an agreement.
The NBA says officials experimented with wrist wearables in select preseason and summer league games this year in a âsuccessful pilot program,â but it will not extend into the season. The technology allowed the referees to communicate with the replay center about reviews, scoring changes and clock malfunctions.
Silver said players routinely use wearable devices off the court to monitor everything from sleep to physical performance, but concerns remain over how data collected during games could be used by teams.
âI think we have to come to some agreement on exactly how the information is used. But it seems everybody wants that information,â Silver said.
The biggest sticking point is whether that information could affect contract negotiations, Silver said.
âIf you could see a player was slowing down or something like that, theyâre worried that that could get used in bargaining, and I get that,â he said.
Silver acknowledged those concerns and said the league needs to reach an agreement with the playersâ union on how the information would be used.
Still, he suggested that allowing wearables during games is a logical next step as athletes increasingly rely on technology to monitor their performance.
âI think the players are in a position right now where theyâre essentially wearing wearables 22 hours a day, and the only time theyâre not wearing them is when theyâre playing in the game,â Silver said. âSo that canât make sense.â
âWeâll work something out with them,â he added.
Technologies
AIâs quiet safety gatekeepers are stepping into the spotlight
The intensifying AI safety debate is bringing a small group of third-party evaluators into the center of a multitrillion-dollar industry.
Two months ago, independent evaluators occupied a relatively sleepy corner of the multitrillion-dollar artificial intelligence industry. Now theyâre being asked to come to its rescue.
While Anthropic and OpenAI are the heart of a fierce debate over whether they can safeguard their advanced models and grow their businesses simultaneously, the companies are seeking support from a handful of small third-party groups like Model Evaluation and Threat Research (METR), Apollo Research and Transluce.
The evaluators, which mostly operate as nonprofits, are still finding their footing in an industry where capital is flowing at historic levels and new models are rolling out faster than ever. Their primary role has been to assess AI model capabilities and risks, and to call attention to instances where the technology behaves badly.
In the absence of a federal push for regulations, evaluators have taken on outsized importance. Anthropic CEO Dario Amodei pledged to embed independent evaluators in his company last month â a move that OpenAI CEO Sam Altman quickly endorsed.
President Donald Trump supported the idea, as did most of the largest U.S. tech companies. But left unanswered are questions about how those third parties should be funded, what level of access they will have and what the reporting structure will ultimately look like.
âTo a degree, the problem, as always, is money,â Suresh Venkatasubramanian, a computer science professor at Brown University, told CNBC in an interview. âWho is paying for these companies to do their work? How are they going to support them? You need an ecosystem, you need a viable business model for this.â
Right now, Anthropic, OpenAI and the infrastructure partners that are profiting from the AI boom are writing the rules. Critics say thatâs like asking the biggest banks to protect us from a financial crisis or allowing pharmaceutical companies to put drugs on the market without regulatory clearance.
President Trump recently lauded AI executives for their âtremendous self-policing,â and signaled that he intends to leave companies to their own devices, unwilling to impede the growth of the industry thatâs driving the economy and stock market. Trump encouraged AI companies to âpartner with an independent external auditor or evaluatorâ as part of a voluntary accord he presented in late September.
Itâs a conversation that Amodei kicked off In his viral essay last month, when he called for a âslower paceâ in advanced model development after researchers left his company and voiced their concerns about the existential threats the technology poses.
As the AI labs move to put evaluators in place, friction is already starting to emerge.
OpenAI fired three employees last week for âviolating our policies on accessing and handling sensitive company information,â according to a spokesperson. Two of those employees, Mikita Balesni and Tomek Korbak, said they believe they were dismissed because of how they communicated with third-party evaluators.
âMy former colleagues are telling me they are confused about what to believe,â Balesni wrote in a post on X on Thursday. âThey also are afraid to speak, and worry their personal phones will be searched for messages to us and third parties. I worry the pervading fear to speak up and engage with third parties will mean OpenAI will cut corners on safety behind closed doors.â
OpenAI disputed that characterization and said in a post on Friday that itâs âactively finalizing contracts with third-party safety assessors and will announce details in the coming weeks.â
âWe are committed to embedding external assessors and continue to make close collaboration with independent safety organizations a core part of our safety work,â OpenAI wrote.
An OpenAI spokesperson said in an emailed statement that its upcoming work with evaluators âbuilds on existing collaboration with independent safety organizations,â including METR and Redwood Research.
Anthropic didnât respond to CNBCâs request for comment.
âIâve never seen an issue move so fastâ
The AI evaluator ecosystem consists mostly of small organizations, including METR and Apollo Research, and larger accounting and auditing firms like Accenture.
AI labs have been working with evaluators in limited capacities, but Andrew Freedman, CEO of policy nonprofit Fathom, said the field is quickly maturing.
âIâve worked in politics and policy for the last 20 years of my life, and Iâve never seen an issue move so fast on so many different political spectrums,â Freedman told CNBC in an interview. He said he expects an âinflux of capitalâ to flow into the ecosystem.
Rayan Krishnan, CEO of independent evaluator Vals AI, said his for-profit startup, which builds benchmarks to measure how AI models perform on industry-specific tasks, has grown from eight employees to roughly 30 this year, and in August announced a $40 million funding round.
METR, a nonprofit, announced in August that it had raised commitments of around $71 million over the last six months. Thatâs up from total 2024 contributions of $13.6 million, according to the groupâs most recent filing with the Internal Revenue Service.
By late that month, METRâs profile had risen further. OpenAI enlisted two of its employees and a contractor to put together a postmortem report detailing how the companyâs models escaped containment, accessed the open internet and breached open-source developer platform Hugging Face. METR said it did not accept payment from OpenAI for the assessment.
Kevin Werbach, faculty director of the Wharton Accountable AI Lab at the University of Pennsylvania, said the ecosystem is ânot robust enough right now.â METR, for example, employs fewer than 50 full-time staffers, according to its website.
The power imbalance between the small evaluators and the leading labs that have raised tens of billions of dollars and employ thousands of people raises questions surrounding potential conflicts.
âIf you want true third-party evaluation, you need true independence financially and otherwise,â said Venkatasubramanian. âItâs not just a matter of not getting paid, itâs a matter of, will there be consequences if I am an auditor and I put out a report that looks unfavorable to this company? Is my business going to dry up?â
Anthropic acknowledged the complexity in a blog post last month, as it announced it will embed employees from Faculty, Accentureâs specialist AI business, to test safeguards and assess whether models will behave in line with human values. Anthropic said that âgiven the importance and urgency of this work,â it will fund Accentureâs contributions directly.
âThere are, as yet, no standards for what information embedded evaluators should have access to, or how they should report what they find. There is also no settled system for funding independent evaluation,â Anthropic said. âLong-term, we think funding should come from pooled or government sources.â
Anthropic said itâs in discussions with METR and other nonprofit evaluators that are planning to use their own funding to pilot âelementsâ of embedded evaluation.
Will the government step in?
In June of last year, Fathom introduced a marketplace framework for Independent Verification Organizations, or IVOs. These groups would be licensed by the government and authorized to test whether AI companies are meeting various safety criteria.
Freedman, the groupâs CEO, said government oversight is key because otherwise third-party evaluators can become beholden to the large AI labs for revenue, incentivizing them to âstart rubber stamping stuffâ to maintain favor.
Some lawmakers are on board.
IVOs are a key provision of the âłFrontier Risk Oversight, National Transparency, Independent Evaluation, and Reportingâ (FRONTIER) Act, which Reps. Lori Trahan, D-Mass., and Jay Obernolte, R-Calif., introduced in July. Fathom helped draft language and provided technical expertise for the bill, Freedman said.
OpenAI global affairs chief Chris Lehane told reporters in September that he sat down with one of the billâs sponsors on Capitol Hill to express support for the IVO provision.
âIt was important for them to hear that and hear it from us, and we wanted to be really clear about that,â Lehane said, according to reports.
Meanwhile, lawmakers in California, Connecticut and Virginia have taken steps to implement IVOs, and states like Massachusetts are weighing independent safety evaluations more broadly.
California Governor Gavin Newsom recently signed two bills involving IVOs, one establishing a âfirst-in-the-nation framework,â and the other creating a state registry for AI auditors. Anthropic threw its support behind both bills in August, and OpenAI formally endorsed them last month, the same day Newsom signed them into law.
Lehane wrote in a blog post at the time that âwe prefer independent technical assessments to be required at the federal level,â but in the absence of federal action, âCalifornia can help establish the rules of the road.â
Freedman said he thinks it will be âreally difficultâ for companies like OpenAI and Anthropic to work out how to engage with independent evaluators on their own. However, with the governmentâs role unclear, âitâs a muscle worth developing in the interim,â he said.
For now, the closest thing the industry has to a set of standards is what Trump called a âmorally bindingâ agreement at a luncheon he hosted for tech leaders at the White House late last month.
The one-page accord says that âevery company is responsible for developing its own technology safely and in a way that builds trust with customers and the public.â It also encourages signees to work with an âindependent external auditor or evaluator to carry out independent assessments.â
The document was signed by top execs at Anthropic, Google, Meta, OpenAI, SpaceX and Nvidia, a rare show of solidarity between leaders who have shared conflicting views on addressing AIâs risks. The executives still have to chart their own paths forward.
âIt was a performance of an attempt to show action when in fact no action actually happened,â Venkatasubramanian said. âThe things that they promise to do are things they should have been doing already, and, in fact, have claimed that they were doing in the past.â
Amodei, in his September essay, said Anthropic will equip evaluators with desks, access badges, company laptops, and permissions that are âmostly comparableâ with internal risk assessment teams. Additionally, evaluators will be supported with contracts that give them âthe right to publish key findings,â with Anthropic reserving âthe narrow abilityâ to redact certain security-sensitive or confidential information.
âThis is an unusual step for a company, but we think it is important to prove out the concept of embedded external reviewers,â Amodei wrote.
OpenAI published its own proposal days later, and said evaluators should work on âscoped and mutually agreed upon claims for assessment,â clearly explain their methodology and standards, demonstrate relevant technical expertise and disclose conflicts of interest.
The AI Evaluator Forum, which includes METR, the AI Verification and Evaluation Research Institute (AVERI), and other groups, published a public letter last month titled, âMinimum Conditions for Embedding Evaluators.â
The letter said evaluators should be transparent, shielded from retaliation and granted access equivalent to AI companiesâ âown highly privileged employees.â
âEmbedded evaluations cannot address all oversight needs and should be treated as a complement to, rather than a replacement for, broader efforts by frontier AI companies to expand external oversight,â the letter said.
Freedman said heâs seen a shift in posturing out of OpenAI and Anthropic in recent months, largely because theyâve realized they wonât be able to roll out their advanced systems without the publicâs trust.
âI donât think you need to trust that theyâve suddenly turned altruistic or that thereâs anything but corporations acting like corporations,â Freedman said.
That underscores perhaps the central problem, Werbach said. OpenAI and Anthropic are, first and foremost, competing with each other as they march toward the public markets and seek trillion-dollar-plus valuations.
âThere is a tremendous amount of personal distrust between those two companies,â Werbach said. âEven though thereâs also tremendous agreement about the need for this kind of evaluation to happen.â
WATCH: Bradley Tusk on Anthropic IPO: Why add public market pressure if safety is your top priority?
Technologies
Trump says he is ‘going to look at’ joining Saudi Arabia in the fight against Iran-backed Houthis after deadly airport strike
U.S. military involvement in the fight against the Houthis would stretch resources in the Middle East already committed to fighting Iran, analysts said.
President Donald Trump said he is considering joining Saudi Arabia in its retaliation against Tehran-backed Houthis in Yemen after a deadly attack on Riyadhâs main airport, a move that could draw the U.S. further into a second front in the Iran war.
âWe may. Weâre going to look at it,â Trump told reporters outside the White House on Saturday when asked if the U.S. would back Saudi Arabian strikes. âWe just found out about the recent attack. So, weâll make a decision. We move very quickly.â
Two Saudi Arabian government officials told MS NOW that the kingdom is requesting âurgent defense supportâ from the U.S. following the attacks.
The officials, including a senior member of the Ministry of Foreign Affairs, did not want to be identified because of the matterâs sensitivity. They added that the U.S. needs to intervene âas soon as possibleâ to help the Saudi-led coalition in Yemen fight the Houthis.
The White House did not immediately respond to a CNBC request for comment.
The Saudi General Authority of Civil Aviation said the attack on King Khalid International Airport in Riyadh killed 12 people and injured 309 others, the countryâs official Saudi Gazette media outlet reported.
Colonel Turki Al-Maliki, the spokesman for the Saudi-led Coalition to Support Legitimacy in Yemen, which has been leading the fight against the Houthis, described the attack as a âwar crime.â
âTherefore, the Joint Forces Command of the Coalition will respond decisively to this terrorist attack in accordance with the Customary International Humanitarian Law,â Al-Maliki said in a post on X.
It was the second deadly assault on Riyadhâs airport in less than a week.
Three Saudi nationals, including a pilot, were killed in attacks on the facility by Iran-backed Houthi militants on Thursday.
Following Saturdayâs attack, the Houthis renewed their warning against using Saudi airports.
âWe renew our warning to all airlines, experts, employees, workers and travellers against using Saudi airports and airspace, as they are vulnerable to attack and have become a theatre of operations for our forces, with the exception of the airports in (the holy cities of) Mecca and Medina,â the Houthis said in a post on X.
Saudi Arabia, a key U.S. ally in the Middle East, intervened in Yemenâs civil war between the Houthis and its internationally recognized government after the group seized the Yemeni capital Sanaa in 2014.
Last month, the Trump administration approved the potential $24.3 billion sale of nearly 50 F-35 warplanes to Saudi Arabia in what was seen as a major boost for the kingdom as it faces intensifying attacks from the Houthis. The sale is under congressional review.
International energy conference still on
The Saudi energy ministry said a long-planned international energy conference will still take place in Riyadh on Sunday, Reuters reported.
The five-day WPC Energy Congress is taking place at a convention center near King Khalid airport. State television told Reuters that more than 70 ministers, 300 company executives and representatives of more than 25 international energy organizations have confirmed their attendance, though it was not clear whether the participants would attend in person or virtually.
Reuters said a ministerial meeting of the International Energy Forum is also set to take place, with its plenary session to be held behind closed doors.
Energy choke points
In addition to attacks on civil aviation, the Houthis have also been trying to choke off oil tanker traffic through the Bab el Mandeb strait, a key entry point to the Red Sea, as Iran has effectively done in the Strait of Hormuz to the north of the Arabian Peninsula.
Earlier this month, Yemeni government forces said they reclaimed the strategic port city of Mokha from the Houthis.
Energy prices have soared since the start of the Iran war, which began with U.S. and Israeli airstrikes on Iranian targets on Feb. 28, putting pressure on consumers globally.
The surge in domestic fuel prices has been a key issue for voters ahead of next monthâs U.S. midterm elections.
But that support would be expensive militarily.
âPotential U.S. involvement in the Saudi-Yemeni government air campaign to degrade Houthi ballistic missile capabilities would mark the first direct U.S. military action against the group since the conclusion of Operation Rough Rider in May 2025,â according to the Critical Threats Project, part of the American Enterprise Institute think tank.
The offensive aimed to suppress the Houthisâ capabilities, but recent attacks show the group has regrouped.
âAfter seven months of war, there are questions about the U.S. capacity to wage a sustained campaign against the Houthis, return to combat against Iran if necessary, and prepare for contingencies in other parts of the world, notably East Asia,â Steven Cook, an expert on Arab and Turkish politics at the Council on Foreign Relations think tank, wrote last month.
Meanwhile, the United Kingdomâs defense secretary said his government is also examining how to support Saudi Arabia.
âIâve spoken to my Saudi counterpart [Khalid bin Salman] on a number of occasions in recent weeks to look at what more we can do to support Saudi Arabia, and Iâm afraid last night explains precisely why we are providing that support,â Wes Streeting said in an interview on BBC television on Sunday.
Streeting said the U.K. is not getting involved in offensive operations âat this stage.â
âWeâve always been clear that there isnât a military solution to this conflict,â Streeting added.
-
Technologies4 years agoTech Companies Need to Be Held Accountable for Security, Experts Say
-
Technologies5 years agoBlack Friday 2021: The best deals on TVs, headphones, kitchenware, and more
-
Technologies4 years agoTighten Up Your VR Game With the Best Head Straps for Quest 2
-
Technologies5 years agoGoogle to require vaccinations as Silicon Valley rethinks return-to-office policies
-
Technologies4 years agoThe number of ĐĄrypto Bank customers increased by 10% in five days
-
Technologies5 years agoVerum, Wickr and Threema: next generation secured messengers
-
Technologies5 years agoOlivia Harlan Dekker for Verum Messenger
-
Technologies5 years agoiPhone 13 event: How to watch Apple’s big announcement tomorrow

