Mediaboss Marketing

This June, in the political battle leading up to the 2024 US presidential primaries, a series of images were released showing Donald Trump embracing one of his former medical advisers, Anthony Fauci. In a few of the shots, Trump is captured awkwardly kissing the face of Fauci, a health official reviled by some US conservatives for promoting masking and vaccines during the COVID-19 pandemic.

It was obvious that they were fakes, says Hany Farid, a computer scientist at the University of California, Berkeley, and one of many specialists who examined the pictures. On close inspection of three of the photos, Trumps hair is strangely blurred, the text in the background is nonsensical, the arms and hands are unnaturally placed and the details of Trumps visible ear are not right. All are hallmarks for now of generative artificial intelligence (AI), also called synthetic AI.

Science and the new age of AI: a Nature special

Such deepfake images and videos, made by text-to-image generators powered by deep learning AI, are now rife. Although fraudsters have long used deception to make a profit, sway opinions or start a war, the speed and ease with which huge volumes of viscerally convincing fakes can now be created and spread paired with a lack of public awareness is a growing threat. People are not used to generative technology. Its not like it evolved gradually; it was like boom, all of a sudden its here. So, you dont have that level of scepticism that you would need, says Cynthia Rudin, an AI computer scientist at Duke University in Durham, North Carolina.

Dozens of systems are now available for unsophisticated users to generate almost any content for any purpose, whether thats creating deepfake Tom Cruise videos on Tik Tok for entertainment; bringing back the likeness of a school-shooting victim to create a video advocating gun regulation; or faking a call for help from a loved one to scam victims out of tens of thousands of dollars. Deepfake videos can be generated in real time on a live video call. Earlier this year, Jerome Powell, chair of the US Federal Reserve, had a video conversation with someone he thought was Ukrainian President Volodymyr Zelenskyy, but wasnt.

The quantity of AI-generated content is unknown, but it is thought to be exploding. Academics commonly quote an estimate that around 90% of all Internet content could be synthetic within a few years1. Everything else would just get drowned out by this noise, says Rudin, which would make it hard to find genuine, useful content. Search engines and social media will just amplify misinformation, she adds. Weve been recommending and circulating all this crap. And now were going to be generating crap.

Although a lot of synthetic media is made for entertainment and fun, such as the viral image of Pope Francis wearing a designer puffer jacket, some is agenda-driven and some malicious including vast amounts of non-consensual pornography, in which someones face is transposed onto another body. Even a single synthetic file can make waves: an AI-generated image of an explosion at the US Pentagon that went viral in May, for example, caused the stock market to dip briefly. The existence of synthetic content also allows bad actors to brush off real evidence of misbehaviour by simply claiming that it is fake.

Peoples ability to really know where they should place their trust is falling away. And thats a real problem for democracy, says psychologist Sophie Nightingale at Lancaster University, UK, who studies the effects of generative AI. We need to act on that, and quite quickly. Its already a huge threat. She adds that this issue will be a big one in the coming year or two, with major elections planned in the United States, Russia and the United Kingdom.

AI-generated fakes could also have huge impacts on science, say some experts. They worry that the rapidly developing abilities of generative AI systems could make it easier for unscrupulous researchers to publish fraudulent data and images (see Scammed science at the end of this article).

For now, some synthetic content contains give-away clues such as images featuring people with six fingers on one hand. But generative AI is getting better every day. Were talking about months until people cant tell the difference with the naked eye, says Wael Abd-Almageed, an information scientist and computer engineer at the University of Southern California in Los Angeles.

An overlay of synthetic faces (left) shows more regularity than one of real images (right).Credit: Hany Farid

All of this has researchers scrambling to work out how to harness the deepfake powers of AI for good, while developing tools to guard against the bad. There are two prongs of technological defence: proactively tagging real or fake content when it is generated; and using detectors to catch fakes after publication. Neither is a perfect solution, but both help by adding hurdles to fakery, says Shyam Sundar, a psychologist and founder of the Media Effects Research Laboratory at Pennsylvania State University in University Park. If youre a dedicated malicious actor, you can certainly go quite far. The idea is to make it difficult for them, he says.

Technology will be crucial in the short term, says Nightingale, but then longer term, maybe we can think more about education, regulation. The European Union is leading the way globally with its AI Act, which was passed by the parliament this June and is awaiting decisions by the two other branches of the EU government. Were going to learn important lessons from it for sure, says Nightingale, whether they get it right or not.

For researchers, generative AI is a powerful tool. It is being used, for example, to create medical data sets that are free of privacy concerns, to help design medicinal molecules and to improve scientific manuscripts and software. Deepfakes are being investigated for their use in anonymising participants of video-based group therapy; creating custom avatars of physicians or teachers that are more compelling for viewers; or allowing for improved control conditions in social-science studies2. Im more hopeful than concerned, says Sundar. I think its transformative as a technology.

But with the spectre of rampant misuse, researchers and ethicists have attempted to lay down rules for AI, including the 2018 Montreal Declaration for the Responsible Development of Artificial Intelligence and the 2019 Recommendation on Artificial Intelligence from the Organisation for Economic Co-operation and Development. An initiative called the Partnership on AI, a non-profit organization that includes major industry partners, fosters dialogue on best practices although some observers and participants have questioned whether it has had any impact.

AI and science: what 1,600 researchers think

All advocate for the principles of transparency and disclosure of synthetic content. Companies are picking that up: in March, for example, TikTok updated its community guidelines to make it mandatory for creators to disclose use of AI in any realistic-looking scene. In July, seven leading technology companies including Meta, Microsoft, Google, OpenAI and Amazon made voluntary commitments to the White House to mark their AI-generated content. And in September, Google declared that starting in mid-November, any AI-generated content used in political ads will have to be declared on its platforms, including YouTube.

One way to tag synthetic images is to watermark them by altering the pixels in a distinctive way thats imperceptible to the eye but notable on analysis. Tweaking every nth pixel so that its colour value is an even number, for example, would create a watermark but a simple one that would disappear after almost any image manipulation, such as applying a colour filter. Some watermarks have been criticized for being too easy to remove. But deeper watermarks can, for instance, insert a wave of dark-to-light shading from one side of an image to the other and layer it on top of several more such patterns, in a way that cant be wiped away by fiddling with individual pixels. These watermarks can be difficult (but not impossible) to remove, says Farid. In August, Google released a watermark for synthetic images, called SynthID, without revealing details about how it works; its unclear yet how robust it is, says Farid.

The companion idea to watermarking is to tag a files metadata with secure provenance information. For photography, such systems start when a photo is taken, with software on the camera device that ensures that an images GPS and time stamps are legitimate, and that the image isnt a photo of another photo, for example. Insurance underwriters use such systems to validate images of assets and damages, and the news agency Reuters has trialled authentication technology to validate photos of the war in Ukraine.

The Coalition for Content Provenance and Authenticity (C2PA), which brings together key industry groups in technology and publishing, released a first version of a set of technical specifications in 2022 for how systems should track provenance information for both synthetic and real imagery. Plenty of C2PA-compliant tools that embed, track and verify provenance data are now available, and many corporate commitments such as Microsofts say they will follow C2PA guidelines. C2PA is going to be very important; its going to help, says Anderson Rocha, an AI researcher at the University of Campinas in Brazil.

Systems that track image provenance should become the workhorse for cutting down the sheer number of dubious files, says Farid, who is on the C2PA steering committee and is a paid consultant for Truepic, a company in San Diego, California, that sells software for tracking authentic photos and videos. But this relies on good actors signing up to a scheme such as C2PA, and things will slip through the cracks, he says. That makes detectors a good complementary tool.

AI will transform science now researchers must tame it

Academic labs and companies have produced many AI-based classifiers. These learn the patterns that can distinguish AI-made media from real photos, and many systems have reported that they can spot fakes more than 90% of the time, while falsely identifying real images as fakes only 1% or less of the time. But these systems can often be defeated3. A bad actor can tweak images so that the detector is more likely to be wrong than right, says Farid.

AI-based tools can be paired with other techniques that lean on human insights to unravel the fake from the real. Farid looks for clues such as lines of perspective that dont follow the rules of physics. Other signs are more subtle. He and his colleagues found that facial profiles made by StyleGAN generators, for example, tend to place the eyes in the exact same position in the photo4, providing a hint as to which faces are fakes. Detectors can be given sophisticated algorithms that can, for example, read a clock somewhere in the photo and check to see whether the lighting in the image matches the recorded time of day. Tech company Intels FakeCatcher analyses videos by looking for expected colour changes in the face that arise from fluctuations in blood flow. Some detectors, says Rocha, look for distinctive noise patterns generated by light sensors in a camera, which so far arent well replicated by AI.

Experts worry about the influence of deepfake videos, such as this one of Barack Obama.Credit: AP via Alamy

The battle between fake-makers and fake-detectives is fierce. Farid recalls a paper by his former student Siwei Lyu, now a computer scientist at the University at Buffalo, New York, that highlighted how some AI videos featured people whose two eyes blinked at different rates5. Generators fixed that problem in weeks, he says. For this reason, even though Farids lab publishes the vast majority of its work, he releases code only on a case-by-case basis to academics who request it. Abd-Almageed takes a similar approach. If we release our tool to the public, people will make their own generation methods even more sophisticated, he says.

Several detection services that have public user interfaces have sprung up, and many academic labs are on the case, including the DeFake project at the Rochester Institute of Technology in New York and the University at Buffalos DeepFake-o-meter. And the US Defense Advanced Research Projects Agency (DARPA) launched its Semantic Forensics (SemaFor) project in 2021, with a broad remit of unearthing the who, what, why and how behind any generative file. A team of nearly 100 academics and corporate researchers have worked together under SemaFor to create more than 150 analytics, says the projects head, Wil Corvey. The bulk are detection algorithms that can be used in isolation or together.

AI tools as science policy advisers? The potential and the pitfalls

Because there are a huge number of both generators and detectors, and every case is different, reported accuracy rates vary wildly. And the arms race between them means that the situation is constantly changing. But for many media types, current success rates are poor. For generated text, a review this year of 14 detection tools found that all were neither accurate nor reliable6. For video, a high-profile competition in 2020 was won by a system that was only about 65% accurate3 (see also go.nature.com/3jvevoc). For images, Rocha says that if the generator is well known, detectors can easily be more than 95% accurate; but if the generator is new or unknown, success rates typically plummet. Using multiple detectors on the same image can increase the success rate, says Corvey.

He adds that detecting whether something is synthetic is only one part of the puzzle: as more users rely on AI to tweak their content, the more important question is not how much of this is synthetic? but rather why was this made?, he says. For this reason, an important part of SemaFors work is to determine the intent behind fakes, by attributing the media to a creator and characterizing its meaning. A parallel DARPA project, the Influence Campaign Awareness and Sensemaking (INCAS), is attempting to develop automated tools to detect the signals of mass misinformation campaigns that might or might not be supported by AI fakery.

SemaFor is now in the third and final stage of its project, in which Corvey is focusing on reaching out to potential users such as social-media sites. We have outreach to a number of companies including Google. To our knowledge, none have taken or are running our algorithms on a constant basis on-site, he says. Meta has collaborated with researchers at Michigan State University in East Lansing on detectors, but hasnt said how it might use them. Farid works with the employment-focused platform LinkedIn, which uses AI-based detectors to help weed out synthetic faces that support fraudulent accounts.

Abd-Almageed is in favour of social-media sites running detectors on all images on their sites, perhaps publishing a warning label on images flagged with a high percentage chance of being fake. But he had no luck when he discussed this a couple of years ago with a company that he would not name. I told a social network platform, take my software and use it, take it for free. And they said, if you cant show me how to make money, we dont care, says Abd-Almageed. Farid argues, however, that automated detectors arent well suited to this kind of use: even a 99% accurate tool would be wrong one time out of 100, which he thinks would completely erode public confidence. Farid argues that detection should be targeted at intensive, human-led investigations of specific cases, rather than trying to police the entire Internet.

Many argue that companies such as publishers and social-media sites will need legislation to push them into responsible behaviour. In June, the European Parliament passed a draft law that would strictly regulate high-risk uses of AI and enforce disclosure of content generated by such tools. The world is watching, because the EU has taken the lead on this, says Nightingale. But experts disagree widely about the acts merits and whether it might quash innovation. In the United States, a few AI bills are pending, including one aimed at preventing deepfakes of intimate images and one about the use of AI in political advertising, but neither is certain to pass.

There is one point that experts agree on: improving tech literacy will help to stop society and democracy from drowning in fakes. We need to get the word out to make people aware of whats happening, says Rocha. When they are informed about it, they can take action. They can demand education in schools.

Even with all the technological and social tools at our disposal, Farid says, its a losing battle to stop or catch all fakery. But its OK, because even in defeat I will have taken this out of the hands of the average person, he says. Then, as with counterfeit money, it will still be possible to fool the world with generative AI fakes but much harder.

Science isnt immune to the problem of AI-generated fakery. One concern is the integrity of biomedical images such as scans, microscopy images and western blots a standard technique in which distinctive bands are created by proteins of various molecular weights as they spread across a gel. Fraudsters have long faked such images using Photoshop or other image-manipulation software, but that is often detectable by a trained eye or by computers that check for image duplication. Generating images entirely with AI presents a bigger detection problem.

Last year, Rongshan Yu and his colleagues at Xiamen University, China, created scientific images using synthetic AI to see how easily it can be done7. They trained one generative AI program to create new western blot images from a training data set of 3,000 images; and used another to insert features of oesophageal cancer into an image of a non-cancerous intestine. They then tested how convincing the western blots were by inviting three specialists to try to spot one fake in a set of four images; two of the experts performed worse than chance, and the third fared better by spotting a visual clue to do with the smoothness between the image and the background. A computer system did slightly better than the specialists.

In a larger study of 14,000 original western blot images and 24,000 synthetic ones, made using four different generators, Anderson Rocha at the University of Campinas, Brazil, and his colleagues found that AI detectors trained on large data sets achieved accuracies of more than 85% (ref. 8). This is just the beginning, where we showed its feasible. Its possible to do way more than that, Rocha says. He and his team have a paper under review that extends their methods beyond western blots, he says.

No one Nature spoke to could provide proof that AI is being used by academics to beef up their papers or funding applications, but experts say its likely. Wael Abd-Almageed, an information scientist and computer engineer at the University of Southern California in Los Angeles, says he knows of specific cases in which academics have used AI to synthesize fakes, but declined to divulge details. Researchers investigating fake paper mills have said they have seen AI-generated images.

I dont know any case yet of a retracted paper that used synthetic creation to illustrate the results in that paper, says Rocha, who is working with the blog Retraction Watch on this issue. But its just a matter of time, I guess.

Publishers are on alert, says Bernd Pulverer, chief editor of the journal EMBO Reports. We are certainly concerned, he says. In the arms race between generative AI and detection tools, he says, we will struggle to detect well-executed AI image manipulation. It is all too easy to weaponize existing AI tools at every level of biological data.

Watermarking technologies could help, Pulverer adds. This idea has been explored previously for medical images not to prevent AI alterations, but to protect privacy and prevent any kind of tampering. Pulverer says he plans to work with instrument makers and image-processing experts to discuss the feasibility of watermarking or provenance-tagging scientific imagery.

Beyond developing technological tools against fraud, he says, the key is to encourage efforts to reproduce data in science. Attempting to replicate results could be the ultimate way to catch out fakery.

Originally posted here:

How to stop AI deepfakes from sinking society and science - Nature.com