[Editor’s Note: Today’s post — co-authored by returning guest blogger and proclaimed Mad Scientist Dr. Jan Kallberg, LTC Todd Arnold, and COL Stephen Hamilton, all from the U.S. Army Cyber Institute at West Point, New York — addresses how we have been socially conditioned to “trust the pixel” in the digital age, creating a vulnerability for adversaries to exploit in future conflicts by manipulating our Soldiers and Leaders, the Home Front, and captured comrades-in-arms via deepfake technology. Read on to see how our adversaries could use deepfakes to negatively influence the American will to fight, and learn what we can do now to build resiliency to these increasingly likely possibilities!]
As technology evolves, it introduces new methods to deliver and consume information. Unfortunately, technology is often dual-use – the technology that is used for positive civilian purposes can be simultaneously used with nefarious intent or military gain. Dual-use technology is not novel or unique to the current information age; mass printing and trains created social mobility and consolidated democracy in America, but conversely was the foundation for the early Soviet totalitarian propaganda machine in the 1920s. Similarly, the radio, which gave all Americans political 
insights and strengthened the bond between the electorate and the elected, was the same technology used by multiple totalitarian governments as the engine for their propaganda machine to suppress the opposition, consolidate control, and retain their grip on power (e.g., 1930’s Germany and Italy). With current technological advances and capabilities, the information society can be informed as events happen in real-time, but the same wide information dissemination can manipulate, disinform, and negatively influence entire populations in unanticipated ways and at unprecedented speed.
Our potential adversaries seek to influence our country’s direction, while advancing their agenda, by distorting American society’s perception of reality and how we relate to our own values. The Russian perception management strategy, adopted from the Soviet Union and duplicated by China and North Korea, aims to make the average citizen feel less loyal to the United States 
through questioning, and thereby undermining, the legitimacy, authority, trust, and confidence in U.S. elected officials. The perception under attack is our belief in our society, our mission, and democracy itself. In a traditional peace time media landscape, Russian and Chinese perception management would struggle to have an audience. However, with great power competition, social media and online content proliferation, and the potential for U.S. Prisoners of War (POWs) being taken in future combat operations, an opportunity unfolds for our adversaries.
POWs are an audience that, due to the physical circumstances and the nature of captivity, cannot shelter themselves from a captor that does not obey international humanitarian law. In previous armed conflicts, authoritarian regimes attempted to exploit American POWs for propaganda gain. These efforts often took the form of video and audio recordings, forced statements, and pictures of the POWs, despite such activities clearly violating the Geneva Conventions. The prospect of advanced digital capabilities, such as deepfakes, presents potential adversaries with a new powerful tool for manipulation and influence in future conflicts. The American military must begin preparing for the prospect of these new technologies being used against its POWs.
In the last decade, images and video have replaced the written word as the standard bearer for facts, entertainment, notions, and emotions. The ubiquity of digital media, tablets, smart phones, and wearables that we watch and interact with has also created a reliance on the available images, footage, and media. The old adage, “a picture is worth a thousand words,” is more relevant 
than ever. The younger generation trusts images and available online footage, while they consume information rapidly, with little or no time to analyze the validity of the information. The inherent trust of visual information elevates the credibility of plausible information, making it an ideal medium for launching active disinformation campaigns (e.g., recent Russian and Chinese activities). So, rather than society becoming more resilient to disinformation campaigns, it is likely that we are gradually building up a vulnerability by trusting the pixel instead of our traditional sources of information.
What are deepfakes? A deepfake is the manipulation of digital video and audio files to make it appear in a video that the person says or does something they never said or did, primarily by face-swapping. The technology can be used to manipulate television news, public statements, audio or film footage; 
virtually any digital medium. The technique utilizes pre-existing audio and video of the target to create a video (potentially even a real-time live video feed) where another person is controlling what is said by the deepfake subject, duplicating the targeted individual’s face, features, speech, and vocal distinction. The end product is often not only plausible, but it can also be highly believable and is becoming increasingly difficult to detect. Several Internet-famous deepfakes have surfaced on social media. The deepfakes of the Belgian visual artist Chris Ume gained international attention as he created compelling manipulated videos featuring what appeared to be Tom Cruise. The person is instead the actor Miles Fisher, whom Mr. Ume altered to look and sound like Tom Cruise. The process required two months to create the deepfakes, but Mr. Ume neither had access to Tom Cruise nor could he call Mr. Cruise to extract voice features to speed up the deepfake creation. Today, a plausible deepfake can be created in as little as five minutes. In a POW or captivity scenario, the captor’s access to the captive makes gaining adequate features to render a deepfake trivial.
From a POW and captive recovery perspective, this technology creates two distinct concerns. The initial concern is the release of a POW deepfake to the public. Even though the creation and release would be considered a violation of the Geneva Conventions, such a deepfake could be manipulated and utilized to create narratives of war crimes, atrocities, rejection of the U.S. war effort, pleadings to end the war, and other propaganda. The deepfakes could be 
rapidly distributed across the American home front on a national scale to undermine the American war effort, the will to fight, stress service members’ families, use tailored messages to politicians by POW constituents, create cleavage in communities, and weaken support for the war.
The systematic attack on the will to fight could seriously impact the support for continued U.S. operations. To expand on this concern and visualize its utilization, we give two examples using deepfakes which demonstrate negatively influencing the American will to fight:
First, a deepfake video is produced where U.S. POWs claim they will be executed for the war crimes ordered by General Jones — and the only way to save them is for General Jones to publicly accept responsibility for their war crimes. The deepfake video, distributed 
widely among U.S. allies, would exacerbate anti-US sentiment in the Third World. The video is shared and becomes viral among social media users, many driven by an anti-US sentiment. Additionally, the adversary may still hold the advantage of having POWs for as long as possible by using deepfake technology to continue generating new content even though the POWs have already been executed or perished in captivity.
Second, a POW is interviewed by the captor’s interrogation team about mundane daily events of no significance; the sound and images are captured during the 30 minute session without the POW’s knowledge or consent. The POW is resisting collaboration and acts according to the U.S. Army Code of Conduct. However, technology could portray the POW as a willing collaborator. Constructing a narrative based on Open Source Intelligence (OSINT) that is plausible would reinforce the idea that the video is genuine. The 30 minutes of audio and images are well beyond what the captor would need to create a few minutes of deepfake video that is a plea from the POW to spare their life and end hostilities. The POW in the deepfake video could falsely detail collateral damage, the deaths of innocent civilians, the absence of a U.S. cause for hostilities, or how the futile American policy will fail.
The second concern is that the captors could manipulate POWs by showing them deepfakes while in captivity. This scenario is neither implausible nor unprecedented; during both the Vietnam and Korean conflicts, the 
Communists dedicated hours of daily indoctrination to sway captured U.S. Soldiers’ and Airmen’s opinions and beliefs. The old-school Soviet propaganda machine was focused on making a lie plausible while undermining the sense of reality and truth that prevailed until that point. For the propagandist, achieving plausibility is easier than fully convincing a target. The length of captivity could be several years, which works in the captors’ favor, since POWs have no other information or sources other than those the captor provides. For the Communists during the conflicts in Vietnam and Korea, the captives were a propaganda asset that, if turned against the United States, would support their cause.
Modern technology gives a variety of effective tools to confuse the captive and undermine their will to resist. The captor could utilize deepfakes to indoctrinate, psychologically destabilize, and manipulate the captive’s mental 
state. The potential effects become more likely in a protracted conflict where captivity may continue for several years. When combined with intentionally harsh conditions, even the most ardent resistance could crumble. Despite the fact that a POW might dismiss each individual deepfake as implausible, it is possible that over time, the isolation and pressure from the surrounding conditions would lead a POW to cave into its manipulation.
In our view, the POW deepfake concerns need to be addressed in advance of potential conflicts where such tactics may be used. Planning and research initiatives should be initiated to address these increasingly likely possibilities. Preliminary efforts should include: 1) Identifying ways to identify deepfakes shortly after their dissemination; 2) Exploring the possibility of preparing a validated genuine video of all service members as an aid to identifying deepfakes, with enough data captured and archived prior to deployment (much like is currently done for ISOPREP); 3) Preparing both the military and public at large in advance to the possibility of deepfakes; and 4) Incorporating deepfake awareness in POW resistance training to prepare the service member for the possibility that deepfakes may be used against them while in captivity.
We find it essential to migrate from a shared notion of what captivity means based on experiences in WWII and the conflicts in Korea and Vietnam. Today, the tools captors can use for their propaganda and manipulation are widely different. Modern-day propaganda can now reach every networked digital device, and our potential adversaries are ready to use that reach to influence our will to fight. Based on the history of abuse demonstrated by our adversaries, our concern is that utilizing the manipulated POWs’ voices and images could be a part of these psychological operations.
If you enjoyed this post, check out Dr. Kallberg’s previous post The Classified Mind – The Cyber Pearl Harbor of 2034, as well as the following related content:
The Exploitation of our Biases through Improved Technology, by Raechel Melling
LikeWar — The Weaponization of Social Media
The Death of Authenticity: New Era Information Warfare
Influence at Machine Speed: The Coming of AI-Powered Propaganda by MAJ Chris Telley
A House Divided: Microtargeting and the next Great American Threat, by 1LT Carlin Keally
Sub-threshold Maneuver and the Flanking of U.S. National Security, by Dr. Russell Glenn
The Erosion of National Will – Implications for the Future Strategist, by Dr. Nick Marsella
Weaponized Information: What We’ve Learned So Far…, Insights from the Mad Scientist Weaponized Information Series of Virtual Events, and all of this series’ associated content and videos [access via a non-DoD network]
Weaponized Information: One Possible Vignette and Three Best Information Warfare Vignettes
About the Authors: Jan Kallberg is a research scientist at the Army Cyber Institute. Todd Arnold is the Lead for the Cyber Operations Research Element (CORE) at the Army Cyber Institute and an Assistant Professor in EECS at the U.S. Military Academy. Stephen Hamilton is the Chief of Staff and Technical Director at the Army Cyber Institute and an Associate Professor in EECS at the U.S. Military Academy. The authors would like to thank Mark Visger for his valuable feedback.
Disclaimer: The views expressed in this blog post do not necessarily reflect those of the Department of Defense, Department of the Army, Army Cyber Institute, the U.S. Military Academy, Army Futures Command (AFC), or U.S. Army Training and Doctrine Command (TRADOC).
