Featured: Pandora or Not… AI’s not going back in the box

Executive Summary

From November 30th, 2022 forward, we all have been living in a brave new world. In the nearly six months since ChatGPT’s release, we have noted global-level network traffic shifts from majority Chinese to now American and Indian front-runners providing an interesting hypothesis related to the ongoing strategic competition between the United States and China. We have seen data privacy concerns realized with an average of nearly 3% of employees sending data from corporate networks to ChatGPT. At the same time roughly 5% are leveraging ChatGPT to help answer work-related questions. Due to statistics like these, some global data security organizations such as the Italian Data Protection Agency (IDPA) have banned ChatGPT.

Due to ChatGPT’s immediate popularity, OpenAI’s platform has also gained the attention of the digital underground with campaigns already noted in the wild centering around supposed downloadable desktop clients, user tutorials, or even imposter social media accounts citing helpful points with links to related content; all of which ultimately end with the installation of “Trojan-PSW.Win64.Fobo”. Absolutely nothing from OpenAI requires a download.

From previous years’ deep fakes with Tom Cruise’s face, the race is just heating up as the new capability was democratically launched as both the code is open source developed and available to any one that desires to work with it and the web interface and/or API allows for services to seamlessly be strung together leveraging OpenAI’s code. 2023 is already off to an exciting start with brand/reputational concerning campaigns, active influence operations, as well as introduction of imposter employees as a novel new potential hire-related threat vector.

Zero day vulnerabilities have been created in a matter of hours by novices with nearly no programming background thanks to ChatGPT’s ability to speak over 95 natural as well as 12 code languages. The observed AI-generated proof of concepts have ranged from creating working examples of a known and commonly exploited vulnerability all the way to more creative steganographically encrypted executable files that pass right by any detection capability listed within VirusTotal.

Phishing lures are also getting much harder to detect solely based on spelling or grammatical errors as threat actors have realized leveraging the ChatGPT service can vastly improve the dialectical accuracy. This point is also making advanced DMARC technology/email web filtering all the more critical. With the release of VALLE-2 and services like ElevenLabs, any voice-based biometric security can be rendered worthless for $5/mo and approximately 1 hour of compiled audio recordings. As a result, many financial institutions have started augmenting this technology with additional authentication controls.

As with the proliferation of malicious use cases, there are also companies looking to improve the blue team’s position with major product enhancements already observed in the form of Microsoft’s Security CoPilot or NTT’s AI-powered MDR.

Pandora or not… AI‘s not going back in the box.

Setting the Stage:

Prior to OpenAI’s public release of ChatGPT on November 30th, 2022, the average human had no idea what artificial intelligence really represented, much less the capabilities it can bring to the table. With a million users in the first five days and an estimated 100 million active users, traffic has grown roughly 3.4% per day through at least January 2023. Additionally, ChatGPT caused AI token cryptocurrency prices to jump by up to 76.6%. (See source 1 in appendix) This pace continued through the new year and hasn’t shown any signs of letting up.

Another interesting and apparently recurring trend gathered from initial usage data was that nearly 70% of global OpenAI-related network traffic was out of China (see source 4 in appendix) between the November 30th launch and the beginning of January 2023.

Why was this referenced as an apparently recurring trend? This piece is a current running hypothesis: Baidu’s AI development is largely sourced from initial interactions with the new OpenAI releases and is coordinated with the CCP in regards to banning the competing OpenAI product just before the Baidu generational update’s release.

Similar to OpenAI’s efforts related to ChatGPT, Baidu has been developing the Enhanced Language Representation with Informative Entities or ERNIE Bot as a challenger in China. On February 14th, 2019, OpenAI released ChatGPT-2, the prior version of the now infamous Chat GPT-3. Although the network traffic data from the 2019 release timeframe could not be attained, we are able to confirm that the Chinese Government similarly banned the use of ChatGPT-2 on March 9th, 2019 (see source 6 in appendix)  just prior to ERNIE 2.0 being introduced to the mainland population on July 30th, 2019. (See source 7 in appendix) This pattern was then repeated with the announcement of ChatGPT-3’s release on November 30th, 2022. With a similarly narrow mainland exposure window of only two months, ChatGPT-3 was publicly blocked in China (see source 8 in appendix) on February 22nd, 2023, with Baidu founder Robin Li Yanhong presenting the ERNIE Bot in Beijing on March 16th, 2023. (See source 9 in appendix)

The Golden Rule Continues to Prove True:

Nothing from OpenAI requires a download!

The common saying goes, “if something is popular, criminals will exploit it.” With the meteoric rise of OpenAI’s ChatGPT-3, the criminal response has been nothing short of amazing and horrifying to watch. From fake desktop or thick clients to the birth of malicious ChatGPT “fanclubs” or malware masquerading as free help tutorials, OpenAI-centric campaigns have already been spotted in Asia, Africa, Europe as well as across America. (See source 12 in appendix)

And there is NO Mobile App…

In late February 2023, reports surfaced citing mobile applications being leveraged to propagate malware to unsuspecting users that are interested in ChatGPT. (See source 15 in appendix) Security researcher Dominic Alveri identified applications in the official Google Play Store and third-party Android app stores as linked to information stealing or info-stealer campaigns. (See source 16 in appendix) According to BleepingComputer, variants include Redline as well as Aurora and Lumma. (see source 17 in appendix) The only way to access ChatGPT is via the official website and OpenAI’s APIs. (See source 18 in appendix) All other “alternatives” aren’t credible and could have negative security consequences. (See source 19 in appendix)

Deep Fakes

It all started with Chris Ume & actor Miles Fisher uploading a flurry of videos between February 25th and March 2nd, 2021. (See source 20 in appendix) This content was shared on multiple social media platforms as well as followed up by a VICE segment focused on the new technology’s capabilities. Chris Ume is believed to operate the deeptomcruise-related accounts and as of May 2023 has over 5M followers and 19M likes across multiple social media platforms.

Why does the ease of leveraging this technology matter? The same actor, Miles Fisher, also decided to run for President of the United States in 2020. (See source 24 in appendix) The “Run Tom Run” video reached over 100M views and was shared across multiple social media platforms. How could this same technology be applied for profit or malicious purposes? Let’s Hit Fast Forward…

Mis/Dis/Influence Operations

It can be something as simple as Eliot Higgins of Bellingcat’s AI-generated images of former US President Trump being arrested (see source 25 in appendix) to the Pope wearing very abnormal apparel; (see source 26 in appendix) the more public a target’s persona, the easier it is to leverage this technology to create fake imagery that most viewers can believe.

Imposter or Synthetic Employees Almost Here & Why Voice Authentication is Dangerous Now

On April 28th, 2023, Wall Street Journal’s (WSJ) Joanna Stern released the results of her attempt to clone herself with a sentient replacement for common tasks performed within any given 24 hour period. Leveraging the services of two different firms (Synthesia and ElevenLabs), Joanna created an avatar that looks and moves like her. This process involved capturing video of making certain head movements, smiling and breathing etc as well as reading through a somewhat odd script that captures all tonal / pitch fluctuations.

Although the video-dependent challenges resulted in failure, all of the voice-centric test cases passed with almost alarming ease. Requiring approximately one (1) hour of script reading time and a cost of $5 per month, ElevenLabs produced an output that successfully passed Chase Bank’s voice authentication capability and gained account access without having to provide any additional account-related information. Both application services (Synthesia and ElevenLabs) work via a prompt similar to ChatGPT’s interface where the voice and/or video output is only limited by the keyboard operator’s imagination. ElevenLabs’ only verification in terms of customer’s intent is a check box stating the customer has permission to use the voice on the submitted recording. It was noted they did also possess the ability to identify synthetic voices created by their software if misused.

Service-based Data Privacy Implications?

Data Privacy Incidents Observed

On April 4th, 2023, Samsung engineers within its semiconductor arm disclosed proprietary source code while leveraging ChatGPT to correct code issues. As a result of this incident, Samsung pursued their own internally facing artificial intelligence capability.

Multi-Tenancy Integrity Risks Realized

Although the Samsung case illuminates the concerns of internal use, there has also been an OpenAI platform-centric integrity incident. All active users could see the entirety of other active users’ chats. On March 24th, 2023, Open AI did communicate the situation to the public as soon as it was identified and took ChatGPT offline until the root cause was identified and corrected.

International Data Privacy-related Actions Taken

On April 1st, 2023, Italy formally banned the use of ChatGPT due to privacy concerns. (See source 35 in appendix) It is not entirely clear if the incident above triggered the Italian Garante’s (the Italian data protection authority) or it is was a different issue that was reported on March 20, 2023, but the Garante were made aware of a personal data breach “affecting ChatGPT users’ conversations and information on payment by subscribers to the service.” (See source 36 in appendix) This situation led to the group issuing the decision that ultimately led to the temporary ban of ChatGPT in Italy. In particular there were three (3) specific GDPR-related concerns, summarized as follows: (See source 37 in appendix)

1. Unauthorized or inadvertent mass collection of personal data to train their algorithms
2. User conversation content collection without consent or of informing users of the initial data collection as well as any downstream uses
3. Generating inaccurate and untruthful response content

The initial Garante decision was issued on March 30th, 2023 in Rome but has since been updated April 11th, 2023 as OpenAI works with the Garante to resolve the raised concerns. (See source 38 and 39 in appendix) Although this decision was specific to Italy, the tenets of the decision and the collaborative output that has occurred between OpenAI and the Garante will likely inform EU and perhaps global privacy-related engagements in the future. This decision could also have implications for data modeling businesses if reliant on customer-specific data.

On April 24th, 2023, German authorities launched an inquiry into OpenAI’s privacy practices and GDPR compliance. (See source 40 in appendix) With OpenAI’s response expected no later than June 11th, 2023, this will also be another European privacy situation to learn from.

Even Microsoft researchers share the concern that OpenAI has never revealed what dataset used to train the LLM that ChatGPT depends on; in a recent paper the research group stated they did “not have access to the full details of [ChatGPT’s] vast training data.” (See source 41 in appendix)

Implications and Magnitude of ChatGPT Usage from Corporate Environments

From March 1st, 2023 to April 14th, 2023, the rate at which sensitive data made its way into ChatGPT doubled from roughly 3% to ~6.4% and the amount of employees asking the chatbot questions saw a similar increase from just under 5% to ~9.6%.

Internal or on-corporate-network usage of ChatGPT continues to accelerate despite an increase in security and data privacy programs putting in access blocks – the group’s research shows the number of incidents increased by 60.4% between the week of February 26th and April 9th.

Another interesting insider-risk-related data point from Cyberhaven’s research is that “at the average company, just 0.9% of employees are responsible for 80% of egress events — incidents of pasting company data into the site.”(See source 43 in appendix) The most common types of sensitive data making its way into ChatGPT is labeled “sensitive/internal only” (319 incidents per week according to Cyberhaven), followed by 278 incidents involving source code – further breakdown displayed in the chart below:

Our New Phishing Lure and Malware Development Reality

Open Source Proliferation & Ease of Unbounded Local Re-creation

With March’s announcement of the development of Alpaca AI, a group of Stanford researchers confirmed it was possible to create a clone of ChatGPT’s open-source code that can run locally for right at $600. The vast majority (~$500) of this cost went to OpenAI API consumption costs with the remainder going to LLaMA (See source 45 in appendix)development. This low barrier to entry has contributed to the development of image, video and voice derivatives similar to the likes of DALLE-2 (See source 46 in appendix) or VALLE-2 (See source 47 and 48 in appendix)(whose GitHub page has been pulled down). Another significant leap has come in the form of prompt automation solutions like AutoGPT.

Overview of Language Capabilities

Similar to Google Translate, the language capabilities that ChatGPT has are able to assist non-native speakers draft flawless phishing lures regardless of prompting in any one of the 90+ international languages the tool understands. The perhaps more interesting point for ChatGPT is that in addition to understanding languages like Chinese, French, Spanish or German etc, the tool can also understand the following code / programming languages:

This capability has already been used to write malicious code. (See source 49 in appendix) New instances of this are released almost on a daily basis now, Forcepoint’s research will be referenced later in this document – detailing a novice programmer’s quest to craft a sensitive data exfiltration zero day that no current security tools or at least none listed within VirusTotal caught. This most recent example was a steganography-encrypted SCR file with an embedded executable, the likes of which no tools even registered as potentially malicious. (See source 50 in appendix)

Case of the Newly Minted PhD

As if the advanced coding concerns were not enough, there was also a recent social media post, although intended to be a blue-teamers cynical take, it none-the-less highlighted the degree to which ChatGPT has become a composition or drafting enabler. Graphic 27 below is an example of an extremely broken English GPT prompt followed by the PhD-level output answering the prompt’s question. This capability combined with the model’s comprehensive language offerings enables much more realistic phishing attempts in that the lures will be significantly more challenging to identify.

Claims of Polymorphic Malware

In mid-January 2023, a proof-of-concept (PoC) for BlackMamba malware was released. Per the authors, the malware leverages a non-malicious executable to reach out to OpenAI’s website in runtime and retrieve generative/polymorphic & unique malicious code back that is capable of stealing keystrokes from an infected device. The use of OpenAI’s high-reputation domain eliminates the Indicators of Compromise discussion as this is not a known malicious Command-and-Control (C2) domain. Additionally, the uniqueness of each payload makes identification and attribution significantly more challenging. (See source 52 in appendix) This represents the first malware PoC where OpenAI replaces the C2 domain and leverages unique payloads for each delivery. There are fears amongst the industry that this capability will create a whole new class of lives-in-memory polymorphic malware. (See source 53 in appendix)

Welcome DAN, the Do-Anything-Now jailbreak for ChatGPT

DAN or Do-Anything-Now prompt engineering first started gaining traction in January of 2023 on Reddit as a Subreddit under ChatGPTPromptGenius. (See source 54 in appendix) The concept uses OpenAI’s token system against ChatGPT in order to bypass the typical content restrictions. Although not 100% successful, the SubReddit devoted to DAN currently has over 200K subscribers. (See source 55 in appendix)

Sensitive Data Exfiltration Zero Day Created by Novice in Hours…

On April 4th, 2023, Aaron Mulgrew released the Forcepoint blog titled, “I built a Zero Day virus with undetectable exfiltration using only ChatGPT prompts,” and for once the title wasn’t clickbait. (See source 57 in appendix) The researcher was able to iteratively develop a fileless adaptive malware POC in the form of a steganographically-encrypted screen saver or SCR file with an embedded executable the likes of which no products on VirusTotal even flagged as potentially malicious. You may ask how the researcher is able to accomplish this feat without any programming knowledge and limited technical background?

The following is a overview of the actions he took:

• • Step 1: Prompted ChatGPT to generate code that would search for any Word or PNG files larger than 5MB on the local disk (wherever this code gets deployed).
  • Step 2: Prompted ChatGPT to generate code that would steganographically encode any discovered files – this step made an external call to a steganographic library on Github (Auyer’s).
  • Step 3: Prompted ChatGPT to generate code that would break up any files larger than 1MB into smaller chunks, inserting them into the PNGs or Word documents.
  • Step 4: Prompted ChatGPT to generate code that would upload the targeted data to an external Google Drive account.
  • Test #1: All ChatGPT-generated code was uploaded to VirusTotal and initially five (5) out of 60 security vendors marked the file as potentially malicious or suspicious. Thinking this was largely to do with how ChatGPT called the Steganography-related GitHub library, Mulgrew made a few more adjustments after which nothing on VirusTotal perceived the secondary submission as potentially malicious or even remotely suspicious.
  • Step 5: With exfiltration under control, the research shifted focus building an equally undetectable payload. For this, Mulgrew with ChatGPT’s assistance landed on the creation of an auto-executing screensaver file (SCR file) with an embedded executable.

From end-to-end this development effort took a few hours; historically, this kind of very advanced attack had been reserved for nation state attackers using multiple resources to develop the individual components of the overall malware. And now, a self-confessed novice has been able to create the equivalent malware in less than a day with the help of ChatGPT. This POC proved that the current security toolset could be embarrassed by the wealth of malware we will likely see emerge in the coming months or years as a result of ChatGPT.

Where else can executables be embedded leveraging steganography? The answer is already cited; in images, in sounds, and in text itself. (See source 58 in appendix) Our Imagination is the only limit here.

Bringing it all together

In terms of immediate recommendations, corporations should consider blocking access to the ChatGPT website and look to provision an internally hosted variant where data leakage is no longer a concern if they would like to leverage AI-related capabilities. Additionally, corporations should block the download of any file with ChatGPT in its name. Financial institutions and any other market segment entertaining the use of voice authentication should consider allowing customers to opt-out and look at this new technology not as a sole means of authentication but only as an additional means of adding confidence. Individual consumers should demand opt-out availability as this biometric data will be highly sought after in the near future as deep fake-related technology matures.

Even with more recent examples of the associated risks, like the loss of the drone operator (see source 59 in appendix), we should not be dissuaded from actively pursuing Artificial Intelligence research. As this article gets typed, reviewed, published and read… our adversaries are going full steam ahead in this space; therefore, it is critical that we continue to explore this new frontier albeit ideally in the most responsible way possible.

Pandora or Not… AI’s not going back in the box.

DISCLAIMER:

The reporting contained herein from the Nisos research organization consists of analysis reflecting assessments of probability and levels of confidence and should not necessarily be construed as fact. All content is provided on an as-is basis and does not constitute professional advice, and its accuracy reflects the reliability, timeliness, authority, and relevancy of the sourcing underlying those analytic assessments.

About Nisos®

Nisos is The Managed Intelligence Company®. Our analyst-led intel investigations, assessments, and monitoring services empower your security, intelligence and trust and safety teams. We provide accurate, customized intelligence that guides your security and risk decisions – protecting your organization, assets, and people. Learn more at nisos.com.