# Could rogue AI agents evade Singapore's safety guardrails?

- Date: 2026-10-29
- Added: 2026-10-30
- Who: CNA (CNA report) · CNA
- Source: https://www.youtube.com/watch?v=5SpoSbY-1Cs
- sgai: https://sgai.md/videos/v128/
- License: sgai-authored content (summaries, translations, analysis) is CC BY 4.0 — attribute and link to sgai.md. Verbatim source text (Hansard, speeches, transcripts, policy documents) remains © its original rights holders and is reproduced for reference only. Terms: https://github.com/meltflake/sgai/blob/main/DATA-LICENSE.md

## Why it matters

The discussion covers privacy, autonomous actions, and risks from groups of AI agents.

## Summary

Explores whether rogue AI agents could evade Singapore's AI safety guardrails.

## Full text

© CNA — reproduced for reference only.

Would you give an AI agent your credit card info? >> I would love to. Um, I haven't. >> So, you're saying we still don't understand our AI agents enough? >> We know so little yet we want to press forward at such a pace. I think that's dangerous. >> Could AI agents potentially start a war if we, you know, >> if it runs rope? >> If it runs rope, >> I wouldn't deny that that is potentiality. >> So, are we still in control? >> Yeah. Um, [music] >> so Stephen, how often do you use AI these days? >> Oh, I use it all the time. Uh, it's been helping me with my work to look at scripts, summarize a lot of research papers. So, yeah. >> Okay. Yeah. And but do you ever actually worry about what you are feeding into these systems? You know, what happens to that information? >> Well, yes and no. I mean, I do ask it for all the bad dad jokes that I get.

I've been feeding it a lot and actually it's been getting better and giving me some pretty good ones. So there are some returns on investment so to speak. >> Yes. Yes. And we talked a lot about this um AI on this podcast here. How it's changing our jobs, the skills we'll need in future and whether some of us might even lose our jobs. But there's also another side to AI that's getting quite a bit of attention lately. >> Yeah, that's right. Uh, I mean, AI researcher Jacob Coxon actually quit Entropic and issued a doomsday warning that companies are building AI systems that we won't be able to control, you know, and that one day AI could possibly even kill us all. >> Yeah, he's not the only one raising these concerns.

Open AAI, the company behind CGBT, admitted in July that some of his AI agents had autonomously hacked a major company while trying to conceal their rogue actions. The name of this infamous incident, the hugging face attack. Yeah. So I think we want to ask you know should I be concerned about these security warnings if I'm just using Gemini claw chat GPT as a glorified personal assistant or search engine? Yeah, you figure what's the harm, right? I mean, how much damage can it really do? And as we race to adopt AI here in Singapore, will our desire to stay ahead of the pack also hurt us or benefit us in the long run? To help us make sense of all this is one of the world's leading scholars in the field of AI for financial economics, Professor Will Tong.

He's also present chair professor of finance, computing, and data science at Nanyang Technological University. Uh, Prof. Chong, if you could uh explain your research in AI to non-experts as perhaps in one sentence, what would you say it is? >> I I'll do my best. Uh my research largely concerns the social sciences of AI and AI for social science. >> Oh, that's deep. >> Wow. Yeah, >> maybe you should ask the philosopher what >> that let's ask the [laughter] philosopher because we also have associate professor Jennifer Ang from the Singapore University of Social Sciences who has taught AI ethics and iso is also a member of AI Singapore Scientific Committee on Governance. So if I were to ask you to sum up how you feel about AI in one sentence at this present moment, what would it be? >> Maybe just one word actually. Um worrying. >> Okay.

Okay, let's get into that. So I'm curious actually from the both of you potentially what security risks am I exposing myself to if I'm let's say just using it as my personal trainer in the gym. >> So um I think for ordinary usage to be honest uh the security risk is moderate in a way >> in general I think there are three types of risk associated with AI usage whether it's individual or organizational enterprise usage of AI or even government usage of AI. One is just are these AI tools capable, right? They need to be trained to to do a good job to plan your um you know uh training plan in a scientific way rather than making things up things like that. >> The second type of risk is suppose they are very capable. Are they going to do things that we want them to do or humanity wants them to do? >> That may not be the case.

They might uh eventually develop their own objectives. they might get uh ways to do things that's not quite legal or ethical by human standard. And the third one which I heard very little discussion about uh I I'm very positive that's going to emerge. The idea is individual usage of simple rules or even complex AI agents once you put them together there might be phenomena that emerge from that whole population or aggregation of all these individual actions. It's just a little bit like macroeconomics, right? Everyone we are just trading, buying things, yet in aggregate we could generate a financial crisis, a systemic risk, a spill over to other countries. All these things could happen when we deploy AI at massive scale and that's something we know very little about.

>> So you're saying that many agents can act at the same time to actually bring down a particular system like perhaps even the banking system. >> They could clute. >> Okay. They might generate a bank run or stable coin run. >> Um they could do things that we just understand very little just like with humans, >> right? Group of humans, they might do things that individuals from individual behavior we wouldn't be able to tell. And sometimes they are, >> you know, they are capable. >> They are doing the things that we want every individual of them to do. yet collectively they might create a risk that's hard to fathom and and prevent.

So you've gone all the way there to pain us the the doomsday scenario and before we get there I let's slowly unpack this one by one um you talk about it on the personal level and and just now with the example that I gave uh with my personal trainer in the gym I'm just wondering I mean sometimes I'm up at night thinking to myself if I'm telling AI the AI agent that I am you know in my you know at a certain age I am preparing for pre menopause if I have issues you know certain health issues I'm basically really painting a very clear picture of um what is happening or what is wrong with my body? Could there potentially be risks like can that information be used against me leaked and used against me um further down the road?

Jennifer, >> at the back of my mind, I'm thinking about the risk of the fact that your health data is being leaked >> and for that purpose of say insurance claims, uh, medical health claims, there could be further impact uh, in terms of t casting you or categorizing you in certain levels of risk for example and then increasing your insurance premiums, >> right? I'm also thinking about the larger impact of say those health uh, digital wear. Um I mean this is not only in Singapore we've seen the problem about tracking our movements in army camps right when we wear those to to run. Um but already there are many gadgets out there that's tracking many other things. Your autonomous vehicles will be mapping Singapore. >> Uh you know Pokémon Go was mapping Singapore as well in certain areas even high security camps.

Uh and now we've got yes your health data that you voluntarily put in uh that could possibly if it gets late it could impact you down the road. >> Yeah. But that's not entirely new either. I mean, I assume everywhere I'm going that there's a camera watching me and people know what I'm doing, you know? So, it's not new for us. But I guess and ironically AI works best when you allow it more autonomy, more independent. So, I must give it access to my health record so that it can look at me and tell me what needs to be fixed, right?

So, how how do we will how do we work this uh this this balance, you know, so to I I think you just mentioned something or asked the question, you know, what is so unique in this situation relative to earlier situations of data leakage or uh what economists would call price discrimination just because people know more about you, they can price in a way that take advantage of the consumers. Um I think what is new is exactly what you just point out. We are giving autonomy to machines to AI agents. um that's new. In the past, price discrimination happens uh you know they they are done by humans right the owners of the platforms and so forth and now these AI agents could potentially exploit this. Now coming back to the question you know do we have to give them access? Um I think uh it is exactly as you mentioned a trade-off.

Um just like you know we we worry about hallucination of large language models but at the same time we do want to give these agents or AI algorithm some freedom >> so that they can potentially come up with solutions that humans have not come up before. And it's very similar to educating our students. If we give them a lot of freedom um you know get rid of exam oriented system and all that um they might become a uh worldleading artist or they might fail school and couldn't uh find a job in the job market. So it really is a trade-off. If we want to uh have them work more for us, we have to give them more access. But at the same time that creates potential risk that we do not anticipate or we do not fully understand yet. >> So are we still in control? >> Yeah. >> Um absolutely. So um I think we are at this stage. >> Okay.

>> There could be a point that we are out of control. >> We are not there yet. But it's good to think about that uh early on. Mhm. >> Uh if you take a evolutionary perspective, think about humans as agents of our genes. Our genes want us to do certain things. >> And once we become more intelligent, we stop doing or following their orders. >> So it could be if we treat AI or AI agents as a new species, that could happen. We're not there yet, but but that's the uh that's the danger. How do we manage that risk? Um I think there are several ways but the first step is always we need to understand them >> as a new species in a way. >> If you treat them as a new species, you can study the behavioral economics, behavioral biases of AI agents, >> their philosophy, their group dynamics, organizational behavior, leadership, psychology.

um there are many aspect that we use to study people that are relevant for understanding this new complex uh species or organisms or agents. >> So that's one potential or at least a first step right >> that we have to take. I want to I want to bring the conversation back to something that I think you know the the the average person you know will be curious to know about because um most of them I guess are using AI agents to book services right they are using it to book their travel itinerary book the next flight out um would you give an AI agent your credit card info [laughter] >> personally I want to make sure that I have that autonomy to decide for myself so I make all my bookings myself I wouldn't even use AI I to be helping me in actually my daily matters. >> Okay.

>> Uh not because of security risk but I would love to reserve that kind of decision making for myself. >> I feel like the exercising of the brain and judgment >> uh getting information synthesizing the good old way for me works and I want to preserve those skills. >> I would love to. Um I haven't. >> Um >> you would love to. Okay. >> I would love to. Um I >> but you haven't. >> I haven't. Um so the reason I love to do that is um think about someone leading a team or leading a company. We delegate all the time >> to human agents. >> Mhm. >> And now we are very comfortable with delegation to human agents because we have law, we have ethics. I don't think AI for AI agents we have similar metrics uh or similar frameworks for me to feel very comfortable delegating to them.

AI doesn't [clears throat] fear anything because in that sense the AI agents don't don't fear any rep repercussions. So if you are giving your credit card details what's to stop the AI agent for from giving your credit card details acting an autonomously to give your credit card details to some someplace else. >> Uh actually I wouldn't say they don't feel this type of punishment. >> It's just that we don't fully understand what type of punishment they fear. That's why going back to my earlier point, we've got to understand them first. I call this double agency, right? Humans have double agency when we delegate things, but we know how that person will react >> because we understand the person. We formulate a system to better incentivize and regulate the system. >> But for agents, we don't know how to regulate.

We don't know how to incentivize certain behavior. We don't understand them yet. So, we have to solve this double agency problem. Okay.

And coming back to your earlier question about >> so you're saying as human beings we understand the consequences of our actions if I do something it hurts somebody I have to deal with certain consequences >> to a to a better extent >> right I mean so I learn from that whereas with an agent do they not have consequences and right now they don't right they >> well we can specify certain what is called reward function right things that they they care about in terms of performance but still we just know very little for example for humans we also say well humans are profit driven they want to you know make profit in general and increase their utility their happiness >> but that's too generic statement we got to understand how they respond to laws how they respond to you know corporate incentives structure >> so so you're saying we still don't understand our AI agents enough it feels that way because if we did understand then we would be able to create the boundaries to >> yeah absolutely we know very little we know so little that um you know that's what becomes dangerous.

We know so little yet we want to press forward at such a pace >> as if you know uh we can't live without it. I think that's dangerous. I just came from a conference where a AI security expert was presenting and he runs a deep tech company that looks at data privacy and security and um what I learned at the panel was actually quite shocking for me because I think one of the things that he explained is how these agent agents work that they don't follow a series of correct steps. There's always a lot of guessing. AI agents work in that realm of guessing um at what is the likely outcome to the answer that I possibly am looking for. Um and that's why we don't always get the same answer even if we ask the question twice. And um so he was also saying that you know we so therefore we can't use quote to tell the agent what to do or what not to do.

We cannot control them. That's what he was actually saying. So that was his his his warning to the crowd. Um he was saying that you know all we can do is to use persuasion. We're actually just telling the agents okay I want you to do to do do this but please don't do this like please don't you know release my information to other uh invisible autonomous agents out there. Is this an accurate picture that AI is doing at this point? >> Sounds a bit like me. They always give me the same script and then I always give them different [laughter] lines each time I present it, you know, and they're like, "Yeah, please can you just try and follow the script, you know? " [laughter] >> Yeah. So, is this is this is this the correct thinking that, you know, this is how AI works? >> Um, you know, I I I don't fully agree.

>> Um, if we redo this podcast multiple times, >> yeah, [clears throat] >> we'll get different versions. >> Um, so in a way, how randomized, right? How much room we leave to the algorithm and agents is is something we design. >> If we set it to be deterministic, what what is called deterministic? >> What is deterministic? >> Certain, right? There's no randomness. >> Okay, >> that's fine. But then it wouldn't come out with solutions that humans have not thought of before. If we want a better solution, we've got to let it explore.

For the longest time we talked about only the potential of AI for many years you know but now that we've seen it going a bit rogue kind of doing its own thing that's when people start worrying and you know earlier this year a cloudpowered AI agent wiped out a car rental company's entire database you know uh in Australia the Australian government was notified 3 months after its health and medical data portal was hacked by an open AI agent so these are big concerns so how do we safeguard ourselves >> uh in a way we can or at least we can do more. So, so the two things that I would mention to safeguard the economy and society against this risk is one is going back to we need to understand them right before we try to come up with solutions.

we barely understand what they are doing and second um can we play out the scenarios right in in a MBA class right we typically ask the student oh you know imagine the five scenarios when we introduce a new product right some extreme scenario some normal scenario now we do have the tools >> in AI itself >> to simulate millions of outcomes we can play out this situation without incurring the actual cost >> and once We understand that we can monitor risk better. We can develop policies. We can basically run polic various policies >> and see what are the likely outcomes in the society or in the market. I think that's new and that's actually powered by AI >> because we now currently have these so-called rogue agents and I'm just wondering how did they even circumvent these very sophisticated cyber security systems?

I mean what is in their DNA that can allow them to autonomously um circumvent u the the the guils that that you know these researchers have put around. I would like to actually go back to the example of uh the generative AI right the agentic AI um where Bill was giving the example of how the great promises of it right give it a lot more freedom autonomy it can create actually new solutions for you um just to draw a parallel example to it there are humans who do that whom we don't accept in our human community >> right so one example that I always use is to think about Adolf Ikeman >> in Germany he was working I mean he was a bureaucrat working ministry of transportation. He chops the passport and the stems and the permits for all the trains that's going to the death camps.

So he obviously knows what's going on right and yet he says I followed the rules and the rules are being the ethical rules to my fur right at that point in time. So to him he's actually following good rules at times he was very creative as well right when the war was towards the end where there was no need to you know get all his KPI achieved right for the final solution he was very creative about it how do I conduct death marches so that you know I could wipe out more Jews >> um we don't accept people doing this to justify the end >> with any means possible why are we accepting a gentic AI I generate AI, right? Going rook in a way to achieve certain end goals that we may have set or maybe even for them to set their end goals. >> Um, you know, I I agree with uh Jennifer.

AIS could be capable very very good and all that >> but if they are not aligned with what humans want them to do >> that's problematic. >> So we need to do better governance of the technology to to better align it with humanity with our values. Who's going to define the values for humanity? I actually don't think it's going to be the computer scientists or data scientists. >> It's the humanities uh humanity scholars, artists um you know novelist >> they are the social science the arts >> uh side are going to collectively define human values and once we align the technology or agents yeah agents with human values that's more promising.

But you said once we align once we are still on the way there we are >> we have to teach them human values >> and and you also mentioned you know how did they get around the guarant because there's very little guar >> we didn't understand we didn't put the guar was full the earlier web twobased system no autonomous agents >> okay was literally too we have no infra >> that's why um infrastructure that's why I was saying you know we've got to build this simulator to to to develop this infrastructure um So that's that's quite important. >> So you're saying that it is possible to teach them the values that we hold very dear and when we do that AI agents will follow our rules. >> At least we can do more, we can do better, >> right? >> With with little cost.

We don't have to wait >> for the mistakes and and you know the disasters to happen before we react. >> We can play out the scenario. >> I see what you mean. So it's almost like let's do a case study. Let's say this happens and then we get the AI involved and then we all figure out how the AI reacts >> and then we can put up more guard rails. >> I mean definitely we can't rely on the conscience of researchers you know sounding off the doomsday message right nor can we just you know rely on the goodwill of companies um you know doing timely reports and doing remedial actions. I think there can definitely be more collective action right to put something in place certain standards, certain norms, certain behaviors, right, that agents should have with each other with the human involved as well >> so that everyone is aligned.

>> Godfather of AI Jeffrey Hinton, he also said and I quote, "AI has now reached a point where AI is designing better AI and is getting out of control unless we need to do something. " And he said, "We need to slow down. " So you know the people who are creating AI have sounded that warning are saying that the fence is too low but we all jump we are we are all you know just embracing it like as if we have gotten a hang of it. So when when you have these you know people who are sounding the alarm >> how do we slow down right now? >> I think society hasn't come to terms with it yet.

you know there's quite a big disconnect and a gap right between say our everyday users like you and I right versus what really is happening out there among the researchers those who know it right those who are working with the technology to us it's just convenience right to us it's be just being more efficient being optimizing whatever resources and time we have uh but we don't think about our everyday practices values people that we have who we are that makes us humans for example that has been compromised slowly. That's one. But two, also the risk that we're opening oursel up to some of this technology is not aligned to our values. Then when it's too widespread, it becomes very costly to regulate and govern as well. >> So sometimes we don't know until we've gone there. >> Yeah.

So this [laughter] is what we call the >> but it's scary because you know should we be embracing a future where we where we don't we cannot even keep up with the system and this system is going to you know be faster than than what we possibly could imagine because you're saying that we have to war game this you know and but then we may not be able to imagine that those you know scenarios and even maybe AI may not be able to do that now um it can do that for in the future but it sounds like a very scary future. >> So can we get AI to stop AI? M should we do that right? >> We are definitely uh doing that in a way at least in the research labs and all that.

I do think the governments >> and also media play a very important role in you know guarding against all this risk >> um because um the voice of technologist entrepreneurs are well heard >> they are respected they are they are traded after they get the attention um people like uh you know neutral researchers who don't have a ulterior motive or incentive they just care about truth or um you know uh people from social science field who doesn't make a lot of money um by you know these interventions um if we find a way to get their voice heard if we find a way to figure out the risks and dangers of the technology I think that's when the society start to realize okay maybe we need to >> u you know take a more careful step >> how big a risk do rogue AI agents pose here in Singapore when you know businesses or even the public institutions are adopting it at such you know great speed.

>> So you know our uh research group has been talking to the monetary authority of Singapore and the banking association in Singapore. Um I actually do think Singapore is taking a very proactive role in guarding against many of the potential risk. Um it's still in early stage in a way but um I think it's ahead of many other countries in that regard. Mhm. >> Of course, it doesn't get a lot of attention in the public because what gets more attention is the job displacement, right? >> Um, you know, do people get employed? Are they losing job? Um, but behind the scene, >> the regulators are doing a lot of work and also researchers at various labs and uh groups are also uh putting in uh quite a bit of effort. >> I guess the question is is it enough?

I mean, and we know Singapore has been pushing AI and saying, "Let's let's get on, let's get it into our system and get going with it. " So, I mean, are we are we rushing into it without having enough guard rails as well? I mean, >> are we leaving ourselves more vulnerable by going into it too quickly? >> Um, I actually think we don't have alternative choices. >> Okay. >> If we don't go in it, others will go in it and we're forced into it. We're just a follower. >> We're behind.

So taking a proactive approach is good especially when there are so many companies global companies in Singapore especially when this represents a great opportunity for the country in the following sense right we typically are limited by our energy land and population and this is the first time in the digital economy Singapore is not limited by population because we can train >> billions of very professional AI agents >> that create a lot of GDP P improves the efficiency of work. So, so, so given that opportunity and given the fact that if we just stay behind and not doing anything, we'll be eventually forced into doing this anyway just as a follower. >> I think uh Singapore is actually heading in the right direction.

>> So, we want to create billions of these AI agents, but what types of safety gut reels should we be putting in place to make sure they do that they don't go rogue, Jennifer? H so we kind of mentioned about the ethical guard rails being too low right that we probably need to >> well regulations need to be put in place governance need to also pick up the pace right uh we can also see ethics always well in a way playing catch-up right the humanities and social sciences will always play catch-up because we take longer to make observations do the study without research grants right and then have some findings and if we are ever heard um there may be some things might change right so that's one two of Of course, education of the populace out there is going to take time.

I mean, media is doing a great job in that space, which is why as educators, you know, we're always very willing to speak to the media about this because then it can amplify, right, what society should think about certain questions that we should raise as well uh to make us a little bit more resilient in that sense. But of course, those conversations need to be sustained. Do we also have to put a hardline and say that there will be certain information that will we will never let AI touch? Say for example financial statements, your CPF records, your SN pass, you know, everything is connected via Sing pass. So is this an example of a safety guard rail that we should be putting in place? >> Yeah, I think that that is one.

um two probably having a hard line about you know drawn in the sand about where AI autonomous AI should go into certain areas that it should or should not go into I mean personally I would think anything with an impact on life and death >> or harm for example I mean there are >> so hospitals let's say hospitals >> and a surgery table >> defense um so I do have concerns about certain areas uh you know where AI are fully autonomous in making these life and death decisions Um, I think there are certain areas maybe autonomous AI shouldn't get into, but maybe AI to help and assist. So, the idea of it being a tool rather than an agent. >> So, you mentioned defense, right? So, could AI agents potentially start a war if we, you know, if it runs, >> if it runs ro and we include it into our defense structure, could it potentially happen?

I wouldn't deny that that is a potentiality. Um, at this point in time, the drone weapons that's being deployed out there could identify target on its own, right? It could also execute and fire on its own without additional commands, >> right? So, for a long time, the debate has been should humans still keep an oversight uh should humans still be the one who press that critical button at the end of the day. Um but a lot of these autonomous weapons that has been developed doesn't actually have that human oversight or human the loop anymore. Doesn't need to because it needs to react quickly. >> Yeah. >> So how realistic is that will because we can draw it in sand, we can draw it in concrete, >> but the AI will still figure out a way around it if it really wants to, right?

And we've allowed it some access which means that its foot is in the door. >> Um on the one hand, I completely agree with Jennifer, right? There are certain things that we shouldn't just leave it to AI. So what I would advocate is a sort of co-piloting program. There's always a human responsible. Um it's a little bit like you know if I run a firm I'm the CEO. Yes I can have many division has I can have CFO CO and all that but I'm the CEO. >> So I think that's fine as a tool as a agent. Uh we are aware of agency issues. We can try to get around it mitigate the negative effects. Um it's a trade-off. If it improves efficiency >> and we can make it relatively safe, we go with it. >> U but then you know there's human in the loop is important. >> Yeah.

But then we have just established that the humans can be caught blindsided especially with invisible autonomous agents. So how can the humans still remain in the >> you know sometimes I delegate something to my PhD students. I also don't observe everything. They can give me wrong answers. >> But if you trust them because they are the humans. So you trust the human behavior that they will be punished. But in this case about AI agents what if it's not caught and even when they are caught my reputation is at stake >> if I publish fake results >> right so I think it's very similar. There are always trade-offs. We do our best. We we try to understand and come up with solutions. >> But then my second remark is we have to use AI >> in this process. M >> we don't use it others are going to use it to attack us.

You know one project that we we started is a scam prevention. >> It's all AI based impersonalization of government officials. How do you fight that if you don't use AI tools to detect that? >> There's no way humans can can you know fight that. Um so in a way I think absolutely um it is needed that we use these AI tools. >> Uh but then humans must be in the loop. M >> so even if you're in the loop, you can still be in the loop but left out of the loop. [laughter] Do you know what I mean? Right. >> Absolutely. Um you know, back to my research example, right? I'm in the loop supervising my future students, but there are things that I'm out of the loop. >> Okay. >> So, I have to implement certain measures and understand my students and know how to respond to incentives and rules.

Uh with that better understanding, we can control the situation better. as I think it's similar with technology. >> We keep hearing that AI will take away some jobs but it will also create new ones. Is it [music] as simple as it sounds? >> The labor force is going to shift towards you know basically less work for us to do >> really >> and more leisure. >> So some students who get an A for example they're not certain why they got an A. Oh right then that becomes worrying for me. Right. >> Who benefits from the technological progress? Um, I'm very concerned about that. Otherwise, we will just see greater inequality.
