Vietnam: A Hacker's Report

Disclaimer: this post contains solely my personal opinions. I do not speak for my employer or anyone else.

Last December I was invited to present on behalf of Google Security Team at Vietnam Information Security Day Conference, a national security event held annually by VNISA. The organizers always pick a theme for the event, and this year can’t be more serious: Information Security and National Sovereignty. The conference has always been, however, a bad joke. It has zero technical content, all presentations are sponsors sales pitches. It's like a lost poor Vietnamese half-brother of the rich spoiled American kid named RSA Conference. One presenter spent 30' literally reading out loud the history of Samsung from his slide deck to an audience that don't-care-just-come-here-to-have-free-lunch-and-to-escape-work. At some point I zoned out and wondered why I was even there. Anyway it’s difficult to reject a free trip to Vietnam, and impossible once I’d told my SO. I also thought I might make the conference a bit more interesting.

<In earlier versions of this post there were three paragraphs in which I described a meeting with some government agency. I was asked to remove them, and I obliged because it was a confidential meeting that I shouldn't have disclosed. I retained the following paragraph because it's relevant to what I presented at the conference>

The discussion changed to information security strategy for Vietnam. I’m no cybersecurity strategist, I didn't really know what I was talking about, but I know that the country needs more and better engineers. I told them Coursera and Udacity and their online courses taught by top professors, and that Vietnam should recognize their certificates as valid credentials as soon as possible. They need to make information security more appealing to students, perhaps by holding a nationwide capture the flag competition with big prizes, or by requiring private and public organizations at certain sizes or in certain sectors to hire at least one full-time security engineer. We chat a bit more, but I don’t remember what it was about.

The next day was the conference, which was bigger and more formal than I expected. Google was one of the gold sponsors, and I was assigned to give the second presentation, right after a government VIP who gave a leadership keynote, and successfully made most of the attendees asleep at 9 in the morning. From my experience and contacts, I know a lot of companies in Vietnam are spending all of their security budget buying turn-key solutions and paying ISO 27001 consultants or PCI DSS auditors, and I want to tell them that they’re doing it wrong.

I started the presentation saying that I had nothing to sell, but I just wanted to share some experiences of how we do security at Google. I showed a photo of taviso@ and asked if anyone knows him. I saw one hand, and it was a friend of mine. I told them about Sophail, and explained why buying more devices or “APT solutions” does not necessarily result in better security. I told them about the recent disclosed vulnerabilities in Cisco ASA, F5 load balancers, and assured them that we've found vulnerabilities in anything that we've ever looked at. I emphasized that software always has vulnerabilities, and security software is just another type of software. If anything, security software is often more complex, and thus has more vulnerabilities.

This would for sure piss off the other sponsors, but it’s exactly what I wanted to do. I wanted to make enemies not friends. I quoted the CISO of Sony in a 2007 interview with CIO Magazine that he would not invest "\$10 million to avoid a possible \$1 million loss”, and explained why risk management by the numbers alone does not work, and why people should not bother with things like ITIL or ISO 27001 unless they need them to win contracts. It's like you don't need a CISSP to become a security engineer, unless your job requires it. I told them essentially what I told the gentlemen in the earlier meeting: they need to spend all of their money on training and recruiting engineers, and they need to do that now. Google won’t hire an army of full-time security engineers, if there’s a off-the-shelf solution for the security problems we’re facing. Of course not every one is Google, but more on that just in a moment.

Every one in the industry understands that breaking in a system is way easier than safeguarding it. I pointed out another asymmetry that many people are unaware of: the defenders understand their systems better than any adversaries. Once the attackers get in, they might trigger a custom trap or trigger a signal that the defenders have spent years building and tuning. No off-the-shelf solutions can do this, and having a team of competent security engineers is the only way to do it. Most small and medium businesses should stand on the shoulder of giants like Google or Amazon, and consider instead of building their own systems leveraging ready-to-use, usually cheaper and safer cloud services. For extremely sensitive data, they might want to use end-to-end encryption to avoid leaking data to third parties.

I thought the talk was well received. Aside from one sales representative from Trend Micro accusing me of lying to sell Google cloud services, many people greeted me afterward and said the presentation was eye-opening for them. I was invited to meet with the security team at Viettel, the largest communications corporation owned by the Ministry of Defense. While I was talking to Viettel, a uniformed man interrupted us, and asked me to have a private conversation with him. We walked outside the main conference room, the man introduced himself as an officer at the Information Security Agency at the Ministry of Defense. I thought I was in trouble, because of things that I wrote on this blog. 

The officer said that he thought the talk was interesting, but immediately got down to business and asked if I can help him identify the bloggers behind some blogs on blogspot.com(!?) “They just posted a few things, then disappeared forever. We’ve tried to track them down, but no luck. Can you give us their IP or email addresses?” I explained that I don’t have the authority to answer this question, but if he emails me I’ll route it to the right person. I said that it’s unlikely that Google will do anything, as “their culture is different from ours.” I don’t know why, but I wanted to pretend that even though I can’t do anything I’m on their side. Luckily, it’s the only question that he had. I went back to the main hall, but it wasn’t more than 5 minutes before I was asked by another uniformed man to have a private conversation with him. He’s another officer at the Ministry of Defense, and he asked me exactly the same question. I played the same trick, and he let me go after explaining that this is a matter of national security, and as a Vietnamese it’s my responsibility to help him. It's not the last time during my trip I was asked to fulfill my citizenship duty by ratting out my own people.

A less serious but funnier encounter was with an engineer, friend of friend, working at the Information Security Agency at the Very Important Ministry that I don't want to call out to protect their identity. Given the sensitive nature of her system, it’s a top secret that the engineer has relied solely on a friend of mine, who is not an employee at that ministry, on everyday operations security. “I’ve got a lot of guidances from the information security agencies at other ministries, but I won’t trust them much because they take themselves too seriously. Everything they sent was stamped Top Secret, even if it was a copy of Microsoft’s announcement that Windows XP was no longer supported”. They told me that their agency has bought all the vulnerability scanners ever made, but they have no idea what they would do with them. “My boss always wants to buy concrete things. It’s easier to convince management that we’ve spent the money well by buying hardware, than doing intangible things like training the staff, but, most importantly, it’s the best way to get kick back”.

So it isn't that we don't have money, but the core of the problem is as old as humanity: people with power are corrupted. Given the raising tension against China, these people, whose one job is protect the country's information infrastructure, see nothing but an opportunity to pocket themselves a handsome share of government money. Most of the colluding vendors, sadly, are from the U.S. One source told me that the standard kick back rate is as high as 30%, not including expensive parties or sexual bribes. I very much want to help defend my motherland, and I know I'm capable of making meaningful contributions, but the mere thought that I have to deal with these people makes me feel sick. Not that had anybody asked me, I think they just don't care that much, or, perhaps, they just don't trust me.

The last interesting bit that I’ve learned was how VCCorp got hacked. The company runs the country’s largest display ad network, and has an interesting portfolio of Internet products and services. They got hacked so bad that for several weeks during October and November most of their websites were down. A few reliable sources told me that the attackers had compromised a web cache proxy at the largest, state-owned ISP, and replaced the cached copies of Adobe Flash on Windows and Mac OS X with trojaned versions. Laptops and workstations of VCCorp’s sysadmins were infected, and from there everything else was hacked. What interesting was A68, code name of the very powerful Information Security Agency at the Ministry of Public Security, told the victim and everyone involved that the incident was a matter of national security. They seized control of all evidences, and forbade any public disclosures, even if millions of people are probably still infected. Why? I think they want to hide something. At any rate, if you travel to Vietnam, never install anything.

The last two weeks in my trip I spent most of my time organizing TetCon Saigon 2015. The conference was a huge success, we attracted 400 attendees and a nice line up of speakers. Organizing a conference is tiresome, raising money, selecting talks, talking to partners, arranging speaker details, etc. all that take a lot of precious time that I'd rather spend with my family. Anyway what should be done needs to be done, and hopefully something good will come out of it. I was and am lucky with my career. I've got chances to see and work on wonderful things, and I take it as my duty to help some of my comrades get the chances that I've got.