24 Sep 2015

Volkswagen and their emission cheating software

Everyone these days is talking about Volkswagen and how they made a software that cheated in vehicle emission tests. Volkswagen's stock price is tanking, CEO has been asked to resign, EU bureaucrats are looking into it and other major engine manufacturers are being investigated as well.

Let me give my opinion on all this affair.

How did they do it?

Quote from EPA violation notice sums it up well enough:

The 'switch' sense whether the vehicle is being tested or not based on various inputs including the position of the steering wheel, vehicle speed, the duration of the engine's operation, and barometric pressure. These inputs precisely track the parameters of the federal test procedure used for emission testing for EPA certification purposes. During EPA emission testing, the vehicles' ECM ran software which produced compliant emission results

So, they added a piece of code to vehicles' ECU block that was able to detect testing mode and then adjust engines' operating parameters. It's very similar to what ECU tuning shops do, except Volkswagen did it to reduce emissions in certain cases and petrolheads do it to achieve best possible performance from their cars.

Whose decision was it?

Some dude on hackaday sees a big ethical issue here:

An engineer, either in Volkswagen or less likely at a subcontractor, signed off on code that would defeat the entire purpose of EPA and Clean Air Act regulations. Someone with the authority to say ‘no’ didn’t, and this code was installed in the electronic control unit of millions of cars.

Say what?

This dude apparently knows nothing about how corporations work. There is no way in hell that some engineer came to his boss and said: "Hey, I just figured out a way to cheat in USA emission tests, do you think it will be useful for our company?".

No. Fucking. Way.

I'm convinced that this decision came from the middle management and was passed down to engineers. Something like: "We don't care how you do it, just make sure our diesel engine passes those tests. Just don't tell us how you managed that." Plausible deniability, you know.

However, dude from hackaday is absolutely right in another aspect - some engineer will likely lose his job over this. It's not because he did something wrong, it's because the company needs a scapegoat. Just like they sacrificed Chief Executive Martin Winterkorn - CEO had nothing to do with a scandal, it's just one of those steps company needs to do to make a good PR.

How did they get caught?

As strange as it sounds, they got caught by accident. International Council on Clean Transportation (ICCT) wanted to convince European bureaucrats to implement strict US standards for diesel emissions in EU. So, they hired West Virginia University’s Center for Alternative Fuels, Engines and Emissions (CAFEE) to run tests in the field. And as interim director of CAFEE explains:

They rented VW diesels, measured their tailpipe emissions on the road and compared them to measurements on the same cars made in the lab. The discrepancies were huge.

So, the scientists made some presentations in 2014, published their research online, and nobody except USA bureaucrats cared about it. Until last week, that is.

Now suddenly everybody is acting as if the world is going to be destroyed by this.

So, how bad it really is?

Let me answer this question with a quote from the original EPA news release:

These violations do not present a safety hazard and the cars remain legal to drive and resell. Owners of cars of these models and years do not need to take any action at this time.

I'll give you a moment to think about that.

482'000 cars in USA alone. 11'000'000 cars in the whole world. 5 years. Exceeding NOx limits 20 times. Affected cars are not a safety hazard. USA cities are not covered in black smog. In fact, nobody noticed anything for 5 years. What does it tell you?

To me, the answer is simple - those NOx limits are fucking bullshit. They make your car more expensive and reduce horsepower of your engine. They don't save the planet. They are there because some bureaucrat needs to justify his puny existance in some environmental agency.

Don't get me wrong - I do care about the environment. But you are not helping the environment much by limiting already small emissions of NOx. Instead, you should rather look at the Asia and their industrial practices. For example, burning down forests in Sumatra - which produce so much smoke that the entire Singapore city (being 80 kilometers away from Sumatra!) has its air quality deteriorating to a "very unhealthy" range. Or look at the half of China's rivers which are polluted with industrial waste and fertilizers. Now, that is something that actually needs fixing!

To sum it all up

Volkswagen knew these regulations are bullshit and won't save the Earth. They knew their engines can't pass them. So, they had balls big enough to give all bureaucrats the finger and cheated their way through.

I say - good for them! In my scorebook it's "Volkswagen 1, Bureaucrats 0".

23 Sep 2015

Why do most antiviruses suck?

Mandatory disclaimer - all views in this article are my own and in no way represent views of my employer or my coworkers.

Last few weeks I noticed several gposts about antiviruses, False Positives and how bad the situation is. For example, this essay from atom0s and this complaint (reg required) by mudlord. And then there is this epic rage by evlncrn8. smile

To understand why antiviruses work this way, you need to consider plenty of factors. So, let's take a quick look.

Why make antiviruses?

It usually starts with a group of skilled guys wanting to save the world. They make a great product, people like it, company makes some money, more people like the product, company grows even more and so on..

But as company grows, priorities change. The bigger and more popular the company gets, the more managers and investors it attracts. Those guys usually have no clue about technology behind antivirus. And they don't care about technology, they only see numbers and dollar signs everywhere.

And then the primary goal of company changes to making profit for shareholders.

What's with the UI?

Let's face it - readers of my blog are not the usual antivirus users. Antiviruses are used by everyone - from extremely skilled IT geeks to Granma Millie living in the retirement home. And this causes second biggest problem - big companies cannot make product just for skilled IT geeks, as nobody else will be able to use it. You can't make a product for the average user either. You need to make something that even Granma Millie can use.

And that's why most software products in recent years get dumbed-down - managers think that they need to do "inclusive designs" - so even the most retarded of users can use the product.

New shiny features.

One of the most common complaint I hear is that all antivirus products are becoming a huge bloatware. There are several reasons for that. First, product managers just don't know any better.They look at all competitors - if Company A has feature X, you need to have feature X, no matter if it actually makes sense or not. Second reason is that company somehow needs to sell new version of product. You can't say - this version is the same as the old one, we just changed colours and moved buttons around. No, you need to have something like "New version, now with features Z and Q!"

It's not the best way but it's certainly the easiest!

AV reviews and tests.

When you are purchasing a new car, you probably search for the reviews online. You probably do the same when you decide to move to new city, plan your vacation or make any other big decision. That's just normal.

And it's the same with antiviruses - most people will either get a recommendation from someone they trust, or they'll search for reviews online. So, the companies need to invest a lot in PR and make sure their product looks good in tests and reviews.

Testing methodologies most of the times are not representative of any real-life experience of ordinary users. Testers take whatever pieces of malware they can find and test AV products against them. They don't distinguish between different types of malware, sample prevalence or geographical distribution.

I'm sure you feel much safer knowing that your antivirus protects you against a worm that is distributed only through Chinese QQ messenger, or that very nasty banker attacking only Brazilian banks. Don't you?

To test False Positive rate, testers check number of files from popular download sites like CNET, Softpedia or PCWorld, or collected from European SMB companies. Of course, AV companies do the same thing and try to make sure they have no false positives on those sites. But if you're a small software dev and distribute your software using other means, or don't target SMB companies - well, bad luck. False Positive on your file doesn't influence test results. smile

It's a load of crap - but every company is still doing it because lots of potential users rely on such "tests" before buying antivirus. Some companies even cheat in tests.

Automation and big data.

Number of new malware and other crap these days is increasing exponentially. According to McAfee Quarterly Threat reports, ~4 million new malware samples appeared in the Q1 2009, ~7mil in the Q1 2012, ~32mil in Q1 2014 and ~48mil in Q1 2015.

Think about it. How can you process 48'000'000 samples?

The answer is simple - automation, automation and more automation. Malware classification is hugely automated process. Does the file look weird? Does it do weird things? Was it sent out in a spammy email? Is it encrypted to prevent automated analysis? Was it protected using stolen Themida? Do other antiviruses think it's bad? Game over, classified as bad!

Sure, sometimes some legitimate software gets classified as bad. In this scale, it's bound to happen.

If automation is not able to classify file, malware researchers will need to analyze it manually. This is where big data software, statistical models and cluster analysis come in. They alert researchers to traffic anomalies, suspiciously similar thousands of files and other "interesting" stuff. Files get prioritized based on prevalence, number of users affected and other factors. And, of course, the bigger the issue, the faster it gets attention from a real human being.

So, if your legitimate software is classified as bad and it affects all your 50 users - it's not because AV company hates you or your product. Really, they don't hate you. They just don't know you even exist. So, the sooner you let the AV company know about the problem, the sooner they will fix the issue.

But hiding your head in sand and saying "I don't have to time to play a cat and mouse game with anti-virus companies" will get you nowhere.

Are we all doomed?

Think about the points I just made. Your product needs to bring company money. You need to make a product Granma Millie can use. Your product needs to behave well in tests. Given the requirements, no matter how skilled the developers and researchers are, the end product will be...

Well, it will be just like the product you're getting now - dumbed-down, feature-bloated money-making piece of software that fares reasonably well in artificial tests.

You're living in the era of globalization and money-making corporations. Deal with it.