Category Archives: Technology

Big Data is Coming, and It’s Going to Get Ugly

Big Data

Image Credit: Camelia.boban
OK folks, in case you weren’t aware: so-called ‘Big Data’ is going to control every important aspect of our lives in relatively short order. Medical diagnosis and treatment, whether you get a loan, where you get into college, how much you pay for goods and services online, whether you get hired and when you get fired, how much you get paid, whether you get a raise or bonus, even how much water you get to use (if you live in California) will all be decided by computer. Sure, a human will be in the chain somewhere, but do you really think they’ll make a decision they’ll have to defend when they have the “…but the computer said so!” excuse as the other option? Yeah, I don’t think so either.

Why this matters, aside from the obvious, is that a (very large) subset of the US population is inevitably going to get permanently screwed. This is important, so let me say that again: a substantial number of Americans will get cut out of opportunities for work, credit, fair prices, and god knows what else, permanently, all thanks to Big Data. Why? Let me explain.

Let’s say you’re an employer who wants to spend $100,000 a year on a software developer. That’s a big chunk of change (you could buy a house for less), so you want to be careful how you invest your money. There are online services that allow you to use Big Data to identify the most productive, most reliable, and least problematic programmers in the world without even posting a job listing. Now you have a top-down list of the best developers available, and all you have to do is pry them away from your competitors with a better offer.

If you’re the poor schmo who’s only a Very Good software developer (maybe even an Excellent developer), and you haven’t already been identified by a Big Data algorithm as a Top Performer, not only won’t you get tapped for the job, you won’t even be able to apply. And you’ll have plenty of company, since about half of all people are below average on any particular skill set, pretty much by definition.

Even better: since general intelligence predicts performance on just about everything, and since (again by definition) most people have an IQ that’s just above average or lower, some folks are going to always end up toward the bottom of the list, regardless of which list it is.

What this means is that there’s a huge number of people out there who will be indefinitely hosed on everything. These people will come from all walks of life: college graduates and high-school dropouts, old and young, rich and poor, all different races and sexual orientations – None of that will matter, unless the algorithm says so. They won’t be able to get a respectable job at a decent rate of pay, an apartment lease (never mind a home loan), a credit card, a car, or even a cell phone contract, because an algorithm has rated them as a greater risk than other potential applicants. If the decision-maker does the logical thing and work from the top of the list down (which is the whole point of creating the list to begin with), these people will never get selected. For anything.

They’ll be like the kids on the playground that no one wants to pick for their team, but for games with much, much higher stakes. Sooner or later, those kids are gonna get really angry, and it’s going to get ugly. I hope we can make some good choices about Big Data before it gets that far.

Tagged , ,