Tag Archives: selection

Big Data is Coming, and It’s Going to Get Ugly

Big Data

Image Credit: Camelia.boban
OK folks, in case you weren’t aware: so-called ‘Big Data’ is going to control every important aspect of our lives in relatively short order. Medical diagnosis and treatment, whether you get a loan, where you get into college, how much you pay for goods and services online, whether you get hired and when you get fired, how much you get paid, whether you get a raise or bonus, even how much water you get to use (if you live in California) will all be decided by computer. Sure, a human will be in the chain somewhere, but do you really think they’ll make a decision they’ll have to defend when they have the “…but the computer said so!” excuse as the other option? Yeah, I don’t think so either.

Why this matters, aside from the obvious, is that a (very large) subset of the US population is inevitably going to get permanently screwed. This is important, so let me say that again: a substantial number of Americans will get cut out of opportunities for work, credit, fair prices, and god knows what else, permanently, all thanks to Big Data. Why? Let me explain.

Let’s say you’re an employer who wants to spend $100,000 a year on a software developer. That’s a big chunk of change (you could buy a house for less), so you want to be careful how you invest your money. There are online services that allow you to use Big Data to identify the most productive, most reliable, and least problematic programmers in the world without even posting a job listing. Now you have a top-down list of the best developers available, and all you have to do is pry them away from your competitors with a better offer.

If you’re the poor schmo who’s only a Very Good software developer (maybe even an Excellent developer), and you haven’t already been identified by a Big Data algorithm as a Top Performer, not only won’t you get tapped for the job, you won’t even be able to apply. And you’ll have plenty of company, since about half of all people are below average on any particular skill set, pretty much by definition.

Even better: since general intelligence predicts performance on just about everything, and since (again by definition) most people have an IQ that’s just above average or lower, some folks are going to always end up toward the bottom of the list, regardless of which list it is.

What this means is that there’s a huge number of people out there who will be indefinitely hosed on everything. These people will come from all walks of life: college graduates and high-school dropouts, old and young, rich and poor, all different races and sexual orientations – None of that will matter, unless the algorithm says so. They won’t be able to get a respectable job at a decent rate of pay, an apartment lease (never mind a home loan), a credit card, a car, or even a cell phone contract, because an algorithm has rated them as a greater risk than other potential applicants. If the decision-maker does the logical thing and work from the top of the list down (which is the whole point of creating the list to begin with), these people will never get selected. For anything.

They’ll be like the kids on the playground that no one wants to pick for their team, but for games with much, much higher stakes. Sooner or later, those kids are gonna get really angry, and it’s going to get ugly. I hope we can make some good choices about Big Data before it gets that far.

Tagged , ,

Google’s Brilliant Employment Strategies

Hiring is a difficult and laborious process, generally because it’s very hard to determine which particular person will actually perform well on the job. (I spent years studying this in grad school, and have spent years implementing it in my office. Believe me – It’s very, very hard.) However, research suggests that there’s one trait that universally predicts job performance better than any other, and that’s general intelligence. Basically, the smarter you are, the better you’re likely to do on performance ratings. As such, companies that select staff based on how smart the applicants are will have employees with higher performance ratings, and if performance ratings are correlated with the marketplace performance of the company, hiring and retaining smart people should result in a stronger bottom line.

Consider for example how Google goes about getting new crew members: They hire brilliant people. Not only do they want to know where you went to school and if you graduated with a degree, they want to know what your GPA was while you were there. (Fortunately for my ego, I haven’t put in a resume.)

Picture of one of Google's self-driving cars

Credit: Google


Once they’re on board, Google keeps their brilliant people entertained with personal projects. “Google screws around with a lot of far out, even cockamamie projects — well, because it can and it’s fun. Two good reasons that few can fault.” It’s not hard to imagine that having one out of every five days of work to devote to your own ideas would be very intellectually stimulating for someone with the brainpower to have brilliant ideas.

Google provides services on campus so that their brilliant people don’t get distracted by the everyday minutiae like getting their oil changed or their hair cut. And they require all employees to get at least 120 hours of training and development every year, to keep them abreast of what’s going on in their field. (There’s also the unlimited sick leave and 27 days of paid vacation every year. And don’t get me started on the free gourmet lunches and dinners.) You put all these factors together, and you get a gigantic company staffed entirely by brilliant people that repeatedly gets acknowledged as one of the best companies to work for in America. ‘Nuff said.


Tagged , ,