Big Data is Coming, and It’s Going to Get Ugly

Big Data

Image Credit: Camelia.boban
OK folks, in case you weren’t aware: so-called ‘Big Data’ is going to control every important aspect of our lives in relatively short order. Medical diagnosis and treatment, whether you get a loan, where you get into college, how much you pay for goods and services online, whether you get hired and when you get fired, how much you get paid, whether you get a raise or bonus, even how much water you get to use (if you live in California) will all be decided by computer. Sure, a human will be in the chain somewhere, but do you really think they’ll make a decision they’ll have to defend when they have the “…but the computer said so!” excuse as the other option? Yeah, I don’t think so either.

Why this matters, aside from the obvious, is that a (very large) subset of the US population is inevitably going to get permanently screwed. This is important, so let me say that again: a substantial number of Americans will get cut out of opportunities for work, credit, fair prices, and god knows what else, permanently, all thanks to Big Data. Why? Let me explain.

Let’s say you’re an employer who wants to spend $100,000 a year on a software developer. That’s a big chunk of change (you could buy a house for less), so you want to be careful how you invest your money. There are online services that allow you to use Big Data to identify the most productive, most reliable, and least problematic programmers in the world without even posting a job listing. Now you have a top-down list of the best developers available, and all you have to do is pry them away from your competitors with a better offer.

If you’re the poor schmo who’s only a Very Good software developer (maybe even an Excellent developer), and you haven’t already been identified by a Big Data algorithm as a Top Performer, not only won’t you get tapped for the job, you won’t even be able to apply. And you’ll have plenty of company, since about half of all people are below average on any particular skill set, pretty much by definition.

Even better: since general intelligence predicts performance on just about everything, and since (again by definition) most people have an IQ that’s just above average or lower, some folks are going to always end up toward the bottom of the list, regardless of which list it is.

What this means is that there’s a huge number of people out there who will be indefinitely hosed on everything. These people will come from all walks of life: college graduates and high-school dropouts, old and young, rich and poor, all different races and sexual orientations – None of that will matter, unless the algorithm says so. They won’t be able to get a respectable job at a decent rate of pay, an apartment lease (never mind a home loan), a credit card, a car, or even a cell phone contract, because an algorithm has rated them as a greater risk than other potential applicants. If the decision-maker does the logical thing and work from the top of the list down (which is the whole point of creating the list to begin with), these people will never get selected. For anything.

They’ll be like the kids on the playground that no one wants to pick for their team, but for games with much, much higher stakes. Sooner or later, those kids are gonna get really angry, and it’s going to get ugly. I hope we can make some good choices about Big Data before it gets that far.

Tagged , ,

A Theory About the Increasing Incidence of Food Allergies

Most people in the US who have children (including yours truly) are concerned about the seemingly inexplicable rise in the morbidity of historically rare childhood illnesses. Autism, diabetes, obesity…Why are our kids suddenly subject to these awful conditions in increasing numbers?

Some of these ailments can be particularly troublesome, because we have to be constantly on the lookout for the triggers. Consider for example food allergies: According to the CDC, the incidence of food allergies in children rose 18% between 1997 and 2006. Popular theories as to why have ranged from them living in “germ free” environments due to overzealous cleaning (the so-called “hygiene hypothesis“), to changes in the typical child’s diet in the last forty years. These certainly have merit, however I’ve got a different idea, which was inspired by the recent article by Kevin Drum highlighting a study which showed the astonishing amount of variance in violent crime rates that can be explain by the rise and fall of the presence of lead in gasoline 23 years prior. The striking thing about this study was how events in the past had such a demonstrably strong influence on those in the future. And it turns out there’s an event in the past that occurred before the current crop of food-sensitive children were even born that might provide a useful explanation as to why food allergies are on the rise: The invention of the EpiPen.

EpiPen Autoinjector

The EpiPen was developed in the sixties and seventies by Shel Kaplan out of military research into devices for treating victims of chemical weapons attacks, and it was patented in 1977. It was made available for use in 1980, about 17 years before the meteoric rise in childhood food allergies. This means that in 1997, people born in 1980 or later were around 17 years old, which puts them roughly four years into puberty. Since we know that allergies are highly heritable, we know that children of people with allergies are much more likely to have allergies themselves. And if EpiPens allowed kids born after 1980 to survive into their child-bearing years, more of them are likely to have had kids, which would mean more kids born with allergies. Ipso facto, a rise in the incidence of food allergies, just about the time you’d expect it.

There’s some historical data to support this idea. Hospital admissions for anaphylaxis are up dramatically, which suggests more people are surviving long enough to be admitted. However, despite a reasonably extensive search, I’ve so far been unable to uncover a reliable source of historical epidemiological data on the mortality of anaphylaxis that would provide a clear indication as to whether the mortality rate did indeed drop significantly after the EpiPen became available. If I do eventually find a source, I’ll post an update here. In the mean time, please leave a comment and let me know what you think.

Tagged , , ,

Why You Should Use a Password Manager

Update 1/18/2016: There is a security flaw in LastPass’ Google Chrome extension. Don’t be fooled! You can mitigate this attack by:

  1. using Firefox with the NoScript addon (which eliminates the cross-site scripting vulnerability), and
  2. using a two-factor authentication solution that’s routed through your phone instead of your computer, like Duo or Transakt. This eliminates the possibility of a man-in-the-middle picking off the two-factor code entered into the same compromised browser session.

Update 10/16/2016: LastPass has been acquired by LogMeIn! Congrats guys! Don’t get cocky.

Update 5/13/2015: Edited for clarity, and for the change introduced in iOS 8 that allows browser extensions in Safari.

Update 5/8/2014: The Wall Street Journal ran another review of password managers.

Update 1/16/2014: Apple now incorporates a password manager, iCloud Keychain, in iOS7 and OSX Mavericks that has many of the same features as LastPass. I’ll stick with LastPass (since it’s a platform-independent solution), but Apple’s solution is probably easier to use for most folks. Here’s ArsTechnica’s coverage of it’s pros and cons.


The Wall Street Journal recently ran a review of PasswordBox, an online password manager. Many people don’t understand why using a password manager is vitally important to maintaining online account security. Consider for example Charleen Larson, who made the following comment:

“OK, so how again is this superior to me keeping a small notebook at home with my passwords? My husband knows where it is and doesn’t have to sign up for yet another account at PasswordBox. We both have too many accounts (free and otherwise) already.”

It was clear from this tidbit, as well as her replies to the responses she received, that she still doesn’t get it.

Password cartoon

Charleen, this one’s for you…

First, a password manager randomly generates passwords for you when you create a new account, and it makes them as long as you want (I use 30 characters.) It can also, of course, generate random passwords on your old accounts if you bother to change them. If you don’t understand why this is important, see this article. Random passwords for each account are especially valuable because they discourage the use of a single password across multiple sites, even for accounts you don’t care about keeping secure. Using the same password across sites is like using the same key for your office, house, car, gym locker, etc. If you don’t understand why this is important, see this article.

Second, you can use password managers from more than one machine at more than one place; you don’t have to be at home. I use LastPass, and have its browser extensions on my computer in Safari and Firefox, and in Safari on my iPhone and iPad (the same company also make an iOS app.) This is like having a copy of my virtual key ring on each device, and makes it easy for me to access my accounts securely from all of those platforms. And if I’m really desperate, I can reach my LastPass vault directly online (though I won’t do so from an untrusted machine except in dire circumstances, such as being forced out-of-town by a natural disaster.)

Third, using a password manager means you don’t have to type the password in every time you access an online account, which helps defend against keylogger attacks. If you don’t understand why this is important, see this article. (If you’re really paranoid, you can copy-and-paste your master password to avoid it getting keylogged, but then you’d better really know what you’re doing as far as security by obfuscation goes, as your password will have to be accessible somewhere in plain text format.)

Fourth, best-in-class password managers offer an optional two-factor authentication mechanism that’s linked to a physical device that’s distinct from the computer itself. If you don’t understand why this is important, see this rather eye-opening article. Use it if it’s available.

And finally Charleen, now that everyone who reads the WSJ article comments knows you keep your passwords in a handy-dandy notebook at home (hint: it’s near the computer, probably in a desk drawer), you’ve got a security issue you can’t easily fix. If you don’t understand why this is important, see this site.

Tagged , ,

Why Socially Responsible Companies Should Have Legal Status

I’ve been giving a lot of thought to “socially responsible companies” lately. A socially responsible company (SRC) has more than a simple fiduciary responsibility to its shareholders (wall-street speak for “make as much money as possible so we can line the pockets of the owners“); it also must give due consideration to the social context in which it operates. The big idea behind SRCs is to give companies the freedom to protect not only their shareholder’s wallets, but also their stakeholder’s welfare.

If publicly-traded SRCs were certified and given official legal status as such, they would have the legal protection needed to consider options other than maximizing profits without regard to the social consequences. In other words, rather than waiting to be sued for having done something wrong, dragging it out through the courts for as long as possible, then settling for an undisclosed sum while not admitting guilt or responsibility, companies could actively pursue doing something right, even if it might cost the shareholders money.

Consider for example the gambling industry; in particular, casinos.

CasinoCasinos make money when people gamble; the more people gamble there, the more money the casino makes. But the tools now exist for a casino to identify likely gambling addicts by way of analyzing the data they collect on their customers. If a casino were a registered, legally-protected SRC, it would have the protection it needed to pursue the development and refinement of these tools, so as to get the customers who might otherwise spend themselves into financial oblivion the help they need, or at least cut them off, even if doing so resulted in financial losses for the casino. This makes an awful lot of sense to me. The way the system works now, the casinos either have to A) gamble that implementing these tools will give them a competitive advantage (“Worry-free gambling here! We’ll cut you off and kick you out if the computer says you have a problem!”), or B) hope the government or the communities in which they operate don’t protest or pass laws mandating these tools, while doing nothing and continue to screw a small percentage of their customers, who are gambling their way into bankruptcy. Guess which one they prefer? (Not sure? Read the article.)


Tagged , , ,

The New Sony Walkman: A Review (Part I)

Disclaimer: I received the Meb Keflezighi Special Edition NWZ-W262 Walkman as a free gift from Sony by way of Klout. As such, you should take this review with a huge grain of salt, even though I’ll do my darnedest to be fair and balanced.

There are a lot of options out there for listening to music while you exercise, most of which involve carrying around a device to which a pair of headphones are attached via a cable. If you’re made of money (and have the right device), you can get bluetooth headphones that are cable free, but you still have to carry around the device. Sony’s solution to this duality problem is its latest generation of the Walkman, the NWZ-W262, which is the subject of this two-part review. In Part I, I’ll unbox the device and give you my impressions as I run down the features and try it at my desk. In Part II, I’ll report on how well it did after I’ve used it under various conditions in the field.


The Walkman arrived in handsome and well-constructed retail packaging, which is pleasing, if not particularly unusual.

The Walkman in its original packaging.

The first thing I did was to flip it over and look at the description on the back. What I saw was disheartening, as I’m an Apple user, and the (ridiculously long and specific) System Requirements indicated that I’d need Windows to make it work.

System Requirements

Fortunately, this turned out not to be the case, as I’ll explain later on.

Opening the box didn’t require any special tools or cutting through plastic (huzzah!). It contained a micro-B USB cable (about 2′ long), the usual array of ear gels in various sizes, a small plastic holder, the typical safety/warranty/quick-start literature, and a separate pink sheet warning that it’s water resistant, not water proof. (The IPX5 rating that it meets is for resistance to jets of water, but not total immersion; that’s IPX7 or higher. Ergo, I wouldn’t recommend using it in the pool.) There was also a nice booklet about training for running, written by Meb Keflezighi, right on top.


The booklet has a lot of information and tips that sound logical to me, but I’m not a distance runner, so I can’t really cast an expert’s eye on it. If you are a runner, however, you’ll probably find it more interesting than you’d expect. (And lest you think some guy in Sony’s marketing department actually did the writing, it’s worth noting that Meb graduated from UCLA, where he majored in communications studies with a business focus.)

The headphones themselves have a bit of weight to them that gives them ‘feel’, which I liked, though it could conceivably be a distraction (I’ll let you know in Part II.) The design is handsome, though not extraordinary – Each side vaguely resembles a bluetooth headset, with a bit of decorative flair.


The wire connecting the two halves rests the whole thing very comfortably around your neck when you take them out of your ears. This beats the dockers off of dealing with dangling earbuds and tangled cables, and I instantly appreciated why they came up with this particular product the first time I took them off.

A Note About the Systems Requirements

As I mentioned, my concerns about the system requirements specified on the box were largely unfounded. After I plugged the Walkman into my MacBook Pro, it showed up as a mass storage device in the Finder, with a folder labeled MUSIC that was pre-populated with set of mp3 files labeled “Tip1_join_a_Team”, “Tip2_Training_Routine”, and the like. (As it turns out, these files were recordings of Meb’s suggestions for being a successful runner. Having no interest in being a runner, I deleted them.) I opened iTunes, selected the songs I wanted in my library, and dragged them to the MUSIC folder in the Finder. They were automatically copied to the Walkman, and all of them played flawlessly without any further manipulation, with one exception. I did some analysis, and it turns out that one exception was copied over in Apple’s lossless ALAC format, rather than AAC, even though it was in the same type of container (MPEG-4) as some of the other songs I transferred that did play. (The box specified that the Walkman “Plays back MP3, WMA, AAC, [and] Linear PCM Audio Files”.) I used iTunes to create an mp3 version of the same song, transferred it over, and wallah! It played, though for some users, this extra step may be an annoyance. I also discovered by experimentation that I could create different playlists (also referred to as “folders” in the instructions) by creating sub-folders inside the MUSIC directory and populating them with songs.


The device has around 2GB of storage available, which strikes me as plenty for its intended purpose. I played songs from a couple of genres (rock and a capella), and the sound quality was excellent from my perspective, though I don’t consider myself to be an audiophile; i.e., your mileage may vary. The controls were very cleverly designed, with the play/pause button and a skip forward/back rocker below the right earpiece, and a playlists/shuffle button and a volume lower/higher rocker on the left.


The clever part is that the buttons are shaped differently from one side to the other, so it’s easy to distinguish by feel which is which. The rockers were also oriented in an intuitive way – Pushing the end of the rocker that’s closest to your chest increases the volume (left ear) or goes forward in the song list (right ear); pushing the end of the rocker that’s toward your back does the opposite. That’s one of those little things that makes a HUGE difference in making it easy to use, at least for me. There’s also a pre-recorded voice that tells you what’s going on when you activate some of the major features (e.g., “Shuffle on”, “Playing next folder”), which I found unobtrusive and infinitely more helpful than simple beeps.

The power button and indicator light are on the inside surface of the right ear piece; you have to remove the Walkman to turn it off entirely. (Note: the power light is off in the picture, even though the power switch is on, because the light comes on about every five seconds rather than staying on. I presume this is to conserve power.) The indicator light is supposed to change color as the batteries get low, but you won’t be able to see it when that happens without removing it from your ears, so I don’t think that’s likely to be useful; time will tell.

Power Switch


The USB socket is on the back of the right earpiece as well. The cover is easy to remove, and replaces snugly.

USB Port

Putting the Walkman on takes some getting used to, as the feel is completely different from popping in earbuds. I found it easiest to hang it around my neck as though I’d just taken it off, grab each side with my thumbs on the bottoms of the earpieces and put them in my ear canals, then flip the wire over the tops of my ears. I suspect putting it on will become second nature in short order. Once it’s on, it feels firm and steady; not loose at all.

That’s it for my initial impressions! If you have any questions, please post them in the comments and I’ll do my best to answer promptly. Look for Part II of this review after I’ve had a chance to put the Walkman through its paces in the field.


Tagged , , ,

Is Heading the Ball Dangerous?

There’s a new study that purports to show that heading the ball in soccer is equivalent to getting punched by an amateur boxer. The question that’s left unanswered, of course, is whether this causes a concussion, and/or does any lasting damage.

Unfortunately, a link to the original research paper isn’t yet available, so it’s hard to draw definitive conclusions about what the study results really mean. However, I play a lot of soccer, and I can tell you the following from direct personal experience:

Heading the ball can indeed cause concussions. I’ve had the distinct displeasure of ‘missing’ while heading a hard-kicked ball. In one particularly memorable instance, a cross-field zinger partly hit my left eye socket, I wasn’t positioned properly despite having plenty of time, and I was seeing stars and feeling woozy afterward. I’ve had enough concussions in my life (at least three that knocked me out entirely, as well as several others) to have a pretty good idea what they feel like, and this was it.

Good technique greatly moderates the effect of the impact. When I’ve headed the ball with proper technique, I’ve noticed a significant decrease in the ‘jarring’ sensation that inevitably accompanies heading a ball. Even under these ideal conditions, however, it’s currently impossible to know with certainty whether the brain is striking the inside of the skull. (Mythbusters – Are you paying attention?)

Good technique isn’t easy, and isn’t implemented effectively by everyone. To learn to head the ball correctly, you need to know what proper technique looks like, and practice it repeatedly under reasonably safe conditions. Most of us learned heading early in life from parent coaches, who may not know good technique, or who may not be able to teach it effectively. And some of us may not have the depth perception and sense of timing needed to head the ball well every time. (I certainly don’t.) This increases our odds of injury when we choose to head the ball.

With all that in mind, consider that…

It’s just a freakin’ game. I love soccer, and I’ve been seriously injured playing it, but unless you’re playing pro ball, it’s better to live with the dirty looks of your teammates than to take a chance on a poorly-executed header that might give you brain damage. And make sure you (and your soccer-loving kids) know proper heading technique.

Update 2013-3-8:  A new study shows even light soccer headers cause declines in cognitive function, at least in the short term. Long term effects are as yet unknown.

Tagged , , ,

Why Not Use Tracers in Fracking?

I know relatively little about fracking. I’m not a geologist, environmental activist, petroleum industry magnate (or employee), or hydrologist. However, I get the impression that there are a lot of people concerned that fracking is putting toxic chemicals into the water supply (among other things.) I’m still agnostic on this issue, but I would like to know with some degree of certainty why some people can ignite the water coming from their faucets, and how many wells are faulty. So why not use chemical tracers in fracking to attribute sources of pollution?

There are a number of companies that provide the needed technology. Consider for example Chemical Tracers Inc, Environmental Tracing Systems LTD, and others. If they can produce tracers that are A) uniquely identifiable, B) not naturally occurring, and C) compatible with fracking fluid mixes, they could be mixed into the fracking fluid at the well site. This would give each well a ‘fingerprint’ that would allow us to ‘see’ whether fluid from any particular well was getting into the water supply by simply looking for the chemical tracers in the water. Wallah! Instant attribution.

All of us should have a stake in implementing this idea. The oil companies want to show that their fracking wells aren’t polluting aquifers, and the rest of us want to know that fracking wells aren’t polluting aquifers. And many of us on either side of this issue are going to have grandkids living near fracking wells some day; I for one would like to know that they’re not going to get sick from one of the best national water systems on the planet.


Tagged , , ,

I’ll Take My Self-Driving Car Now, Please

I can’t wait to own a self-driving vehicle. I wish Google (and the competition coming a bit later to the game) would hurry their asses up and get them on the market. I know some people don’t like giving up the sense of control, or are worried about being involved in a massive pile-up due to getting BoD’d, but to my way of thinking, the ability to read the paper or burn through some email on the way into work is the ultimate in luxury living.


Consider for example the years I spent commuting to downtown DC on a train, the original self-driving car. Every day, I’d take Amtrak from the station in Laurel down to Union Station, right in the heart of the capitol, and I loved every minute of it. Wide, comfy seats with lots of legroom, and I didn’t have to worry about traffic conditions,  accidents, or getting there on time – In my experience, the trains were rarely off schedule. And the best part was being able to do a crossword, read the news, chat with the person next to me, or whatever else I felt like doing within the confines of the seat and public propriety. (I’d have done stuff on my iPad, but this was before they existed.)

As far as I’m concerned, the day I can relax and let a computer do the driving can’t come soon enough.

photo credit: MarkDoliner via photopin cc

Tagged , , ,

Why Teaching To The Test May Not Be So Bad After All

Despite my background in tests and measures, I’ve been virulently opposed to the whole concept of standards-based education in public schools. To my way of thinking, that approach is about teaching kids how to do well on tests, rather than how to do well in life, which are two completely different things. Worse, if the US is going to compete internationally on the basis of test results, we will lose. It’s impossible for us to overcome the phenomenal discipline parents in other cultures apply to drive their children toward test success. Americans (by and large) just aren’t that way, and that’s served us to our advantage time and time again. (For a clear, if fictional, illustration of this principle, see Kirk’s answer to the Kobayashi Maru. If you need a ‘real life’ example, look to MacAruthur’s return to the Phillipines.)

Exam Sign

However, I’ve just had an insight from some recent reading. It might, possiblymay be that teaching to the test isn’t all bad, if it gives teachers a challenge; a purpose; a result on which to focus their efforts, one which is more concrete and apparent than producing ‘a well-rounded student’.

Consider for example the profile and tweets of the Sylvania Schools’ Assistant Director of Curriculum, Julie Sanford. I realized after reading her statement that “CURRICULUM IS FUN!” (capitalization is hers) that having a meaningful educational agenda to work on, even if that curriculum is oriented toward the result of producing test-compliant kids, may supply a sense of purpose for teachers in the classroom. This, in turn, may yield the well-rounded students we’re actually trying to produce. The last time we (as a country) had such clarity and urgency in teaching, especially in the areas of science, math, and engineering, was when we were in the Space Race with the Soviet Union, and look how that turned out.

photo credit: Blue Square Thing via photopin cc

Tagged , , ,

On Newtown, and Non-Lethal School Defenses

Here’s an email I sent recently to Brad Rieger, the Superintendent of Sylvania Schools (and an all around pleasant and very smart fellow.) I thought it was worth sharing:

Hello Brad,

I’ve been giving some thought to the Newtown tragedy, and to the debate about whether to allow arming of teachers or posting of armed security guards in schools. In my opinion as a parent and CCW permit holder, there may be safer and more cost-effective means of defending a school against an armed assailant than firearms. I thought I’d run this suggestion by you to get a professional educator’s opinion, and in the hopes you’d spread this idea among your colleagues to solicit their opinions as well.

In particular, I’m thinking of devices such as the Torch, the Inferno (also see Wired’s article), the Dazzler (or an open-source variant), “less lethal” weapons or paintball markers, and others. These devices can be quite inexpensive — For example, the Torch is $199 and puts out 4400 lumens. (350 lumens is enough to blind an attacker for a second or two after exposure, even in broad daylight.) They also have other notable advantages, in particular, they are easier to aim for effect than a firearm, they require little (if any) specialized training, and they pose little or no threat of permanent injury or death to the students. If devices like these were placed in strategic locations throughout a school, the entire staff and/or student body could actively participate in their own defense against an armed intruder.

Please let me know what you think about this general idea when you get the chance. I’ll be looking forward to hearing from you.


Update: The WP put up an article a few days back on Pennsylvania schools hiring armed security guards.

Update: The Toledo Blade put up a similar article about the schools in Montpelier, Ohio arming their janitors.

Newtown Memorial

photo credit: via photopin cc