This is not how I do it

I could never get a good distribution this way, so I just read them one page at a time.

What do we know?

I am naturally somewhat of a skeptic, but it is hard to not read something and think about how smart the scientists are, and how much they know, and what will soon be possible. One of the things that is really interesting is brain research. Suppose we could know how we work?

All kinds of new tools allow researchers to "watch" your brain work. Or can they? How do they "know"?

Here some interesting reading about the difficulties in measuring, knowing and statistics.

Here is another that makes you think twice when we assume that science is a clean process that follows a straight line to the truth, getting it right along the way.

And here's a cartoon view of how this works (or doesn't work).

Cartoon with statistics

This cartoon site has quite a bit of "geek" humor, so I like it. It also has an interesting feature where if you move the mouse over the cartoon, a hidden message appears. This one has a comment about "the mother of all sampling biases." It may not seem funny, but it is a really great example. Some of his other cartoons are pretty good. This one makes me wonder about teaching statistics.

Complex software

One of the areas I have worked in is thinking about modeling complex manufacturing systems to try to analyze the cost functions that could be used to approximate the behavior of the system. This required lots of machinery, and reviewers were at a huge disadvantage, wondering what was going on with all my software. I tried to think of ways to "open" the code and the methods, but it was quite difficult. Getting the paper published was not picnic.

The global warming software debate is very similar. I don't know whose software is more complicated (maybe not mine?). Here's an article that talks about the lack of (and need for) software reviewing. Academe needs to confront this issue seriously.

Simpson's paradox

Averages seem like simple things, but not always. Consider this simple baseball example:

Tony and Joe are competitive friends and so they compare batting averages. At the All-Star break, Tony is batting .300 and Joe is only batting .290. Joe mentions that batting in the second half of the season is more important, and so he and Tony agree to compare their batting averages for the second half of the season (and only the second half). When they finally meet, it turns out that Tony batted .390 in the second half of the season. Joe did better, too, but only batted .375. Tony wins both halves of the season.

Question: who's batting average was higher for the entire year? Turns out we don't know, and it could very easily be Joe! (I'll post an example later)

What the paradox states is that averages for subgroups can demonstrate relationships that are inconsistent with averages for different subgroups or the overall averages. So Tony could win both halves of the season, but have a lower batting average for the entire season.

I do not know the details about Climategate, but it is very interesting to me, with all the statistics. Are average temperatures going up or down or whatever. Here's an article that about midway through mentions this batting average paradox. I wonder if I have a new example of the paradox involving averages.

Telling a story with a graph

This website has some interesting graphs, and I especially like this one. You have to think for a second before it makes sense, but once it does, it provides a nice explanation for some of the mess people have gotten themselves into (they believed the fiction).

Financial collapse

First, know that I am not especially savvy about financial instruments. I sort of understand puts and calls, but not some of the financial instruments that are all over the news. And you can read lots of articles from people who think these people or those people are responsible for the mess. And some that civilization is coming to an end in deflation or hyperinflation.

Second, know that I am not a stockholder in JPMorgan Chase, although I do have one of their credit cards.

But I read this (long) letter from the CEO of JPMorgan Chase to his shareholders and came away feeling like it was one of the more honest assessments I've read in a while. The part that interested me starts on page nine. I think I will come back to it in a year or two and see how Mr. Dimon's analysis has held up.

Bubble charts

New charts that are popular these days are called bubble charts. They show data over time, usually over a couple of axes by using bubbles of different sizes. Now I don't know about the source data for this page, but the graph is pretty cool. And a little scary.

Means and medians

Here is an article from the Wall Street Journal about the housing market in Detroit. Here's how the story starts:

On a grassy lot on a quiet block on a graceful boulevard stands the answer to a perplexing question: Why does the typical house in Detroit sell for $7,100?

The brick-and-stucco home at 1626 W. Boston Blvd. has watched almost a century of Detroit's ups and downs, through industrial brilliance and racial discord, economic decline and financial collapse. Its owners have played a part in it all. There was the engineer whose innovation elevated auto makers into kings; the teacher who watched fellow whites flee to the suburbs; the black plumber who broke the color barrier; the cop driven out by crime.

The last individual owner was a subprime borrower, who lost the house when investors foreclosed.

Then the article sites some statistics:

And the median selling price for a home stood at a paltry $7,100 as of July, according to First American CoreLogic Inc., a real-estate research firm -- down from $73,000 three years earlier. A typical house in Cleveland sells for $65,000. One in St. Louis goes for $120,000.

Now I understand that median house prices are the standard way of talking about what is "typical." Using the mean creates problems when most houses are of one value, but there are some outlier, high-priced houses.

But in this article, is median misleading in a different way? Did the house on Boston Blvd sell for $7,100? How many houses were sold in Detroit? Voluntarily? Can the dynamics associated with foreclosures make the median a poor choice to report?

Often we try to summarize with a few statistics, and that can be useful. But here I think there is need for more information to really understand this situation. The reporter probably had access to that data, if they wanted to report it.

Decision making when you know you will be second guessed

I teach quite a bit about structuring problems to help make decisions. Usually models can help illuminate the tradeoffs, and sometimes it becomes much more clear what the right strategy is.

But sometimes the downside of the "right" decision comes to dominate the thinking of the decision maker. This is a fundamental part of the dilemma facing the contestant on the game show Let's Make a Deal when they are offered the chance to change their original decision. Even if they are better off changing, the emotional pain of changing and then losing makes them stay with their first choice. They can hear their friends saying "You won the car and then you gave it away."

Here's an interesting article about football coaches going for it on fourth down. I am not sure about all the details of the study, but it is probably true that the thought of what the sports writers and talk radio people will say play an important part of the calculations of when to try or punt.

Statistics versus anecdotes

When you report on something and you have statistics and anecdotes, which should win? Which should be the basis for your conclusions? If you had data on a large part of the US population and then you talked to your brother-in-law, which would be more important to report?

Check out this NYTimes article about the Exodus from Facebook, and think about how statistics and andecdotes are used. Ouch.

Fun with probabilities

Here's a real life application of probability theory. It is similar to the sports betting company business model:

You get an mailing list (email, it is cheaper), and you pick a big football game. You send half the list an advisory that one of the teams will win, and you send the other half an advisory that the other will win, along with the opportunity to subscribe to your newsletter for a low, low price.

To the half of the people that got the correct prediction, you send another letter, picking another big game, and sending half the prediction that one team will win, half the prediction that the other team will win.

People who get two correct predictions in two weeks will be amazed and will be more likely to subscribe to your newsletter.

But you can split that group in half, and send out a third prediction, then split the people who got three weeks in a row right and send out a fourth letter, and so on. You might recycle some of the losers, too, so that some people see that your accuracy is 5 out of 6 weeks and so on.

Imagine getting a letter where the predictions were right five weeks in a row! What is the probability of that? This newsletter must be really good!

Or the newsletter writer knows some probability...

Dilbert economics and a decision tree

Scott Adams has an interesting blog where he tries to reason his way through some of the issues of the day (a bit more seriously than in his cartoons). He has been blogging about all of the decisions associated with building his new house, and one of the big decisions is whether or not to use solar power.

Here is a post where he recognizes that the usual analysis (decision) about solar is just a choice of yes or no. This sounds okay, because you're in the middle of building the house and have to make a decision, right?

Well, there's another branch that people often overlook - wait and retrofit solar later. Nice observation, especially since that branch may be the best one.

I also can't help but smile when I read the summary:

My new home will have solar power. It was a city requirement. I plan to brag about it to people who are passionate about the environment and bad at math.

Sometimes a really smart person talks about things

Ok, so there's lots of blogging and arguing and anxiety about health care. Most of what I read is relatively unhelpful, and some of it is downright misleading. I don't know whether the mis-leaders are doing it on purpose or not (they might not know either).

Dean Kamen, on the other hand, is a pretty smart guy. I think you'll have a different appreciation for the debate after reading this Popular Mechanics piece.

NYTimes article about statistics

I have to link to this article, For Today's Graduate, Just One Word: Statistics mostly because of this quote:

“I keep saying that the sexy job in the next 10 years will be statisticians,” said Hal Varian, chief economist at Google. “And I’m not kidding.”

Works for me. There's also a nice example at the end of the article about how just finding relationships in data is not always enough:

For example, in the late 1940s, before there was a polio vaccine, public health experts in America noted that polio cases increased in step with the consumption of ice cream and soft drinks, according to David Alan Grier, a historian and statistician at George Washington University. Eliminating such treats was even recommended as part of an anti-polio diet. It turned out that polio outbreaks were most common in the hot months of summer, when people naturally ate more ice cream, showing only an association, Mr. Grier said.

Maybe some economic modeling next time first?

Here's a good quiz. I thought of another, operations version:

You're the gate agent for Delta and the flight is overbooked by two people. You announce in the gate area that anyone who wants to take a later flight should get in line. You'll give everyone who volunteers a $1000 voucher for future travel on Delta. After you've given four vouchers, and you realize there are ten people still in line, do you call you boss to see if you can continue to give out vouchers? Or do you realize you probably should have offered $100 vouchers?

Just wondering.

Queueing theory and Atlanta traffic

Now, I cannot vouch for the details of this article, which claims that Jerks help traffic move faster and keep jams from happening - hard to believe, eh?

Well imagine that at the busy grocery store, people stood twenty feet away from the end of the checkout counter and waited for the person in front of them to leave before they started putting their food on the checkout counter. The cashier would be standing their with nothing to do during the time between customers, and we can all see that the waiting time for all the other shoppers would go up, right?

Now next time you're stuck in traffic and notice an empty piece of road in front of you. Many car lengths of bare pavement. Think about the fact that in order to process traffic, each segment of road is like a cashier waiting to "check you out" as you pass by. Cashiers with nothing to do means that there will be a longer wait for you, further back in line.

Now there is a "safe" distance between cars, so I am not advocating tailgating (we all know how badly traffic is affected by rear-end collisions). But excessive amounts of "idle road" means that everyone behind that road space is going to wait longer. So do your part to keep the road filled in.

But try not to seem like too much of a jerk when you're doing it.

Markets and medicine

There isn't a magic solution, but I do think that markets would help in bringing down medical costs. If I have insurance and can go to a doctor for free (no co-pay insurance from my company) or stop by the Minute Clinic and see someone who can give me the same care (or send me to the doctor if they are not sure) but I have to pay $20, I know I'll choose the doctor. Even though I know the social costs are much, much higher, and that by going to the doctor, I'm making it harder for the Minute Clinic to stay in business and reduce health care costs.

But the idea that someone can decide all this stuff for me, so I don't have to think about it, or worry that I made a mistake and chose the wrong thing, is pretty appealing to many (me?). So I understand the appeal of centralizing the process, either with an insurance company bureaucracy or the intervention of some government. But I also know that centralization can never work very well (information economics) and will stifle innovation (a hard to determine but real cost).

Here's a reasonable article arguing for the market solution. But I think there is a way to explain this even more simply and convincingly. I hope someone tries.

The world changes too fast

Here's a post from Wired magazine that is similar to the annual professor lists of the historical events that we share, but that happened before our students were born. These are some of the technologies that no longer exist (but that we thought were cool). I actually remember 24" data packs that held (wait for it) 5MB of data. Must have weighed 10-15 pounds each. I also remember when you needed to not only know the email address of your recipient, but also the internet nodes along the way to that person. So email addresses got long and complicated.