What happens after the elections?

Have you ever wondered how our politicians act once they’re elected? Do they live up to their promises? Do they just vote for whatever their party tells them to vote for or do they have a mind of their own? If so, are there Republicans out there who vote more with Democrats than with their own party and vice versa? Who is really a maverick politican? Or what states are the most polarized politically?

This post will answer some of these questions and more for the Senate of the 109th Congress. That is the Congress from January 2005 to January 2007. The data is thus a little bit dated but on the plus side we’ll have such big names as Obama, Santorum or Clinton in our list of Senators.

The idea to write this post came from the lab exercise of an online class I’m taking. The data and most of the methodology is thus due to the course organizers at Brown. The analysis and conclusions are my own.

The methdology is simple: we have data on all the votes each Senator cast. Basically, if two Senators’ votes coincide for the same question, their “similarity” is increased by one. If they’re opposed to each other, then their “similarity” is reduced by one. If one or both of them abstain, then their “similarity” does not change. For a more detailed mathematical explanation, see the end of the post.

Senators voted on 46 questions in this period, so a similarity score of 46 means perfect agreement (and no abstentions), whereas a similarity score of -46 means perfect disagreement (and no abstentions). Let us see what questions we can attempt to answer with this type of analysis:

Who is the most average Democrat and Republican? In other words, which Senators’ votes coincide most with the average of their own party. The most average Democrats are Biden (D-DE), Durbin (D-IL), Sarbanes (D-MD) and Dodd (D-CT). These people’s votes are the closest to the general consensus within the Democratic party. They are the ones best representing the average Democrat view in a sense. Their similarity scores with all Democrat Senators lie between 34 and 35.

As for the Republicans, a set of five Senators, Allen (R-VA), Bond (R-MO), Grassley (R-IA), Roberts (R-KS), Talent (R-MO), all voted the same way for everything. They are the most average Republicans too. Their similarity score with the Republicans is 39.6.

Who is the most Republican Democrat, the most Democrat Republican? In other words, which Democrats tend to vote most with the Republicans? Do they vote more often with the Republicans than with their own party? There are actually six Democrat Senators who voted more with Republicans than with their own party. Ben Nelson (D-NE), Landrieu (D-LA), Pryor (D-AR), Carper (D-DE), Bill Nelson (D-FL) and Salazar (D-CO). Ben Nelson (D-NE) actually voted more with Republicans than the average Republican did, Landrieu (D-LA) voted with them about as often as the average Republican did.

On the other hand, there are three Republicans who voted more with Democrats than they did with Republicans. Chafee (R-RI), Collins (R-ME) and Snowe (R-ME) are the ones. Collins (R-ME) and Snowe (R-ME) both voted slightly more with the Democrats than the average Democrat did.

Is current Secretary of Defense Chuck Hagel really a Democrat at heart? No, Hagel (R-NE) is a full blown Republican. He voted significantly more with the Republicans than the average Republican did (similarity score of 38.8) and less with the Democrats than the average Republican did.

Which party is more partisan? On average Democrats have a similarity score of 30.7 with other Democrats. Republicans have a similarity score of 35.9 with other Republicans on average (higher than the highest Democrat’s similarity score with their own party!). In other words, it is clearly Republicans that are much more partisan in the Senate. Democrats tend to vote more outside party lines.

Who hates his party the most? Chafee (R-RI) and Coburn (R-OK) are the Republicans who voted the least with their own party. As for Democrats, Ben Nelson (D-NE) leads the list with Lieberman (D-CT), Landrieu (D-LA) and Feingold (D-WI) also voted outside of party lines quite often. (These are the people who have the lowest similarity score with their own party, regardless of what similarity score they have with the other party.)

Who’s a maverick? Based on the data we can define maverickness in two ways:

1. Compare each Senator to all other senators (both R and D) and take the average of the similarity scores.
2. Sum each Senators’ average similarity to Democrats and their average similarity to Republicans.

In both cases, the lower score indicates more maverickness and independence. High scores indicate agreeing with the establishment all the time. Using either measure the biggest maverick is by far Feingold (D-WI, who happens to be the only Senator to have voted against the USA PATRIOT Act during the first vote on it), Coburn (R-OK, who is known as “Dr. No” in the Senate) comes in second and Sununu (R-NH, whose voting record was 100% fiscally conservative in 2005-06 according to the Club for Growth) third. The top 3 have a comfortable lead over the rest of the pack.

The supposed maverick McCain (R-AZ) comes in on 13th place with measure #1, and on 6th place with measure #2. Either position is quite good seeing as there are 99 Senators in the sample. Meaning that yes, McCain (R-AZ) is indeed a maverick albeit not the biggest one.

Who’s Mr. Establishment? By both measures it is Lugar (R-IN), Murkowski (R-AK), Smith (R-OR) and Warner (R-VA) who tend to give in to peer pressure. Well, at least their votes tend to coincide with how the average Senator votes.

How does maverickness play with party affiliation? If we assume that those with a maverickness score of less than 25 are mavericks of some sort (remember a lower maverickness score indicates that the Senator is a “bigger maverick”), we call those with a score of more than 30 “establishment types”, and anyone between 25 and 30 can be considered neither, then this is how it looks by party:

This tends to support the idea that Democrats are more likely vote outside party lines whereas Republicans tend to avoid doing that.

Which states are the most polarized politically? Here, I compare the voting record of the two Senators representing the same state. If their voting record is not similar, this would suggest that quite different people can get elected in the same state, politically speaking. This would suggest that the given state has a large number of supporters of both political sides, making the state quite polarized in this sense. This is opposed to states where both Senators seem to vote in the same manner and the electorate (or at least the majority) is thus more homogeneous politically.

Using this measure, the most polarized state is by far Iowa, then Nevada. Afterwards, the list gets a little more saturated but Oregon comes in third, Indiana fourth and Minnesota fifth. The least polarized states tend to be red states: Missouri, Georgia, Alaska, Tennessee, Mississippi and Maine lead the pack. The picture below illustrates this as well. The closer to white a state is, the less polarized it is; the closer to red/orange it is, the more polarized it is. Black means no data for the state (that’s New Jersey):

Which two Senators are the least and most alike? The least alike Senator pair is Feingold (D-WI) and Inhofe (R-OK). The two have a similarity score of -3. Matter of fact, the 10 most opposing (i.e. least alike) Senator pairs all involve Feingold (D-WI) and some Republican. As for the most alike Senator pairs, there are 20 pairs of Senators with the same score on top. All but one are Republican pairs. Their similarity scores are 46. The only Democrat pair that has this similarity score is Durbin (D-IL) and Sarbanes (D-MD).

Who are then the most alike Senators from a different party? Ben Nelson (D-NE), who is also the most Republican Democrat, has a similarity score of 44 (the third highest) with five Republican Senators. Then with a similarity score of 43 comes again Ben Nelson (D-NE) and Santorum (R-PA). And with a similarity score of 42 comes Pryor (D-AR) and DeWine (R-OH),  Johnson (D-SD) and Collins (R-ME), Specter (R-PA) and Landrieu (D-LA) and of course Ben Nelson (D-NE) with six other Republicans.

Finally, to flip the coin, who are the most opposing pairs from the same party? It is perhaps no surprise that in the Democratic party it is Ben Nelson (D-NE) and Feingold (D-WI) with a similarity score of only 4, then comes Feingold (D-WI) and Landrieu (D-LA) with 9. Only after that come the most opposing Republican pairs: Chafee (R-RI) has a similarity score of only 12 with Coburn (R-NE) and with Sununu (R-NH).

How about some “famous” pairs? Below are some selected Senator pairs and their similarity scores. The table seems to indicate among others that Obama (D-IL) is more to the left than Clinton (D-NY) is, that McCain (R-AZ) really is much more to the right than Santorum (R-PA) is, that Obama is not an extremist (after all, his similarity with McCain is much higher than that of Dodd (D-CT) or Biden (D-DE), it’s about as high as Chafee’s (R-RI)) but McCain is (his similarity scores with Democrats are extremely low, especially when compared with Santorum’s (R-PA)), and that Ben Nelson (D-NE) and Chafee (R-RI) should probably consider changing parties (note: Chafee already did).

How polarized is the Senate as a whole? One could use all kinds of analyses to determine this. But if we just look at each Senator’s similarity score with each party, one finds that even Democrats have a similarity score of ~21 with Republicans and the same applies vice versa. That is positive 21. Also, if one looks at Senator pairs as mentioned above, the most opposing Senators have a similarity score of -3. Whereas -46 would be the minimum they could have. In other words, the Senate does not seem to be polarized. They seem to agree with each other more than disagree. This is also not due to a high number of abstentions. On average, each Senator abstained about 0.8 times out of the 46 votes. So this isn’t significant.

I am not familiar with the way the Senate works in detail. So it is possible that there is ample consultation and discussion before each vote and that thus mostly only those things get voted on that have some kind of a consensus behind them. In this case, the above analysis may not be a good indicator of how polarized the Senate is as a whole.

This concludes our overall analysis. I hope you found it as interesting as I did. I would like to thank again the organizers of the Coding the Matrix course on Coursera for the data and for providing the outline of the methodology for this analysis. Note again that the above analysis pertains to the Senate of the 109th Congress, which was in session from January 2005 to January 2007. The polarization of states, the partisanship of the two major parties and the behavior of the Senators could have changed since then.

Mathematical appendix

The analysis is simply based on the notion of the dot product. To illustrate how the above numbers were calculated, let me show you an example. Consider that there are two Senators and they each voted on four questions. Let us represent “yes” votes by +1, “no” votes by -1 and abstentions by 0. Then we can arrange the votes of the two Senators in two vectors,

$\mathbf{a} = [0, +1, -1, -1],$

$\mathbf{b} = [-1, +1, -1, +1].$

By definition, the dot product of two n-vectors, u and v, is

$\mathbf{u} \cdot \mathbf{v} = \sum \limits_{i=1}^{n} u_i v_i,$

where u_i and v_i of course are the ith element of the u and v vectors, respectively. In words, this means that you multiply the corresponding elements of the two vectors (first element of u by first element of v, second element of u by second element of v, etc.) and then sum the resulting numbers. For our example:

$\mathbf{a} \cdot \mathbf{b} = 0 \cdot (-1) + 1 \cdot 1 + (-1) \cdot (-1) + (-1) \cdot 1 = 1.$

Clearly, if the ith element of either voting record is 0, then the contribution from the ith question to the above sum will be 0. If the ith element of both voting records is +1, then the ith term’s contribution to the sum will be +1. The same applies when both are -1. When the votes are opposing (one is +1, the other is -1), then their product will of course be -1, and thus that element will contribute negatively to the overall sum.

Of course if one has 99 Senators that voted on 46 questions each, it is way too time consuming to calculate these by hand. But luckily, Python (or some other language) can help us make a great number of calculations in no time. With the method above in mind, anyone with a decent knowledge of Python can do it. If you’re interested, one excellent way is presented in the Coding the Matrix course on Coursera.