Why?

"Does it mean the same thing when others look at it?"

Existing Tools

Online Palette Generator with API

• http://www.colourlovers.com/ (with colourlovers R interface)
• http://www.pictaculous.com/ (the results are nice. Yet, the API has a 500kb image size limit)

Key R Packages

• RColorBrewer by Erich Neuwirth - been using this since very first days
• colorRamps by Tim Keitt - another package that I have been using for a long time
• colorspace by Ross Ihaka et al. - important package for HCL colours
• colortools by Gaston Sanchez - for HSV colours
• munsell by Charlotte Wickham - very useful for exploring and using Munsell colour systems

Funky R Packages and Posts:

• wesanderson by Karthik Ram - love this! Give this a go and if you haven't tried it yet.
• RSkittleBrewer by Alyssa Frazee - funky Skittle and M&M colour schemes
• Further points on crayon colors by Karl Broman - another interesting set of colours!

Other Languages:

• Color Thief by Lokesh Dhakar (JavaScript)

The Plan

1. Extracting colours from images (local or online).
2. Selecting and (adjusting if needed) colours with web design and colour blindness in mind.
3. Arranging colours based on colour theory.
4. Evaluating the aesthetic of a palette systematically (quantifying beauty).
5. Sharing the palette with friends easily (think the publish( ) and load_gist( ) functions in Shiny, rCharts etc).

First function - rPlotter :: extract_colours( )

Example One - R Logo

Example Two - Kill Bill

Example Three - Palette Tarantino

Example Four - Palette Simpsons

Going Forward

useR! 2014

Acknowledgement

If you are lost and feel alone, circumnavigate the globe (For You, Coldplay)

You can not consider yourself a R-blogger until you do an analysis of Twitter using twitteR package. Everybody knows it. So here I go.

Inspired by the fabulous work of Jonathan Harris I decided to compare human emotions of people living (or twittering in this case) in different cities. My plan was analysing tweets generated in different locations of USA and UK with one thing in common: all of them must contain the string “I FEEL”. These are the main steps I followed:

• Locate cities I want to analyze using world cities database of maps package
• Download tweets around these locations using searchTwitter function of twitteR package.
• Cross tweets with positive and negative lists of words and calculate a simple scoring for each tweet as number of positive words – number of negative words
• Calculate how many tweets have non-zero scoring; since these tweets put into words some emotion I call them sentimental tweets
• Represent cities in a bubble chart where x-axis is percentage of sentimental tweets, y-axis is average scoring and size of bubble is population

This is the result of my experiment:

These are my conclusions (please, do not take it seriously):

• USA cities seem to have better vibrations and are more sentimental than UK ones
• Capital city is the happiest one for both countries
• San Francisco (USA) is the most sentimental city of the analysis; on the other hand, Liverpool (UK) is the coldest one
• The more sentimental, the better vibrations

From my point of view, this analysis has some important limitations:

• It strongly depends on particular events (i.e. local football team wins the championship)
• I have no idea of what kind of people is behind tweets
• According to my experience, searchTwitter only works well for a small number of searches (no more than 300); for larger number of tweets to return, it use to give malformed JSON response error from server

Anyway, I hope it will serve as starting point of some other analysis in the future. At least, I learned interesting things about R doing it.

Here you have the code:

library(twitteR)
library(RCurl)
library(maps)
library(plyr)
library(stringr)
library(bitops)
library(scales)
#Register
if (!file.exists('cacert.perm'))
{
  download.file(url = 'http://curl.haxx.se/ca/cacert.pem', destfile='cacert.perm')
}
requestURL="https://api.twitter.com/oauth/request_token"
accessURL="https://api.twitter.com/oauth/access_token"
authURL="https://api.twitter.com/oauth/authorize"
consumerKey = "YOUR CONSUMER KEY HERE"
consumerSecret = "YOUR CONSUMER SECRET HERE"
Cred <- OAuthFactory$new(consumerKey=consumerKey,
                         consumerSecret=consumerSecret,
                         requestURL=requestURL,
                         accessURL=accessURL,
                         authURL=authURL)
Cred$handshake(cainfo=system.file("CurlSSL", "cacert.pem", package="RCurl"))
#Save credentials
save(Cred, file="twitter authentification.Rdata")
load("twitter authentification.Rdata")
registerTwitterOAuth(Cred)
options(RCurlOptions = list(cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl")))
#Cities to analyze
cities=data.frame(
  CITY=c('Edinburgh', 'London', 'Glasgow', 'Birmingham', 'Liverpool', 'Manchester',
         'New York', 'Washington', 'Las Vegas', 'San Francisco', 'Chicago','Los Angeles'),
  COUNTRY=c("UK", "UK", "UK", "UK", "UK", "UK", "USA", "USA", "USA", "USA", "USA", "USA"))
data(world.cities)
cities2=world.cities[which(!is.na(match(
str_trim(paste(world.cities$name, world.cities$country.etc, sep=",")),
str_trim(paste(cities$CITY, cities$COUNTRY, sep=","))
))),]
cities2$SEARCH=paste(cities2$lat, cities2$long, "10mi", sep = ",")
cities2$CITY=cities2$name
#Download tweets
tweets=data.frame()
for (i in 1:nrow(cities2))
{
  tw=searchTwitter("I FEEL", n=400, geocode=cities2[i,]$SEARCH)
  tweets=rbind(merge(cities[i,], twListToDF(tw),all=TRUE), tweets)
}
#Save tweets
write.csv(tweets, file="tweets.csv", row.names=FALSE)
#Import csv file
city.tweets=read.csv("tweets.csv")
#Download lexicon from http://www.cs.uic.edu/~liub/FBS/opinion-lexicon-English.rar
hu.liu.pos = scan('lexicon/positive-words.txt',  what='character', comment.char=';')
hu.liu.neg = scan('lexicon/negative-words.txt',  what='character', comment.char=';')
#Function to clean and score tweets
score.sentiment=function(sentences, pos.words, neg.words, .progress='none')
{
  require(plyr)
  require(stringr)
  scores=laply(sentences, function(sentence, pos.word, neg.words) {
    sentence=gsub('[[:punct:]]','',sentence)
    sentence=gsub('[[:cntrl:]]','',sentence)
    sentence=gsub('\\d+','',sentence)
    sentence=tolower(sentence)
    word.list=str_split(sentence, '\\s+')
    words=unlist(word.list)
    pos.matches=match(words, pos.words)
    neg.matches=match(words, neg.words)
    pos.matches=!is.na(pos.matches)
    neg.matches=!is.na(neg.matches)
    score=sum(pos.matches) - sum(neg.matches)
    return(score)
  }, pos.words, neg.words, .progress=.progress)
  scores.df=data.frame(score=scores, text=sentences)
  return(scores.df)
}
cities.scores=score.sentiment(city.tweets[1:nrow(city.tweets),], hu.liu.pos, hu.liu.neg, .progress='text')
cities.scores$pos2=apply(cities.scores, 1, function(x) regexpr(",",x[2])[1]-1)
cities.scores$CITY=apply(cities.scores, 1, function(x) substr(x[2], 1, x[3]))
cities.scores=merge(x=cities.scores, y=cities, by='CITY')
df1=aggregate(cities.scores["score"], by=cities.scores[c language="("CITY")"][/c], FUN=length)
names(df1)=c("CITY", "TWEETS")
cities.scores2=cities.scores[abs(cities.scores$score)>0,]
df2=aggregate(cities.scores2["score"], by=cities.scores2[c language="("CITY")"][/c], FUN=length)
names(df2)=c("CITY", "TWEETS.SENT")
df3=aggregate(cities.scores2["score"], by=cities.scores2[c language="("CITY")"][/c], FUN=mean)
names(df3)=c("CITY", "TWEETS.SENT.SCORING")
#Data frame with results
df.result=join_all(list(df1,df2,df3,cities2), by = 'CITY', type='full')
#Plot results
radius <- sqrt(df.result$pop/pi)
symbols(100*df.result$TWEETS.SENT/df.result$TWEETS, df.result$TWEETS.SENT.SCORING, circles=radius,
        inches=0.85, fg="white", bg="gold", xlab="Sentimental Tweets", ylab="Scoring Of Sentimental Tweets (Average)",
        main="How Do Cities Feel?")
text(100*df.result$TWEETS.SENT/df.result$TWEETS, df.result$TWEETS.SENT.SCORING, paste(df.result$CITY, df.result$country.etc, sep="-"), cex=1, col="gray50")

This is the first of a two-part series. Part 1 sets up the story and goes into how to discover, digest & reformat the necessary data. Part 2 will show how to perform some basic visualizations and then how to build beautiful & informative density maps from the data and offer some suggestions as to how to prevent potential tracking.

Xfinity has a Wi-Fi hotspot offering that they offer through a partnership with BSG Wireless. Customers of Xfinity get access to the hotspots for “free” and you can pay for access to them if you aren’t already a customer. I used the service a while back in near area where I live (which is just southwest of the middle of nowhere) when I needed internet access and 3G/4G connectivity was non-existent.

Since that time, I started noticing regular associations to Xfinity hotspots and also indicators saying it was available (i.e. when Wi-Fi was “off” on my phone but not really off). When driving, that caused some hiccups with streaming live audio since I wasn’t in a particular roaming area long enough to associate and grab data, but was often in range just long enough to temporarily disrupt the stream.

On a recent family+school trip to D.C., I noticed nigh pervasive coverage of Xfinity Wi-Fi as we went around the sights (with varied levels of efficacy when connecting to them). That finally triggered a “Hrm. Somewhere in their vast database, they know I was in Maine a little while ago and now am in D.C.”. There have been plenty of articles over the years on the privacy issues of hotspots, but this made me want to dig into just how pervasive the potential for tracking was on Xfinity Wi-Fi.

DISCLAIMER I have no proof—nor am I suggesting—that Xfinity or BSG Wireless is actually maintaining records of associations or probes from mobile devices. However, the ToS & privacy pages on each of their sites did not leave me with any tpye of warm/fuzzy feeling that this data is not—in fact—being used for tracking purposes.

Digging for data

Since the Xfinity Wi-Fi site suggests using their mobible app to find hotspots, I decided to grab it for both my iPhone & Galaxy S3 and see what type of data might be available. I first zoomed out to the seacoast region to get a feel for the Xfinity Wi-Fi coverage:

Yikes! If BSG (or any similar service) is, indeed, recording all associations & probes, it looks like there’s almost nowhere to go in the seacoast area without being tracked.

Not wanting to use a tiny screen to continue my investigations, I decided to poke around the app a bit to see if there might be any way to get the locations of the hotspots to work with in R. Sure enough, there was:

I fired up Burp Proxy, reconfigured my devices to use it and recorded session as I poked around the mobile app/tool. There were “are you there?” checks before almost every API call, but I was able to see calls to a “discovery” service as well as the URLs for the region datasets.

The following Burp Proxy intercept shows that the app requesting data from the “discovery” API and receiving a JSON response:

REQUEST

(Host: http://datafeed.bsgwireless.com)

POST /ajax/finderDataService/discover.php HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Length: 40
Content-Type: application/x-www-form-urlencoded
Host: datafeed.bsgwireless.com
Connection: Keep-Alive

api_key=API_KEY_STRING_FROM_BURP_INTERCEPT

RESPONSE

HTTP/1.1 200 OK
Date: Sat, 31 May 2014 16:20:41 GMT
Server: Apache/2.2.22 (Debian)
X-Powered-By: PHP/5.4.4-14+deb7u9
Set-Cookie: PHPSESSID=mci434v907571ihq7d16vtmce0; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Length: 1306
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html
        {"success":true,"results":{"baseURL":"http:\/\/comcast.datafeed.bsgwireless.com\/data\/comcast","fileList":[{"id":"45","name":"metadata.sqlite","title":"Metadata","description":null,"lastUpdated":"20140513","fileSize":"11264","txSize":"3210","isMeta":true},{"id":"51","name":"finder_comcast_matlantic.sqlite","title":"Mid-Atlantic","description":"DC, DE, MD, NJ, PA, VA, WV","lastUpdated":"20140513","fileSize":"9963520","txSize":"2839603","isMeta":false},{"id":"52","name":"finder_comcast_west.sqlite","title":"West","description":"AK, AZ, CA, CO, HI, ID, MT, NV, NM, ND, OR, SD, UT, WA, WY","lastUpdated":"20140513","fileSize":"5770240","txSize":"1644518","isMeta":false},{"id":"53","name":"finder_comcast_midwest.sqlite","title":"Midwest","description":"AR, IL, IN, IA, KS, KY, MI, MN, MO, NE, OH, OK, WI","lastUpdated":"20140513","fileSize":"3235840","txSize":"922214","isMeta":false},{"id":"54","name":"finder_comcast_nengland.sqlite","title":"Northeast","description":"CT, ME, MA, NH, NY, RI, VT","lastUpdated":"20140513","fileSize":"10811392","txSize":"3081246","isMeta":false},{"id":"55","name":"finder_comcast_south.sqlite","title":"South","description":"AL, FL, GA, LA, MS, NC, SC, TN, TX","lastUpdated":"20140513","fileSize":"5476352","txSize":"1560760","isMeta":false}],"generated":1401553245}}

We can use R to make the same request and also turn the JSON into R objects that we can work with via the jsonlite library:

library(RCurl)
library(jsonlite)

# post the same form/query via RCurl

resp <- postForm("http://datafeed.bsgwireless.com/ajax/finderDataService/discover.php", 
                 api_key="API_KEY_STRING_FROM_BURP_INTERCEPT")

# convert the JSON response to R objects

resp <- fromJSON(as.character(resp))

# take a peek at what we've got

print(resp)
## $success
## [1] TRUE
## 
## $results
## $results$baseURL
## [1] "http://comcast.datafeed.bsgwireless.com/data/comcast"
## 
## $results$fileList
##   id                            name        title
## 1 45                 metadata.sqlite     Metadata
## 2 51 finder_comcast_matlantic.sqlite Mid-Atlantic
## 3 52      finder_comcast_west.sqlite         West
## 4 53   finder_comcast_midwest.sqlite      Midwest
## 5 54  finder_comcast_nengland.sqlite    Northeast
## 6 55     finder_comcast_south.sqlite        South
##                                                  description lastUpdated
## 1                                                       <NA>    20140513
## 2                                 DC, DE, MD, NJ, PA, VA, WV    20140513
## 3 AK, AZ, CA, CO, HI, ID, MT, NV, NM, ND, OR, SD, UT, WA, WY    20140513
## 4         AR, IL, IN, IA, KS, KY, MI, MN, MO, NE, OH, OK, WI    20140513
## 5                                 CT, ME, MA, NH, NY, RI, VT    20140513
## 6                         AL, FL, GA, LA, MS, NC, SC, TN, TX    20140513
##   fileSize  txSize isMeta
## 1    11264    3210   TRUE
## 2  9963520 2839603  FALSE
## 3  5770240 1644518  FALSE
## 4  3235840  922214  FALSE
## 5 10811392 3081246  FALSE
## 6  5476352 1560760  FALSE
## 
## $results$generated
## [1] 1401553861

We can see that each region (from the app screen capture) has an entry in the resp$results$fileList data frame that obviously corresponds to a SQLite database for that region. Furthermore, each one also shows when it was last updated (which you can then use to determine if you need to re-download it). There’s also a metadata.sqlite file that might be interesting to poke around at as well.

The API also gives us the base URL which matches the request from the Burp Proxy session (when retrieving an individal dataset file). The following is the Burp Proxy request capture from the iOS app:

GET /data/comcast/finder_comcast_nengland.sqlite HTTP/1.1
Host: comcast.datafeed.bsgwireless.com
Pragma: no-cache
Proxy-Connection: keep-alive
Accept: */*
User-Agent: XFINITY%20WiFi/232 CFNetwork/672.1.14 Darwin/14.0.0
Accept-Language: en-us
Accept-Encoding: gzip
Connection: keep-alive

The Android version of the app sends somewhat different request headers, including an Authorization header that Base64 decodes to csl:123456 (and isn’t used by the API):

GET /data/comcast/finder_comcast_midwest.sqlite HTTP/1.1
Accept-Encoding: gzip
Host: comcast.datafeed.bsgwireless.com
Connection: Keep-Alive
User-Agent: Apache-HttpClient/UNAVAILABLE (java 1.4)
Authorization: Basic Y3NsOjEyMzQ1Ng==

Given that there are no special requirements for downloading the data files (even the User-Agent isn’t standardized between operating system versions), we can use plain ol’ download.file from the “built-in” utils package to handle retrieval:

# plyr isn't truly necessary, but I like the syntax standardization it provides

library(plyr)

l_ply(resp$results$fileList$name, function(x) {
  download.file(sprintf("http://comcast.datafeed.bsgwireless.com/data/comcast/%s", x),
                sprintf("data/%s",x))
})

NOTE: As you can see in the example, I’m storing all the data files in a data subdirectory of the project I started for this exaple.

While the metadata.sqlite file is interesting, the data really isn’t all that useful for this post since the Xfinity app doesn’t use most of it (and is very US-centric). I suspect that data is far more interesting in the full BSG hotspot data set (which we aren’t using here). Therefore, we’ll just focus on taking a look at the hotspot data, specifically the sites table:

CREATE TABLE "sites" (
  "siteUID"               integer PRIMARY KEY NOT NULL DEFAULT null, 
  "siteTypeUID"           integer NOT NULL DEFAULT null,
  "siteCategory"          integer DEFAULT null, 
  "siteName"              varchar NOT NULL DEFAULT null, 
  "address1"              varchar DEFAULT null, 
  "address2"              varchar DEFAULT null, 
  "town"                  varchar,
  "county"                varchar, 
  "postcode"              varchar, 
  "countryUID"            integer DEFAULT null, 
  "latitude"              double NOT NULL, 
  "longitude"             double NOT NULL, 
  "siteDescription"       text DEFAULT null, 
  "siteWebsite"           varchar DEFAULT null, 
  "sitePhone"             varchar DEFAULT null, 
  "operatorUID"           integer NOT NULL DEFAULT null, 
  "ssid"                  varchar(50), 
  "connectionTypeUID"     integer DEFAULT null, 
  "serviceProviderBrand"  varchar(50) DEFAULT null, 
  "additionalSearchTerms" varchar);

You can get an overview on how to use the SQLite command line tool in the SQLite CLI documentation if you’re unfamiliar with SQL/SQLite.

The app most likely uses individual databases to save device space and bandwith, but it would be helpful if we had all the hotspot data in one data frame. We can do this pretty easily in R since we can work with SQLite databases via the RSQLite package and use ldply to combine results for us:

library(RSQLite)
library(sqldf)

# the 'grep' is here since we don't want to process the 'metadata' file

xfin <- ldply(grep("metadata.sqlite", 
                   resp$results$fileList$name, 
                   invert=TRUE, value=TRUE), function(x) {

  db <- dbConnect(SQLite(), dbname=sprintf("data/%s", x))

  query <- "SELECT siteCategory, siteName, address1,  town, county, 
                   postcode, latitude, longitude, siteWebsite, sitePhone
            FROM sites"

  results <- dbSendQuery(db, query)

  # this makes a data frame from the entirety of the results

  aps <- fetch(results, -1)

  # the operation can take a little while, so this just shows progress
  # and also whether we retrieved all the results from the query for each call
  # by using message() you can use suppressMessages() to disable the
  # "debugging" messages

  message("Loading [", x, "]... ", ifelse(dbHasCompleted(results), "successful!", "unsuccessful :-("))

  dbClearResult(results)

  return(aps)

})

I had intended to use more than just latitude & longitude with this post, but ended up not using it. I left it in the query since a future post might use it and also as an example for those unfamiliar with using SQLite/RSQLite.

The function in the ldply combines each region’s data frame into one. We can get a quick overview of what it looks like:

str(xfin) 
## 'data.frame':    261365 obs. of  10 variables:
##  $ siteCategory: int  2 2 2 2 2 2 2 2 3 2 ...
##  $ siteName    : chr  "CableWiFi" "CableWiFi" "CableWiFi" "CableWiFi" ...
##  $ address1    : chr  "7 ELM ST" "603 BROADWAY" "6501 HUDSON AVE" "1607 CORLIES AVE" ...
##  $ town        : chr  "Morristown" "Bayonne" "West New York" "Neptune" ...
##  $ county      : chr  "New Jersey" "New Jersey" "New Jersey" "New Jersey" ...
##  $ postcode    : chr  "07960" "07002" "07093" "07753" ...
##  $ latitude    : num  40.8 40.7 40.8 40.2 40.9 ...
##  $ longitude   : num  -74.5 -74.1 -74 -74 -74.6 ...
##  $ siteWebsite : chr  "" "" "" "" ...
##  $ sitePhone   : chr  "" "" "" "" ...

Now that we have the data into the proper format, we’ll cover how to visualize it in the second and final part of the series.

16 days to go for submissions in the DataVis contest at useR!2014 (see contest webpage).
The contest is focused on PISA data and students’ skills. The main variables that reflect pupil skills in math / reading / science are plausible values e.g. columns PV1MATH, PV1READ, PV1SCIE in the dataset.
But, these values are normalized to have mean 500 and sd 100. And it is not that easy to understand what the skill level 600 means and is 12 points in average a big difference. To overcome this PISA has introduced seven proficiency levels (from 0 to 6, see http://nces.ed.gov/pubs2014/2014024_tables.pdf) that base on plausible values with cutoffs 358, 420, 482, 545, 607, 669.
It is assumed that, for example, at level 6 ,,students can conceptualize, generalize, and utilize information based on their investigations and modeling of complex problem situations, and can use their knowledge in relatively non-standard contexts”.

So, instead of looking at means we can now take a look at fractions of students at given proficiency level. To have some fun we use sp and rworldmap and RColorBrewer packages to have country shapes instead of bars and dots that are supposed to represent pupils that take part in the study. The down side is that area does not correspond to height so it might be confusing. We add horizontal lines to expose the height.

And here is the R code

library(ggplot2)
library(reshape2)
library(rworldmap)
library(RColorBrewer)
map.world <- map_data(map = "world")
cols <- brewer.pal(n=7, "PiYG")
 
# read students data from PISA 2012
# directly from URL
con <- url("http://beta.icm.edu.pl/PISAcontest/data/student2012.rda")
load(con)
prof.scores <- c(0, 358, 420, 482, 545, 607, 669, 1000)
prof.levels <- cut(student2012$PV1MATH, prof.scores, paste("level", 1:7))
 
plotCountry <- function(cntname = "Poland", cntname2 = cntname) {
  props <- prop.table(tapply(student2012$W_FSTUWT[student2012$CNT == cntname],
         prof.levels[student2012$CNT == cntname], 
         sum))
  cntlevels <- rep(1:7, times=round(props*5000))
  cntcontour <- map.world[map.world$region == cntname2,]
  cntcontour <- cntcontour[cntcontour$group == names(which.max(table(cntcontour$group))), ]
  wspx <- range(cntcontour[,1])
  wspy <- range(cntcontour[,2])
  N <- length(cntlevels)
  px <- runif(N) * diff(wspx) + wspx[1]
  py <- sort(runif(N) * diff(wspy) + wspy[1])
  sel <- which(point.in.polygon(px, py, cntcontour[,1], cntcontour[,2], mode.checked=FALSE) == 1)
  df <- data.frame(long = px[sel], lat = py[sel], level=cntlevels[sel])  
  par(pty="s", mar=c(0,0,4,0))
  plot(df$long, df$lat, col=cols[df$level], pch=19, cex=3,
       bty="n", xaxt="n", yaxt="n", xlab="", ylab="")
}
 
par(mfrow=c(1,7))
#
# PISA and World maps are using differnt country names,
# thus in some cases we need to give two names
plotCountry(cntname = "Korea", cntname2 = "South Korea")
plotCountry(cntname = "Japan", cntname2 = "Japan")
plotCountry(cntname = "Finland")
plotCountry(cntname = "Poland")
plotCountry(cntname = "France", cntname2 = "France")
plotCountry(cntname = "Italy", cntname2 = "Italy")
plotCountry(cntname = "United States of America", cntname2 = "USA")

This is the second of a two-part series. Part 1 set up the story and goes into how to discover, digest & reformat the necessary data. This concluding segment will show how to perform some basic visualizations and then how to build beautiful & informative density maps from the data and offer some suggestions as to how to prevent potential tracking.

I’ll start with the disclaimer from the previous article:

DISCLAIMER I have no proof—nor am I suggesting—that Xfinity or BSG Wireless is actually maintaining records of associations or probes from mobile devices. However, the ToS & privacy pages on each of their sites did not leave me with any tpye of warm/fuzzy feeling that this data is not—in fact—being used for tracking purposes.

Purely by coincidence, @NPRNews‘ Steve Henn also decided to poke at Wi-Fi networks during their cyber series this week and noted other potential insecurities of Comcast’s hotspot network. That means along with tracking, you could also be leaking a great deal of information as you go from node to node. Let’s see just how pervasive these nodes are.

Visualizing Hotspots

Now, you don’t need the smartphone app to see the hotspots. Xfinity has a web-based hotspot finder based on Google Maps:

Those “dots” are actually bitmap tiles (even as you zoom in). Xfinity either did that to “protect” the data, save bandwidth or speed up load-time (creating 260K+ points can take a few, noticeable seconds). We can reproduce this in R without (and with) Google Maps pretty easily:

library(maptools)
library(maps)
library(rgeos)
library(ggcounty)

# you can grab ggcounty via:
# install.packages("devtools")
# install_github("hrbrmstr/ggcounty")

# grab the US map with counties

us <- ggcounty.us(color="#777777", size=0.125)

# plot the points in "Xfinity red" with a 
# reasonable alpha setting & point size

gg <- us$gg
gg <- gg %+% xfin + aes(x=longitude, y=latitude)
gg <- gg + geom_point(color="#c90318", size=1, alpha=1/20)
gg <- gg + coord_map(projection="mercator")
gg <- gg + xlim(range(us$map$long))
gg <- gg + ylim(range(us$map$lat))
gg <- gg + labs(x="", y="")
gg <- gg + theme_bw()

# the map tends to stand out beter on a non-white background
# but the panel background color isn't truly "necessary"

gg <- gg + theme(panel.background=element_rect(fill="#878787"))
gg <- gg + theme(panel.grid=element_blank())
gg <- gg + theme(panel.border=element_blank())
gg <- gg + theme(axis.ticks.x=element_blank())
gg <- gg + theme(axis.ticks.y=element_blank())
gg <- gg + theme(axis.text.x=element_blank())
gg <- gg + theme(axis.text.y=element_blank())
gg <- gg + theme(legend.position="none")
gg

library(ggmap)

x_map <- get_map(location = 'united states', zoom = 4, maptype="terrain", source = 'google')
xmap_gg <- ggmap(x_map)

gg <- xmap_gg %+% xfin + aes(x=longitude, y=latitude)
gg <- gg %+% xfin + aes(x=longitude, y=latitude)
gg <- gg + geom_point(color="#c90318", size=1.5, alpha=1/50)
gg <- gg + coord_map(projection="mercator")
gg <- gg + xlim(range(us$map$long))
gg <- gg + ylim(range(us$map$lat))
gg <- gg + labs(x="", y="")
gg <- gg + theme_bw()
gg <- gg + theme(panel.grid=element_blank())
gg <- gg + theme(panel.border=element_blank())
gg <- gg + theme(axis.ticks.x=element_blank())
gg <- gg + theme(axis.ticks.y=element_blank())
gg <- gg + theme(axis.text.x=element_blank())
gg <- gg + theme(axis.text.y=element_blank())
gg <- gg + theme(legend.position="none")
gg

It’s a bit interesting that they claim over a million hotspots but the database has less then 300K entries.

I made the dots a bit smaller and used a fairly reasonable alpha setting for them. However, the macro- (i.e. the view of the whole U.S.) plus dot-view really doesn’t give a good feel for the true scope of the coverage (or possible tracking). For that, we can turn to state-based density maps.

There are many ways to generate/display density maps. Since we’ll still want to display the individual hotspot points as well as get a feel for the area, we’ll use one that outlines and gradient fills in the regions, then plot the individual points on top of them.

library(ggcounty)

l_ply(grep("Idaho", unique(xfin$county), value=TRUE, invert=TRUE), function(state) {

  print(state) # lets us know progress as this takes a few seconds/state

  gg.c <- ggcounty(state, color="#737373", fill="#f0f0f0", size=0.175)

  gg <- gg.c$gg
  gg <- gg %+% xfin[xfin$county==state,] + aes(x=longitude, y=latitude)
  gg <- gg + stat_density2d(aes(fill=..level.., alpha=..level..), 
                            size=0.01, bins=100, geom='polygon')
  gg <- gg + scale_fill_gradient(low="#fddbc7", high="#67001f")
  gg <- gg + scale_alpha_continuous(limits=c(100), 
                                    breaks=seq(0, 100, by=1.0), guide=FALSE)
  gg <- gg + geom_density2d(color="#d6604d", size=0.2, alpha=0.5, bins=100)
  gg <- gg + geom_point(color="#1a1a1a", size=0.5, alpha=1/30)
  gg <- gg + coord_map(projection="mercator")
  gg <- gg + xlim(range(gg.c$map$long))
  gg <- gg + ylim(range(gg.c$map$lat))
  gg <- gg + labs(x="", y="")
  gg <- gg + theme_bw()
  gg <- gg + theme(panel.grid=element_blank())
  gg <- gg + theme(panel.border=element_blank())
  gg <- gg + theme(axis.ticks.x=element_blank())
  gg <- gg + theme(axis.ticks.y=element_blank())
  gg <- gg + theme(axis.text.x=element_blank())
  gg <- gg + theme(axis.text.y=element_blank())
  gg <- gg + theme(legend.position="none")

  ggsave(sprintf("output/%s.svg", gsub(" ", "", state)), gg, width=8, height=8, units="in", dpi=140)
  ggsave(sprintf("output/%s.png", gsub(" ", "", state)), gg, width=6, height=6, units="in", dpi=140)

})

The preceeding code will produce a density map per state. Below is an abbreviated gallery of (IMO) the most interesting states. You can click on each for a larger (SVG) version.

Some of SVGs have a hefty file size, so they might take a few seconds to load.

You can also single out your own state for examination:

Now, these are just basic density maps. They don’t take into account Wi-Fi range, so the areas are larger than actual signal coverage. The purpose was to show just how widespread (or minimal) the coverage is vs convey discrete tracking precision. As you jump from association to association, it would be trivial for any provider to “connect the dots”.

Covering Your Tracks

Comcast (Xfinity) and AT&T aren’t the only places where this tracking can occur. CreepyDOL was demoed at BlackHat in 2013 (making it pretty simple for almost anyone to setup tracking). Stores already use your Wi-Fi associations to track you. Navizon has a whole product/service based on the concept.

Apple is trying to help with a new feature in iOS 8 that will randomize MAC addresses when probing for access points and David Schuetz has advocated deleting preferred networks from your iOS networks list.

What can you do while you wait for iOS (and wait even longer for the framented Android world to catch up)? Android users can give AVG’s new PrivacyFix a go, but one of your only direct controls is to disable Wi-Fi, but that might not truly help if your mobile operating system does not deal well with passive Wi-Fi probes. Another option (as mentioned above) is to regularly purge the list of previously associated networks. You could even go so far as to bundle up your phone and stop all signales coming in and out, but that somewhat defeats the purpose of having your mobile with you.

Remain aware that the tracking can happen invisibly anywhere and, perhaps more importantly, the dangers that open Wi-Fi networks pose in general. Use a VPN service like Cloak to at least ensure all your transmissions are free from local prying eyes so the trackers have as little data to associate with you as possible.

Finally, keep putting pressure on the FTC to help with this privacy issue. While FTC/FCC efforts won’t stop malicious actors, it might help reign in businesses and encourage more privacy innovation on the part of Apple/Android/Microsoft.

For DataCamp, February is one of the most interesting months so-far in terms of user data, as we added two new and free online interactive courses to our curriculum: Data Analysis and Statistical Inference and Introduction to Computational Finance. Courses that are/were also used as interactive R complements to the like-named Coursera courses. In February we welcomed over 14,000 new R enthusiasts, from a total of 163 countries. Our servers handled peak traffic of 1,000 requests per minute, and hundreds of concurrent users. Other insights that you will find in the presentation are:

• Number of chapters started and finished by course
• Geographical distribution of the DataCamp user base
• Spillover effect across courses
• …

Make sure to have a look, and if you want more information send your requests to info@datacamp.com.

Sometimes you may want to plot maps of the whole world,nthat little bluenspinning sphere thensurface of which provides a home for us all.nCreating maps of smaller areasnis covered in antutorialnI helped create called ‘Introduction to visualising spatial data in R’,nhosted with data and code on angithub repository.nThere are a range of options for plotting the world, including packages callednmaps,na function called map_data fromnggplot2 package and rworldmap.

In this post we will use the latter two (newer) optionsnto show how maps of the entire worldncan easily be produced in R and overlaid with shortest-linenpaths called great circles. Amazingly, in each package, the geographicndata for the world and many of its subregionsnare included, saving thenneed to download and store files of unknown quality from the internet.

Plotting continents and great circle lines in base graphics

The first stage is to load the packages we’ll be using:

x <- c("rworldmap", "geosphere", "ggmap")nlapply(x, require, character.only = T)n

Let us proceed by loading an entire map of the world fromnthe rworldmap function getMap:

s <- getMap() # load the map datanclass(s) # what type of are we dealing with?n

## [1] "SpatialPolygonsDataFrame"n## attr(,"package")n## [1] "sp"n

nrow(s) # n. polygonsn

## [1] 244n

plot(s) # the data plotted (not shown)nbbox(s) # the bounding box... of the entire worldn

##    min    maxn## x -180 180.00n## y  -90  83.65n

The above shows that in single line of code we have loadedns, which represents the entire world and all its countries.nThis impressive in itself,nand we can easily add further details like colour based onnthe countries’ attributes (incidentally, you can seenthe attribute data by typing s@data).

Adding points randomly scattered over the face of the Earth

But what if we want to add up points to the map ofnthe world and join them up? This can be done innthe same way as we’d add points to any R graphic.nUsing our knowledge of bbox we can define the limitsnof random numbers (from runif) to scatter points randomlynover the surface of the earth in terms of longitude. Note the use ofncos(abs(l)) to avoid oversampling at the poles,nwhich have a much lower surface area than the equator, pernline of longitude.

set.seed(1984)nn = 20nx <- runif(n=n, min=bbox(s)[1,1], max = bbox(s)[1,2] )nl <- seq(from = -90, to = 90, by = 0.01)ny <- sample(l, size = n, prob = cos(abs(l) * pi / 180))np <- SpatialPoints(matrix(cbind(x,y), ncol=2), proj4string=CRS("+proj=longlat +datum=WGS84"))nplot(s)npoints(p, col = "red")n

Joining the dots

So how to join these randomly scattered points on the planet?nA first approximation would be to join them with straight lines.nLet’s join point 1, for example, to all others to test this method:

plot(s)nsegments(x0 = rep(coordinates(p[1,])[1], n), y0 = rep(coordinates(p[1,])[2], n),n         x1 = coordinates(p)[,1], y1 = coordinates(p)[,2])n

(Incidentally, isn’t the use of segments here rather clunky - any suggestionsnof a more elegant way to do this welcome.)nThe lines certainly do join up, but something doesn’t seem right in the map, right?nWell the fact that you have perfectly straight lines in the image means bendynlines over the Earth’s surface: these are not the shortest,ngreat circle lines.nTo add these great circle lines, we must use the geosphere package:

head(gcIntermediate(p[1,], p[2]), 2) # take a look at the output of the gcIntermediate functionn

##        lon    latn## [1,] 55.16 -37.47n## [2,] 53.16 -37.25n

plot(s)nlines(gcIntermediate(p[1,], p[2,]), col = "blue", lwd = 3)n# for loop to plot all lines going to zone 5nfor(i in 1:length(p)){n  lines(gcIntermediate(p[1,], p[i,]), col = "green")n}n

Fantastic. Now we have great circle lines represented on anmap with a geographic coordinate system (CRS)n(as opposed to a projected CRS, which approximates Euclidean distance).

Beautifying the map

The maps we created so far are not exactly beautiful.nLet’s try to make the map look a little nicer:

names(s@data)n

##  [1] "ScaleRank"    "LabelRank"    "FeatureCla"   "SOVEREIGNT"  n##  [5] "SOV_A3"       "ADM0_DIF"     "LEVEL"        "TYPE"        n##  [9] "ADMIN"        "ADM0_A3"      "GEOU_DIF"     "GEOUNIT"     n## ...n

library(rgdal)n

# s <- spTransform(s, CRSobj=CRS("+proj=robin +lon_0=0 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs"))nrcols <- terrain.colors(length(unique(s$REGION)))ns$col <- as.numeric(factor(s$REGION))npar(bg = 'lightblue')nplot(s, col = rcols[s$col], xlim = c(-180, 180))npoints(p, col = "red")nfor(i in 1:length(p)){n  lines(gcIntermediate(p[5,], p[i,]), col = "black")n}n

par(bg = 'white')n

Doing it in ggplot2

The ‘beautified’ map above certainly is more interesting visually, with addedncolours. But it’s difficult to call it truly beautiful. For that, as withnso many things in R plotting, we turn to ggplot2.

s <- map_data("world")nm <- ggplot(s, aes(x=long, y=lat, group=group)) +n  geom_polygon(fill="green", colour="black")nmn

When we add the lines in projected maps (i.e. with a Euclidean coordinate system)nbased solely on origins and destinations, this works fine, butnas with the previous example, generates incorrectnshortest path lines:

# adding linesn# for all combinations of lines, use this coden# p1 <- do.call(rbind, rep(list(coordinates(p)),n ))n# p2 <- cbind(rep(coordinates(p)[,1], each=n ), rep(coordinates(p)[,2], each=n ))n# for all lines goint to point 5:np1 <- coordinates(p[5,])[rep(1, n),]np2 <- coordinates(p)n# test plotting the linesn# ggplot() + geom_segment(aes(x = p1[,1], y = p1[,2], xend = p2[,1], yend = p2[,2]))nggplot() + geom_polygon(data = s,aes(x=long, y=lat, group=group), n  fill="green", colour="black") +  n  geom_segment(aes(x = p1[,1], y = p1[,2], xend = p2[,1], yend = p2[,2]))n

Adding great circle lines to ggplot2 maps

Adding great circle lines in ggplot2 is similar, but we mustnsave all of the coordinates of the paths in advance before plotting,nbecause ggplot2 like to add all its layers in one function: youncannot iteratively add to the map using a for loop as we didnin the base graphics example above.

To create the for loop, first create a data frame of a single line.nThe iterate for all zones and use rbind to place one data frame onntop of the next:

paths <- gcIntermediate(p[5,], p[1,])npaths <- data.frame(paths)npaths$group <- 1nsel <- setdiff(2:length(p), 5)nfor(i in sel){n  paths.tmp <- gcIntermediate(p[5,], p[i,])n  paths.tmp <- data.frame(paths.tmp)n  paths.tmp$group <- in  paths <- rbind(paths, paths.tmp)n}n

To plot multiple paths, we can use the geom_segment command.nBefore plotting the lines on the map, it’s sometimes best to firstnplot them on their own to ensure that everything is working.nNote the use of the command ggplot(), which initiates annempty ggplot2 instances, ready to be filled with layers.nThis is more flexible than stating the data at the outset.

ggplot() + geom_polygon(data = s, aes(x=long, y=lat, group=group), n  fill = "green", colour="black") +n  geom_path(data = paths, aes(lon, lat , group = group)) +n  theme(panel.background = element_rect(fill = 'lightblue'))n

Changing projection in ggplot

ggplot2 has inbuilt map projection functionality with thenfunction coord_map. This distorts the Euclidean axis of thenmap and allows some truly extraodinary shapes (thesentransformations can also be done in base graphics, ne.g. by using spTransform). However,nas shown in the examples below, the library is currently buggynfor plotting polygons.

# to see the range of projections available using this method, try ?mapprojectnm <- last_plot()nm + coord_map()n

# remove fill as this clearly causes problems:nm <- ggplot() + geom_path(data = s, aes(x=long, y=lat, group=group), colour="black") +n  geom_path(data = paths, aes(lon, lat , group = group)) n# m + coord_map("bicentric", lon = 0)n# m + coord_map("bonne", lat= 0)nm + coord_map("ortho", orientation=c(41, -74, 0)) # for ortho mapsn

Conclusion

We’ve seen 2 ways of plotting maps of the world and overlayingn‘great circles’ lines on them. There are probably more, butnthese two options seem to work well, except withnthe bugs in ggplot2 for plotting polygons innmany map projections. The two methods are not incompatiblen(see fortify for plotting sp objects in ggplot2)nand can be combined in many other ways.

For more information on plotting spatial data in R,nI recommend checking out R’s range ofnspatial packages.nFor an introductory tutorial on visualising spatial datanin R, you could do much worse than start withnVisualising Spatial Data in Rnby James Cheshire and myself.

Introduction

Definitions of Gulf Stream location sometimes centre on thermal signature, but it might make sense to work with dynamic height instead. This is illustrated here, using a model for , with the distance along the transect. The idea is to select the halfway point in the function, where the slope is maximum and where therefore the inferred geostrophic velocity peaks.

Methods and results

1

library(oce)n

## Loading required package: methodsn## Loading required package: mapprojn## Loading required package: mapsn

 1n 2n 3n 4n 5n 6n 7n 8n 9n10n11n12n13n14n15n16n17n18n19n20n21n22n23n24n25n26n27n28n29n30n31n32n33n34n35n36n37n38n39

data(section)n## Extract Gulf Stream (and reverse station order)nGS <- subset(section, 109<=stationId & stationId<=129)nGS <- sectionSort(GS, by="longitude")nGS <- sectionGrid(GS)n## Compute and plot normalized dynamic heightndh <- swDynamicHeight(GS)nh <- dh$heightnx <- dh$distancennpar(mfrow=c(1, 3), mar=c(3, 3, 1, 1), mgp=c(2, 0.7, 0))nplot(x, h, xlab="Distance [km]", ylab="Dynamic Height [m]")nn## tanh fitnm <- nls(h~a+b*(1+tanh((x-x0)/L)), start=list(a=0,b=1,x0=100,L=100))nhp <- predict(m)nlines(x, hp, col='blue')nx0 <- coef(m)[["x0"]]nnplot(GS, which="temperature")nabline(v=x0, col='blue')nn## Determine and plot lon and lat of midpointsnlon <- GS[["longitude", "byStation"]]nlat <- GS[["latitude", "byStation"]]ndistance <- geodDist(lon, lat, alongPath=TRUE)nlat0 <- approxfun(lat~distance)(x0)nlon0 <- approxfun(lon~distance)(x0)nplot(GS, which="map",n     map.xlim=lon0+c(-6,6), map.ylim=lat0+c(-6, 6))npoints(lon0, lat0, pch=1, cex=2, col='blue')ndata(topoWorld)n## Show isobathsndepth <- -topoWorld[["z"]]ncontour(topoWorld[["longitude"]]-360, topoWorld[["latitude"]], depth,n        level=1000*1:5, add=TRUE, col=gray(0.4))n## Show Drinkwater September climatological North Wall of Gulf Stream.ndata("gs", package="ocedata")nlines(gs$longitude, gs$latitude[,9], col='blue', lwd=2, lty='dotted')n

Exercises

From the map, work out a scale factor for correcting geostrophic velocity from cross-section to along-stream, assuming the Drinkwater (1994) climatology to be relevant.

Resources

n
• n

  Source code: 2014-06-22-gulf-stream-center.R

  n
n
• n

  K. F. Drinkwater, R. A Myers, R. G. Pettipas and T. L. Wright, 1994.n Climatic data for the northwest Atlantic: the position of the shelf/slopen front and the northern boundary of the Gulf Stream between 50W and 75W,n 1973-1992. Canadian Data Report of Fisheries and Ocean Sciences 125.n Department of Fisheries and Oceans, Canada.

  n
n

The version control system Git is an amazing piece of software for tracking every change that you make to a project and saving its entire history. It is incredibly useful, for users of R and other programming languages, leading it shoot from 0 market share in 2005 (when it was first released) to market domination in one short decade.

However, Git can cause confusion. Even (or at times especially) when used in conjunction with a nice graphical user interface such as that provided by GitHub, the main online repository of Git projects worldwide and home to over 10 million projects, Git can cause chaos. Like Linux (the operating system was incidentally created by the same prolific person), Git assumes you know what you’re doing. If you do not, watch out!

Partly knowing what I was doing (but not fully) I set up a repository to host a tutorial on making maps in R. I was pretty relaxed about what went in there and soon, the repository grew to an unwieldy 60 Mb in size and over 20 Mb just to download the automatically created zip file. (It is now a sprightly 2.6 Mb Zipped, wahey!) Needless to say this did not help my aim of making R accessible to everyone, a tool for empowerment (as this inspiring article about R for blind people shows it can be).

So I decided to act to clean things up. In the hope it’ll be useful to others, what follows is a description of the main steps I took to sort things out.

Step 1: delete files in the current project

The first stage was simply to identify and delete excessively sized files in the current version of the project. For this there is no better program than Baobab, which shows you where bloat exists on your system.

That was only part of the problem though: as shown in the image of disk usage from Baobab below, most (80%, almost 50 Mb) of the space was taken up by the .Git folder itself. This meant files I’d changed in the past were taking up the most space and. Git is not designed to allow you change the past but to save it…

Step 2: use the BGF

Next up is the BFG ‘repo cleaner’. This is just a small java program that cleans up unwieldy commits using a command line interface.

In order for it to work, you need to mirror your repository, using the --mirror flag when you clone. The first step was thus:

    $ git clone --mirror git@github.com:Robinlovelace/Creating-maps-in-R.git

Next, you run this (in a Linux terminal, as illustrated by the $ sign), changing the size depending on what you want to keep:

   $ java -jar ~/programs/bfg-1.11.7.jar  --strip-blobs-bigger-than 1M  .git

This successful cut the size of the project in half, making it far more accessible, as shown in the figure below. Note, the changes made by the BFG only translate into disk space savings after running the following commands (suggested in the BFG usage section):

    $ cd Creating-maps-in-R.git/
    $ git reflog expire --expire=now --all
    $ git gc --prune=now --aggressive

One issue

The only issue I encountered was this message:

    ! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref)

Although this was repeated several times, it didn’t seem to influence the success of the operation: I’ve halved the size of my GitHub repo and roughly 1/8thed the size of the zip file people need download to run the tutorial code. So the issue seems to be a non-issue in the grand scheme of things.

Conclusion

Ideally we’d all be like Linus Torvalds and make no mistakes. But unfortunately we are human and prone to mistakes, which are actually one of the best ways of learning. Thanks to software like BFG and many helping hands through the open source community, 99 times out of 100 these mistakes are no big deal. I hope this post will help others to shrink unwieldy git repositories and uncrustify their lives. More importantly I hope this leads to better design from the outset: the experience has certainly made me think about project design carefully including saving giant .RData files externally and keeping new objects in a project to a minimum. According to Joseph Tainter, the marginal costs of added complexity now outweigh the benefits for industrial civilization. Lets hope R users and other programmers, at the very least, can simplify our lives sufficiently to avoid collapse. Hopefully then the rest of society will follow!

Everybody loves a good map. Even if you don’t have any reason to make one, your boss will love it when you do, so check this out and get yourself a pay rise (possibly).

First, this set of diagrams via ONS Geographies on Twitter, showing how the different terminologies of UK administrative geography overlap and nest. It looks horrible, like it was made by NSA staffers after their PowerPoint refresher day, but it does the trick, and I haven’t seen this pulled together in one simple way like this before.

Second, this nice presentation from the most recent LondonR meeting, by Simon Hailstone. He shows the value of proper mapping tools inside R with some real-life ambulance service data. Croydon, Romford, Kingston, West End, OK. Heathrow Airport is a bit of a surprise.

Naming conventions in R are famously anarchic, with no clear winner and multiple conventions in use simultaneously in the same package. This has been written about before, in a lucid article in the R Journal, a detailed exploration of names in R source code hosted on CRAN and general discussion on stackoverflow.

Basically, there are 5 naming conventions to choose from:

• alllowercase: e.g. adjustcolor
• period.separated: e.g. plot.new
• underscore_separated: e.g. numeric_version
• lowerCamelCase: e.g. addTaskCallback
• UpperCamelCase: e.g. SignatureMethod

There are clear advantages to choosing one naming convention and sticking to it, regardless which one it is:

“Use common sense and BE CONSISTENT”

The Google Style Guide is ironically written in a rather inconsistent way (mixing capitals with lowercase in a single sentence surely breaks their own rule!)

But which one to choose? Read below to find out about the thorny issue of naming conventions in R, based on a tutorial on geo-spatial data handling in R.

Naming convention chaos

I recently encountered this question when I looked at the CRAN hosted version of the tutorial I co-authored ‘Introduction to visualising spatial data in R’. To my dismay, this document was littered with inconsistencies: here are just a few of the object names used, breaking almost every naming convention:

• Partic_Per: This variable is trying to be simultaneously UpperCamelBack and underscore_separated: a new naming convention I’d like to coin Upper_Underscore_Separated (joke). Here’s another example: Spatial_DistrictName These styles should not be mixed according to Hadley Wickham and Colin Gillespie.
• sport.wgs84: An example of period.separation
• crimeDat$MajorText: lowerCamelBack and UpperCamelBack in the same object!
• ons_label: a rare example of a consistent use of a naming convention, although this was in a variable name, not an object.

Does any of your code look like this? If so I suggest sorting it out. Ironically, we had a section on typographic conventions in the error strewn document. This states that:

“it is a good idea to get into the habit of consistent and clear writing in any language, and R is no exception”.

It was time to follow our own advice!

A trigger to remedy chaotic code

The tutorial was used as the basis for a workshop delivered at the Free and Open Source Software for Geo-spatial (FOSS4G) conference in Bremen. The event is affiliated with the The Open Source Geospatial Foundation (OSGeo), who are big advocates of consistency and standards. With many experienced programmers at the event, it was the perfect opportunity to update the tutorial on the project’s github repository.

Which naming convention?

We decided to use the underscore_separated naming convention. Why? It wasn’t because we love typing underscores (which can cause problems in some contexts), but because of more fundamental issues with the other options:

• alllowercase names are difficult to read, especially for non-native readers.
• period.separated names are confusing for users of Python and other languages in which dots are meaningful.
• UpperCamelBack is ugly and requires excessive use of the shift button.

There are also a couple of reasons why we positively like underscores:

• Underscores are fast to read: 10% to 20% faster than camelBack, which is especially confusing to non-native English readers, according to one article.
• Underscores are recommended by some prominent R users, including Hadley Wickham, Colin Gillespie and Andrew Gellman.

Implementing a consistent coding convention

After overcoming the mental inertia to decide on a new naming convention, actually implementing it should be the easy part. A series of regex commands could help, including the following (the ‘Regex’ tickbox must be enabled if you’re searching in RStudio):

[a-z]\.[a-z] # will search for dots between lowercase chars (period.separation)
[a-z][A-Z] # find camelBack code

Unfortunately, these commands will also find many R commands that use these naming convention, so just re-reading the code may be just as fast.

The below image shows the github diff of a typical change as part of a renaming strategy. Note in this example that not only are we implementing a consistent naming convention, we also added a new comment in this commit, improving the code’s ‘understandability’. Implementing a naming convention can be part of a wider campaign to improve your R projects. This could include adding comments, removing redundant information from large projects and reformatting code, perhaps using the formatR package.

Conclusion

It is important to think about style in writing any languages, especially if your code will be read by others:

“What could help might be to raise awareness in the R community about naming conventions; writers of books and tutorials on R could make a difference here by treating naming conventions when introducing the R language.”

In conclusion, it is lazy and irresponsible to write and maintain messy code that is difficult to read. By contrast, consistent, clear and well-commented code will help you and others use your code and ensure its longevity. Adoption of a clearly defined naming convention such as the underscore_separation adopted in our tutorial can be an easy step one can take now towards this aim.

The only question that remains is which naming convention WiLL.U_uSe!

I’m often typing the same bits of code over and over. Those bits of code really should be made into functions.

For example, I’m still using base graphics. (ggplot2 is on my “to do” list, really!) Often some things will be drawn with a slight overlap of the border of the plotting region. And in heatmaps with image, the border is often obscured. I want a nice black rectangle around the outside.

So I’ll write the following:

u <- par("usr")
rect(u[1], u[3], u[2], u[4])

I don’t know how many times I’ve typed that! Today I realized that I should put those two lines in a function add_border(). And then I added add_border() to my R/broman package.

It was a bit more work adding the Roxygen2 comments for the documentation, but now I’ve got a proper function that is easier to use and much more clear.

Update: @tpoi pointed out that box() does the same thing as my add_border(). My general point still stands, and this raises the additional point: twitter + blog → education.

I want to add, “I’m an idiot” but I think I’ll just say that there’s always more that I can learn about R. And I’ll remove add_border from R/broman and just use box().

I was browsing the Social Security Administration’s website and found a link for the open government initiative (http://www.ssa.gov/open/data/).  There seems to be a fair amount of interesting data here, but I grabbed the names of people born in the US since 1910 (http://www.ssa.gov/oact/babynames/limits.html).  Each state has a data file that lists the number of births under a given name by year in that state and the gender of the child.

There’s a lot of interesting analysis that could be done with this data, but I’m going to start by just plotting the most popular name by state by gender across the entire dataset (after 1910).

Here is the plot for males:

We can see that John is most popular in the Mid-Atlantic (PA, NY, etc.)  Robert is most popular in the Midwest and the northeastern states.  James dominates large portions of the South while Michael is most popular in the West, Southwest, and Florida.

Here is the plot for females:

Mary was the most popular name basically everywhere in the country (with the exceptions of CA and NV where there were more Jennifers).

It’s interesting to see how dominant Mary is across the entire country while the males names seem to have more regional dominance.  It is particularly unusual because states tended to have many more distinct female names than male names.

More analysis will follow, but here is the code…

###### Settings
library(plyr)
library(maps)
setwd("C:/Blog/StateName")
files<-list.files()
files<-files[grepl(".TXT",files)]
files<-files[files!="DC.TXT"]
 
###### State structure
regions1=c("alabama","arizona","arkansas","california","colorado","connecticut","delaware",
	"florida","georgia","idaho","illinois","indiana","iowa","kansas",
	"kentucky","louisiana","maine","maryland","massachusetts:main","michigan:south","minnesota",
	"mississippi","missouri","montana","nebraska","nevada","new hampshire","new jersey",
	"new mexico","new york:main","north carolina:main","north dakota","ohio","oklahoma",
	"oregon","pennsylvania","rhode island","south carolina","south dakota","tennessee",
	"texas","utah","vermont","virginia:main","washington:main","west virginia",
	"wisconsin","wyoming")
 
mat<-as.data.frame(cbind(regions1,NA,NA))
mat$V2<-as.character(mat$V2)
mat$V3<-as.character(mat$V3)
 
###### Reading files
for (i in 1:length(files))
	{
	data<-read.csv(files[i],header=F)
	colnames(data)<-c("State","Gender","Year","Name","People")
	data1<-ddply(data,.(Name,Gender),summarise,SUM=sum(People))
	male1<-data1[data1$Gender=="M",]
	female1<-data1[data1$Gender=="F",]
	male1<-male1[order(male1$SUM,decreasing=TRUE),]
	female1<-female1[order(female1$SUM,decreasing=TRUE),]
 
	mat$V2[grep(tolower(state.name[grep(data$State[1], state.abb)]),mat$regions)]<-as.character(male1$Name[1])
	mat$V3[grep(tolower(state.name[grep(data$State[1], state.abb)]),mat$regions)]<-as.character(female1$Name[1])
	}
 
jpeg("Male.jpeg",width=1200,height=800,quality=100)
map("state",fill=TRUE,col="skyblue")
map.text(add=TRUE,"state",regions=regions1,labels=mat$V2)
title("Most Popular Male Name (since 1910) by State")
dev.off()
 
jpeg("Female.jpeg",width=1200,height=800,quality=100)
map("state",fill=TRUE,col="pink")
map.text(add=TRUE,"state",regions=regions1,labels=mat$V3)
title("Most Popular Female Name (since 1910) by State")
dev.off()

Created by Pretty R at inside-R.org