Fun with Data - Basics of R - Part 3 Visualization

In this section we'll cover some basic visualizations with R using the built-in plot function and using the library ggplot2. Before we start, here's a quick review of the topics we covered in previous two sections.

Part 1 - Getting R and R-Studio in your machine, understanding the structure of R-Studio and loading data into R. We also covered some basic functions for viewing the content of loaded dataset.

Part 2 - Understanding prompts in R-Studio, variables, vectors and data frames. Also, summary, mean and length functions.

Now that we can load data into R and see what's inside, let's try to visualize it. Visualization is a great way to actually make the meaning of data visible, especially when we're dealing with large amount of data. For example, we're planning a sports event at school and trying to group students based on their heights. We can see the distribution by looking at the data, but when we create a graph it is easy to comprehend which proportion falls in which category based on height. 

Today we'll see how to create scatter plot using both basic plot function and ggplot2. R comes with many freely available datasets, which you can view by typing data(). We'll use the airquality, mtcars and pressure datasets to create sample graphs. Since they're already in R, we don't have to load them, but look into the content by using some of the functions we covered in previous sections, or by simply typing their names in the console.

airquality()

mtcars()

pressure()

Splitting the View Window

You can split the view window to see more than one graph or plot in the screen. For example, par(mfrow = c(1, 2)) splits the window into 2. Changing it to c(2, 2) will split it into 4. 

Install ggplot2 Package

install.packages("ggplot2")  # installs the package

library(ggplot2)                    # loads the library into the workspace 

Scatter Plot

In the mtcars dataset we can see weight of a car and how many miles it runs per gallon. We can find out if there's any correlation between these two by creating a scatter plot.

plot(mtcars$wt, mtcars$mpg, main="Scatterplot Example", xlab="Car Weight ", ylab="Miles Per Gallon ", pch=19)

I'll explain what each of the parameters mean. 

main -> Give a name to the plot

xlab -> Name of X-axis

ylab -> Name of Y-axis

pch -> Type of symbol for the graph. You can see the full list from this page.

Now, this alone can be little difficult to understand. So, you can add fit lines to this to see how many data are positively or negatively correlated. 

abline(lm(mtcars$mpg~mtcars$wt), col="red")  # regression line (y~x)

lines(lowess(mtcars$mpg~mtcars$wt), col="green")   # lowess line (x, y)

You can create the same graph using ggplot2, which is visually more pleasing and provides a lot of options to ornate your graph. (Check out the package documentation for further information)

# Create the same graph using ggplot2

qplot(mtcars$wt, mtcars$mpg, xlab = "Car Weight", ylab = "Mile/Gallon")

# Following syntax can be used if the two vectors are already in the same data frame.

qplot(wt, mpg, data=mtcars, xlab = "Car Weight", ylab = "Mile/Gallon")

We can also add colors to see correlation to another variable. In this example, cylinder size of cars. 

qplot(wt, mpg, data=mtcars, xlab = "Car Weight", ylab = "Mile/Gallon", color =cyl)

From this graph we can make assumption that light weight cars with smaller cylinder size usually run more miles per gallon. Similarly, we can create different visualizations by comparing other variables or in other words, by considering which information we're trying to provide to our audience. 

A great reference to learn about visualizations with R is R Graphics Cookbook by Winston Chang- 

Chang, W. (2012). R graphics cookbook. " O'Reilly Media, Inc.".

Fun with Data - Basics of R - Part 2

In our first post on R programming language I covered how to download R and R Studio, understanding the structure of R Studio, and loading datasets into R. I meant to continue the series (as I always do), but couldn't get back to it any sooner. Recently I've started working on a workshop where I'll be teaching R to the beginners. So, I thought this would be the best time to add more content to this series as well. 

In this Part 2 I'll write about the following - Understanding prompts of R Studio, doing basic calculations in R, all about variables, functions, the concept of vector in R, and data frame. Some of these I should have covered in Part 1, but better late than never!

Prompts in R Studio

• In console a new line starts with >, means it is waiting for us to communicate
• If we give it an incomplete command then it returns +. Press esc button to return to a new line.
• To quit R type q()

Doing Basic Calculations in R

• The order of arithmetic operations is  (left [done first] to right [done last]) : ^ / * - +
• ^ is used for raised to the power of, followed by division, multiplication, subtraction and addition.
• At the prompt, we enter the expression that we want evaluated and when we hit enter, it will compute the result for us. For example: > 10 + 22 will return [1] 32

All About Variables 

• Variables are the symbols that store assigned values. We can store a computation under a new variable or change the existing value of an old variable.
• Variable names in R are case sensitive (upper or lower case).
• It is a good practice to assign meaningful variable names that helps to refer to easily for complex calculations.

To assign a value: variable_name <- value

Example: x <- 100

ALERT! Reserved Symbols!

In all programming languages certain symbols are reserved for specific purposes. The reserved symbols in R are - c q t C D F I T (So, don't use them for your personal variables ^-^)

Functions

A function is a sub-program that performs a specific task. For example, to find a square root of a given value. It helps to avoid repetition and easy execution in future. 

Try this code to understand how functions work -

firstFunction <- function(n){n*n}

This function named firstFunction is supposed to return square of any integer. Test it out by assigning different values to the function. Think of what other functions you can possibly write.

Vector

Vector has different meanings in different contexts. In math and physics, a vector is an element with both value and direction. But in R, vector is a sequence of data elements of the same basic type. It can be defined by concatenating the members in a set c(). Example: x <- c(1, 2, 4, 5).

Once we have a vector of numbers we can apply certain built-in functions to them to get useful summaries. For example:

> sum(x)        ## sums the values in the vector

> length(x)    ## produces the number of values in the vector, ie its length

> mean(x)     ## the average (mean)

Data Frame

A data frame can be created by defining different variables for each column as vectors and then joining them together.

Example: Let us assume we have a list of different fruits with their names, colors and size.

> name      <- c("apple", "banana", "peach", "watermelon", "grape")

> color      <- c("red", "yellow", "peach", "green", "red")

> size_cm <- c(10, 15, 8, 40, 2)

Then we add these three columns together to create the data frame names fruits.data.

> fruits.data <- data.frame(name, color, size_cm)

To see the values of the data frame -

> fruits.data

        name      color       size_cm

1      apple        red            10

2     banana      yellow      15

3      peach       peach         8

4 watermelon  green         40

5      grape       red             2

--------------------------------------------------------------------------------------------------------------------------

I think we've covered a lot of basics concepts already, so I'll stop here today. In the next post of this series, I'll write about setting work directory, manipulating datasets, and playing around with some plots/visualizations, and hope I can make it sometime soon! 

Solution to Ruby's REST Client File Corruption Error

One of my projects involves working with REST API to upload binary files and associated metadata, and we are using Ruby to write our programs here. There are multiple ways to work with REST API in Ruby, some of the popular ones are -

1. Ruby's own HTTP client API Net::HTTP

2. REST Client gem

3. Faraday gem

Among three of these, I found REST Client to be comparatively easy to use as it has simpler syntax, and has advanced enough options to get the work done. But I was having a strange corrupted file error while using this, where only PDF files were uploaded fine but the other file formats were corrupted when uploaded using REST Client. Our system is a bit more complicated where it downloads the files from different source, and then ingests them into DSpace repository using their REST API. So, we were not sure at first in which step of the whole process the files are getting corrupted. Our primary assumption was, the files were probably sent for uploading into DSpace before they were fully downloaded, hence they were broken. But they seemed to be downloaded fine, instead something was going wrong in the upload process using REST Client. 

Here is the method for POSTing file mentioned in the original documentation of REST Client- 

RestClient.post( url,

  {

      :transfer => {

      :path => '/foo/bar',

      :owner => 'that_guy',

      :group => 'those_guys'

    },

      :upload => {

      :file => File.new(path, 'rb')

    }

  })

After digging little bit I found that some other people had similar problem using REST multipart POST - librelist archives - uploaded pictures are in a bad shape. According to one solution, form encoding of the payload could be the main reason for corruption. Though I was unable to open other file formats, after checking the content of an uploaded JSON-LD file I could figure out what was happening inside. This is how the corrupted file looks like when uploaded using multipart POST.

--653361

Content-Disposition: form-data; name="transfer[type]"

bitstream

--653361

Content-Disposition: form-data; name="upload[file]"; filename="Filetype Check.jsonld"

Content-Type: text/plain

[ Main content of the original JSON-LD]

--653361--

It creates a wrapper around the main content by including the information provided in payload following the original method, and it does the same to other file formats as well. Thus corrupting the files. And here are two solutions that I have found - not using multipart form in REST Client and using Net::HTP.

1. Using REST Client without multipart form

open('filepath') do |fh|

response = RestClient.post(

"url/to/post/file", fh,

{ :content_type => 'application/json', :accept => 'application/json'})

end

p "#{response}"

Or, simply -

RestClient.post("url/to/post/file", File.new('filepath', 'rb'), {:content_type => 'application/json', :accept => 'application/json'})

2. Using Net::HTTP

data = File.read('filepath')

url = "url/to/post/file"

uri = URI.parse(url)

http = Net::HTTP.new(uri.host, uri.port)

http.use_ssl = true                                          # include this if the connection is secured, otherwise fails

request = Net::HTTP::Post.new(uri.request_uri)

request["header"] = "header_info" # if any additional header is required

request.body = data

request.content_type = 'image/jpg'

res = http.request(request)

puts res.body

Both of these methods can upload file without corrupting them by creating that strange wrapper. Hope this would be helpful for any Ruby programmers struggling with the same problem.

Fun with Data - Basics of R - Part 1

It's been a while since I wrote my last technical post, actually I need to finish writing my database series. When I started grad school, my plan was to write post on every new exciting thing I learn for my own reference. But due to time constraint couldn't make it. Recently I'm learning to use statistical software R for data analytics, which is a widely used tool, and I thought it'd be useful to write about the basics for anyone who'd like to learn it from scratch. I'll keep these posts short and write as I learn more.

Using R on your machine

I'm using R with the software R Studio, which requires to have R installed on your computer. Both are supported for all platforms such as Windows, MacOS and Ubuntu. Since I'm using MacOS, I'll write about how it works for this platform. But it'll be pretty similar for other ones. 

First of all, make sure you have R installed. In your terminal type which r, which will return the path of where R is installed, or simply type r, which will start r console. If you don't have it installed then you can easily install it from the following link - https://cran.r-project.org/bin/macosx/

Then download and install R Studio from the following link- https://www.rstudio.com/products/rstudio/download/

If the installation is completed properly then you can run the software, which will look like the screen below, but without any data of course. I've marked the fields as A, B, C, and D to explain their meaning and usage.

The top left field A is to write the scripts like any other text editor. You can write your script and select any portion to run. Or you can use the bottom left field B, which is R console to directly input the scripts and see output. It works similar to terminal or iTerm.

The top right field C is where the data frames are shown once any data frame or table is imported or created. As for me, I've three tables in my working directory now. And in the bottom right field D all the graphical outputs are shown, such as bar plot, scatterplot, etc. 

Loading data into R

The commonly used data formats are .csv or .txt, and usually gathered from other data sources and then loaded into R. Once you have your file saved on your machine, get the full path of the file, which can be done by viewing the information or properties of the file. For example, my ihis_0005.csv file is saved under /Documents/fall2015_classes/SODA_EVD directory. The command to read csv file is simply read.csv() and the file path goes within the parentheses. You would like to give the table a name, which is ihis_data in my case. So the command will be,

your_table_name <- read.csv("your_file_path")

Once you have your data loaded, it'll show up on the top right section. R is very helpful to show the summary of the data by just typing the command summary(your_table_name), which is useful for quantitative variables. It returns minimum and maximum value, mean, median, 1st quartile, 3rd quartile values for each column. For categorical variables using the table() command is more useful. To see the actual distribution of values for any certain column with categorical variable, type table(your_table_name$column_name). The $ sign denotes columns for that table.

You can also see all the data frames in your working directory by ls() command, where ls means list. And delete any data frame by rm(your_table_name), where rm means remove. To delete multiple tables at ones list all the table names you would like to delete like this-
rm(list = c("table1", "table2", ...)

These are the preparatory steps before we can actually go ahead and play with our data. I'll write more about how to actually have fun with it in my following post (which I believe will happen soon!) :)