POP77001 Computer Programming for Social Scientists
Earliest known (Robson, 2003) tabular account (Puzrish-Dagan, Ur, c.2028 BCE):
Source: (Sigrist, 1984)
| 3 | 3 | 3 | 2 | 1 | lambs |
| 93 | 93 | 93 | 6[2] | 31 | first-rate sheep |
| 6 | 6 | 6 | 4 | 2 | billy goats |
| 102 | 102 | 102 | 68 | [3]4 |
Source: (Reinhardt & Rogoff, 2010) and (Herndon, Ash and Pollin, 2018)
Extra
pandas library has become the de facto standard for data manipulation.pandas is built upon (and often used in conjunction with) other computational libraries:
scipy (linear algebra), statsmodels (statistical models) and scikit-learn (machine learning)pandas Object Typespandas. fruit weight berry
0 apple 150.0 False
1 banana 120.0 True
2 watermelon 3000.0 True
DataFrame.iloc() provides method for index locationDataFrame.loc() provides method for label location| Expression | Selection Operation |
|---|---|
df[val] |
Column or sequence of columns +convenience (e.g. slice) |
df.loc[lab_i] |
Row or subset of rows by label |
df.loc[:, lab_j] |
Column or subset of columns by label |
df.loc[lab_i, lab_j] |
Both rows and columns by label |
df.iloc[i] |
Row or subset of rows by integer position |
df.iloc[:, j] |
Column or subset of columns by integer position |
df.iloc[i, j] |
Both rows and columns by integer position |
df.at[lab_i, lab_j] |
Single scalar value by row and column label |
df.iat[i, j] |
Single scalar value by row and column integer position |
fruit weight berry
0 apple 150.0 False
1 banana 120.0 True
Index(['fruit', 'weight', 'berry'], dtype='str')
fruit weight berry
0 apple 150.0 False
| Format | Readability | Platform | Speed | Compression | Persistence |
|---|---|---|---|---|---|
csv |
✅ Human-readable | ✅ Cross-platform | ❌ Slow | ❌ No | ✅ Long-term |
rds |
❌ Binary | ❌ R only | ✅ Fast | ✅ Yes | ✅ Long-term |
pickle |
❌ Binary | ❌ Python only | ✅ Fast | ✅ Yes | ✅ Long-term |
parquet |
❌ Binary | ✅ Cross-platform | ✅ Fast | ✅ Yes | ✅ Long-term |
feather |
❌ Binary | ✅ Cross-platform | ✅ Fast | ✅ Yes | ❌ Short-term |
.csv (Comma-separated value)
read.csv()/write.csv() - base R functionsreadr::read_csv()/readr::write_csv() - functions from readr package in tidyverse.rds (R data serialization)
readRDS()/writeRDS() - base R functionsreadr::read_rds()/readr::write_rds() - functions from readr (no default compression).rda (R data)
save()/load() - base R functions.feather/.parquet
arrow::read_feather()/arrow::write_feather() - functions fromarrow::read_parquet()/arrow::write_parquet() - arrow package in Apache Arrowpandaspandas provides high-level methods that takes care of file connections.read_<format> and to_<format> name patterns.<variable_name> = pd.read_<format>(<filepath>)
<variable_name>.to_<format>(<filepath>)
open():<variable_name> = open(<filepath>, <mode>)
r)ead a file (default);w)rite an object to a file;x)clusively create, failing if a file exists;a)ppend to a file;r+ mode if you need to read and write to file.with statement can be used:data.frame() function with named vectors as input.'data.frame': 4 obs. of 3 variables:
$ x: int 1 2 3 4
$ y: chr "a" "b" "c" "d"
$ z: logi TRUE FALSE FALSE TRUE
$x
[1] 1 2 3 4 5
$y
[1] "a" "b" "c" "d" "e"
$z
[1] TRUE FALSE TRUE FALSE TRUE
data.frame[[index]]
data.frame[index]
data.frame$name
data.frame[vector_1, vector_2]
tibble from tibble package (part of tidyverse package ecosystem)data.table from data.tabletibble provides features enhancing user experience (readability, ease of manipulation)data.table provides speeddata.table() might be a bit confusing at first.tidyverse packagestidyverse package ecosystem - rich collection of data science packages.readr - data input/output (also readxl for spreadsheets, haven for SPSS/Stata)dplyr - data manipulation (also tidyr for pivoting)ggplot2 - data visualisationlubridate - working with dates and timetibble - enhanced data frametidyverse ecosystem.tibble [4 × 3] (S3: tbl_df/tbl/data.frame)
$ x: int [1:4] 1 2 3 4
$ y: chr [1:4] "a" "b" "c" "d"
$ z: logi [1:4] TRUE FALSE FALSE TRUE
dplyrdplyr - is one of the core packages for data manipulation in tidyversefilter() - subset rows from datamutate() - add new/modify existing variablesrename() - rename existing variableselect() - subset columns from dataarrange() - order data by some variablegroup_by() - aggregate data by some variablesummarise() - create a summary of aggregated variablesdplyr: Subsettingdplyr: Modifying Columns%>% Operatortidyverse packages are encouraged to use pipe operator %>%.|> but it is still not used as widely.<result> <- <input> %>%
<function_name>(., arg_1, arg_2, ..., arg_n)
<result> <- <input> %>%
<function_name>(arg_1, arg_2, ..., arg_n)
%>% Operator: Example# A tibble: 4 × 4
x y z random
<int> <chr> <lgl> <dbl>
1 1 a TRUE 0.102
2 2 b FALSE 0.290
3 3 c FALSE 0.333
4 4 d TRUE -1.17
%>% vs |> Operator|> (pipe) operator.tidyr::pivot_wider())tidyr::pivot_longer()) pivot_wider()
pivot_longer()
# A tibble: 2 × 3
country `1999` `2000`
<chr> <dbl> <dbl>
1 Afghanistan 745 37737
2 Brazil 2666 80488
# A tibble: 4 × 3
country year cases
<chr> <chr> <dbl>
1 Afghanistan 1999 745
2 Afghanistan 2000 37737
3 Brazil 1999 2666
4 Brazil 2000 80488