validata

library(validata)
library(tidyselect)

Distinct

Confirm Distinct

In data analysis tasks we often have data sets with multiple possible ID columns, but it’s not always clear which combination uniquely identifies each row.

sample_data1 has 125 row with 3 ID type columns and 3 value columns.

head(sample_data1)
#> # A tibble: 6 × 6
#>   ID_COL1 ID_COL2 ID_COL3     VAL1   VAL2   VAL3
#>     <dbl>   <dbl>   <dbl>    <dbl>  <dbl>  <dbl>
#> 1    2413    1034    1014 -0.0639  -1.16  -0.302
#> 2    2413    1034    1322  0.363    1.62   0.165
#> 3    2413    1034    2999 -0.00466  1.23   0.819
#> 4    2413    1034    3544  1.83    -2.58  -0.525
#> 5    2413    1034    9901  0.837   -0.442 -0.341
#> 6    2413    1122    1014 -0.894   -1.11   0.768

Let’s use confirm_distinct iteratively to find the uniquely identifying columns of sample_data1.

sample_data1 %>% 
  confirm_distinct(ID_COL1)
#> database has 120 duplicates at ID_COL1
sample_data1 %>% 
  confirm_distinct(ID_COL1, ID_COL2)
#> database has 100 duplicates at ID_COL1, ID_COL2
sample_data1 %>% 
  confirm_distinct(ID_COL1, ID_COL2, ID_COL3)
#> database is distinct at ID_COL1, ID_COL2, ID_COL3

Here we can conclude that the combination of 3 ID columns is the primary key for the data.

Determine Distinct

These steps can be automated with the wrapper function determine distinct.

sample_data1 %>% 
  determine_distinct(matches("ID"))
#> Warning: There was 1 warning in `dplyr::summarize()`.
#> ℹ In argument: `dplyr::across(.fns = ~dplyr::n_distinct(.) == rws)`.
#> Caused by warning:
#> ! Using `across()` without supplying `.cols` was deprecated in dplyr 1.1.0.
#> ℹ Please supply `.cols` instead.

Mapping

confirm_mapping tells you the mapping between two columns in a data frame:

  • 1 - 1 mapping
  • 1 - many mapping
  • many - 1 mapping
  • many - many mapping

Confirm mapping

confirm_mapping gives the option to view which type of mapping is associated with each individual row.

sample_data1 %>% 
  confirm_mapping(ID_COL1, ID_COL2, view = F)
#> many - many mapping between ID_COL1 and ID_COL2

Determine mapping

sample_data1 %>% 
  determine_mapping(everything())

Overlap

The overlap functions give a venn style description of the values in 2 columns. This is especially useful before performing a join function, and you want to confirm that the dataframes have matching keys.

Confirm Overlap

confirm_overlap is different from the other confirm functions in that it takes 2 vectors as arguments, instead of a data frame. This is to allow the user to test overlap between different dataframes, or arbitrary vectors if necessary


confirm_overlap(iris$Sepal.Width, iris$Petal.Length) -> iris_overlap
#> # A tibble: 1 × 5
#>   only_in_iris_Sepal.Width only_in_iris_Petal.Length shared_names total_names
#>                      <int>                     <int>        <int>       <int>
#> 1                       12                        32           11          55
#> # ℹ 1 more variable: pct_shared <chr>

confirm_overlap returns a summary data frame invisibly allowing you to access individual elements using the helper functions.

print(iris_overlap)
#> # A tibble: 55 × 4
#>        x iris_Sepal.Width iris_Petal.Length both_flags
#>    <dbl>            <dbl>             <dbl>      <dbl>
#>  1   3.5                1                 1          2
#>  2   3                  1                 1          2
#>  3   3.2                1                 0          1
#>  4   3.1                1                 0          1
#>  5   3.6                1                 1          2
#>  6   3.9                1                 1          2
#>  7   3.4                1                 0          1
#>  8   2.9                1                 0          1
#>  9   3.7                1                 1          2
#> 10   4                  1                 1          2
#> # ℹ 45 more rows

Find the elements unique to the first column

iris_overlap %>% 
  co_find_only_in_1() %>% 
  head()
#> # A tibble: 6 × 1
#>   iris_Sepal.Width
#>              <dbl>
#> 1              3.2
#> 2              3.1
#> 3              3.4
#> 4              2.9
#> 5              2.3
#> 6              2.8

Find the elements unique to the second column

iris_overlap %>% 
  co_find_only_in_2() %>% 
  head()
#> # A tibble: 6 × 1
#>   iris_Petal.Length
#>               <dbl>
#> 1               1.4
#> 2               1.3
#> 3               1.5
#> 4               1.7
#> 5               1.6
#> 6               1.1

Find the elements shared by both columns

iris_overlap %>% 
  co_find_in_both() %>% 
  head()
#> # A tibble: 6 × 1
#>       x
#>   <dbl>
#> 1   3.5
#> 2   3  
#> 3   3.6
#> 4   3.9
#> 5   3.7
#> 6   4

Determine Overlap

determine_overlap takes a dataframe and a tidyselect specification, and returns a tibble summarizing all of the pairwise overlaps. Only pairs with matching types are tested.

Note that the overlap functions only test pairwise overlaps. For multi-column and large-scale overlap testing, see Complex Upset Plots

string length

confirm string length

Get a frequency table of string lengths in a character column. Table is printed while the original df is returned invisibly with a column indicating the string lengths.

iris %>% 
  confirm_strlen(Species) -> species_len
#>  Species_chr_len  n percent
#>                6 50   33.3%
#>                9 50   33.3%
#>               10 50   33.3%

output is a dataframe

head(species_len)
#> # A tibble: 6 × 6
#>   Sepal.Length Sepal.Width Petal.Length Petal.Width Species Species_chr_len
#>          <dbl>       <dbl>        <dbl>       <dbl> <fct>             <int>
#> 1          5.1         3.5          1.4         0.2 setosa                6
#> 2          4.9         3            1.4         0.2 setosa                6
#> 3          4.7         3.2          1.3         0.2 setosa                6
#> 4          4.6         3.1          1.5         0.2 setosa                6
#> 5          5           3.6          1.4         0.2 setosa                6
#> 6          5.4         3.9          1.7         0.4 setosa                6

choose string length

A helped function for the output of confirm_strlen that filters the database for chosen string lengths.

species_len %>% 
  choose_strlen(len = 6) %>% 
  head()
#> # A tibble: 6 × 6
#>   Sepal.Length Sepal.Width Petal.Length Petal.Width Species Species_chr_len
#>          <dbl>       <dbl>        <dbl>       <dbl> <fct>             <int>
#> 1          5.1         3.5          1.4         0.2 setosa                6
#> 2          4.9         3            1.4         0.2 setosa                6
#> 3          4.7         3.2          1.3         0.2 setosa                6
#> 4          4.6         3.1          1.5         0.2 setosa                6
#> 5          5           3.6          1.4         0.2 setosa                6
#> 6          5.4         3.9          1.7         0.4 setosa                6

diagnose

Reproduction of diagnose from the dlookr package. Usually a good choice for first analyzing a data set.

iris %>% 
  diagnose()
#> # A tibble: 5 × 6
#>   variables    types   missing_count missing_percent unique_count unique_rate
#>   <chr>        <chr>           <int>           <dbl>        <int>       <dbl>
#> 1 Sepal.Length numeric             0               0           35       0.233
#> 2 Sepal.Width  numeric             0               0           23       0.153
#> 3 Petal.Length numeric             0               0           43       0.287
#> 4 Petal.Width  numeric             0               0           22       0.147
#> 5 Species      factor              0               0            3       0.02