Column Focus View

The Column Focus View has two modes depending on the number of columns currently in focus. When one column is in focus, the Column Focus View displays column details. When two or more columns are in focus, this view renders different visualizations to compare values and metadata across these columns.

Column Details Mode

In Column Details Mode, this view displays information about a specific column, including metadata, value distribution, data quality metrics, and sample values. It is vertically divided into three sections, a navigation bar at the top, a column overview section below it taking up roughly half the space, and a value count distribution visualization taking up the other half of the space.

alt text When one column is in focus, the Column Focus View displays column details. This example show details on a column containing the number of workers requested for H2A visa applications, taken from the California H-2A Visa Applications tutorial.

The navigation bar the top contains the name of the column in focus on the left side and a button group on the right side. The button group has two toggle buttons that allow you to focus the column to the left or right of the current focused column, enabling you to easily switch between columns. The button on the far right is a close button that will unfocus the column and close this view.

alt text

Column overview

The column overview section consists of three different tabs: distribution, metadata, and quality. The distribution tab is enabled only when the column type is numeric.

Distribution table

The distribution tab displays the distribution of values via a Kernel Density Estimate (KDE) plot. It is a smoothed visualization of the distribution of values, similar to a histogram but without discrete bins. The KDE plot is useful for understanding the overall shape of the distribution and identifying patterns in the data, such as skewness or multimodality.

alt text The distribution tab of the Column Focus View, showing a KDE plot of values in a numeric column, taken from the California H-2A Visa Applications tutorial.

This visualization include additional UI elements to update the KDE plot:

  • Bandwidth slider: This slider allows you to manually tune the bandwidth parameter, which controls the level of smoothing applied to the distribution. A smaller bandwidth will result in a more detailed plot that captures more of the variability in the data, while a larger bandwidth will produce a smoother plot that may obscure some of the finer details.
  • Reset: This button resets the bandwidth parameter to Silverman’s rule of thumb, which is a commonly used default bandwidth for KDE plots. This allows you to quickly return to a reasonable default view of the distribution after adjusting the bandwidth slider.
  • Show histogram checkbox: This checkbox toggles the display of a histogram overlaid on top of the KDE plot. The histogram provides a discrete representation of the distribution of values, which can be useful for understanding the frequency of values within specific ranges and how they contribute to the overall shape of the distribution.
  • Range slider: The range slider allows you to filter the values included in the KDE plot by specifying a minimum and maximum value. This can be useful for focusing on specific regions of the distribution, such as the tails or the center, and for excluding outliers that may skew the visualization.

Metadata tab

The metadata tab displays the metadata of the column in focus, including its name, type, and any other relevant information.

alt text

The metadata tab of the Column Focus View, taken from the California H-2A Visa Applications tutorial.

This can help users understand the context of the data in the column and how it relates to other columns in their tables. Some of these fields are specific to the type of column. For example, numeric columns will display the measure of central tendency. If a metric is not applicable to the column in focus, it will contain an “N/A” value. Below is a list of the different fields in the metadata tab:

  • Parent table: The name of the table that contains this column.
  • Column Type: The data type of the column, e.g. categorical or numeric.
  • Count: The number of values in the column.
  • Mode: The most frequently occuring value in this column and its count.
  • Unique Values: The number of unique values in this column.
  • Min: The minimum value in this column (only for numeric columns).
  • Max: The maximum value in this column (only for numeric columns).
  • Average: The average value in this column (only for numeric columns).
  • Median: The median value in this column (only for numeric columns).

Quality tab

The quality tabs displays data quality metrics for the column in focus. This views are designed to help the user develop a deeper understand of the underlying column data, not to support addressing data quality issues via column transformations. Therefore, we provide two basic data quality metrics, completeness and uniqueness, along with visualizations to help users understand these metrics.

alt text The quality tab of the Column Focus View, taken from the California H-2A Visa Applications tutorial.

We provide to composite data quality metrics, completeness and uniqueness. Each metric includes a donut chart along with the numeric values for each part of the whole.

  • Completeness: The ratio of null to non-null values in the column. A higher completeness score indicates the column has fewer missing or null values.
  • Uniqueness: The ratio of unique value to duplicate values in the column. A higher uniqueness score indicates the column has more unique values and fewer duplicate values.

Value count distribution visualization

The value count distribution visualization displays the count of each value in the column in focus.

alt text The value count distribution visualization in the Column Focus View, taken from the California H-2A Visa Applications tutorial. This view displays the count of each unique value in horizontal bars, where the range of counts is encoded in horizontal bars along the x-axis and the unique values are encoded vertically along the y-axis. The value label is inset inside each bar to allow users to easily identify which value each bar corresponds to.

This is the only visualization in Column Details Mode where users can inspect the actual values in the column. This view support grouping column values along two dimensions, selectable in the combobox in the top right of the visualization labeled, “Group by”. We do not include an non-grouped view of the value counts, a non-grouped view can be seen by selecting the column and examining values in the Table Rows View. The grouping options are:

  • Unique Values: This option groups values by their unique value.
  • Value Length: This option groups values by their length, i.e. the number of characters in the value. This can be useful for surfacing data quality issues. Hovering over the bars when data is grouped in this manner will display a tooltip with the unique values that correspond to each value length.

This view is scrollable and additional results are paginated in as the user scrolls to the bottom of the container. Text at the bottom of the view indicates how many values have been loaded against the total number of unique values in the column.

Column Comparison Mode

Focusing on two or more columns will switch the Column Focus View into Column Comparison Mode. In this mode, the view renders different visualizations to supporting comparing columns by value and by metadata. The interface in this mode is divided into three sections vertically. The top section contains a table listing columns currently in focus with high-level metadata, the middle section contains metadata comparison visualizations, and the bottom section contains value comparison visualizations.

alt text Overview of the Column Focus View in Column Comparison Mode, showing the column table with metadata for each column in focus, a metadata comparison visualization comparing the average value across columns, and a value comparison visualization comparing the distribution of values across columns. This example is taken from the California H-2A Visa Applications tutorial.

Columns table

The column tables displays the column name, parent table, index of this column in the parent table, and the column type for each column currently in focus.

alt text The column table in the Column Focus View, taken from the California H-2A Visa Applications tutorial. This table lists the columns currently in focus along with high-level metadata for each column.

This table is designed to provide a high-level overview of the columns currently in focus to help users orient themselves when comparing multiple columns. Oftentimes, columns will have identical names, so the additional metadata fields of parent table and column index can help users disambiguate between these columns. The column type is also important to understand when comparing columns, as different visualizations and metrics are applicable to different column types.

Metadata comparison

The metadata comparison visualization allows users to compare column metadata across focused columns.

Metadata metrics

The available metadata metrics depend on the types of the columns currently in focus. In the example above, all columns in focus are numeric, so we are able to compare the average value across these columns. If there were categorical columns in focus, we would not be able to compare the average value, but we may be able to compare the mode or number of unique values across columns.

alt text The metadata comparison visualization in the Column Focus View, taken from the California H-2A Visa Applications tutorial. This visualization compares the average value across columns currently in focus.

All column types

Several of the option in the combo box are applicable to all column types. These display a bar chart visualizing the metadata value for each column in focus, where the x-axis encodes the column name and the y-axis encodes the metadata value. The available metadata metrics that are applicable to all column types are:

  • Counts: The number of values in each column.
  • Unique Counts: The number of unique values in each column.
  • Null Counts: The number of null values in each column.
Numeric columns

When all columns in focus are numeric, we can compare them based on their distribution.

alt text To compare the distribution of values across numeric columns, we use a box-and-whisker plot, which displays the median, quartiles, and outliers for each column. This visualization allows us to easily compare the central tendency and variability of values across columns.

In the Column Detail View, we use a KDE plot to visualize the distribution of values in a single column. However, when comparing the distribution of values across multiple numeric columns, we use a box-and-whisker plot instead. This is because metrics to render a box-and-whisker plots, such as the median and quartiles, are pre-computed when columns are ingested into Roundup, so we can render this visualization more efficiently than a KDE plot, which requires us to compute the bandwidth and density estimation at runtime. Additionally, box-and-whisker plots are a common visualization for comparing the distribution of values across multiple groups, so we opted to use this visualization to align with user expectations.

Value comparison

The value comparison visualization allows user to scan all unique values present in either column, i.e. the union of unique values across each set of unique values in each column, and investigate the presence of these values across columns.

alt text The value comparison visualization in the Column Focus View, taken from the California H-2A Visa Applications tutorial. This visualization allows users to compare the presence of unique values across columns currently in focus. Each row corresponds to a unique value present in at least one of the columns in focus, and each column corresponds to a column in focus. The presence of a value in a column is indicated by a filled circle, while the absence of a value is indicated by an empty circle.

This visualization support two interactions, filtering and sorting. The filter interaction allows users to filter the unique values included in the visualization by typing in the text input labeled “Filter values” in the top right of the visualization. This can be useful for focusing on specific values of interest, especially when there are many unique values across columns.

The sort interaction allows users to sort the unique values by their presence across columns by clicking the combobox. This visualization can be sorted in ascending or descending order by these options:

  • Degree: This sorting option sorts values by the number of columns in which they are present, i.e. the degree of the value across columns. Values with a higher degree, or presence in more columns, are sorted higher than values with a lower degree. The range of degrees is ([1, n]), where (n) is the number of columns in focus.
  • Value: This sorting option sorts values alphabetically by their value, independent of their presence across columns.