1. Importing data from CSV files for baytrends
Importing CSV files that have censored data into R dataframes for analysis in baytrends is a critical data preparation step and involves two lines of code: 1) importing the CSV data into R and 2) converting water quality measurements with censored data to Surv objects (see below code).
Example code
The remainder of this section discusses the above lines of code by way of example.
Discussion
Figure 1 represents a portion of data from an example CSV file. Note, there is no requirement for the CSV file to have the data stored in the column order shown. In this example, the three required fields: station, date, and layer are stored in the first three columns ([A]-[C]). Missing values for these columns are not permissable. The user can optionally include additional fields such as depth, time, etc.
Figure 1. Example CSV file.
Fields with no censoring. In the above example, Secchi depth (secchi), salinity (salinity), dissolved oxygen (do), water temperature (wtemp), total suspended solids (tss) are stored in the next group of columns as un-censored data. While possible, censoring is less common for these types of water quality variables and as a result the data are represented with a single column in the CSV file. The values entered in these columns should be numeric values with no symbols. Values shown as “NA” are un-reported entries. Un-reported values can happen for a number of reasons including not being measured, equipment failure, etc. CSV files can use “NA” or use blanks to represent un-reported entries. Depending on the situation and the user’s sensitivity to fully evaluating censored data, a variable such as total suspended solids could be treated as an un-censored variable as shown in this example or could be treated as a variable with censored data as discussed next.
Fields with censoring. To the right of the break in Figure 1, water quality fields include lower and upper bounds to represent censoring. Note, the field naming syntax using a suffix of “_lo” and “_hi”. For example, columns [U] and [V] show the lower and upper bounds for total dissolved phosphorus as tdp_lo and tdp_hi, respectively. The upper bound must be greater than or equal to the lower bound. When the upper bound is equal to the lower bound, it indicates that observation did not have censoring.
loadData
The code df_in <- loadData(file = "ExampleData.csv")
uses a baytrends’ built-in function to load data from a CSV file.
Experienced R programmers can use other functions to load data
(e.g. read.csv, read.table); however,
loadData outputs meta data about the number of columns and
rows read and performs some useful steps such as converting date fields
into an R date field (i.e., POSIXct) and is advantageous
for further analyses with baytrends and other situations. There is also
a companion baytrends function, loadExcel, with similar
functionality to loadData and can be used to load data from
an Excel workbook. See help (i.e., ??loadData and ??loadExcel) for more
information.
makeSurvDF
After importing the above CSV file into an R dataframe, the next step
is to convert fields with censored data to Surv objects as depicted in
Figure 2 using the code df_Surv <- makeSurvDF(df_in).
Fields shaded in green or blue are column to column comparable to the
CSV file. Columns to the right of the break, such as the yellow
highlighted column, tdp, is a Surv object and is derived from the two
corresponding columns, tdp_lo and tdp_hi in the CSV file (Figure 1).
Again, experienced R programmers can make Surv objects independently
from baytrends by using the following syntax for each field with
censored data using the total dissolved phosphorus data as an
example.
The baytends function, makeSurvDF, peforms this process
for all fields in a dataframe with censored data. By default,
makeSurvDF assumes the dataframe employs a suffix of “_lo”
to represent the lower bound and “_hi” to represent the upper bound
although this can be changed by the user. See help (i.e., ??makeSurvDF)
for more information. The makeSurvDF function also performs
a quality assurance review of the data to make sure the upper bound is
greater than or equal to the lower bound. The function outputs the
number of data integrity issues by parameter and sets those observations
to missing (NA) so the code can continue whilst the analyst determines
the appropriate corrective action.
Figure 2. Example dataframe with Surv objects.
