Productionizing with Civis Scripts

An Example Script

The post method creates the job and returns a list of metadata about it, including its type.

source <- c('
 print("Hello World!")
')
job <- scripts_post_r(name = 'Hello!', source = source)

Each script can be uniquely identified by its job id. If you have a job id but don’t know what kind of script it is, you can do jobs_get(id).

Each script type is associated with its own API endpoints. For instance, to post a job of each script type, you need scripts_post_r, scripts_post_containers, scripts_post_custom, or templates_post_scripts.

This job hasn’t been run yet. To kick off a run do:

run <- scripts_post_r_runs(job$id)

# check the status
scripts_get_r_runs(job$id, run$id)

# automatically poll until the job completes
await(scripts_get_r_runs, id = job$id, run_id = run$id)

Since kicking off a job and polling until it completes is a really common task for this guide, let’s make it a function:

run_script <- function(source, name = 'Cool') {
  job <- scripts_post_r(name = name, source = source)
  run <- scripts_post_r_runs(job$id)
  await(scripts_get_r_runs, id = job$id, run_id = run$id)
}

Run Outputs

This script isn’t very useful because it doesn’t produce any output that we can access. To add an output to a job, we can use scripts_post_r_runs_outputs. The two most common types of run outputs are Files and JSONValues.

source <- c("
 library(civis)
 data(iris)
 write.csv(iris, 'iris.csv')
 job_id <- as.numeric(Sys.getenv('CIVIS_JOB_ID'))
 run_id <- as.numeric(Sys.getenv('CIVIS_RUN_ID'))
 file_id <- write_civis_file('iris.csv')
 scripts_post_r_runs_outputs(job_id, run_id, object_type = 'File', object_id = file_id)
")
run <- run_script(source)

source <- c("
 library(civis)
 data(iris)
 write.csv(iris, 'iris.csv')
 write_job_output('iris.csv')
")
run <- run_script(source)

source <- c("
 library(civis)
 library(jsonlite)
 my_farm <- list(cows = 1, ducks = list(mallard = 2, goldeneye = 1))
 json_values_post(jsonlite::toJSON(my_farm), name = 'my_farm.json')
")
run_farm <- run_script(source)

out <- scripts_list_r_runs_outputs(run$rId, run$id)
iris <- read_civis(out$objectId, using = read.csv)

# get csv run outputs
iris <- read_civis(civis_script(run$rId), regex = '.csv', using = read.csv)

# get JSONValues
my_farm <- read_civis(civis_script(run_farm$rId))

Script Parameters

Scripts are more useful if their behavior can be configured by the user, which can be done with script parameters. Script parameters are placeholders for input by the user. Specific values of the parameters input by the user are called arguments. Here, we modify run_script to automatically add a parameter, and simultaneously take a value of that parameter provided by the user. In the script itself, we can access the parameter as an environment variable.

# Add 'params' and 'arguments' to run_script
run_script <- function(source, args, name = 'Cool') {
  params <- list(          # params is a list of individual parameters
    list(
      name = 'PET_NAME',   # name of the environment variable with the user value
      label = 'Pet Name',  # name displaayed to the user
      type = 'string',     # type 
      required = TRUE      # required?
    )
  )
  job <- scripts_post_r(name = name, 
                        source = source, 
                        params = params, 
                        arguments = args)
  run <- scripts_post_r_runs(job$id)
  await(scripts_get_r_runs, id = job$id, run_id = run$id)
}

# Access the PET_NAME variable
source <- c('
  library(civis)
  pet_name <- Sys.getenv("PET_NAME")
  msg <- paste0("Hello", pet_name, "!")
  print(msg)
')

# Let's run it! Here we pass the argument 'Fitzgerald' to the 
# parameter 'PET_NAME' that we created.
run_script(source, name = 'Pet Greeting', args = list(PET_NAME = 'Fitzgerald'))

Custom Scripts

scripts_post_custom creates an instance of a template that inherits all changes made to the template. We can now make a simple program to call and run an instance of the template.

job <- scripts_post_custom(id, arguments = arguments, ...)
run <- scripts_post_custom_runs(job$id)
await(scripts_get_custom_runs, id = job$id, run_id = run$id)

Conveniently, run_template does exactly this and is already provided in civis. It returns the output file ids of the job for you to use later on in your program.

out <- run_template(template$id, arguments = list(PET_NAME = 'CHARLES'))

To stay organized, let’s automatically add the script to an existing project:

# We might need to find the project id first
search_list(type = 'project', 'My project Name')
out <- run_template(template$id, arguments = list(PET_NAME = 'CHARLES'),
                    target_project_id = project_id)

Making Changes

To make changes to the template note or name, use templates_patch_scripts.

templates_patch_scripts(template_id$id, note = new_note)

To change the behavior, name, or parameters of the script, update the backing script using scripts_patch_r.

• Note: It is not recommended to make breaking changes to the API of a script by adding a required parameter, changing a parameter default, or removing a run output. This will break workflows of your users. Instead of making breaking changes, release a new version of the script.

source <- c('
  library(civis)
  pet_name <- Sys.getenv("PET_NAME")
  msg <- paste0("Hello ", pet_name, "! Would you care for a sandwich?")
  print(msg)
')
scripts_patch_r(id = job$id, name = 'Pet Greeter',
                source = source,
                params = params)

Discoverability

To help share your template with others, use this link: https://platform.civisanalytics.com/spa/#/scripts/new/{your template id}.

This link will automatically direct the user to a new instance of the template.

It’s a good idea to archive unused templates so that it’s easy for users to find the right template quickly. This is important if you automatically deploy your templates.

Let’s clean up our experiment by archiving our Pet Greeter Template:

templates_patch_scripts(template$id, archived = TRUE)