Introduction

User friendly automation is hard to write, and is hard to get right.

Minimum viable product (MVP) is not user friendly.

This book describes different dimensions that influence the user friendliness of software, and introduces a new library – choochoo – that tries to support writing software that is more user friendly.

Workflow Scenario

Consider the deployment steps for a standalone web application backed by a database.

The following steps represents this workflow:

Dimensions

In this book, a dimension is defined as a particular nature of something. Dimensions may be independent, and may be measurable. Some dimensions may be derived from and may be influenced by others.

For example, an object's dimensions are:

• Size
• Mass
• Temperature
• Colour

Software Dimensions

We will look at the following software dimensions, and show how choochoo supports building software with higher maturity levels of user friendliness.

1. Path Handling: How exhaustive code path coverage is, and how each path is handled.
2. State Visibility: What state is it in now, progress tracking.
3. Understandability: Presentation / verbosity.
4. Recoverability / Repeatability: Can execution resume from halfway.
5. Performance / Efficiency: How well is execution scheduled, time spent, and resources used.
6. Psychology: Did the user request what they intended?

Path Handling

How exhaustive code path coverage is, and how each path is handled.

Maturity Levels

#! /bin/bash
curl http://somewhere.com/app.zip -o app.zip

# Much better with:
# set -euo pipefail

void AppDownload()
{
    var webClient = new WebClient();
    webClient.DownloadFile("https://somewhere.com/app.zip", "app.zip");
}

fn app_download() {
    let bytes = reqwest::blocking::get("https://somewhere.com/app.zip")
        .unwrap()
        .bytes()
        .unwrap();

    let mut file = File::create("app.zip").unwrap();
    file.write_all(bytes).unwrap();
}

void AppDownload()
{
    var webClient = new WebClient();

    try { webClient.DownloadFile("https://somewhere.com/app.zip", "app.zip"); }
    catch (Exception e)
    {
        throw new System.InvalidOperationException("Failed to download app.zip", e);
    }
}

fn app_download() -> Result<(), Box<dyn std::error::Error>> {
    let bytes = reqwest::blocking::get("https://somewhere.com/app.zip")?.bytes()?;

    let mut file = File::create("app.zip")?;
    file.write_all(bytes)?;
}

void AppDownload()
{
    var webClient = new WebClient();
    var download_failed = new System.InvalidOperationException("Failed to download app.zip", e);

    // https://docs.microsoft.com/en-us/dotnet/api/system.net.webclient.downloadfile?view=net-5.0
    try { webClient.DownloadFile("https://somewhere.com/app.zip", "app.zip"); }
    catch (ArgumentNullException e) { throw download_failed; }
    catch (WebException e)          { throw download_failed; }
    catch (NotSupportedException e) { throw download_failed; }
}

enum Error {
    Connect(reqwest::Error),
    Download(reqwest::Error),
    AppCreateFile(std::io::Error),
    AppWriteToDisk(std::io::Error),
}

fn app_download() -> Result<(), Error> {
    let bytes = reqwest::blocking::get("https://somewhere.com/app.zip")
        .map_err(Error::Connect)?
        .bytes()
        .map_err(Error::Download)?;

    let mut file = File::create("app.zip").map_err(Error::AppCreateFile)?;
    file.write_all(bytes).map_err(Error::AppWriteToDisk)?;
}

API Implications

• Any possible failing functions must return Result.
• So, the return type of consumer-implemented functions must be type parameterized.
• Handle interrupts and stop gracefully.

choochoo constrains users to provide srcerr:SourceError<E>, which requires users to implement:

• Strongly typed error codes.
• Provide the error information that codespan-reporting can consume.

Importance

Arguably the most important dimension for robustness – ensuring that a system is not left in an unknown / unrecoverable state.

Live Demo

• Artifact server:

  mkdir /tmp/server && cd /tmp/server
  

  simple-http-server --nocache -u --ip 127.0.0.1 --port 8000 -l 100000000
  

• Build agent:

  mkdir -p /tmp/choochoo/demo/station_a/ && cd /tmp/choochoo/demo/station_a/
  

  for i in {1..200000}; do printf "application contents $i"; done | gzip -f > app.zip
  

  rm -rf /tmp/choochoo/demo/station_{b,c,d,e,f,g,h} /tmp/server/app.zip
  

• choochoo:

  time ./target/release/examples/demo
  

Scenarios:

• Happy Path.
• Cannot connect to artifact server.
• Stop server before downloading.

State Visibility

What state is it in now, progress tracking.

Is enough information captured and presented.

Maturity Levels

API Implications

• Need overall execution plan steps.
• Additional steps discovered at runtime are "sub-steps".
• Elapsed duration is essential.
• Accurate estimated duration remaining is nice, but can be difficult to provide.

Live Demo

• All tasks are listed.
• Each task has progress limit.
• Colours for failure / success / in progress.

Understandability

Presentation / verbosity.

Show information when it's relevant, hide information when it's not.

Maturity Levels

2021-02-21T15:13:01.123 | Debug | my::krate::module | Starting application.
2021-02-21T15:13:01.325 | Info | another::krate::module | Connecting to server.
2021-02-21T15:13:01.500 | Debug | third::krate::module | Resolving DNS for `some.domain.com`.
2021-02-21T15:13:01.531 | Debug | someones::krate::module | Looking up name server for `some.domain.com`
2021-02-21T15:13:01.570 | Debug | third::krate::module | DNS Resolved to 1.2.3.4
..

❯ ./target/release/examples/demo --concurrent
✅ a Upload App         [████████████████████████████████████████] 26.01KiB/26.01KiB (0s Ok!)
⠦⠦b Create DB          [███████████████████████████░░░░░░░░░░░░░] 68B/100B (0s 1s)
✅ c Download App       [████████████████████████████████████████] 26.01KiB/26.01KiB (0s Ok!)
⏳ d Link App to DB     [                                        ] 0/100 (queued)
⏳ e Run App            [                                        ] 0/100 (queued)
⠤⠤f Allocate Domain    [██████████████████████████░░░░░░░░░░░░░░] 67B/100B (0s 1s)
⏳ g Attach Domain      [                                        ] 0/100 (queued)
⏳ h Notify Completion  [                                        ] 0/100 (queued)

❯ ./target/release/examples/demo --concurrent
❌ a Upload App         [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 0B/772.42KiB (0s)
☠️ b Create DB          [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 0/100 (parent failed)
☠️ c Download App       [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 0/100 (parent failed)
☠️ d Link App to DB     [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 0/100 (parent failed)
☠️ e Run App            [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 0/100 (parent failed)
✅ f Allocate Domain    [████████████████████████████████████████] 100B/100B (0s Unchanged)
☠️ g Attach Domain      [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 0/100 (parent failed)
☠️ h Notify Completion  [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 0/100 (parent failed)
error[E06]: Failed to connect to artifact server.
  ┌─ artifact_server_address:1:8
  │
1 │ http://127.0.0.1:8000
  │        ^^^^^^^^^^^^^^ failed to connect to server
  │
  = Try running `simple-http-server --nocache -u --ip 127.0.0.1 --port 8000`.

API Implications

• Provide mechanism to specify end limit and progress.

• Collect information, present it in a useful way:

  • Real time interactive CLI presentation.
  • Useful errors.

choochoo currently uses indicatif to display progress, and codespan-reporting to display errors.

indicatif provides:

• Progress rendering
• Elapsed time
• Estimated time remaining
• Styling

Recoverability / Repeatability

Can execution resume from halfway.

Maturity Levels

API Implications

• Implementors should provide a "check" function, and a "do it" function.
• The check function checks if the task is in the desired state.
• If not, the do it function is called.

choochoo:

• Requires a visit_fn ("do it" function).
• check_fn is run if present, otherwise it always runs the visit_fn.
• Bonus: The check function is run after the visit function to detect if the logic is correct.

Live Demo

• Run it twice.
• Replay last 2 steps.
• Update the file.

time ./target/release/examples/demo

rm -rf /tmp/choochoo/demo/station_{g,h}

• 514 KB, 1 second

  for i in {1..200000}; do printf "application contents $i"; done | gzip -f > app.zip
  

• 5 MB, 7 seconds

  for i in {1..2000000}; do printf "application contents $i"; done | gzip -f > app.zip
  

Performance / Efficiency

How well is execution scheduled, time spent, and resources used.

Maturity Levels

API Implications

• Use async to operate concurrently.
• Use multiple threads for parallelism.
• Shared data must be behind appropriate RwLocks.
• Can hurt API ergonomics.
• Potential for deadlock if not designed well.

choochoo:

• Uses Pin<Box<dyn Future .. >> for the function return types – tasks cannot be worked on by different threads.
• Uses tokio as the async executor.

Live Demo

rm -rf /tmp/choochoo/demo/station_{b,c,d,e,f,g,h} /tmp/server/app.zip

time ./target/release/examples/demo

time ./target/release/examples/demo --concurrent

Psychology

Did the user request what they intended?

Does the software understand the intent of the user?

a.k.a. Can the developer predict intent ahead of time?

Maturity Levels

# User: "create 100 servers"
choochoo deploy --server-count 100

❯ choochoo deploy --server-count 100
warning[W02]: High server count.
  ┌─ command:1:16
  │
1 │ choochoo deploy --server-count 100
  │                 ^^^^^^^^^^^^^^^^^^ specified here
  │
  = This can incur high costs.
  = Specify the `--allow-high-server-count` flag to bypass this check.

  🚂 Please type `pancakes` to proceed:
  ❯ pancakes_

API Implications

• Provide types with value limit checking.
• Detect long elapsed time, and support graceful interrupt handling.
• Design for dry run – show what the software would do.

choochoo doesn't do this yet.

Future

To Do

• Dry run.
• Make API ergonomic.
• More granular execution.
• Refine shared data access.
• Handle signals (like SIGINT).

More Ideas

• Use choochoo not just for the deployment tool, but the agent running on the web server.
• Split core functionality, have web frontend.
• Serialize state to disk – save / load execution.

Questions, Feedback, Comments

Links

• choochoo repository
• ops_ux repository (this book)

Common Issues

• Execution fails halfway -- "like rolling the dice."
  • Flakey connections / services.
  • Misspecified configuration / parameters.
• User does incorrect action / hard to troubleshoot.
  • Too much information presented to user.
  • Not enough information presented to user.
  • No access to information.
  • Not knowing of access to information.
  • User may not understand information.
• Hacked automation.
  • Tools calling tools.
  • No proper API to invoke / receive information.
• Parallel / concurrent execution interferes with each other.
  • Inadequate namespacing.
• Cost / unused resources.
  • Forgot to clean up.
  • No means to clean up.
  • Unaware that resource is allocated but not freed.