In an effort to understand the new Rust async/await syntax, I made a super-simple app that simply responds to all HTTP requests with Hello! and deployed on Heroku.

Update: If you just want to create a webservice in Rust and deploy on Heroku, I recommend next blog post: rust on heroku with hyper http. This blog post focuses on the details of how the underlying request and response is handled with async/await, on stable Rust since 11/2019.

The full source code and README instructions can be found on github.com/ultrasaurus/hello-heroku-rust, tokio-only branch

Rust “hello world” app

Make a new project with cargo

cargo new hello_rust --bin
cd hello_rust
git init
git add .
git commit -m “cargo new hello_rust —bin”

cargo run

output:

   Compiling hello_rust v0.1.0 (/Users/sallen/src/rust/hello_rust)
    Finished dev [unoptimized + debuginfo] target(s) in 1.47s
     Running `target/debug/hello_rust`
Hello, world!

Heroku setup

Rust isn’t officially supported by Heroku yet, but there are lots of “buildpacks” which help to deploy a Rust app. I picked emk/heroku-buildpack-rust — most stars, most forks & recently updated!

We need the heroku CLI. I already had it and just did heroku update to sync to latest version (7.35.1). Then to set up the app on heroku:

heroku create --buildpack emk/rust

output provides a unique hostname by default:

Creating app... done, ⬢ peaceful-gorge-05620
Setting buildpack to emk/rust... done
https://peaceful-gorge-05620.herokuapp.com/ | https://git.heroku.com/peaceful-gorge-05620.git

We need a Procfile so heroku knows our entrypoint

echo "web: ./target/release/hello_rust" >> Procfile

Write the app

Add crate dependencies to Cargo.toml and add code to main.rs (and other files as with any Rust app). The emk/rust buildpack takes care of building everything as part of the heroku deploy.

The following lines (in Cargo.toml) will add all of tokio features:

[dependencies]
tokio = { version = "0.2", features = ["full"] }

I’d rather specify only what’s needed, but ran into something I couldn’t debug myself (issue#2050)

The core of the app accepts the sockets connections, but doesn’t read/write:

use std::env;
use tokio::net::TcpListener;

#[tokio::main]
async fn main() {
    // Get the port number to listen on (required for heroku deployment).
    let port = env::var("PORT").unwrap_or_else(|_| "1234".to_string());

    let addr = format!("0.0.0.0:{}", port);
    let mut listener = TcpListener::bind(addr).await.unwrap();

    loop {
        println!("listening on port {}...", port);
        let result = listener.accept().await;
        match result {
            Err(e) => println!("listen.accept() failed, err: {:?}", e),
            Ok(listen) => {
                let (socket, addr) = listen;
                println!("socket connection accepted, {}", addr);
                println!("not doing anything yet");
            }
        }
    }
}

Deploy on heroku

The above code will build and deploy, by simply pushing the code to heroku:

heroku push origin master

We can see what it is doing with heroku logs --tail:

Here’s where it starts the build and then kills the old app:

2020-01-05T03:45:31.000000+00:00 app[api]: Build started by user ...
2020-01-05T03:45:50.450898+00:00 heroku[web.1]: Restarting
2020-01-05T03:45:50.454311+00:00 heroku[web.1]: State changed from up to starting
2020-01-05T03:45:50.244579+00:00 app[api]: Deploy 399e1c85 by user ...
2020-01-05T03:45:50.244579+00:00 app[api]: Release v24 created by user ...
2020-01-05T03:45:50.701533+00:00 heroku[web.1]: Starting process with command `./target/release/hello_rust`
2020-01-05T03:45:51.741040+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2020-01-05T03:45:51.819864+00:00 heroku[web.1]: Process exited with status 143

Oddly, it seems to start the app before “State changed from starting to up” but it will fail if we’re not listening on the right port, so maybe that is as expected:

2020-01-05T03:45:52.343368+00:00 app[web.1]: listening on port 49517...
2020-01-05T03:45:53.322238+00:00 heroku[web.1]: State changed from starting to up
2020-01-05T03:45:53.303486+00:00 app[web.1]: socket connection accepted, 10.171.202.59:17201
2020-01-05T03:45:53.303545+00:00 app[web.1]: not doing anything yet
2020-01-05T03:45:53.303619+00:00 app[web.1]: listening on port 49517...
2020-01-05T03:45:53.313259+00:00 app[web.1]: socket connection accepted, 172.17.146.217:43686
2020-01-05T03:45:53.313285+00:00 app[web.1]: not doing anything yet
2020-01-05T03:45:53.313370+00:00 app[web.1]: listening on port 49517...
2020-01-05T03:46:28.000000+00:00 app[api]: Build succeeded
2020-01-05T03:46:48.251168+00:00 heroku[router]: at=error code=H13 desc="Connection closed without response" method=GET path="/" host=peaceful-gorge-05620.herokuapp.com request_id=a0d630d9-790a-47db-87af-67e680b27907 fwd="69.181.194.59" dyno=web.1 connect=1ms service=1ms status=503 bytes=0 protocol=https

So, the first socket connection above is some internal heroku checker, then when I attempt to go to the app URL in the browser, it fails (as expected).

Async read and write

I tried to keep the code clear with as little magic as possible. It’s a bit verbose (without even handling HTTP in any general way), but I found it helpful to see the details of read and write.

Note that adding use tokio::prelude::*; allows calling of read_line (defined in tokio::io::AsyncBufReadExt) and write_all (defined in tokio::io::AsyncWriteExt).
The additional code reads the bytes from the socket line by line until we get the the end of the HTTP Request (signalled by a blank line). So we look for two CLRFs (one at the end of the last header line and one for the blank line).

tokio::spawn(async move makes it so sure we can read/write from one socket while also listening for additional connections. tokio::spawn will allow the program execution to continue, while concurrently allowing our async function process_socket to read and write from the socket. Because we added #[tokio::main] above our async fn main entry point, tokio will set up an executor which will wait for all of our spawned tasks to complete before exiting.

use std::env;
use tokio::net::TcpListener;
use tokio::prelude::*;

#[tokio::main]
async fn main() {
    // Get the port number to listen on (required for heroku deployment).
    let port = env::var("PORT").unwrap_or_else(|_| "1234".to_string());

    let addr = format!("0.0.0.0:{}", port);
    let mut listener = TcpListener::bind(addr).await.unwrap();

    loop {
        println!("listening on port {}...", port);
        let result = listener.accept().await;
        match result {
            Err(e) => println!("listen.accept() failed, err: {:?}", e),
            Ok(listen) => {
                let (socket, addr) = listen;
                println!("socket connection accepted, {}", addr);
                // Process each socket concurrently.
                tokio::spawn(async move {
                    let mut buffed_socket = tokio::io::BufReader::new(socket);
                    let mut request = String::new();
                    let mut result;
                    loop {
                        result = buffed_socket.read_line(&mut request).await;
                        if let Ok(num_bytes) = result {
                            if num_bytes > 0 && request.len() >= 4 {
                                let end_chars = &request[request.len() - 4..];
                                if end_chars == "\r\n\r\n" {
                                    break;
                                };
                            }
                        }
                    }
                    if let Err(e) = result {
                        println!("failed to read from socket, err: {}", e);
                        return;
                    }
                    let html = "<h1>Hello!</h1>";
                    println!("request: {}", request);
                    let response = format!(
                        "HTTP/1.1 200\r\nContent-Length: {}\r\n\r\n{}",
                        html.len(),
                        html
                    );
                    let write_result = buffed_socket.write_all(response.as_bytes()).await;
                    if let Err(e) = write_result {
                        println!("failed to write, err: {}", e);
                    }
                });
            }
        }
    }
}

Background

Here’s my environment info (rustup show):

stable-x86_64-apple-darwin (default)
rustc 1.39.0 (4560ea788 2019-11-04)

Reference docs

  • https://docs.rs/tokio/0.2.6/tokio/net/struct.TcpListener.html
  • https://docs.rs/tokio/0.2.6/tokio/net/struct.TcpStream.html
  • https://docs.rs/tokio/0.2.6/tokio/task/fn.spawn.html

“There are things that you don’t even realize that you can do.” In a recent podcast, B. Mure tells about graphic facilitation:

I didn’t really know it was a skill to have: to listen to people and very immediately draw something related to what they were talking about and present their ideas in a visual way.

Sometimes this is also called making “sketch notes.” He goes on to talk about a general phenomenon where you discover that you can do a thing that you never thought was a thing:

There are so many things that, if you are not given the opportunity, if somebody doesn’t see that within you, or thinks maybe you should just come along and try this thing that I’ve organized, there are things that you don’t even realize that you can do.

I think there are two parts of this that are transformational, that I’ve experienced myself and occasionally been able to spark in others.

  1. Naming a pattern of actions or behavior: this is an act of creating that is rare and powerful. Discovering something is a skill that one can be good at and apply with intention is aided by that skill having a name.
  2. Recognizing the spark within someone else: seeing a capability, especially latent potential, within someone and naming it, inviting that person to experiment with a new skill, encouraging creative action in the world.

There should be a word for the making of words that is more than coining a term, where the naming of a thing helps people do (or not do) whatever that is.

I’ve thought a lot about how the term mansplaining has helped a generation notice and often modify behavior. I learned about restorative justice as a framework for transforming guilt into responsibility. I remember when, at an early RailsBridge workshop, I applied lessons from my kid’s preschool to the challenge of how to teach without grabbing the keyboard (“Use your words”).

I love the idea of a word for a skill providing a path to developing that skill, connecting to a community, and finding paid work.


The whole podcast with Dan Berry and B. Mure is worth listening to for any creative folk or if you enjoy comics or visual storytelling and want a glimpse of that world.

Image from website of B. Mure: the artist in sunshine looking skyward with arms spread wide upward in front of mountains and blue sky
https://bmuredraws.com/

As I think about developing new Internet-connected software, I worry about the safety of the people who use it. By 2021, most Web browsers won’t allow native code extensions, which will eliminate a lot of potential issues, while a hug swath of creative animations and interactives will disappear from the Web. I spent some time this summer, thinking about what I could learn from the security vulnerabilities in the Flash Player that has been much maligned in recent years.

Flash CVEs (2001-2009)

I looked at the Common Vulnerabilities and Exposure List (CVE List hosted by Mitre with all reports 2001-2019. I found 1172 Flash Player vulnerabilities, which sounds huge, but in context of vulnerabilities reported in Web Browsers, doesn’t look that bad:

  • 1172 Flash Player
  • 1999 Internet Explorer
  • 2033 Chrome
  • 2442 Firefox

Note: these numbers don’t necessarily tell us that Firefox had more vulnerabilities than Internet Explorer. It could mean that Firefox was more rigorously open in reporting vulnerabilities, which seems likely.

Understanding attack vectors

Vulnerabilities in the Flash Player were particularly dangerous because Flash was installed on all of the Web Browsers, so any flaw in Flash was much easier to exploit than a flaw in a specific Web Browser. To understand this, one needs to understand that the primary attack vector enabling a hacker to take advantage of a vulnerability in Flash Player was to create a malicious Flash application or movie that would distract the user while doing something illicit or intentionally trigger a crash and then exploiting that crash to execute native code with access to the user’s machine.

In the larger context of a specific attack, a vulnerability in the Flash Player would typically need to be combined with something else:
* deceptive emails (aka phishing)
* deceptive websites
* “man in the middle” attacks (replace real Web content with malicious content that appears identical)

Categorizing vulnerabilities

I conducted a rough cut analysis of matching terms by reading the list of CVEs and creating categories that might provide instructive value in thinking through how to avoid similar issues in the future.

pie chart illustrating data in table below

  • 802 memory safety
  • 42 other code execution
  • 58 XSS, CORS, CLRF
  • 61 parsing / validation
  • 13 clickjacking
  • 91 bypass sandbox
  • 105 other

Memory safety (~70%)

The vast majority of issues (“memory safety”) resulted from coding errors, which can now be avoided with modern programming languages. For a long time, we’ve been able to use languages like Erlang/Elixir, Java, Python, Ruby, and Go for server-side coding with memory safety features. Even C++ has language features and libraries (though you must choose to use them). Now, for low-profile client software we can use Rust or WebAssembly when we need something higher performance or less memory-hungry than JavaScript.

Escaping the “sandbox” (~15%)

If we develop code that runs in a Web browser, we can trust the browser’s “sandbox” — our apps can only use a restricted set of APIs. If we’re writing a Web browser or any other Internet-connected software used by humans or machines, then it is a good idea to carefully isolate our code that can access the operating system to write files or make network calls.

From my CVE analysis, coding errors in this category resulted in just over 15% of CVEs (other code execution, bypassing sandbox, and XSS, CORS, CLRF issues). Of course, the biggest thing you can do is not include the code that does powerful things you don’t want to allow. However, sometimes you do need to load and execute a shared library, accessing the filesystem and the network.

Parsing / Validation (~5% / ~15% excluding memory safety)

Parsing and validation of input (mostly reading a file or parameter) is another common coding error pattern which can result in a serious vulnerability. Having to fix these kinds of issues causes me to be very careful when adding parsing code to any app or library. If we exclude memory safety errors, parsing and validation errors are larger than any identified class of error.

pie chart excluding memory safety shows 16% parsing / validation

Clickjacking and “other”

Clickjacking is noteworthy for anyone developing a client app with extensions where 3rd party developers (or other users via content sharing) can present information to the user and allow interaction. This class of attack uses features that are designed to empower users to present compelling content to be instead used to trick people into doing something unintended. For example, there were bugs that allowed Flash content to overlay other web pages or browser UI, thereby tricking the user into clicking or typing in a way to provide privileged access.

Perhaps “other” deserves a closer look, but I didn’t find clear patterns and suspect that contains many smaller categories.

Parsing is hard

In my experience, many programmers recognize that implementing an extension mechanism that allows for user interaction or providing a “sandbox” for 3rd party code can be very tricky to get right and will exercise great care in writing or using that kind of code. However, I have often interacted with programmers who don’t seem to believe that writing code to parse text is difficult. Writing code that performs the intended action is not hard, but writing code that has no unintended effects requires very careful coding and a little imagination.

Looking toward open source code for some examples to learn from, here are a few examples of URL parsing libraries where bugs were found (and fixed) after vulnerabilities were discovered in the field:

  • https://github.com/envoyproxy/envoy/issues/7728 (Envoy proxy)
  • https://go-review.googlesource.com/c/go/+/189258/ (Go)
  • https://www.cvedetails.com/cve/CVE-2018-3774/ (url-parse Node library)

The results of this analysis were included as part of Code Mesh LDN 19 talk, A landscape of unintended consequences (video, slides). The data and methodology is available at on github: ultrasaurus/flash-cve-analysis.