Reading & writing files

Our Markdown compiler is now at a point where it can accept command-line input and reliably displays some information based on the input provided.

In this chapter, we will configure our tool to read and parse a Markdown file, translate it into HTML, then write that HTML to a new file.

By the end of this chapter, we will have a working (and naive) Markdown compiler written in Rust.

# Opening & reading files

The primary function that will contain the logic of our Markdown compiler is parse_markdown_file(), which takes a single argument: a string slice (&str) that corresponds to the Markdown file we want to create.

Let’s go ahead and create that dummy file right now. Copy all of the below text into a new file called test.md, and save it in the root of your Rust project. The root of your Rust project also contains the manifest file (Cargo.toml), so just make sure test.md and Cargo.toml are in the same directory.

If you would rather just download test.md directly, you can right-click and save a copy via this link

# My favorite author

This is a report about my favorite writer. His name is Jesse Lawson.

# Jesse's favorite food

Jesse really likes enchiladas and any kind of sushi. 

# Jesse's favorite drink

Jesse likes to drink coffee in the mornings and iced tea throughout the day. Sometimes, he even drinks water. 

# Jesse's favorite hobbies

Jesse likes to write about computer programming and game design, and when he is not hunched over a computer, you can find him out on a run and listening to a podcast or the serenity of mother nature. 
 

As you can see, one of my most endearing qualities is my humble modesty.

With test.md in the root of our project folder, we now have an actual file to open and read from.

# Designing our parser

At this point in our program, file is our way of accessing the file passed into the function. With the file open successfully, we can start planning how we will read and parse it.

While this is not a course on compiler design, we need to think about how a compiler looks at a file. When the program gets to the first character in the file, which is a # (indicating a heading tag), it needs to know that the rest of the line will contain all the text within a heading tag.

To illustrate, here is the first line of our test.md, annotated:

# My favorite author 
^ ^________________^ ^
|         |          |
|         |          +--- end of line; output </h1>
|         |
|         +--- put this inside the <h1></h1> tags
|
+--- start of heading; output <h1>

To keep track of what stage the parser is at, we create two kinds of state variables called flags. The first flag is whether a header tag is open. If a header tag is open, that means our HTML file would have a <h1> but no corresponding </h1> yet.

The second flag we will keep track of is whether a paragraph tag is open. Remember: for the purposes of this tutorial, our Markdown compiler can only handle first-level headings (#) and paragraphs.

Let’s create two mutable boolean variables to keep track of these states:

let mut ptag: bool = false; // keep track of paragraph tags
let mut htag: bool = false; // keep track of h1 tags

Notice that I’m using an underscore (_) here to precede their names. In Rust, if you instantiate a variable like this but never use the value you use to instantiate it, you will get a warning about unused variable assignments. Rust gets deeply concerned with the fact that you have set a value to a variable that is essentially garbage (since the falseness of these flags is just their default value going into the for-loop). You can safely remove the underscore, but every time you run cargo build (or cargo run which triggers a build), you will get a harmless warning about an unused variable assignment.

With these two flags we can track what kind of tags we are currently reading for, as well as what kind of tag needs to be closed before moving on to the next line in the file.

Speaking of lines, we need a way to store the resultant HTML dynamically, so that we can add to it before writing it all to a file. What kind of variable type do you think we could use for this?

The way we will do it here is through a vector of strings. I’ll call this vector tokens, to help me remember what’s in here. In compiler design, a token is a keyword, operator, separator, or string literal (or some other kind of lowest form of understandable syntax) that the compiler understands. Our vector elements are each going to be a String, and each string element will be made up of either a heading or a paragraph in HTML.

Let’s instantiate our tokens vector:

let mut tokens: Vec<String> = Vec::new();

We make it mutable because we want to add to it; we use Vec::new() to create an empty vector object.

Now we are ready to read the file line by line. Once again, we’re going to call upon one of Rust’s tools for doing just that: BufReader, which is, as you can derive from the name, a buffered file reader. Reading files from the filesystem can be a daunting task for programs, especially if these files are gigantic. To ensure that a 1gb Markdown file will be read about as fast as a 1mb Markdown file, we buffer the input so it reads the file in chunks. Don’t worry: all the heavy lifting happens behind the scenes.

Let’s go to the top of main.rs and add two new entries to the block of use statements:

use std::io::{BufRead, BufReader};

Note that we can combine calls from the same namespace by adding them to a comma-separated list between curly brackets. Neat!

Both BufReader and BufRead are necessary to do what we want to do. The first one lets us buffer a file into memory; the second lets us read those buffers line by line.

Back where we left off in the parse_markdown_file() function, let’s create a variable named reader that will open our file:

let reader = BufReader::new(file);

There! Not so bad, right? The variable reader is now essentially our window into a memory-optimized lens through which we can read the file.

At this point, your parse_markdown_file() function should look something like this:

fn parse_markdown_file(_filename: &str) { 
  print_short_banner();
  println!("[ INFO ] Starting parser!");
  
  // Create a path variable from the filename
  let input_filename = Path::new(_filename);

  // Try to open the file
  let file = File::open(&input_filename)
             .expect("[ ERROR ] Failed to open file!");

  let mut ptag: bool = false;
  let mut htag: bool = false;

  // Create a place to store all our tokens
  let mut tokens: Vec<String> = Vec::new();

  // Read the file line-by-line
  let reader = BufReader::new(file);

  // ...

With the file now open, we can iterate through it one line at a time and start translating Markdown to valid HTML.

# Making a Path variable

Rust goes to great lengths to make sure that your program is cross-platform. One of the ways it does this is by providing the Path module, which helps format strings and string slices into OS-specific path types. The official Rust docs recommend to use a Path variable instead of a string slice anytime you’re working with filenames. This is one of several special tools from Rust that comes from Rust’s std library–which, for the purposes of this tutorial, you can think of as like a namespace.

In Rust, we can tell our program that we want to use Path objects by going to the top of main.rs and adding:

use std::path::Path;

Now we can create a Path object from the argument passed to the parse_markdown_file() function:

fn parse_markdown_file(_filename: &str) {
  print_short_banner();
  println!("[ INFO ] Starting parser!");
  
  // Create a path variable from the filename
  let input_filename = Path::new(_filename);

The call to Path::new() creates a new Path object for us. Now, input_filename is a Path that we can try to open. Again, don’t use a String object to hold a filename, since the Path object is specificially designed to play well with Rust’s other tools for opening, reading, and writing to files.

With our path now an official Rust Path variable, let’s pull in another tool to be able to open the file that the path points to: File.

Just like when we created the Path variable, we will first need to declare that we want to use it by adding the following to the top of main.rs:

use std::path::Path;
use std::fs::File;

Notice that File comes from std::fs, whereas Path comes from std::path.

Back to our parsing function, after declaring the input_filename path variable, let’s create a new File variable with it:

  // Create a path variable from the filename
  let input_filename = Path::new(_filename);

  // Try to open the file
  let file = File::open(&input_filename)
             .expect("[ ERROR ] Failed to open file!");

Interesting! Now we have a semantic way to open a file using the File::open() function, to which we pass a reference to input_filename, and then to which we chain the .expect() function. You will encounter the .expect() function a lot in your Rust development; it’s used to remove the verbosity around Rust’s Result type.

In a nutshell, many functions in Rust do not just return a value, they return a Result. A Result in Rust has two parts: Ok() and Err(). When you call a function that returns a Result type, you generally need to check whether the function was successful (and thus returned an Ok()) or not successful (and thus returned an Err()). It’s akin to exception handling in other languages, except here it’s baked into the function itself.

What the .expect() does is tell Rust to unwrap the return value and pass along the Ok()—except upon failure, in which case the program quits and emits a string ("Couldn't open file").

So when we write something like this:

let file = File::open(&input_filename).expect("Couldn't open file");

We’re basically writing a less verbose version of this:

use std::error::Error;
// ...
let file = match File::open(&input_filename) {
  Err(err) => panic!("Couldn't open file: {}", err.description()),
  Ok(value) => value,
};

There are a lot of new things in the verbose example above, but since you have some experience interpreting match blocks in Rust already, try to see what’s going on. Notice, for example, how we can assign a value to a variable based on the result of a match block.

# Line-by-line parsing

Now it’s time to actually do something with every file line that we read. To do that, we want to loop through the lines in the file that we are given with the reader variable. This reader variable, being assigned the value of the BufReader::new(file) call, now gives us a .lines() method that we can iterate over with a simple for-loop.

In Rust, a for-loop’s syntax is “for x in y”, like this:

for line in reader.lines() {

Perfect: now each line buffered into reader can be parsed.

We’re not quite ready to read the line yet, though. Remember when we did that condensed and verbose example of File::open() because it returned a Result object? Well, line here is also a Result object! So how do we get the value of the line–the actual contents–out?

You might think we need to use a match block, like the verbose example for File::open(). While you can do it this way, there’s really nothing we care to do if the line is empty (i.e., if the Result object produces an Err()). Think about it: if the file was opened correctly, and the contents were buffered correctly, what could cause the line variable within this for-loop to be an error?

Well, it might be the end of the file! So instead of doing error checking on this line every time, we’ll just do a Rust trick called unwrapping. When you unwrap an Result object, you are telling Rust that you 1) expect the value to be available, and 2) don’t care if the value is garbage.

If you wanted to do it the verbose way, you might write:

let line_contents = match line {
      Ok(contents) => contents,
      Err(e) => panic!("Garbage: {}", e.description())
};

Notice how we would have to match each of the Result elements (Ok() and Err()). We don’t really need all this because, frankly, it’s a bit overkill for just reading the contents of a line in a toy markdown compiler. So instead of manually unwrapping the Result object, we can just use Rust’s .unwrap() method:

// Verbose way:
/*let line_contents = match line {
      Ok(contents) => contents,
      Err(e) => panic!("Garbage: {}", e.description())
};*/

// Condensed way:
let line_contents = line.unwrap();

Let’s look at what we have so far for the for-loop, complete with some comments:

// Loop through the reader lines
for line in reader.lines() {
  // For each line, unwrap it
  let line_contents = line.unwrap();

You’re doing great! Get a drink of water, stretch your back and legs, and come back when you’re ready.

# Getting the first character of a line

With a line from the Markdown file represented as a string variable called line_contents, we can now figure out what kind of line this is. There are really only two kinds of lines we care about for this naive Markdown compiler: first-order headings and paragraphs.

If a line starts with a hash (#) symbol, then it’s a first-order heading. If, however, it starts with any other alphanumeric value, it’s a paragraph.

You might think that we would do something like line_contents[0] to get the first character of line_contents, but Rust doesn’t store strings as a sequence of characters in memory. Instead, we have to first convert the variable into a sequence of characters, take the first one from that sequence, and then convert it into something that we can use.

To do this, we will create a new variable called first_char which will (you guessed it) hold the first character of line_contents. The variable type will be a vector of characters, and we will create it like this:

let mut first_char: Vec<char> = line_contents.chars().take(1).collect();

Let’s look at the right-hand one piece at a time:

• line_contents.chars() says, “Get the line_contents variable and convert it to a sequence of characters.” (Rust will create an Iterator object of characters)
• .take(1) says, “Now take the first element of that iterable object.” (Rust will convert this Iterator into a Take<char> object, a special kind of iterator)
• .collect() says, “Now convert everything I have retrieved up to this point into a Collection–something that I can subsequently use–of the type matching the left-hand variable (which is Vec<char>)

It’s a lot of steps, but the more you read them from left to right the more they will make sense. Rust is a language that feels like C but behaves like Ruby; there is a lot going on behind the scenes! At the end of the day, the most important part of the above expression is that collect() takes all the work we did of pulling out the first character and collects up the result in a vector for us.

At this point we have line_contents, which holds the entire contents of the line, and first_char, which has the first character of line_contents. Neat!

We’re about to start doing different things based on what the first character of the line is, but before we do, let’s create a new string variable that will hold the valid HTML that the current line we are reading (line_contents) will translate to:

let mut output_line = String::new();

Just like when we created a Path variable by using Path::new(), we’re using String::new() to declare a mutable variable named output_line.

Our main output variable is tokens; that’s a vector a strings that will contain one string object for every line that comes through from the file. When we are done processing line_contents based on the value of first_char (whether it is a # or not), we will write valid HTML to output_line and then push output_line into tokens. Then, in the last section of this chapter, we will iterate through the strings in tokens and write them all to an output file.

# Compiling Markdown to HTML

Let’s now write some HTML based on what the first character of the line is.

Remember that first_char is a vector that only has one element. To see it, we will use the pop() method that Rust provides to vectors, which will not return the first character of the line, but rather, an Option object.

Just like Result, Option is made up of two pieces–but they’re called Some() and None() instead of Result’s Ok() and Err().

When you pop and element from a vector in Rust, you will either get some value or none. We really only care if we are getting some character that looks like #–or, put another way, Some('#').

The match block has the same kind of syntax that you may recall from before, this time with the default case (_) and Some("#"):

match first_char.pop() {
  Some('#') => {},// The first character is #
  _ => {} // The first character is not # 
}

Let’s work on the first-order heading matches first.

What are the things that need to happen when we arrive at a line that starts with the # character? Don’t think about the lines in sequence; think about the problem and try to come up with what would need to happen algorithmically, regardless of which line we are on.

What do we know about the # character? Well, we know it corresponds to a <h1> tag–an opening tag for a first-order heading. We also know that these should not be nested; in no case should a <h1> tag be followed by a <p> tag, nor should a <p> tag be followed by a <h1> tag. So the first thing we should do is check whether either of these tags are open.

Remember those boolean flag variables we created earlier? They’re finally coming into play!

We’re going to do the Some('#') match block first, and then the default case, denoted by an underscore character (_).

We’ll first check if the ptag is set, since we don’t want to start a new heading tag without first closing an open <p> tag.

If it’s set, we’ll unset it (by marking it false), then send a closing </p> tag and a newline character to the output_line string–which is the string we are creating inside this loop iteration that will be pushed to tokens when we are done processing line_contents:

Some('#') => {
  if ptag {
    ptag = false;
    output_line.push_str("</p>\n");
  }

Other than the syntax for the if block, there’s nothing new here. We are setting ptag to false and then using .push_str() to append a string literal onto the end of output_line.

Next, we are going to perform the same check for htag:

Some('#') => {
  if ptag {
    ptag = false;
    output_line.push_str("</p>\n");
  }

  if htag {
    htag = false;
    output_line.push_str("</h1>\n");
  }

At this point, we have accounted for the two kinds of tags that our compiler knows about; we checked for open paragraph and first-order heading tags, and if we found an open one, we closed it properly. The next thing to do is to set the htag flag to true, then push a new heading tag to output_line.

How do you think we will do that?

htag = true;
output_line.push_str("<h1>");

htag = true;
output_line.push_str("\n\n<h1>");

htag = true;
output_line.push_str("\n\n<h1 class=\"report-title\">");

The easiest way is to just use the .push_str() method from way back in Chapter 3:

htag = true;
output_line.push_str("\n\n<h1>");

At this point, we are almost done processing this line! The last step of this iteration is to actually push the contents of line_contents minus the starting character # and the space next to it onto output_line.

We want line_contents minus the first two characters. Here’s why:

# Sometext
^^
||
|+--- This space is unnecessary for <h1>Sometext</h1>
|
+--- This character converts to <h1>

To get all of line_contents except the first two characters, we can use a special string slice generation method that will feel very familiar to Python developers: &line_contents[2..]. The [2..] says “Take a string slice of line_contents starting at the element in index 2 (so, the third character) and go all the way until the end of the string.”

To push this to output_line, we just pass it by reference to .push_str():

  htag = true;
  output_line.push_str("\n\n<h1>");
  output_line.push_str(&line_contents[2..]); // Get all but the first two characters
}, // end of the Some('#') => { ... } block

The Some('#') match block is complete, but we have one more case to account for before closing it completely: the default case.

For this, the only check we care about is whether there is no ptag. If we read a line that doesn’t start with a #, then what are our priorities? Well, to start a paragraph tag if one isn’t already open! So we’ll check if ptag is false, and if it is, we’ll set it to true and then push <p> to output_line. When we’re done, we will push all of line_contents to output_line.

With those parameters, see if you can finish the match block’s default case by yourself, then open the solution below to see how you did.

 _ => {
    
    if !ptag {                      // If ptag is false,
      ptag = true;                  // set it to true, then 
      output_line.push_str("<p>");  // push a <p> to the output line.
    } 

    output_line.push_str(&line_contents); // Push the whole line to the output line.
    
  }
};

At this point, the match block is finished!

Here’s about what it should look like so far:

 1match first_char.pop() {
 2    Some('#') => {
 3      if ptag {
 4        ptag = false;
 5        output_line.push_str("</p>\n"); // adding \n for instructional clarity
 6      } 
 7      if htag {
 8        htag = false;
 9        output_line.push_str("</h1>\n"); // close it if we're already open
10      }
11
12      htag = true;
13      output_line.push_str("<h1>");
14      output_line.push_str(&line_contents[2..]); // Get all but the first two characters
15    },
16
17    _ => {
18      
19      if !ptag {
20        ptag = true;
21        output_line.push_str("<p>");
22      } 
23
24      output_line.push_str(&line_contents);
25      
26    }
27};

We’re almost finished with this section. Before we can push output_line into the tokens vector (which we will ultimately be writing to the output file), we have three more checks we have to do.

At this point in the program, we have completed checking whether the line is a heading or a paragraph, and we have both opened the appropriate tag and pulled in the contents of the line appropriately. The final three checks we need to do are to 1) check if there’s still an open paragraph tag, 2) check if there’s still an open heading tag, and 3) check if the line is empty, since we don’t really care to write an empty set of tags to the output file.

Right after the closing bracket of the match block, we’ll do our first check: if the paragraph tag is open, close it and push a closing HTML tag.

  if ptag {
    ptag = false;
    output_line.push_str("</p>\n");
  }

Next, we’ll do the same for the heading tag:

  if htag {
    htag = false;
    output_line.push_str("</h1>\n");      
  }

Finally, we’ll avoid pushing blank lines by making sure output_line is not equal to two empty paragraph tags:

  if output_line != "<p></p>\n" {
    tokens.push(output_line);
  }

} // end of "for line in reader.lines()" block

Notice that after the final check above, we are closing out the for loop. Finally!

At this point, our compiler successfully reads and parses paragraph and first-order headings from Markdown to HTML. In the next section, we will derive the name of our output file based on the name of the input file, then write the contents of tokens (which, remember, holds all of the output_line iterations from the for-loop) to our output file.

Let’s add a quick loop that will iterate over tokens and then print out the value of each element–which will give us the resultant HTML that will be written to the output file in the next section.

Try to write a for-loop that iterates over tokens and calls println!() on each element, then check your work against the solution below.

for t in &tokens {
  println!("{}", t);
}

You may be wondering about that open reader that we never closed. In other languages it’s often necessary to close file pointers. Thankfully, Rust will do this automatically when reader falls out of scope.

Let’s continue on and the finishing touches on the parse_markdown_file() function—writing our results to a file.

# Writing to a file

Remember when we had to include those Rust libraries when we wanted to open and read a file into a buffer? We will have to do the same thing for creating and writing to a file. Fortunately, it’s only one addition.

At the top of main.rs, add a use block for the Write library:

use std::io::Write;

Now let’s head back to the bottom of the parse_markdown_file() function, and think about all we need to do.

By this point, the tokens vector has a bunch of string elements that need to be written to a file. Before we can do that, we need to know which file we are going to be writing to. Most tools have a way of specifying the output file, but since ours is a naive one, we will just have it automatically derive the output file name from whatever we passed as the input file.

Recall that the argument variable for the filename is _filename, and since we are passing it a file called test.md, then the value for _filename is also test.md.

Let’s have our output file be the same name as the input file, minus the last three characters (“.md”) and plus five new ones (“.html”). Our first task will be to get the name of the file without the extension. Again, we’ll be assuming that the only kind of file being passed is a *.md. I’ll leave it to you as a future challenge to accept different filetypes (like *.markdown).

Back in Chapter 4, we learned that we can access specific parts of a string slice by using brackets. For example, if we wanted all but the first three characters of a string slice called example, we could get them like this:

&example[3..]

Likewise, if we wanted to get all but the last three characters, you might think we could do it like this–&example[..-3]–but this is incorrect. In Rust, the bracket notation for string slices must be an unsigned integer that is equal to or less than the length of the string slice.

If we think about our intent with this code ([..-3]–which, again, is NOT valid Rust syntax), we are basically asking for the entire length of the slice minus the last three characters. Since Rust cannot infer that this is what we want, we can explicitly tell it how many characters to use by passing it the length of the string slice first.

The length of example would be example.len(), and to get all but the last three characters, we would put the call to .len() right in the brackets: &example[..example.len()-3].

Substituting out example for _filename, how do you think we might create a mutable String variable called output_filename from a reference to _filename containing all of _filename except the length of _filename minus three characters?

Try it first, then check your work against the solution below.

  // Create an output file based on the input file, minus ".md"
  let mut output_filename = String::from(&_filename[.._filename.len()-3]);

At this point, if we passed in test.md as the filename argument, the value of output_filename would be test. What are we missing?

The .html!

Do you remember how to push a string onto the end of a String object?

output_filename.push_str(".html");

With the name of our file ready, we now need to create the actual file. To do this, we will use File::create(), which returns a Result object. Instead of unpacking the Result object, though, we’re going to continue to use .expect():

let mut outfile = File::create(output_filename)
  .expect("[ ERROR ] Could not create output file!");

By this point, I hope you’re comfortable reading the above code. We’re creating a mutable variable outfile equal to the result of File::create(), into which we pass the output_filename. The call to .expect() will trigger only if there was an error creating the file.

With the file created, we are FINALLY ready to loop through tokens and write each element to the output file. Assuming a successfully created outfile, we now have access to a byte-writer called .write_all(). The way it works is like this: for each line in tokens, write each line as a byte sequence to the outfile.

Here it is in code:

for line in &tokens {
  outfile.write_all(line.as_bytes())
    .expect("[ ERROR ] Could not write to output file!");
}

So outfile.write_all() takes a string as bytes (line.as_bytes()) and stuffs it into the output file. Neat!

Remember that we borrow a reference to the tokens vector (like this: &tokens) because of Rust’s ownership rules. If we didn’t include that &, the value of each element in tokens would be moved into the for-loop and removed from outside of it–and we don’t want that!

Remember, too, that outfile will automatically be closed once it falls out of scope–which will be the at end of the parse_markdown_file() function.

Now that we have finished writing the tokens vector to the output file, let’s add some helpful output to let the user know that the parsing is finished:

println!("[ INFO ] Parsing complete!");

We can now put the closing bracket on our parsing function and test our compiler. You should already have a file called test.md in the root of your project (the same directory as the manifest file). We can trigger a build and a quiet run of the tool by running:

$ cargo run -q test.md

You should see something like this:

$ cargo run -q test.md
tinymd (v0.1.0), A tiny markdown compiler based on Jesse's tutorials.
[ INFO ] Starting parser!
[ INFO ] Parsing complete!
$

Now check the root of your project directory for a new file called test.html. If you open it in your editor, you should see valid HTML:

<h1>Welcome</h1>
<p>Welcome to my website.</p>
<p>Here's what I like: enchiladas, coffee, and plants.</p>

And with that, we are finished!

We have successfully built a tiny Markdown compiler in Rust!

In this chapter, we built a Markdown compiler! We opened a file, read it into memory, parsed it one line at a time, then wrote results to a file. Great job!

Try creating some Markdown files and passing them to the compiler. While you’re doing that, ask yourself:

• Is it working as expected?
• What happens when you use a Markdown tag that isn’t supported yet?
• How could you implement support for a new tag?

The only thing left to do now that our compiler has been constructed is to build a release version—which we are going to do in the next and final chapter of this book.

“Perseverance is not a long race; it is many short races one after another.”

— Walter Elliot

📦 All the code up to this point is available on GitHub.