Macros - The Rust Programming Language (2023)

Macros

We’ve used macros like println! throughout this book, but we haven’t fullyexplored what a macro is and how it works. The term macro refers to a familyof features in Rust: declarative macros with macro_rules! and three kindsof procedural macros:

• Custom #[derive] macros that specify code added with the derive attributeused on structs and enums
• Attribute-like macros that define custom attributes usable on any item
• Function-like macros that look like function calls but operate on the tokensspecified as their argument

We’ll talk about each of these in turn, but first, let’s look at why we evenneed macros when we already have functions.

The Difference Between Macros and Functions

Fundamentally, macros are a way of writing code that writes other code, whichis known as metaprogramming. In Appendix C, we discuss the deriveattribute, which generates an implementation of various traits for you. We’vealso used the println! and vec! macros throughout the book. All of thesemacros expand to produce more code than the code you’ve written manually.

Metaprogramming is useful for reducing the amount of code you have to write andmaintain, which is also one of the roles of functions. However, macros havesome additional powers that functions don’t.

A function signature must declare the number and type of parameters thefunction has. Macros, on the other hand, can take a variable number ofparameters: we can call println!("hello") with one argument orprintln!("hello {}", name) with two arguments. Also, macros are expandedbefore the compiler interprets the meaning of the code, so a macro can, forexample, implement a trait on a given type. A function can’t, because it getscalled at runtime and a trait needs to be implemented at compile time.

The downside to implementing a macro instead of a function is that macrodefinitions are more complex than function definitions because you’re writingRust code that writes Rust code. Due to this indirection, macro definitions aregenerally more difficult to read, understand, and maintain than functiondefinitions.

Another important difference between macros and functions is that you mustdefine macros or bring them into scope before you call them in a file, asopposed to functions you can define anywhere and call anywhere.

Declarative Macros with macro_rules! for General Metaprogramming

The most widely used form of macros in Rust is the declarative macro. Theseare also sometimes referred to as “macros by example,” “macro_rules! macros,”or just plain “macros.” At their core, declarative macros allow you to writesomething similar to a Rust match expression. As discussed in Chapter 6,match expressions are control structures that take an expression, compare theresulting value of the expression to patterns, and then run the code associatedwith the matching pattern. Macros also compare a value to patterns that areassociated with particular code: in this situation, the value is the literalRust source code passed to the macro; the patterns are compared with thestructure of that source code; and the code associated with each pattern, whenmatched, replaces the code passed to the macro. This all happens duringcompilation.

To define a macro, you use the macro_rules! construct. Let’s explore how touse macro_rules! by looking at how the vec! macro is defined. Chapter 8covered how we can use the vec! macro to create a new vector with particularvalues. For example, the following macro creates a new vector containing threeintegers:

#![allow(unused)]fn main() {let v: Vec<u32> = vec![1, 2, 3];}

We could also use the vec! macro to make a vector of two integers or a vectorof five string slices. We wouldn’t be able to use a function to do the samebecause we wouldn’t know the number or type of values up front.

Listing 19-28 shows a slightly simplified definition of the vec! macro.

Filename: src/lib.rs

#[macro_export]macro_rules! vec { ( $( $x:expr ),* ) => { { let mut temp_vec = Vec::new(); $( temp_vec.push($x); )* temp_vec } };}

Listing 19-28: A simplified version of the vec! macrodefinition

Note: The actual definition of the vec! macro in the standard libraryincludes code to preallocate the correct amount of memory up front. That codeis an optimization that we don’t include here to make the example simpler.

The #[macro_export] annotation indicates that this macro should be madeavailable whenever the crate in which the macro is defined is brought intoscope. Without this annotation, the macro can’t be brought into scope.

We then start the macro definition with macro_rules! and the name of themacro we’re defining without the exclamation mark. The name, in this casevec, is followed by curly brackets denoting the body of the macro definition.

The structure in the vec! body is similar to the structure of a matchexpression. Here we have one arm with the pattern ( $( $x:expr ),* ),followed by => and the block of code associated with this pattern. If thepattern matches, the associated block of code will be emitted. Given that thisis the only pattern in this macro, there is only one valid way to match; anyother pattern will result in an error. More complex macros will have more thanone arm.

Valid pattern syntax in macro definitions is different than the pattern syntaxcovered in Chapter 18 because macro patterns are matched against Rust codestructure rather than values. Let’s walk through what the pattern pieces inListing 19-28 mean; for the full macro pattern syntax, see the RustReference.

First, we use a set of parentheses to encompass the whole pattern. We use adollar sign ($) to declare a variable in the macro system that will containthe Rust code matching the pattern. The dollar sign makes it clear this is amacro variable as opposed to a regular Rust variable. Next comes a set ofparentheses that captures values that match the pattern within the parenthesesfor use in the replacement code. Within $() is $x:expr, which matches anyRust expression and gives the expression the name $x.

The comma following $() indicates that a literal comma separator charactercould optionally appear after the code that matches the code in $(). The *specifies that the pattern matches zero or more of whatever precedes the *.

When we call this macro with vec![1, 2, 3];, the $x pattern matches threetimes with the three expressions 1, 2, and 3.

Now let’s look at the pattern in the body of the code associated with this arm:temp_vec.push() within $()* is generated for each part that matches $()in the pattern zero or more times depending on how many times the patternmatches. The $x is replaced with each expression matched. When we call thismacro with vec![1, 2, 3];, the code generated that replaces this macro callwill be the following:

{ let mut temp_vec = Vec::new(); temp_vec.push(1); temp_vec.push(2); temp_vec.push(3); temp_vec}

We’ve defined a macro that can take any number of arguments of any type and cangenerate code to create a vector containing the specified elements.

To learn more about how to write macros, consult the online documentation orother resources, such as “The Little Book of Rust Macros” started byDaniel Keep and continued by Lukas Wirth.

Procedural Macros for Generating Code from Attributes

The second form of macros is the procedural macro, which acts more like afunction (and is a type of procedure). Procedural macros accept some code as aninput, operate on that code, and produce some code as an output rather thanmatching against patterns and replacing the code with other code as declarativemacros do. The three kinds of procedural macros are custom derive,attribute-like, and function-like, and all work in a similar fashion.

When creating procedural macros, the definitions must reside in their own cratewith a special crate type. This is for complex technical reasons that we hopeto eliminate in the future. In Listing 19-29, we show how to define aprocedural macro, where some_attribute is a placeholder for using a specificmacro variety.

Filename: src/lib.rs

use proc_macro;#[some_attribute]pub fn some_name(input: TokenStream) -> TokenStream {}

Listing 19-29: An example of defining a proceduralmacro

The function that defines a procedural macro takes a TokenStream as an inputand produces a TokenStream as an output. The TokenStream type is defined bythe proc_macro crate that is included with Rust and represents a sequence oftokens. This is the core of the macro: the source code that the macro isoperating on makes up the input TokenStream, and the code the macro producesis the output TokenStream. The function also has an attribute attached to itthat specifies which kind of procedural macro we’re creating. We can havemultiple kinds of procedural macros in the same crate.

Let’s look at the different kinds of procedural macros. We’ll start with acustom derive macro and then explain the small dissimilarities that make theother forms different.

How to Write a Custom derive Macro

Let’s create a crate named hello_macro that defines a trait namedHelloMacro with one associated function named hello_macro. Rather thanmaking our users implement the HelloMacro trait for each of their types,we’ll provide a procedural macro so users can annotate their type with#[derive(HelloMacro)] to get a default implementation of the hello_macrofunction. The default implementation will print Hello, Macro! My name is TypeName! where TypeName is the name of the type on which this trait hasbeen defined. In other words, we’ll write a crate that enables anotherprogrammer to write code like Listing 19-30 using our crate.

Filename: src/main.rs

use hello_macro::HelloMacro;use hello_macro_derive::HelloMacro;#[derive(HelloMacro)]struct Pancakes;fn main() { Pancakes::hello_macro();}

Listing 19-30: The code a user of our crate will be ableto write when using our procedural macro

This code will print Hello, Macro! My name is Pancakes! when we’re done. Thefirst step is to make a new library crate, like this:

$ cargo new hello_macro --lib

Next, we’ll define the HelloMacro trait and its associated function:

Filename: src/lib.rs

pub trait HelloMacro { fn hello_macro();}

We have a trait and its function. At this point, our crate user could implementthe trait to achieve the desired functionality, like so:

use hello_macro::HelloMacro;struct Pancakes;impl HelloMacro for Pancakes { fn hello_macro() { println!("Hello, Macro! My name is Pancakes!"); }}fn main() { Pancakes::hello_macro();}

However, they would need to write the implementation block for each type theywanted to use with hello_macro; we want to spare them from having to do thiswork.

Additionally, we can’t yet provide the hello_macro function with defaultimplementation that will print the name of the type the trait is implementedon: Rust doesn’t have reflection capabilities, so it can’t look up the type’sname at runtime. We need a macro to generate code at compile time.

The next step is to define the procedural macro. At the time of this writing,procedural macros need to be in their own crate. Eventually, this restrictionmight be lifted. The convention for structuring crates and macro crates is asfollows: for a crate named foo, a custom derive procedural macro crate iscalled foo_derive. Let’s start a new crate called hello_macro_derive insideour hello_macro project:

$ cargo new hello_macro_derive --lib

Our two crates are tightly related, so we create the procedural macro cratewithin the directory of our hello_macro crate. If we change the traitdefinition in hello_macro, we’ll have to change the implementation of theprocedural macro in hello_macro_derive as well. The two crates will need tobe published separately, and programmers using these crates will need to addboth as dependencies and bring them both into scope. We could instead have thehello_macro crate use hello_macro_derive as a dependency and re-export theprocedural macro code. However, the way we’ve structured the project makes itpossible for programmers to use hello_macro even if they don’t want thederive functionality.

We need to declare the hello_macro_derive crate as a procedural macro crate.We’ll also need functionality from the syn and quote crates, as you’ll seein a moment, so we need to add them as dependencies. Add the following to theCargo.toml file for hello_macro_derive:

Filename: hello_macro_derive/Cargo.toml

[lib]proc-macro = true[dependencies]syn = "1.0"quote = "1.0"

To start defining the procedural macro, place the code in Listing 19-31 intoyour src/lib.rs file for the hello_macro_derive crate. Note that this codewon’t compile until we add a definition for the impl_hello_macro function.

Filename: hello_macro_derive/src/lib.rs

use proc_macro::TokenStream;use quote::quote;use syn;#[proc_macro_derive(HelloMacro)]pub fn hello_macro_derive(input: TokenStream) -> TokenStream { // Construct a representation of Rust code as a syntax tree // that we can manipulate let ast = syn::parse(input).unwrap(); // Build the trait implementation impl_hello_macro(&ast)}

Listing 19-31: Code that most procedural macro crateswill require in order to process Rust code

Notice that we’ve split the code into the hello_macro_derive function, whichis responsible for parsing the TokenStream, and the impl_hello_macrofunction, which is responsible for transforming the syntax tree: this makeswriting a procedural macro more convenient. The code in the outer function(hello_macro_derive in this case) will be the same for almost everyprocedural macro crate you see or create. The code you specify in the body ofthe inner function (impl_hello_macro in this case) will be differentdepending on your procedural macro’s purpose.

We’ve introduced three new crates: proc_macro, syn, and quote. Theproc_macro crate comes with Rust, so we didn’t need to add that to thedependencies in Cargo.toml. The proc_macro crate is the compiler’s API thatallows us to read and manipulate Rust code from our code.

The syn crate parses Rust code from a string into a data structure that wecan perform operations on. The quote crate turns syn data structures backinto Rust code. These crates make it much simpler to parse any sort of Rustcode we might want to handle: writing a full parser for Rust code is no simpletask.

The hello_macro_derive function will be called when a user of our libraryspecifies #[derive(HelloMacro)] on a type. This is possible because we’veannotated the hello_macro_derive function here with proc_macro_derive andspecified the name HelloMacro, which matches our trait name; this is theconvention most procedural macros follow.

The hello_macro_derive function first converts the input from aTokenStream to a data structure that we can then interpret and performoperations on. This is where syn comes into play. The parse function insyn takes a TokenStream and returns a DeriveInput struct representing theparsed Rust code. Listing 19-32 shows the relevant parts of the DeriveInputstruct we get from parsing the struct Pancakes; string:

DeriveInput { // --snip-- ident: Ident { ident: "Pancakes", span: #0 bytes(95..103) }, data: Struct( DataStruct { struct_token: Struct, fields: Unit, semi_token: Some( Semi ) } )}

Listing 19-32: The DeriveInput instance we get whenparsing the code that has the macro’s attribute in Listing 19-30

The fields of this struct show that the Rust code we’ve parsed is a unit structwith the ident (identifier, meaning the name) of Pancakes. There are morefields on this struct for describing all sorts of Rust code; check the syndocumentation for DeriveInput for more information.

Soon we’ll define the impl_hello_macro function, which is where we’ll buildthe new Rust code we want to include. But before we do, note that the outputfor our derive macro is also a TokenStream. The returned TokenStream isadded to the code that our crate users write, so when they compile their crate,they’ll get the extra functionality that we provide in the modifiedTokenStream.

You might have noticed that we’re calling unwrap to cause thehello_macro_derive function to panic if the call to the syn::parse functionfails here. It’s necessary for our procedural macro to panic on errors becauseproc_macro_derive functions must return TokenStream rather than Result toconform to the procedural macro API. We’ve simplified this example by usingunwrap; in production code, you should provide more specific error messagesabout what went wrong by using panic! or expect.

Now that we have the code to turn the annotated Rust code from a TokenStreaminto a DeriveInput instance, let’s generate the code that implements theHelloMacro trait on the annotated type, as shown in Listing 19-33.

Filename: hello_macro_derive/src/lib.rs

use proc_macro::TokenStream;use quote::quote;use syn;#[proc_macro_derive(HelloMacro)]pub fn hello_macro_derive(input: TokenStream) -> TokenStream { // Construct a representation of Rust code as a syntax tree // that we can manipulate let ast = syn::parse(input).unwrap(); // Build the trait implementation impl_hello_macro(&ast)}fn impl_hello_macro(ast: &syn::DeriveInput) -> TokenStream { let name = &ast.ident; let gen = quote! { impl HelloMacro for #name { fn hello_macro() { println!("Hello, Macro! My name is {}!", stringify!(#name)); } } }; gen.into()}

Listing 19-33: Implementing the HelloMacro trait usingthe parsed Rust code

We get an Ident struct instance containing the name (identifier) of theannotated type using ast.ident. The struct in Listing 19-32 shows that whenwe run the impl_hello_macro function on the code in Listing 19-30, theident we get will have the ident field with a value of "Pancakes". Thus,the name variable in Listing 19-33 will contain an Ident struct instancethat, when printed, will be the string "Pancakes", the name of the struct inListing 19-30.

The quote! macro lets us define the Rust code that we want to return. Thecompiler expects something different to the direct result of the quote!macro’s execution, so we need to convert it to a TokenStream. We do this bycalling the into method, which consumes this intermediate representation andreturns a value of the required TokenStream type.

The quote! macro also provides some very cool templating mechanics: we canenter #name, and quote! will replace it with the value in the variablename. You can even do some repetition similar to the way regular macros work.Check out the quote crate’s docs for a thorough introduction.

We want our procedural macro to generate an implementation of our HelloMacrotrait for the type the user annotated, which we can get by using #name. Thetrait implementation has the one function hello_macro, whose body contains thefunctionality we want to provide: printing Hello, Macro! My name is and thenthe name of the annotated type.

The stringify! macro used here is built into Rust. It takes a Rustexpression, such as 1 + 2, and at compile time turns the expression into astring literal, such as "1 + 2". This is different than format! orprintln!, macros which evaluate the expression and then turn the result intoa String. There is a possibility that the #name input might be anexpression to print literally, so we use stringify!. Using stringify! alsosaves an allocation by converting #name to a string literal at compile time.

At this point, cargo build should complete successfully in both hello_macroand hello_macro_derive. Let’s hook up these crates to the code in Listing19-30 to see the procedural macro in action! Create a new binary project inyour projects directory using cargo new pancakes. We need to addhello_macro and hello_macro_derive as dependencies in the pancakescrate’s Cargo.toml. If you’re publishing your versions of hello_macro andhello_macro_derive to crates.io, they would be regulardependencies; if not, you can specify them as path dependencies as follows:

hello_macro = { path = "../hello_macro" }hello_macro_derive = { path = "../hello_macro/hello_macro_derive" }

Put the code in Listing 19-30 into src/main.rs, and run cargo run: itshould print Hello, Macro! My name is Pancakes! The implementation of theHelloMacro trait from the procedural macro was included without thepancakes crate needing to implement it; the #[derive(HelloMacro)] added thetrait implementation.

Next, let’s explore how the other kinds of procedural macros differ from customderive macros.

Attribute-like macros

Attribute-like macros are similar to custom derive macros, but instead ofgenerating code for the derive attribute, they allow you to create newattributes. They’re also more flexible: derive only works for structs andenums; attributes can be applied to other items as well, such as functions.Here’s an example of using an attribute-like macro: say you have an attributenamed route that annotates functions when using a web application framework:

#[route(GET, "/")]fn index() {

This #[route] attribute would be defined by the framework as a proceduralmacro. The signature of the macro definition function would look like this:

#[proc_macro_attribute]pub fn route(attr: TokenStream, item: TokenStream) -> TokenStream {

Here, we have two parameters of type TokenStream. The first is for thecontents of the attribute: the GET, "/" part. The second is the body of theitem the attribute is attached to: in this case, fn index() {} and the restof the function’s body.

Other than that, attribute-like macros work the same way as custom derivemacros: you create a crate with the proc-macro crate type and implement afunction that generates the code you want!

Function-like macros

Function-like macros define macros that look like function calls. Similarly tomacro_rules! macros, they’re more flexible than functions; for example, theycan take an unknown number of arguments. However, macro_rules! macros can bedefined only using the match-like syntax we discussed in the section“Declarative Macros with macro_rules! for GeneralMetaprogramming” earlier. Function-like macros take aTokenStream parameter and their definition manipulates that TokenStreamusing Rust code as the other two types of procedural macros do. An example of afunction-like macro is an sql! macro that might be called like so:

let sql = sql!(SELECT * FROM posts WHERE id=1);

This macro would parse the SQL statement inside it and check that it’ssyntactically correct, which is much more complex processing than amacro_rules! macro can do. The sql! macro would be defined like this:

#[proc_macro]pub fn sql(input: TokenStream) -> TokenStream {

This definition is similar to the custom derive macro’s signature: we receivethe tokens that are inside the parentheses and return the code we wanted togenerate.

Summary

Whew! Now you have some Rust features in your toolbox that you likely won’t useoften, but you’ll know they’re available in very particular circumstances.We’ve introduced several complex topics so that when you encounter them inerror message suggestions or in other peoples’ code, you’ll be able torecognize these concepts and syntax. Use this chapter as a reference to guideyou to solutions.

Next, we’ll put everything we’ve discussed throughout the book into practiceand do one more project!