napi

What Did I Learn While Creating feed-rs

Muhammed Kadir Tan

Nov 15, 2023 • 7 min read

I am building an application involving RSS feeds, and at some point, I needed an RSS parser. Even though there are existing NPM packages that can parse RSS, I wanted something more robust and performant.

I've found a great Rust crate that can parse RSS feeds seamlessly: feed_rs. But, I wanted to build my project in Node.js, where I am more experienced. Switching language for a package didn't seem feasible, so I've decided to create Node.js bindings for feed_rs.

It's been a challenge for me, and I wanted to share my learnings so you can transfer your favorite Rust crates to the Node.js ecosystem.

What is a Binding?

Bindings allow developers to use code from other languages. In this case, feed_rs is written using Rust, but I want to use it in a Node.js project. You can achieve this using bindings.

Most commonly, you can find bindings for C libraries in many languages. This is usually achieved using a technique called FFI ( Foreign Function Interface ).

Nowadays, it is more popular to create bindings for Rust libraries due to the performance and safety guarantees.

Luckily, Node.js provides a more robust way of implementing bindings. It is called N-API ( Node.js API ). Using N-API, you can create Node.js addons built with other languages that can interact with Node.js objects. Pretty awesome, if you ask me.

NAPI-RS

It may sound easy in theory, but creating Node.js addons or using N-API is not a simple task. NAPI-RS makes this process a breeze.

Basically, you can write Rust code, sprinkle some magical #[napi] macros in your code, and suddenly create a native Node.js addon that can be run as ordinary JavaScript code.

However, if you are not a seasoned Rustacean, as I am not, some struggles are awaiting you.

Creating the Project

First, you need to log into your GitHub account and create a new project using the https://github.com/napi-rs/package-template template.

Check out the repository and open up a terminal in the project folder. Run the following code as suggested in the readme:

yarn install
npx napi rename -n [name]

The rename command changes the project name throughout the repository. However, there is a catch to it. If you are going to publish your package under an NPM organization as in @nooptoday/feed-rs , the naming structure causes trouble.

The Name Problem for Organization Packages

The / character in the package name causes your build files to be generated under @nooptoday and some commands can't find the artifacts. To work around this issue:

Run the command npx napi rename -n @nooptoday/feed-rs
Change all the package.json files under npm directory as follows:

   "cpu": [ "arm" ],
-  "main": "@nooptoday/feed-rs.android-arm-eabi.node",
+  "main": "feed-rs.android-arm-eabi.node",
   "files": [
-    "@nooptoday/feed-rs.android-arm-eabi.node"
+    "feed-rs.android-arm-eabi.node"
   ],
   "description": "Node.js bindings for feed_rs",

A similar change for the same reason needs to be made in the CI.yml file. It is where your GitHub actions are defined and sits under ProjectFolder/.github/workflows/CI.yml.

I highly suggest you read the file and try to understand what it does. At the beginning of the file, you can see the environment variables definition:

name: CI
env:
  DEBUG: napi:*
  APP_NAME: '@nooptoday/feed-rs'
  MACOSX_DEPLOYMENT_TARGET: '10.13'

The problem here is that the APP_NAME variable is also used to find the artifact file. So, you should also change it as follows:

  name: CI
  env:
    DEBUG: napi:*
+   APP_NAME: 'feed-rs'
-   APP_NAME: '@nooptoday/feed-rs'
    MACOSX_DEPLOYMENT_TARGET: '10.13'

The Optional Dependencies Problem of Modern Yarn

I've set up my project and pushed my changes to see how things work out. Sadly, I've got an error:

➤ YN0000: · Yarn 4.0.1
➤ YN0000: ┌ Resolution step
➤ YN0035: │ @nooptoday/feed-rs-android-arm-eabi@npm:1.0.0: Package not found
➤ YN0035: │   Response Code: 404 (Not Found)
➤ YN0035: │   Request Method: GET
➤ YN0035: │   Request URL: https://registry.yarnpkg.com/@nooptoday%2ffeed-rs-android-arm-eabi
➤ YN0000: └ Completed in 0s 900ms
➤ YN0000: · Failed with errors in 0s 910ms

This package, along with other packages is listed in the optinalDependencies. However, they are non-existent yet. If you ask me, that should be considered okay, and yarn should continue with the installation process. NPM doesn't cause an error, and with the classic yarn, you can just run:

yarn install --ignore-optional

The modern yarn doesn't support this option, and there is no alternative option to disable optionalDependencies.

This issue can be fixed by switching to npm or classic yarn. Or, you can just remove the optionalDependencies from package.json. Which, I did. It might cause some inconvenience to some users, but it is okay for the initial release. The napi-rs provides an explanation about why they use optionalDependencies

The other problem is how to deliver prebuild binary to users. Downloading it in postinstall script is a common way that most packages do it right now. The problem with this solution is it introduced many other packages to download binary that has not been used by runtime codes. The other problem is some users may not easily download the binary from GitHub/CDN if they are behind a private network (But in most cases, they have a private NPM mirror).
In this package, we choose a better way to solve this problem. We release different npm packages for different platforms. And add it to optionalDependencies before releasing the Major package to npm.
- https://github.com/napi-rs/package-template/tree/main#release

With these issues solved, the project is good to go, and I started writing Rust 🤓

Exposing A Rust Library to Node.js

NAPI-RS is advertised as being easy to use. And the examples were looking very easy. Here is an official example:

use napi_derive::napi;
 
#[napi]
fn fibonacci(n: u32) -> u32 {
  match n {
    1 | 2 => 1,
    _ => fibonacci(n - 1) + fibonacci(n - 2),
  }
}

I only wanted to expose a single function from feed_rs and my plan was simply:

use feed_rs::parser;
use feed_rs::models::Feed
use napi_derive::napi;

#[napi]
pub fn parse(feed_string: String, feed_source: Option<String>) -> Feed {
  parser::parse_with_uri(
    feed_string.as_bytes(),
    feed_source.as_ref().map(|source| source.as_str()),
  );
}

It didn't work, what a surprise 😂 But what is the problem here?

The Orphan Rule

Well, the problem is the #[napi] macro expects a return type that can be converted into NapiValue. For that, the Feed struct should implement the ToNapiValue trait.

I am okay with these errors, I understand there should be a mapping between Rust data structures and the Javascript data structures.

However, the solution is not very straightforward. As a newbie, my first attempt was to encapsulate the data as follows:

use feed_rs::models::Feed
use napi_derive::napi;

#[napi]
struct FeedEncapsulated {
  feed: Feed
}

This didn't work, because the Feed struct is from another module and you can't implement traits for structs on external crates 😥 Or in Rust words:

only traits defined in the current crate can be implemented for arbitrary types

Then, I did some research and found out the same issue exists for the serde crate and they offer some solutions in the documentation. Their solution is to do the inevitable: create your own struct and convert the external struct to your own struct.

This is the struct from my own crate:

#[napi(object)]
pub struct Feed {
  pub feed_type: FeedType,
  pub id: String,
  pub updated: Option<i64>,
  pub authors: Vec<String>,
  pub description: Option<String>,
  pub links: Vec<String>,
  pub categories: Vec<String>,
  pub contributors: Vec<String>,
  pub generator: Option<String>,
  pub icon: Option<Image>,
  pub language: Option<String>,
  pub logo: Option<Image>,
  pub published: Option<i64>,
  pub rating: Option<MediaRating>,
  pub rights: Option<String>,
  pub ttl: Option<u32>,
  pub entries: Vec<Entry>,
}

own crate

This is the struct from the external crate:

pub struct Feed {
    pub feed_type: FeedType,
    pub id: String,
    pub title: Option<Text>,
    pub updated: Option<DateTime<Utc>>,
    pub authors: Vec<Person>,
    pub description: Option<Text>,
    pub links: Vec<Link>,
    pub categories: Vec<Category>,
    pub contributors: Vec<Person>,
    pub generator: Option<Generator>,
    pub icon: Option<Image>,
    pub language: Option<String>,
    pub logo: Option<Image>,
    pub published: Option<DateTime<Utc>>,
    pub rating: Option<MediaRating>,
    pub rights: Option<Text>,
    pub ttl: Option<u32>,
    pub entries: Vec<Entry>,
}

external crate

I simplified some structs to simple fields, but this will be fixed in the following versions. It was a trade-off to launch the package's first version more quickly.

An important change between types is that JavaScript doesn't have u64 any other unsigned integer equivalent. And, you can't convert them to i64 versions safely because the u64 might contain numbers greater than what the i64 can hold. Another alternative might be to use i128 or String.

For now, I simply converted them i64 and hoped the number wouldn't exceed the capacity. In the next version, I can conceptually make sure that those numbers are not expected to be greater than i64, e.g. utc timestamp milliseconds doesn't exceed i64 capacity.

Next Steps & Final Thoughts

Match the external Feed struct with the internal Feed struct
Expose errors correctly, there is no error handling atm.
Benchmark: Compare memory usage and performance with other popular libraries.

I believe, after these updates, the package becomes stable and it can just follow along with the upstream updates. Luckily, these standards don't change too often and the upstream is relatively stable.

Update 28-11-2023

The next steps are completed:

Feed structs are in sync! ✅
Benchmarks were added to the repository, and here! 🚀
Correct errors are exposed to the Node.js! ✅

The following benchmarks are a comparison of parsing performance. As a real-world example, feed data of Noop Today was used at https://nooptoday.com/rss

feed-rs          2367 ops/s, ±0.39%   | fastest
fast-xml-parser  1198 ops/s, ±0.26%   | 49.39% slower
rss-parser:      125 ops/s,  ±2.27%   | slowest, 94.72% slower

NPM's fastest feed parser