rust

Dogfeeding Your Library | Case: feed-rs

Muhammed Kadir Tan

Dec 25, 2023 • 4 min read

Last month, I announced the release of feed-rs, and now is a good time to talk about why the package was needed, and what my experience was with it.

As someone not familiar with the term, dogfeeding initially struck me as a bit odd. It is a term used to describe that when someone uses their product themselves, so they have a better idea about possible improvements or bugs.

I believe dogfeeding is great and it aligns with the philosophy that good products are the ones built out of our own needs.

Why I Created the feed-rs

I'm on a quest for content – lots of it. And when you find someone craving data in large quantities, you can bet there's some data science wizardry in the making.

Recently, I've manifested a future for the way we consume content. I imagine a future in which people will be able to run their algorithms on content websites. Which will specifically be important for social media websites.

I am building a website to validate my idea, and for that, I need a lot of content.

It is possible to collect content from most of the blogs via RSS. It is one of the Web 2.0 standards that is still alive and well today. So, a library like feed-rs plays a very crucial role in collecting data from the Web.

feed-rs is the fastest feed parser in the NPM. However, its main strength is the ability to recognize and parse all kinds of different feeds into a single format. Shoutout to the authors of the underlying Rust package.

Dogfeeding Results

Not surprisingly, feed-rs is a very performant library. So, I am happy on that end.

But, there was a strange behavior about the dates. While inspecting collected feed data, I realized almost all of the published and updated dates are the same.

Dates were truncated when parsed with feed-rs. But when visiting the feed from the browser, the dates were correct.

I am using napi.rs for creating Node.js bindings on the Rust package. Conversion between Rust types and Node.js types is not always one-to-one. For example, Rust has i64 type, which corresponds to BigInt on Node.js. But, if you try to convert it to a number type, as I did, you get truncated numbers on the Node.js side.

Working with BigInt on Node.js is not very pleasant. So, I decided to shy away from numbers and use Date object to represent... Dates.

-  published: value
-    .published
-    .map(|published| published.timestamp_millis()),
+  published: value.published.map(|published| {
+    f64::from_i64(published.timestamp_millis())
+      .and_then(|timestamp| env.create_date(timestamp).ok())
+      .unwrap()
+  }),

This change alone seemed enough. However, some explanations here might help others who try to create Node.js bindings for Rust crates.

Napi.rs Intricacies: Env for Creating JsDate

Though a very good project, I find it lacking some documentation on most of the use cases. Maybe it is a skill issue, I'm a noob Rustacean after all. To return a Date object to Node.js side, I changed the Feed struct as follows:

use napi::{JsDate};

#[derive(Debug)]
#[napi]
pub struct Feed {
  // ...
  published: Option<JsDate>
  // ...
}

The first error I got was that JsDate didn't implement Debug and since it was a foreign type, I couldn't implement it either. I didn't want to deal with this issue again, so I quickly fixed the problem by commenting out #[derive(Debug)].

Then, the following challenge was to create the JsDate. Unfortunately, there was no method such as JsDate::from_timestamp. There were no creation methods for the JsDate struct.

No joke, the following is the exact amount of documentation on napi.rs about the JsDate:

`JsDate`

Represent Date object in JavaScript. JavaScript Date objects are described in Section 20.3(opens in a new tab) of the ECMAScript Language Specification.

Anyway, I soon discovered that to create javascript values/objects, you should use something called Env. It is a representation of the underlying N-Api. Simply adding a parameter on the lib.rs worked out great:

use napi::{Env, Error};

#[napi]
pub fn parse(
  env: Env, // <-- this is the important line
  feed_string: String,
  feed_source: Option<String>,
) -> Result<models::Feed, Error> {
  let result = parser::parse_with_uri(
    feed_string.as_bytes(),
    feed_source.as_ref().map(|source| source.as_str()),
  );

  match result {
    Ok(feed) => Ok(models::Feed::from(env, feed)),
    Err(err) => Err(Error::from_reason(err.to_string())),
  }
}

However, there was a problem with the From trait. It expects a single parameter on the from function, however, I needed to pass the env for JsDate creation. Simply, removing the trait and adding a function with the same name saved the day:

impl Feed {
  pub fn from(env: Env, feed: model::Feed) -> Self {
    Self {
      // ...
      published: feed.published.map(|published| {
        f64::from_i64(published.timestamp_millis())
          .and_then(|timestamp| env.create_date(timestamp).ok())
          .unwrap()
      }),
      // ...
    }
  }
}

Summary

With the date puzzle solved and a smooth sail through other aspects of feed-rs, I proudly declare it as version 1.0-worthy! 🎉

If you haven't already, go check it out!