Using Hublink.cloud to Wirelessly Transfer FED3 Data

There seems to be a perennial problem amongst data logging enthusiasts: those data get stored on an SD card but have to be retrieved manually. The explosive growth of Feeding Experimentation Device 3 (FED3) has compounded this issue for over 100 research labs, with over 3,000 FED3 devices in the wild: that’s a lot of SD cards!

I began speaking with the FED3 developers and power users, and it was clear that transmitting every behavioral event was (1) hard to manage and (2) power-hungry. Real-time data was also not a huge deal, but faithfully transferring all the data every ~24 hours would be sufficient to enable morning-meeting dashboard discussions of all the running experiments.

The Problem: transfer SD card data to the cloud at semi-regular intervals.

A Wireless FED3

The first task was to get some wireless module into the FED3. This has been achieved using an ESP32-S3 dongle for uploading behavioral metadata to Google Sheets. However, the more streamlined solution was replacing the microcontroller with an Adafruit ESP32-S3 Feather that uses the original FED3 pin layout. This creates two new problems:

1. Now, the micro-SD from the original Adalogger is gone.
2. The FED3 code does not natively support alternative architectures (e.g., ESP32).

Firstly, I developed an SD card “backpack” module that co-opted pin A0 for the chip select (mapped initially to the BNC connector), with the option to perform chip select and detent detection over I2C.

Secondly, I created a multi-architecture fork of the FED3 Library. Notably, this gracefully handles GPIO assignments and interrupts and uses the native ESP32-S3 light sleep functionality in place of the Arduino Low Power library.

Selecting a Wireless Protocol

The ESP32-S3 exposes a 2.4GHz radio that supports WiFi and Bluetooth Low Energy (BLE; and a few other niche wireless protocols), so which one should we use?

WiFi would be an excellent solution for a single device with an open network, and you don’t mind embedding API keys and file handling directly in the firmware (take a look at this GitHub gist for WiFi Data Transfer to understand the failure points of hard-coding). However, BLE wins in terms of power consumption, reconfigurability, and offline behavior. The BLE approach is superior in modularizing each system element, such that integration with something like the FED3 is lightweight and portable.

Creating an Internet Gateway

Getting the data from a BLE device to the internet is best approached by first assuming that the gateway acts as the “central” device and the node (or FED3) acts as the “peripheral.” This is typically schematized as a one-to-many system, but indeed, multiple gateways can be used in a many-to-many model to expand coverage.

I chose the Raspberry Pi 5 as a gateway platform as a low-cost, reliable Linux machine that can create the link between BLE and the internet. The gateway runs its own Dockerized container to enable a seamless deployment and update mechanism. In the spirit of modularization, the gateway partitions BLE communication from internet access, which can be configured through a wireless network or physical ethernet.

The gateway is responsible for connecting to BLE nodes and collecting data with as little overhead as possible, after which the gateway manages where the data goes. The gateway is also another data repository, ensuring your files live in multiple places.

A Coordinated Effort

As we begin talking about managing n >> 5 BLE nodes, you start to appreciate why you need some central gateway to coordinate and manage data transfer. Ideally, it would be best if you had some control over the behavior of each node, including the ability to change where the data ends up in the cloud and how files are named and archived. As you scale from one location to many (e.g., separated rooms or a large lab space), gateways should work in a seamless, coordinated manner to push data to the cloud.

---

Introducing: Hublink

The Hublink architecture is a full-stack solution linking the embedded node to the cloud.

1. Hublink Node: a FED3 device integrated with the Hublink Node Library.
2. Hublink Gateway: a Raspberry Pi running the Hublink Gateway container.
3. Hublink.cloud: an endpoint and dashboard to efficiently manage, sync, and monitor data.
4. Hublink Client: the Hublink Client Library demonstrates to fetch/sync your data.

The Hublink Web Application

Hublink uses a risk-aware security model, acknowledging that things like API keys and your data are vitally private but that your ability to use, interact with, and share access is paramount. Therefore, Hublink uses a “secret URL” that exposes an account dashboard; the probability that someone guesses your secret URL is one in 47.7 quadrillion.

---

Integrating Hublink and FED3

Once you have the Hublink hardware, the Hublink Node Library outlines how to integrate it into any ESP32 device using the Arduino IDE.

I’ve created a simple Fixed Ratio 1 example on the FED3 Hublink fork. From a high level:

1. Include FED3.h and HublinkNode.h.
2. Initialize FED3 hardware with a sketch identifier.
3. Set up HublinkNode with BLE advertising name and callbacks.
4. Define BLE sub-loop to manage connections and advertising.
5. Use loop to periodically run BLE sub-loop, FED3 logic, and pellet delivery.

Current Consumption

You will be responsible for when your device exposes itself over BLE. The Hublink gateway defaults to scanning for your devices every 30 seconds, but this can be modified.

Current consumption was tested with a 10-10 second connection interval, where a FED3 would enter the Hublink BLE subloop every 10 seconds for 10 seconds, then return to the FED3 business logic.

Mode	FED3 Feed/Sleep	Hublink BLE
Average Current	2.5 mA	56 mA

A Focus on Modularized Efficiency

The Hublink gateway has features to assist in optimal and efficient data transfer:

• “Round-robin” BLE Connections: when the gateway scans and finds your devices, it will service the one that was least recently connected.
• Local Storage: Each BLE scan and data transfer gets its own folder on the gateway that is tracked in a local database; if your internet network is down, it will attempt to upload all those files when it comes back up. This storage is partitioned to an external SD card that can be removed and archived with rules for when or if local storage is pruned to make room for new data.
• Coordination with Hublink.cloud: Before every scan attempt, the gateway pins the hublink.cloud endpoint to get the most recent user settings. After each upload, it sends a sync command to ensure that the Data page reflects what files are currently in Amazon S3.

The Future of Hublink

Hublink.cloud was created to address a clear need, and as those needs evolve, our mission stays constant: to empower the research community with an intuitive platform for data collection, streamlining the process of observing, debugging, and analyzing experiments to enhance efficiency and insight.

Currently, the node and client software are the only open-source components. The node software, in particular, is designed to benefit from community collaboration and experimentation. Why isn’t everything open-source?

1. Infrastructure Costs: Robust backend services come at a price. To ensure reliability and performance, we have to extend the costs of good tooling to the end user.
2. Support is de-incentivized by open-source projects. Unless there is a primary backer, there is no fire lit under the core team to maintain excellent standards in support and platform uptime.
3. Feature Development: Features flourish when users are invested. While friends may suggest ideas, customers provide actionable insights into what they truly need. This helps refine the Hublink roadmap and prioritize impactful features.

Tell us More

We’re interested in your use case! Use the form to tell us more about you and your needs, and we’ll stay in touch along the way.