opensource | Craig Hesling

SafetyFast

Sun, 03 Jun 2018 07:46:59 -0500

Put thread-safety first, with the performance of safety last.

This is a Go library that implements synchronization primitives over Intel TSX (Intel’s hardware transactional primitives). The library exposes interfaces to make use of Intel HLE and Intel RTM.

Usage Patterns

go get github.com/linux4life798/safetyfast

RTM Context

package main

import "github.com/linux4life798/safetyfast"

func main() {
 r := safetyfast.NewRTMContexDefault()
 r.Atomic(func() {
 btc.Touch(index) // Any action
 })
}

HLE Spin Lock

package main

import (
 "sync"
 "github.com/linux4life798/safetyfast"
)

func main() {
 var m sync.Locker = new(safetyfast.SpinHLEMutex)
 m.Lock()
 btc.Touch(index) // Any action
 m.Unlock()
}

Poster

Download PDF

A preview of the PDF should appear below. If it does not show up soon, please refresh the page.

This browser does not support PDFs. Please download the PDF to view it: Download PDF

GOBF

Fri, 04 May 2018 00:00:00 +0000

Intro

I have always been fascinated by compilers. I think the idea of representing one language’s structure inside of another language is just downright cool. Furthermore, the ability to analyze another language and optimize it really peaks my interest.

Recently, I worked on a project that would allow you to more precisely manage which processor core a set of Goroutines would run on. The idea was that Goroutines that communicate more than they process independently would benefit from sharing that same processor core’s cache. I called them M-Groups. The implementation called for modifying the runtime package and the Go scheduler. While ripping through the Go internals, I stumbled upon quite a bit of the Go compiler code, including the SSA library. As I ripped into it more, I started to form an itch to make my own compiler of some kind.

Fast forward a few months and I found myself staring at some examples of my favorite simple esoteric language, BF. BF is the perfect language to write a compiler / optimizer for because of its extremely minimal syntax. It only has 8 possible syntax items(called commands), <>+-.,[], but those simple syntax items quickly form complex programs (not in the sense of awesome functionality).

Here is Hello World in BF

++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>->>+[<]<-]>>.>---.+++++++..+++.>>.<-.<.+++.------.--------.>>+.>++.

GOBF Overview

Introducing GOBF, an interpreter/optimizer/compiler/code_generator for the BF language. It has two primary modes of operation, interpreting/running (as-is) and converting BF to optimized Go code.

Interpreting

Interpreting and running was the first mode of operation that really just served to solidify my understanding of the BF language. It simply sets up a virtual environment for the BF program and acts on each syntax item as they come. There is no optimization here.

Optimizing / Compiling

Next, I wanted to focus on how I would be able to generate super fast binaries from my BF programs. I considered lots of approaches, but the two most prominent ones were to directly assemble a binary from the BF commands and convert BF to equivalent Go code.

Directly assemble an x86 binary from the the BF commands seems like a natural extension, since BF forces the programmer to act somewhat like a state machine. The downside to this approach is that it would lack portability and would loose out on further architectural optimizations that are found in modern compilers.

I ultimately chose the ladder solution, where I convert the BF commands into equivalent Go code. Although it seems like a bulky implementation strategy, the program’s functionality is quite limited and we gain the builtin compiler inlining.

I have designed GOBF to parse BF programs and generate a internal intermediate representation(IR) that resembles a tree. The next major observation that BF contains lots of repetitive actions, like ++++++++, which increments the local accumulator eight times, only to save the value 8. These repetitive actions can easily be smashed into a single operation, which can yield massive speedups. The GOBF optimizer traverses the IR tree and coalesces these repetitive actions into a more descriptive single action.

When the optimizer finishes, GOBF generates Go code from the IR and optionally compiles it.

Benchmarking

I ran some of the typical BF programs you can find on the internet.

BF Program	Run Time
mandelbrot.b	8.162s
hanoi.b	5.401s
helloworld.b	0.003s

Running on an Intel(R) Xeon(R) CPU E3-1505M v5 @ 2.80GHz

Future Work

I would like to create a reverse code generator that takes a high-level language, such as Go, and generates BF.

Links

GOBF on Github

dproto

Sun, 03 Dec 2017 07:46:59 -0500

An on the fly Google protobuf decoder/encoder library.

Summary

This library allows you to use Protobufs (Google Protocol Buffers) without needing to generate protobuf code and compile it into your project. This means you don’t need to compile .proto files.

This library is designed for marshaling and unmarshalling protobufs in a dynamic way. It is designed to interface with any standard protobuf library and is tested against the official protobuf C++ and Golang bindings, in addition to the C Nanopdb library.

The intent of this library was to allow creating long running services that can interpret and interface with clients using new/unknown protobuf messages.

The basic idea is that you construct a ProtoFieldMap that contains any protobuf field number to protobuf type associations you are interested in and then you call DecodeBUffer on the protobuf payload. DecodeBuffer returns an array of FieldValues that it decoded from the payload. Each FieldValue specifies the protobuf field number and value decoded as a Golang primitive(must be inside a interface{}).

Example Usage

// Encodes and Decodes a protobuf buffer that would interface with
// a .proto file message containing "bool status =1;" and
// "int64 intensity = 2;".
func ExampleNewProtoFieldMap() {
 // Setup the ProtoFieldMap to interface with the
 // following .proto file:
 // message LightStatus {
 // bool status = 1;
 // int64 intensity = 2;
 // }

 // Setup a FieldMap that holds the type-fieldnumber association
 // This is effectively the information held in a .proto file
 fm := dproto.NewProtoFieldMap()

 // Add Protobuf bool as field 1 ("bool status = 1;")
 if !fm.Add(1, descriptor.FieldDescriptorProto_TYPE_BOOL) {
 panic("Failed to add bool field 1")
 }
 // Add Protobuf int64 as field 2 ("int64 intensity = 2;")
 if !fm.Add(2, descriptor.FieldDescriptorProto_TYPE_INT64) {
 panic("Failed to add bool field 1")
 }

 // Provide some values for our "status" and "intensity"
 values := []dproto.FieldValue{
 dproto.FieldValue{
 Field: 1, // status field number
 Value: bool(true),
 },
 dproto.FieldValue{
 Field: 2, // intensity field number
 Value: int64(10),
 },
 }

 // Encode out values into the protobuf message described in fm
 bytes, err := fm.EncodeBuffer(values)
 if err != nil {
 panic("Error Encoding: " + err.Error())
 }
 fmt.Printf("ProtobufBinary: [ %# x ]\n", bytes)

 // Decode all protobuf fields
 decodedValues, err := fm.DecodeBuffer(bytes)
 if err != nil {
 panic("Error Decoding: " + err.Error())
 }
 for _, val := range decodedValues {
 fmt.Printf("%v: %v\n", val.Field, val.Value)
 }
 // Unordered output:
 // ProtobufBinary: [ 0x08 0x01 0x10 0x0a ]
 // 1: true
 // 2: 10
}

Easybits Serialization Service

Sun, 03 Dec 2017 07:46:59 -0500

Summary

Easybits is an on-the-fly Google protobuf stream decoder/encoder for OpenChirp. This service is one of the core services that an embedded device may need in it’s OpenChirp data pipeline.

The idea is that your device would use Google Protobufs to serialize (and deserialize) it’s data and sends it to OpenChirp. The Easybits service is then invoked, which deserializes your data and publishes human readable values back to OpenChirp.

Easybits just needs the user to describe what protobuf types were declared and where to place the data.

How it works

Google Protobufs are great for creating compact serialized data packets for typical sensor values.

Protobufs requires you to create a .proto file that describes what data you intend to send back and forth. It then compiles this .proto file into a set of language specific interface files, which containing functions and objects to represent and aggregate your data.

Here is an example .proto file description for a smart light bulb.

// SmartBulb.proto
syntax = "proto3";
message LightStatus {
 bool status = 1;
 int64 intensity = 2;
}

Due to this native language integration, protobufs are easy for end users to pack up and send data messages, but it is non-trivial to create long running services that can interpret all possible custom protobuf messages, without recompilation.

To get around the Protobuf constraint of recompiling for every new protobuf message, Easybits uses dproto to adapt to new user described messages, without even restarting.

Links

Easybits on Github

OpenChirp

Sun, 03 Dec 2017 07:46:59 -0500

Summary

We are creating a community-driven low-power wireless connectivity solution for IoT.

My Contributions

Designed, built, and installed outdoor gateways
Created custom Debian packages for gateway software and configuration - ocgw
Created gateway system monitoring service - SysMonitor Device
Created the Golang framework for Users, Devices, and Services - framework and Example Service
Created the LoRaWAN integration service
Created data serialization/deserialization services - ByteTranslator Service and Easybits Service
Created prototype trigger service - Trigger Service
Designed and built ultra low-power LoRaWAN client devices - The LoRaBug
Created the hardware abstraction layer and adapted the necessary firmware stacks for the LoRaBug - LoRaBug Software
Created serial bootloader tool to flash the LoRaBug over micro USB - ccbootutil

Checkout the OpenChirp.io Website

The GridBallast Controller

Sun, 03 Dec 2017 07:46:59 -0500

FIXME - Add the gallery line back here.

Summary

This is a modular demand response controller for household or commercial appliances that tend to consume a lot of power. The original goal was to intelligently control water heaters.

Water heaters tax the power grid heavily during peek times (morning and afternoon), when people are using the most hot water. These spikes in power consumption require power providers to plan ahead and preemptively start up temporary power plant that may not meet the typical efficiency and cleanliness standards.

Fortunately, water heaters store enough hot water for most normal household tasks, including taking a shower. Whether they turn on during your morning shower or not, makes no noticeable difference. Unfortunately, hot water heaters are controlled by mechanical switches that activate the water heaters when the internal water temperature drops below a certain point. This causes the heaters to activate during events like your shower in the morning, when it might not be necessary. If the water heater was to delay it’s activation to some random time during the middle of the day, we could alleviate this peak demand issue. This controller is designed to monitor the water heater vitals, observe it’s usage patterns, and intelligently control it’s activation.

Schematic and Board

The Controller

Download PDF

A preview of the PDF should appear below. If it does not show up soon, please refresh the page.

This browser does not support PDFs. Please download the PDF to view it: Download PDF

The CTA2045 Adapter

Download PDF

A preview of the PDF should appear below. If it does not show up soon, please refresh the page.

This browser does not support PDFs. Please download the PDF to view it: Download PDF

Simulation

This board required some custom analog design. I simulated some of the more complex circuits in LTspice. The circuits and simulation are shown on the GitHub Simulations Page.

System Monitor Device (for OpenChirp)

Mon, 06 Nov 2017 00:00:00 +0000

Summary

How it works

Links

sysmonitor-device on Github

The LoRaBug

Sat, 03 Dec 2016 07:46:59 -0500

Summary

The LoRaBug is a general purpose low-power sensor platform. It tightly integrates an ARM Cortex M3 based MCU, BLE, and a LoRa radio.

Checkout the Github repository for more detail.

Key Features

TI CC2650 MCU/Transciever - ARM Cortex M3, Bluetooth+BLE, and 2.4GHz 802.15.4
Semtech SX1276 radio - 137MHz to 1020MHz supporting LoRa, FSK, GFSK, MSK, GMSK, and OOK
Micro USB - Bootloading and Console
Onboard 500mA 3.3V regulator

Software

I have adapted the HAL of the official LoRaWAN stack for the CC2650 with TI-RTOS - LoRaWAN MAC
A spinoff from my example firmware can be found - LoRaBug_Firmware.
I have created a cross-platform serial bootloader tool that allows you to flash the LoRaBug using just a micro usb cable - ccbootutil

Schematic and Board

Download PDF

A preview of the PDF should appear below. If it does not show up soon, please refresh the page.

This browser does not support PDFs. Please download the PDF to view it: Download PDF