golang | Craig Hesling

SafetyFast

Sun, 03 Jun 2018 07:46:59 -0500

Put thread-safety first, with the performance of safety last.

This is a Go library that implements synchronization primitives over Intel TSX (Intel’s hardware transactional primitives). The library exposes interfaces to make use of Intel HLE and Intel RTM.

Usage Patterns

go get github.com/linux4life798/safetyfast

RTM Context

package main

import "github.com/linux4life798/safetyfast"

func main() {
 r := safetyfast.NewRTMContexDefault()
 r.Atomic(func() {
 btc.Touch(index) // Any action
 })
}

HLE Spin Lock

package main

import (
 "sync"
 "github.com/linux4life798/safetyfast"
)

func main() {
 var m sync.Locker = new(safetyfast.SpinHLEMutex)
 m.Lock()
 btc.Touch(index) // Any action
 m.Unlock()
}

Poster

Download PDF

A preview of the PDF should appear below. If it does not show up soon, please refresh the page.

This browser does not support PDFs. Please download the PDF to view it: Download PDF

GOBF

Fri, 04 May 2018 00:00:00 +0000

Intro

I have always been fascinated by compilers. I think the idea of representing one language’s structure inside of another language is just downright cool. Furthermore, the ability to analyze another language and optimize it really peaks my interest.

Recently, I worked on a project that would allow you to more precisely manage which processor core a set of Goroutines would run on. The idea was that Goroutines that communicate more than they process independently would benefit from sharing that same processor core’s cache. I called them M-Groups. The implementation called for modifying the runtime package and the Go scheduler. While ripping through the Go internals, I stumbled upon quite a bit of the Go compiler code, including the SSA library. As I ripped into it more, I started to form an itch to make my own compiler of some kind.

Fast forward a few months and I found myself staring at some examples of my favorite simple esoteric language, BF. BF is the perfect language to write a compiler / optimizer for because of its extremely minimal syntax. It only has 8 possible syntax items(called commands), <>+-.,[], but those simple syntax items quickly form complex programs (not in the sense of awesome functionality).

Here is Hello World in BF

++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>->>+[<]<-]>>.>---.+++++++..+++.>>.<-.<.+++.------.--------.>>+.>++.

GOBF Overview

Introducing GOBF, an interpreter/optimizer/compiler/code_generator for the BF language. It has two primary modes of operation, interpreting/running (as-is) and converting BF to optimized Go code.

Interpreting

Interpreting and running was the first mode of operation that really just served to solidify my understanding of the BF language. It simply sets up a virtual environment for the BF program and acts on each syntax item as they come. There is no optimization here.

Optimizing / Compiling

Next, I wanted to focus on how I would be able to generate super fast binaries from my BF programs. I considered lots of approaches, but the two most prominent ones were to directly assemble a binary from the BF commands and convert BF to equivalent Go code.

Directly assemble an x86 binary from the the BF commands seems like a natural extension, since BF forces the programmer to act somewhat like a state machine. The downside to this approach is that it would lack portability and would loose out on further architectural optimizations that are found in modern compilers.

I ultimately chose the ladder solution, where I convert the BF commands into equivalent Go code. Although it seems like a bulky implementation strategy, the program’s functionality is quite limited and we gain the builtin compiler inlining.

I have designed GOBF to parse BF programs and generate a internal intermediate representation(IR) that resembles a tree. The next major observation that BF contains lots of repetitive actions, like ++++++++, which increments the local accumulator eight times, only to save the value 8. These repetitive actions can easily be smashed into a single operation, which can yield massive speedups. The GOBF optimizer traverses the IR tree and coalesces these repetitive actions into a more descriptive single action.

When the optimizer finishes, GOBF generates Go code from the IR and optionally compiles it.

Benchmarking

I ran some of the typical BF programs you can find on the internet.

BF Program	Run Time
mandelbrot.b	8.162s
hanoi.b	5.401s
helloworld.b	0.003s

Running on an Intel(R) Xeon(R) CPU E3-1505M v5 @ 2.80GHz

Future Work

I would like to create a reverse code generator that takes a high-level language, such as Go, and generates BF.

Links

GOBF on Github

dproto

Sun, 03 Dec 2017 07:46:59 -0500

An on the fly Google protobuf decoder/encoder library.

Summary

This library allows you to use Protobufs (Google Protocol Buffers) without needing to generate protobuf code and compile it into your project. This means you don’t need to compile .proto files.

This library is designed for marshaling and unmarshalling protobufs in a dynamic way. It is designed to interface with any standard protobuf library and is tested against the official protobuf C++ and Golang bindings, in addition to the C Nanopdb library.

The intent of this library was to allow creating long running services that can interpret and interface with clients using new/unknown protobuf messages.

The basic idea is that you construct a ProtoFieldMap that contains any protobuf field number to protobuf type associations you are interested in and then you call DecodeBUffer on the protobuf payload. DecodeBuffer returns an array of FieldValues that it decoded from the payload. Each FieldValue specifies the protobuf field number and value decoded as a Golang primitive(must be inside a interface{}).

Example Usage

// Encodes and Decodes a protobuf buffer that would interface with
// a .proto file message containing "bool status =1;" and
// "int64 intensity = 2;".
func ExampleNewProtoFieldMap() {
 // Setup the ProtoFieldMap to interface with the
 // following .proto file:
 // message LightStatus {
 // bool status = 1;
 // int64 intensity = 2;
 // }

 // Setup a FieldMap that holds the type-fieldnumber association
 // This is effectively the information held in a .proto file
 fm := dproto.NewProtoFieldMap()

 // Add Protobuf bool as field 1 ("bool status = 1;")
 if !fm.Add(1, descriptor.FieldDescriptorProto_TYPE_BOOL) {
 panic("Failed to add bool field 1")
 }
 // Add Protobuf int64 as field 2 ("int64 intensity = 2;")
 if !fm.Add(2, descriptor.FieldDescriptorProto_TYPE_INT64) {
 panic("Failed to add bool field 1")
 }

 // Provide some values for our "status" and "intensity"
 values := []dproto.FieldValue{
 dproto.FieldValue{
 Field: 1, // status field number
 Value: bool(true),
 },
 dproto.FieldValue{
 Field: 2, // intensity field number
 Value: int64(10),
 },
 }

 // Encode out values into the protobuf message described in fm
 bytes, err := fm.EncodeBuffer(values)
 if err != nil {
 panic("Error Encoding: " + err.Error())
 }
 fmt.Printf("ProtobufBinary: [ %# x ]\n", bytes)

 // Decode all protobuf fields
 decodedValues, err := fm.DecodeBuffer(bytes)
 if err != nil {
 panic("Error Decoding: " + err.Error())
 }
 for _, val := range decodedValues {
 fmt.Printf("%v: %v\n", val.Field, val.Value)
 }
 // Unordered output:
 // ProtobufBinary: [ 0x08 0x01 0x10 0x0a ]
 // 1: true
 // 2: 10
}

Easybits Serialization Service

Sun, 03 Dec 2017 07:46:59 -0500

Summary

Easybits is an on-the-fly Google protobuf stream decoder/encoder for OpenChirp. This service is one of the core services that an embedded device may need in it’s OpenChirp data pipeline.

The idea is that your device would use Google Protobufs to serialize (and deserialize) it’s data and sends it to OpenChirp. The Easybits service is then invoked, which deserializes your data and publishes human readable values back to OpenChirp.

Easybits just needs the user to describe what protobuf types were declared and where to place the data.

How it works

Google Protobufs are great for creating compact serialized data packets for typical sensor values.

Protobufs requires you to create a .proto file that describes what data you intend to send back and forth. It then compiles this .proto file into a set of language specific interface files, which containing functions and objects to represent and aggregate your data.

Here is an example .proto file description for a smart light bulb.

// SmartBulb.proto
syntax = "proto3";
message LightStatus {
 bool status = 1;
 int64 intensity = 2;
}

Due to this native language integration, protobufs are easy for end users to pack up and send data messages, but it is non-trivial to create long running services that can interpret all possible custom protobuf messages, without recompilation.

To get around the Protobuf constraint of recompiling for every new protobuf message, Easybits uses dproto to adapt to new user described messages, without even restarting.

Links

Easybits on Github

OpenChirp

Sun, 03 Dec 2017 07:46:59 -0500

Summary

We are creating a community-driven low-power wireless connectivity solution for IoT.

My Contributions

Designed, built, and installed outdoor gateways
Created custom Debian packages for gateway software and configuration - ocgw
Created gateway system monitoring service - SysMonitor Device
Created the Golang framework for Users, Devices, and Services - framework and Example Service
Created the LoRaWAN integration service
Created data serialization/deserialization services - ByteTranslator Service and Easybits Service
Created prototype trigger service - Trigger Service
Designed and built ultra low-power LoRaWAN client devices - The LoRaBug
Created the hardware abstraction layer and adapted the necessary firmware stacks for the LoRaBug - LoRaBug Software
Created serial bootloader tool to flash the LoRaBug over micro USB - ccbootutil

Checkout the OpenChirp.io Website

System Monitor Device (for OpenChirp)

Mon, 06 Nov 2017 00:00:00 +0000

Summary

How it works

Links

sysmonitor-device on Github

TI CC2538/CC26xx Serial Bootloader Utility on Linux

Tue, 11 Apr 2017 11:30:41 -0500

If you have ever been curious about using the TI CC2650 and other similar SimpleLink devices’ serial bootloader, wonder no longer.

I recently implemented the TI CC2538/CC26xx Serial Bootloader protocol in a Go library, called ccboot. This simply uses standard IO operations that can be used with a generic serial library.

Since a library is no fun without a command line interface, I have also implemented ccbootutil, which exposes most of the low level bootloader primitives(which can be shell scripted) and has a complete program sequence that reads ELF binaries, flashes, and resets the chip. You simply run the following command and ccbootutil did the rest:

cbootutil /dev/ttyUSB0 prgm YourCompiledProgramInStandardELFFormat.out

When implementing ccbootutil, I used a Go serial port interface library that indicated that it should work cross platform, so Windows and OSX users should not feel left out of the command line and scriptable goodness.

I have compiled the utility for each of the platforms in the repository builds/ directory.

Notes

I have found that the serial bootloader can flash the CC2650 faster than an XDS110 over cJTAG.
TI Smart RF Flash Programmer 2 certainly works, but is strictly for Windows and is not scriptable.

Links

Please let me know if you find any issues. Happy coding!