Remote Work As Innovation

The recent obsession with capital “I” Innovation that has been rapidly gaining momentum the last couple of years has permeated across a majority of large, traditional companies with business models, technologies, and processes seemingly stuck in the past. Fear of competition and thus declining profits and stock prices appears to be the primary motivator behind this obsession, with many of these companies desiring to be more nimble and forward-thinking. This has given rise to the so-called Innovation Group within large organizations, corporate partnerships with startups, and hiring of expensive consultants and consultancies to come in and tell these companies “you should emphatically embrace our way.” These actions, often made very publicly as to cast off any prior signifier of “old,” seem to be a race to avoid being last to adoption. Through looking to startups (or more nimble companies), these old titans hope to bathe in the fountain of youth and emerge automatically renewed.

This post is prefaced with skepticism not out of discouragement; quite the opposite. I think it’s a great thing that companies, whether they’re “old school” or not, are acknowledging that they can’t keep doing things the way they always have been lest they be eaten alive in today’s market. These innovation initiatives are essentially a self-reflection exercise. However, I question these initiatives because they so often seem to miss an opportunity ripe for innovation.

While the idea so far has been to follow what the startups are doing to create innovative services and products (or what the consultants are telling you to do), I posit a new theory – both types of organizations are overlooking a major area to innovate on: their working model.

Too many companies require the same, century old working model of showing up at 9 in the same office location or handful of different office locations, communicating in real-time throughout the day, then going home at 5.

Remote work changes this. It’s an innovation on the old model and a ride into the future. And for companies that are already investing so much into self-reflection in the hopes of finding new ways to gain market share, they might as well examine all aspects of the company while the body is already cut open before it’s sewn back up.

The obvious should be stated that not all roles and not all companies will be a good fit for remote work. But if you are a software company – which most companies are nowadays, even if it’s not the company’s primary product – it’s worth considering adoption.

Smart companies will realize that many, many developers want to work remotely (more than just work-from-home every Thursday or Friday) if they aren’t already. Smart companies will dispense with the micro-managerial mandates of heavily synchronous, remote-unfriendly practices such as full-time pair programming, only using [insert tool here], and “always-on” Slack/IM/whatever. And smart companies will understand that developing more flexible environments for their employees will attract the best ones.

The way we’ve worked for the last century is ripe for change as it no longer makes sense in a globalized and techno-superfluous world. All the product, business, and technology innovation will be all for naught if your employee model is not updated as well to match that of the current working world, indeed if it inhibits working in such a world. Remote work policies, thoughtfully considered and adopted, are not just ends unto themselves but can be an opportunity to inspect and improve upon the rest of the working experience.

Adopting remote work is innovation too.

 

10 Observations after Working Remote for 6 Months

I turned my “on-premise” job into a remote one 6 months ago.  Here are ten observations from the experience so far, with the goal that these assist others transitioning or thinking about transitioning to remote work:

1. Able to eat a lot healthier

This was the most unexpected benefit of working remote.  It’s been much easier to cook (almost) all meals at home while remote.  When commuting to and working from an office it was always difficult to prep meals the night before, but working from home it’s easy to cook while working.  Rather than paying for expensive cafeteria or restaurant food that is lacking in nutrition, cooking meals at home with good food from grocery stores or markets is easy.

2. Need to plan out work locations

It’s been easy to fall into a pattern of working from home and not going out to coffee shops, libraries, coworking spaces, etc. to do work.  Planning out the day in terms of which locations I’m going to start or end the day at has been helpful in avoiding this trap.  Usually a change of scenery has greatly helped critical thinking when stuck on a problem.

3. Easy to forget timezones

If you’re working with a distributed team, chances are there is still a home office for your company that several of your team members work from.  Usually the home office time zone becomes the de facto time within which meetings are scheduled.  This was a slight adjustment at first, and planning things like trying to eat lunch an hour earlier took some getting used to.  Even now it’s still easy to forget this time difference sometimes.

4. Able to get more done, more effectively

Productivity is often one of the biggest concerns companies have when considering a move to a remote work model.  It may be accurate to say that doubts around effectiveness are the main factor behind this concern around productivity.  First, it should be noted that if you’re hiring employees you don’t trust to get work done in any location, then you’re hiring the wrong people to begin with.  Second, and most importantly, the transition to a remote model is certainly no easy effort but if it’s done right and team members have proper support, productivity can actually receive quite a boost.  Speaking anecdotally, I’m working with a team now that is entirely distributed across several states and continents.  I’ve noticed my personal output has greatly increased, with this increase mostly coming in quality (but quantity as well).  Our team has consistently completed work on time and even sometimes ahead of schedule, helping assuage any doubts our company might have head about productivity in the remote model.

5. Being the first truly remote employee has its pros/cons

Being the first truly remote employee within my organization, it’s been something of a pioneering experience.  The organization within which I work has had offshore contractors for many years but no employees who work remotely full-time (from another state entirely).  The biggest advantage of this for me has been that I’ve gotten to prove that the model works and move the organization ahead.  “Innovation” is incessantly paid lip service to by any and all companies these days, and while many of them may be pursuing legitimate innovations in product or processes, innovation around work itself is often neglected.  Remote work is the future and is slowly but surely gaining adoption.  Being able to help guide my company into this new frontier has provided both myself and the company with a lot of benefits.

Of course being the first has its downsides, the main being that the organization is still adapting to this kind of work.  Fortunately I’ve received support for it at high levels within the organization.

6. Can feel singled out, but this can be a positive thing

Continuing from the last observation, it can be easy to feel like the “remote guy” when you’re the only one.  Now that I’ve proven the model though, others within my organization are starting to join the ranks of the remote workforce.  Done right, you might find that others within your company respect you more for it as you’re essentially setting yourself apart from your coworkers.  Others will see that you’re trusted and respected enough to be granted this privilege.

7. Work is still work

The stereotypical image for remote work is some guy on a laptop working from the beach, but it’s not in any way that idyllic or cinematic.  Work is still work and if you’re on a tight deadline or putting out some fire, no grandiose scenery is going to change that.

8. Whiteboarding suffers

Depending on the type of work you do, not having a whiteboard can seem debilitating.  It’s a highly effective and quick tool for communication.  Similarly, being in meetings and not seeing what is being drawn on the whiteboard contributes to the communication gap.  An easy solution for this is to setup a video conference (either through Skype on someone’s laptop or a room VC if you have one) and point the camera at the whiteboard.

9. Having more frequent 1×1’s is helpful

I’ve had more frequent 1×1’s with coworkers.  This has helped fill the place of hallway conversations that you don’t get not being in the office.

10. A lot of other remote workers are out there

Working from coffee shops and coworking spaces I’ve seen a lot of other people in these spaces that are also working remotely.  These people have all varied in age, so it’s not just old or young employees pursuing this working model.  I suspect more and more spaces will open up, and existing ones catering to the remote worker as this becomes the future of work.

 

Calm Technology – CTA train tracker


This is the first in a series of projects that will be developed around the idea of applying “calm technology” to product design. Coined by Mark Weiser and John Seely Brown at Xerox PARC in 1995, calm technology can de defined at a high level by the following:

First, by placing things in the periphery we are able to attune to many more things than we could if everything had to be at the center. Things in the periphery are attuned to by the large portion of our brains devoted to peripheral (sensory) processing. Thus the periphery is informing without overburdening.
Second, by recentering something formerly in the periphery we take control of it. Peripherally we may become aware that something is not quite right, as when awkward sentences leave a reader tired and discomforted without knowing why. By moving sentence construction from periphery to center we are empowered to act, either by finding better literature or accepting the source of the unease and continuing. Without centering the periphery might be a source of frantic following of fashion; with centering the periphery is a fundamental enabler of calm through increased awareness and power. – Designing Calm Technology

The goal of calm technology is to have technology serve humans, not the other way around. In today’s world of being bombarded by ads, smartphone distractions, and apps that compete for our attention, it should be a goal to reduce technology to its essentials and push it to the background, while also allowing it to come to the foreground when it needs to. I think this is going to be a big thing in the coming years, partly as a reaction to the poorly designed technology experiences we are forced to deal with today.

Building a better train tracker

One area I lose a lot of time to technology is in figuring out when the next CTA (Chicago Transit Authority) train will be arriving. The timetables are not real-time so do not account for delays and having to pull out my phone to see when the next train is arriving and when I should leave in order to catch it inevitably leads to being distracted by the gateway of information a smartphone provides. This project was built so that I could easily and almost sub-consciously check when I should leave from my home to catch my next train.

Design considerations

The device is built using an Arduino MKR1000 powered externally and a single WS2812B LED, so these were the hardware parameters that were designed around.

Several considerations were given to solutioning for this design:

    How many trains should this track? From which station, and in which direction?
    How should the notification system operate? At what frequency?
    How should this be designed so it operates at the periphery?
    What technology choices should be made so that this device is a good “technical citizen”?

In order to quickly get this built and test its efficacy, the decision was made to just track one train line, station, and direction for now (the nearest line and station to my residence, and the direction I am most often heading in).

As for the notification system light and color were chosen due to their ambient, peripheral qualities. Obviously having an additional or alternative notification system that relies on a different medium (color choices, sound, etc.) would be necessary for those that are blind or color-blind.

Now there were several considerations given to frequency (times) of the light being displayed and which colors to use. The decision was made to calculate time based on travel time from my residence to the station, thereby reducing one more cognitive step (addition of travel time to station plus arrival time). A train tracker will tell you when the next train is arriving but does not usually include the buffer time you need to give yourself to leave from your location. It was a very conscious decision to avoid using red, yellow, and green as these colors already have meaning in relation to transportation and time. If the LED was set to red would that mean the train was delayed? Stopped on the tracks? Would it mean I should stop? If it was green would that mean I should leave? Would it mean the train is going and has already passed my target station? Note here that as the station is about a 6 minute walk away and I don’t want to wait at the station for that long, any train that will be arriving within the next 6-8 minutes will be “time to leave,” with 5-6 minutes being “I might make it in time” and less than 5 minutes being “too late”. The LED used is programmable, so there is only one light source and the colors settled on were blue (neutral) for “time to leave,” purple (neutral, but with enough red in the hue that it suggests importance or urgency) and nothing (LED off). The original design had the LED turn on to a brownish yellow if I had missed the time to leave for the train, but in order to reduce the cognitive processing needed by the user and to not distract, it was changed to just turn the LED off. The LED stays on and at its current color until a change in state of the train’s arrival time. So the design ends up looking as below:

Don’t leave yet

Time to leave

Might make it if you leave now

While periphery choices have already been discussed, it is important to note that as this device is designed to be running in the background all the time when it is turned on, sound as a notification was decided against. It would get annoying very quickly hearing even pleasant “time to leave” sounds due to the frequency at which trains arrive and leave the station. Light is more ambient and less disruptive. Also, the LED used is just small enough to not stand out, but bright enough to be noticed when you want to check if it’s time to leave.

In order to provide timely estimates, the Arduino fetches arrival time data from the CTA Arrivals API every 10 seconds. This seemed like the right balance between betwork bandwidth with getting timely data. Bandwidth used is already tiny (data is returned in JSON format in a few bytes) and the code is optimized to account for small Arduino RAM and storage size. And to turn off the device when it’s not needed, the on-board Arduino power button is used. It can also be unplugged from power and will restart automatically when it’s plugged back in.

Next steps/future improvements

As this is the first iteration of this product, several ideas may be implemented in the future. A priority is designing a physical case for this device to be housed in, removing wires from the user’s view and further reducing it to its bare essentials aesthetically. There might be several devices that a user has in their home/office/wherever this device might be installed that similarly rely solely on light as notifications, so some differentiation would be required here. Whether this is done through changing the light patterns, or probably more effectively, adding some simple signifier to the physical design of the housing, it should be obvious to the user what the purpose of this device is while still remaining on the periphery. Another improvement would be around power, as the device currently runs while plugged into AC. Moving to a long-lasting battery would make it easier for the user to move the device to another location while also further hiding the technology itself (wires) from view.

Lastly, getting this in the hands of real users and doing user testing would be valuable for product improvements. I’ve got some ideas for adding sensors to this to have some internal metrics running for product feedback too.

 

Monopoly on Empathy

Look through any User Experience (UX) conference presentation description, UX profile on Linkedin, or go to any UX meetup and the word you will hear most often is “empathy.” As UX has become more formalized as a discipline over the last few years this word has been a rallying cry for the industry. “We fight for the user!” and “We bring empathy to software applications!” are commonly heard. But this word is problematic for many reasons and it’s a massive oversight by the profession that it continues to be used. It is not so much because of the word itself but moreso because of its application and implications. While “empathy” is the most commonly used, it’s certainly not the only one in this tale of poor language choices. A pass at slaughtering these sacred cows will be made too. I aim to point out how this language problem is ultimately counter to the goals of UX (both the end user’s experience, as well as UX as a profession). For want of avoiding making this seem like a fastidious excursion into semantics, I will also tie this into how it impacts UX’s standing within the corporation at large.

The foremost problem of using “empathy” to define your profession’s existence is that you are essentially stating that you as a UX professional have a monopoly on empathy. You are implying that developers, marketers, business people, and anyone else peripheral to the UX craft are apathetic, indeed the opposite of empathetic. It is implied that without you, there would be no empathy for the user, as if the rest of the project team would, whether through menace or ignorance, create and deliver an application that confuses and demoralizes the user. Of course, this is absurd; no one will argue with you that wanting to ensure your users can easily use your application and accomplish their goals is a positive thing. And it should be made clear that UX as a group does not have a monopoly on empathy, just as it does not have a monopoly on the user. Other groups are perfectly capable of wanting to ensure the application works well and easily for users. This is all without even mentioning that “empathy” is a very presumptuous term altogether.

Perhaps the second-most offensive word in our UX existentialist vocabulary is “delight”. Within an application measuring delight is subjective at best, and I can guarantee that no user study that utilizes the encroaching practice of video recording will be able to suss out or give any reliable data whatsoever regarding delight. The misguided pursuit of delight has led to unnecessary animations, uncalibrated and perhaps even overly personal/conversational help text and copywriting, among other things. While delight might be nice in video games or entertainment, I would argue it is not something that is even needed in all applications. Do users even want or need to be delighted? Delight should mean they accomplished their goal or task with the application, not that they saw some pretty colors and slick button animations (…Material Design). Or receiving confirmation emails like “you’re totally awesome for signing up dude, surf’s up!”

The final piece of language we will pick apart is certainly the easiest to do so, due in part to its lesser frequency compared to the prior, but also due to the extremity of its claims – “changing the user’s life.” If “empathy” or “delight” were pretentious or presumptuous, laying claim to “changing the user’s life” through designing experiences is mega-pretentious. Too often this language is a mask for signalling of the offending UX professional’s importance, not the user’s. “We are changing the user’s life” is how this is most often heard. The language has become self-serving, ironic given that UX is meant to not be about the designers themselves but about the user.

Why are we seeing such language be used, and used so boisterously? As mentioned in the opening paragraph, UX has only recently become an “official” group within most companies. While UX has been around for a long time, it’s acknowledgment by the corporation is more recent. Thus the profession has had to utilize language that displays confidence in an effort to establish and formalize itself as part of its entry into the corporate landscape. There is without question a degree of pretentiousness and egoism exhibited by some in the design and UX professions. Is the self-righteous language we are examining the brainchild of those offenders? Perhaps, but the more plausible cause is this birth-stage of the UX profession.

Any discipline at early stages is both burdened and liberated by being able to define themselves. Software engineering is a couple decades ahead of UX and thus has had to work to establish itself within the corporation for much longer, so observing the software engineering profession can serve as a cautionary tale for UX as I believe the isolating language that developers often use has resulted in the profession shooting itself in the foot time and again with regards to its standing within the corporate hierarchy.

A common attribute of the language discussed above is moralism (or even self-aggrandizing). Whether or not this is intended, what we can see is a two-fold effect: through moralizing, you start to create a division between you and other groups (them below, you above); through using this language to establish a jargon of its own for UX, you start to become other. Creating barriers between you and other groups – isolation – is existential death within the corporation. Don’t follow the mistakes that the software engineering profession has made by rattling off technical jargon and buzzwords, feeling shocked at the business’ lack of understanding (or more accurately, caring). UX does NOT want to get the same treatment while making remarks like “yeah we just need to boost the level of empathy with this onboarding flow so that we can delight our customers a little bit more and the user’s life will be forever changed.” Do this and UX becomes “those guys creating experiences and delight for our customers… or something like that, I don’t know but they’re in the basement somewhere” rather than a partner of the business.

Furthermore, empathy and delight are not easily understand in business terms. Something like “helping remove roadblocks for users so that our products are better and get more people through the funnel” – now that is something that the business can understand AND it works in service of the user. Hopefully it is evident that vague and vacuous terms like empathy and delight are terrible word choices for a profession that is working to establish more respect for itself within an organization. The reality is that sometimes business goals cannot be ignored, and designers unfortunately cannot always make the decision that is in the best interest of the user. They can “fight for it!” sure, but it can’t always happen and this is ok.

My recommendation: get rid of the moralizing and self-aggrandizing attitude and use language that puts you on equal footing with the business while explaining the importance of paying attention to user needs. Language puffed up to display a confident attitude may have helped when UX was working to establish itself within the corporation, but it can only do a disservice now. Language plays a larger role in business than many of us wish to acknowledge, so change this before it’s too late. Before UX becomes a group that is relegated to lower rungs of the organization, rather than seen as a business partner. Understand whether users actually need empathy or delight. Dispensing with this language can only help UX in its goal of being seen as a crucial element of the business. Developers have their own self-alienating language they use, and have suffered for it. UX professionals should stop now and avoid it before it’s too late. Take a more research-driven approach to design and gather metrics (where you can). Focus less on capital UX and more on lowercase ux, that is focus less on the profession and its existentialism and more on user experience. And if you truly believe that a software application can be “life-changing,” then it is ultimately the application itself, that is to say, the product, that affects this change. That product is everybody’s job, not just UX’s.

 

Setting up Chrome Extensions for use with ES6

First time setup of Chrome Extensions can be painful if you’ve never done it before. Add to that setting them up for use with ES6 and you can end up spinning your wheels longer than writing code. I recently went through this while creating Reading List, which makes heavy use of ES6 as well as Ramda for the functional work. While Babel setup is fairly easy, the module loading presented some challenges. Having originally gone with SystemJS I faced a lot of difficulty in getting the tests to run. After switching to Webpack, for all the horror stories I had heard about it, the issues I was facing were resolved within the hour.

TLDR: You can see an example of the setup here https://github.com/coreyc/reading-list. It is somewhat barebones – intentionally – as so many JavaScript developers waste their time with tool configuration these days. This setup is meant to get you off the ground ASAP.

We’ll step through the setup as follows:

  • Transpiling – Babel
  • ES6 module bundling & loading – Webpack
  • Setting up the Chrome extension
  • Setting up unit tests

Transpiling – Babel

This part is pretty simple. Install the Babel tools we’ll need with the command below:

What does this install? Because Babel can compile several ECMAScript specs we need to install the preset for the version we want to use, in this case ES2015 (ES6). If we wanted ES7 we could install a preset for that too. We also need to install babel-loader so that we can integrate with Webpack. Lastly, babel-register is needed so that we can run our Mocha tests.

Next step is to tell Babel what presets we want to enable. Create a .babelrc config file if you haven’t already and add the following:

And of course if you want to use ES7 features you would add the ES7 preset to this config.

That takes care of Babel.

ES6 module bundling & loading – Webpack

We’ll be using the import / export statements from ES6, formatting our modules as ES6 rather than AMD or CommonJS. Webpack will bundle these modules up for loading in the browser. Install with:

Next we need to add a webpack.config.js file to the root of our project. Configure it like so:

The entry point for our app contains imports of the other files used in the project. It might look something like this:

bundle.js is the output of our modules after they’ve been run through Babel and Webpack. If you have any 3rd party libraries, include them in the externals property so that they won’t be included in the bundle. Otherwise all the code for that library will get bundled up and dramatically increase the file size.

From the command line, run the following in order to actually create the bundle and it’s source map:

Now we need to configure our npm run start command so that it does this bundling and serves up the files in one step. Add this to package.json:

That takes care of Webpack.

Setting up the Chrome extension

Chrome extensions have a config file of their own, manifest.json. Here’s the one from my project:

I won’t go into too much detail as this config can get really complex, but the main things to know are that you specify the icon, the HTML file you want to run when you click the extension icon, what Chrome API’s you need under permissions, then add your content scripts, which are scripts needed by the HTML file you specify. Disclaimer: you can also specify background scripts that run, but I did not make use of these. This setup is not tested for use with background scripts, although they may run just fine.

We take the bundle file output from Webpack and use it as our content script. An important thing to note is that you can specify when this file should run using "run_at". This is especially useful for when you need to use DOM events such as DOMContentLoaded, as extensions seem to block this event from firing. The "run_at" property is useful because it tells the script to execute when you specify, in this case at the start of the document load.

Next we need to add the bundle file to our HTML:

A side note here: I had to add the Ramda library to the HTML even though it was specified in the Webpack config as an external library. Not sure if this is the correct way or not, but it works. YMMV.

That takes care of Chrome.

Setting up unit tests

Now we just need to set up our unit tests. If you don’t already have mocha installed, run npm install --save-dev mocha . Add this to the “scripts” property of the package.json file:

Most info you’ll find on setup will recommend
"test": "mocha --compilers js:babel-core/register test pattern here"
but this seems to be outdated and the Mocha docs recommend just using –require babel-register. From the docs:
“If your ES6 modules have extension .js, you can npm install –save-dev babel-register and use mocha –require babel-register; –compilers is only necessary if you need to specify a file extension.”

Run npm run test and watch your tests run.
That takes care of unit tests.

 

Why is `compose` right to left?

I’ve been working my way through the excellent Mostly Adequate Guide to Functional Programming recently. Although I’ve been working with compose for a bit now, I needed to review why it operates from right-to-left (also known as “right associative”). A review of mathematical theory will help answer this question.

Function Composition

functional composition

The image above is read as “g ∘ f”, or “g composed with f”, or “g-compose-f”. For example, (g ∘ f )(c) = #. If we change ‘c’ to ‘x’, this function would read as g(f(x)). As you’ll remember from high school algebra and composite functions, this means you plug in something for ‘x’, then plug that value into ‘f’, then plug that value into ‘g’. This evaluation occurs from right to left, hence why compose is right associative.

To illustrate compose, consider the following:
f(x) = 2x + 2 and g(x) = –x + 3, find (g o f)(1)
Solution:
(g o f)(1) = g(f(1))
f(1) = 2(1) + 2 = 4
g(4) = -4 + 3 = -1 (remember, 4 was return value of f(1))
…and -1 is our answer from this function composition.

Left to Right

Developers don’t always think in right-to-left fashion, so if you want the reverse pipe (or sequence depending on the library), is an alternative to compose that operates from left-to-right.

 

Uploading Node.js package to AWS Lambda

Quick tip: for those developing AWS Lambda applications using Node.js, if you’re uploading a zip package rather than editing inline, something you might get stuck on while trying to test your function is the below error:

Unable to import module 'index': Error at Function.Module._resolveFilename (module.js:325:15) at Function.Module._load (module.js:276:25) at Module.require (module.js:353:17) at require (internal/module.js:12:17)

First, make sure the name of your handler in the AWS console matches the name of your “main” JavaScript file (the one containing your exports.handler function). Screen Shot 2016-07-06 at 11.01.28 AM

If your file with the exports.handler function is named “index.js”, then in the AWS console, name it as “index.handler”.

Next, something that really tripped me up was not having this index.js file in the root of my .zip. This was what ultimately led to the Unable to import module 'index' error I kept getting. So make sure this file is in the root of the package.

 

Hartford Hackster.io Edison Hackathon

IMG_0584

Intel Edison Virtual Reality

This weekend I developed a project (github source here) as part of the Hartford Hackster.io June 25th, 2016 Hackathon. You can view projects created by other participants here. Intel and Seeed provided Intel Edison and Grove Starter kits to all participants. This project demonstrates the use of the Edison as a sensor gateway, connecting to AWS IOT service for use by a client utilizing Google Cardboard VR glasses.

The Edison takes sensor readings which are then published to a topic bound to AWS IOT. This service in turn takes all sensor readings received and, through the rule engine, publishes them onto a queue (SQS). For the web app, the ThreeJS library provides the graphics and stereoscopic view needed for the Cardboard glasses. The client is using the AWS SDK for JavaScript in the Browser to poll the queue to get sensor readings, which are used to affect how fast the “strobe” is spinning in the scene. You can view the client in a web browser on your phone, placed inside the Cardboard.

This project was an exercise to learn more about ThreeJS, Virtual Reality, and how the real, physical world can be used as inputs to a constructed, virtual world.

Some Findings

  • Initially I was using the AWS IOT rule engine to route all messages received to DynamoDB, using the ${timestamp()} ‘wildcard’ as the hash key to keep all entries unique. However, Amazon Web Services DynamoDB does not provide a way to query the last element added, so I ran into issues when trying to poll the data from the web application (which is using the data to affect the VR world). Unfortunately, DynamoDB is currently the only database that the IOT rule engine supports, otherwise I likely could have gone with RDS (Relational Database Service). I also considered using S3 (Simple Storage Service), but each message would end up in the S3 bucket as an individual JSON file, making querying and pulling the data difficult. Another alternative would have been setting up DynamoDB ‘triggers’ using the Lambda service to respond to database changes, but this still felt kind of hacky. Because my data did not need to be persisted, Simple Queue Service (SQS) provided a viable alternative, and that was what I ended up going with.
  • SQS is not time-ordered. I’m not sure if any queueing systems are time-ordered, but I found out that due to the way SQS is distributed across AWS servers, getting your message perfectly in order is not possible. For my purposes, the sequencing was close enough.
  • SQS has a purge limit of 60 seconds, and because I was reading from the queue every half second, I was not able to immediately delete the message after reading it. If I stick with SQS, an option might be to set the message retention period to match how often I’m reading the queue, although given some latency at various points in my system, it might be better to set the retention period to twice that of the read frequency.
  • Because I did not need to do anything server-side with the messages stored in SQS, I chose to poll the queue directly from the client code. You can use the ‘AWS SDK for JavaScript in the Broswer’ for this. If you only have unauthenticated users accessing the application, the code to authenticate the application to AWS is as simple as below:
  • AWS Identity and Access Management can be pretty confusing. In order to setup the app-level authentication, you have to go to the ‘Cognito’ service, and create a new federated identity. Then use the pool id from there. The service is nice enough to give you the code to drop in.

Future State

AWS is supremely powerful, but as I improve my project, I’d like to try using a different MQTT client for the publishing and subscribing functionality and potentially remove AWS from the equation altogether. Because I would be subscribing to the topic from the web app, I would have to find a MQTT client that can subscribe from a browser. Going with this approach would limit me from the functionality and services AWS provides, but it may be a cleaner approach for the use case of this project.

 

ES6: Mutability of ‘const’

When first hearing about const in ES6, I was excited about the possibility of having immutability in native JavaScript. For developers programming in the functional style, this would have come in handy, but it turns out const is not actually immutable. It allows mutable properties. For example, all of the below is valid:

while the below is not valid:

So the object cannot be reassigned, but the value of the property can be changed and properties can be added and removed. It seems so similar to immutability, but it’s not and it’s an important distinction to make.

It’s probably known by now that if you need to make an object’s values immutable, you can use Object.freeze(), but be aware that this freezes the object entirely. You can’t add more properties or remove properties after it’s been frozen.

It’s probably a good idea to use const as much as possible as it discourages unnecessary re-assignment and forces the developer to think about how the variable will be used. If you need true immutability, you can use the Immutable library by Facebook.

This post is part of a continuing series on practical application of ES6 features. To view the first in this series, check out this link. More are on the way.

 

Functional Programming as the Paradigm for IOT

FP-iot

As the Internet of Things reaches maturation and begins to become commonplace in our lives, the technology used to support IOT must be chosen well. With potentially millions of devices connected, developing applications to support these devices and the data they produce, while converting that data into something meaningful, will require thoughtful attention to technology choices. In building any system, attention to architecture and technology stacks is important, but if IOT delivers on its promise of scale, the technology implications will be very different than what we’ve had to solution and develop for before. It will not be enough to simply “use what we’ve always used” and to keep building things the way we’ve been building them. The challenges are far too complex to not take a step back and look at other options.

What challenges does IOT present?

Scalability and concurrency are likely the two biggest challenges that IOT will bring about. Think of the scale of data that these devices will produce and the number of applications that will be developed to handle those devices and their data; there is the potential for great complexity in designing these systems. While scaling problems can sometimes be solved by adding more infrastructure, this solution won’t apply to the potentially massive amount of Internet-connected devices. And concurrency is an even bigger problem. Millions of devices and real-time communication amongst these devices and consumer-end applications means millions of concurrent connections. Thread-locking and race conditions get hairy fast. Great strides have been made in recent years with non-blocking technology such as Node.js, but this of course won’t be nor should it be the only solution used.

As systems become more complex, so does the underlying codebase, and so we can consider code readability to be just as important as the other two factors.

Functional Programming as the paradigm

Functional programming is well-suited to help solve these challenges. The properties of functional programming – preference for immutability, function composition, avoiding side-effects, less code, etc. – will help avoid many of the pitfalls of an IOT world. Immutable data helps solve the concurrency issue as locks can be avoided. Real-time communication is also better supported by FP. As an aside here, it should be noted that not all FP languages are strictly immutable (for example Haskell has mutable data structures). Furthermore, not all FP languages are created equal when it comes to concurrency management – some perform better than others. This is important to keep in mind when selecting the right language for your application use-case.

Another benefit is side-effect free functions.  While some FP languages are more liberal than others in their allowance of side-effects, FP as a whole favors side-effect free.  This is of great use when programming IOT applications as it makes scaling easier while making the code easier to reason about.  Functions without side-effects can be run in parallel much easier than functions with side-effects as functions that only take inputs and produce outputs only care about their individual inputs and outputs, not other operations like database calls.  This same reason is why side-effect free functions also have the benefit of being able to be better optimized.

Lastly, with FP there is just less code to write, which means less bugs, which means better programs.

Precursors

What IOT-like applications are currently using FP languages?

Erlang

  • RabbitMQ
  • WhatsApp
  • Chef
  • League of Legends chat
  • Facebook chat (first version, now using C++)
  • Numerous game servers (Call of Duty, Battlestar online)

Clojure

  • Netflix
  • Walmart

Elixir

  • Senseware
  • CargoSense

Haskell

  • IMVU
  • Numerous trading/financial companies

As you can see, many of these applications above have similar challenges as those posed by IOT, namely many concurrent connections (chat, WhatsApp, games servers) and scale (all of the above). FP has proven itself in the above applications, furthering the argument that it is a prime candidate for IOT.

There’s still room for OOP

There’s still room at the table for Object-Oriented programming, although it probably shouldn’t be the dominant paradigm. It’s called Internet of Things for a reason, and OOP can still be useful in describing and reasoning about those things. However, central to IOT is data and communication, and it’s easier to reason about this with FP than OOP.

A better glue

Out of the box, Internet-connected devices will rely on the applications and systems that support them to maintain this connectedness and communication, and so these supporting systems must have a good way of doing so. As John Hughes worded it in his “Why Functional Programming Matters” paper, “… a language must provide good glue.” Functional programming is that “good glue” that will help enable technologists to solve many of the challenges brought about by IOT.