Tried TDD and didn’t realize the benefits? Try it the next time you get writer’s block

Have you ever tried Test-Driven Development (TDD) thinking it would be the “holy grail” it’s often made out to be, only to end up feeling like it was pointless?

Maybe it didn’t add any benefit to your code. Maybe writing your test first, then the code after felt uninspiring or limiting, or just the wrong way to do things, especially since the way programming is taught is code-first, not the other way around. Or maybe it just felt like a chore.

All the best developers seem to talk about TDD like it’s the only way to code, and if you’re not doing it that way, you’re wrong. So you really want to like it. But if you’ve tried it and you didn’t like it for any of the multitude of possible reasons, what is the point in practicing it at all? If only you could have that “aha moment” that made TDD make sense, you might actually enjoy it and feel like a “real developer.”

I think the adoption of TDD is more often than not encouraged in the wrong way, and I’m going to show you how to think about it in a different way that will help you realize the benefits more quickly. Not because industry knowledge dictates it’s what you’re “supposed to do” or your team looks down on you if you don’t develop that way, but because it can be one of the best tools in your toolbox to help you when you get stuck.

How it’s usually encouraged

Much has been written about the value of TDD (writing your tests first, then writing the code). The usual benefits touted by TDD-adoption are:

  • less bugs
  • faster overall delivery
  • smaller, single-responsibility functions

Less bugs, faster overall delivery, smaller functions – awesome. Some developers/teams really struggle with this, and for them the benefits will probably click more easily. But it still may not make sense to you why you should do it if you don’t have many bugs or problems delivering code quickly and your functions are single-responsibility already.

And the “why you should do TDD” argument as above, while certainly developer-oriented (especially the last bullet), is more targeted at management. I’ve seen managers who haven’t coded in forever, if at all, herald TDD as the “fix-all”, suddenly mandating it as the new development-style, which ends up turning it into something that’s been chosen for you, rather than something you’ve chosen. This doesn’t help.

Thus, TDD can become something you feel you should do because:

  • Your boss told you to
  • The industry tells you to
  • You’re looked down upon by your peers if you don’t
  • You look down on yourself if you don’t

Or maybe you don’t have any of these pressures at all – you just don’t get TDD. Maybe the long-term benefits of less bugs and easier to read/write functions is just too abstract right now.

But attempting to adopt TDD with this mindset, it can turn TDD into more of an adversary as opposed to something you do because it helps you – you as a developer who is under the gun to deliver on a feature.

Something more relatable

Instead of understanding TDD from a “best practices” perspective, I’ve found it easier to understand in more concrete terms, something more relatable.

As stated in the title of this post – to get TDD to “click” try using it the next time you are faced with writer’s block, called “coder’s block” from here on.

What is coder’s block?

Have you ever been in a situation where you’ve been completely stuck trying to figure out how you’re going to implement a particular piece of code? Maybe a deadline is approaching and you’re really screwed if you don’t get that code written, but you keep hitting coder’s block and don’t know how to start. Before I started using TDD to push through these block I used to just browse Reddit, HackerNews, etc. as a way of procrastinating. Either I was overwhelmed by a really hard problem and I didn’t know where to start breaking it down, or it was just one of those days.

While “best practices” is abstract, I bet you’ve encountered coder’s block lots of times. But you can use TDD here to help you out of that situation. Not because someone told you you’re not a good developer if you don’t, but because it helps you.

Side note: I’m not a TDD-purist. I understand it doesn’t always make sense to write tests first (R&D work, initial proof-of-concepts/sketches, pure DOM/view code, etc.). But TDD as removing writer’s/coder’s block has been invaluable for me, which is why I recommend it here.

How to do TDD next time you get stuck

In order to demonstrate how you’d go about this, let’s imagine a simplified scenario. You have a feature for an online shopping application you’re working on in which the requirements are:

  • Customer must be able to enter their preferences under a “Profile” tab
  • Customer preferences must be saved
  • Preference input fields must match some regex

Imagining you’re stuck and aren’t sure where to start, you could think about what the very first test you could write would be.

There are several requirements here, but you know you’ve got to manage the state of the selected/entered preferences, so that’s a good place to start. The test, assuming the application is in JavaScript, might look like:

This might not seem like much, but it’s actually quite a lot. We’ve already figured out what shape our state/preferences need to be in, which is a meaningful part of the implementation. And more importantly, we began by not knowing where to start at all.

An example implementation of the code for that test might be:

Cool, now another test:

And that code:

Start with one unit test, then the code. Another test, another piece of code. Now you’re probably already over that block you had when you started.

Wrapping up

Thinking about TDD in this way will hopefully help you realize the power of it. A lot of getting TDD to “click” is getting into a rhythm. And more importantly, using it as tool to help you, not something that’s a “best practice” you’re following.

When you get going and get over that block it will start to make more sense. Just like how you break something down by writing a todo list, then you do the things on that list – using TDD to overcome coder’s block and seemingly overwhelming features is the same mechanism.

This will ultimately be what makes you a better developer – overcoming blocks by learning to understand requirements and breaking down the problem into manageable parts. Not only will you spend more time coding – which itself will make you a better developer – but you’ll know how to make things manageable.

The next time you’re stuck try writing just one test before you write the code. Only one. Even by figuring out a starting point, this will greatly help in getting unstuck and give some direction, and even if you don’t use test-first after those first few tests, you’ll have figured out a path to implementation.

JavaScript developers – want to try this the next time you get stuck, but need more guidance on unit testing? Sign up below and I’ll send you a few links that will help you get started. After you’ve signed up I’ll send you my new posts every week or two, and I’ve got plenty of unit testing and TDD content planned.

Subscribe for more unit testing in JS content


Using the `debug` module to avoid polluting your application logs with logs from Node modules

Have you ever added logging to your custom Node modules, thinking you’ll benefit from all that extra information, only to find that when you add the modules as dependencies, run npm install and npm start, your application logs start looking like the below?

How were you able to find any of the application information you actually needed? Information like which users logged in and when they logged in, and not log info coming from your application’s dependencies:

Similarly, have you ever been in a war room scenario at work when you suspected production was down due to a change your team made in a Node module dependency, but the only logs you had to troubleshoot looked like:

This is what log pollution looks like from a bird’s eye view:

What do we want

The dream logging scenario is being able to keep your module logs out of your application/service logging, but being able to turn them on for troubleshooting purposes when you need to.

To achieve this, I’m going to show you how you can use the debug module to toggle on/off logs in your Node modules.

OFF for when you don’t need them, and your service that consumes the module is running fine.
ON for when you’re facing issues and need as much logging information as you can get.

Overview of the debug module

Functionality-wise, think of the debug module not in the way you’d usually think of as a debugger – like the kind you’d set breakpoints with in your IDE – but think of it more like a modified console module. One that lets you turn on and off your logs depending on what you need. If you clicked the link to the GitHub page, you might have encountered various other features it has, like namespacing for example. Don’t worry about those for now, just know it’s a tool to save you in production (and lower environments too).

Implementing debug in Node modules you wrote yourself

Let’s look at an example of using debug in code. Imagine the below is a module:

Implementing debug is pretty easy. All we have to do is import/require the module then create a logger – const log = debug('http:server');. What is that ‘http:server’? It’s simply the flag/namespace for that specific logger that you will pass to the DEBUG environment variable in order to turn ON the debugging.

When you start your application – the one that’s importing this module – you will start it like so:

DEBUG=http:server node app-entrypoint.js

What if we want separate loggers in order to toggle some logs but not others? For example, what if we want to toggle only errors? It’s as simple as creating another logger, like so:

So turning on only errors would look like:

DEBUG=http:error node app-entrypoint.js

And when you want to log all namespaces (in this case, http:server and http:error), simply pass the wildcard * flag.

DEBUG=http:* node app-entrypoint.js

Let’s see what those logs look like now:

Then, when we load the page:

As a heads-up, debug writes to stderr, not stdout, so if you have a log router that picks up logs based on different sources, be aware these debug logs will end up in the stderr stream.

Lastly, you’ll also have a choice to make:

  • you can log everything you’re going to log in the modules you’ve written using debug
  • OR you can still log some things using console.log() / console.error() and others to debug

If you have mandatory things that absolutely need to be logged – and that matter to the application that’s using the module – log those with console. But if you don’t, I’d log everything using debug.

Turning on debug for third-party modules

So far what we’ve covered is great if you (or your team) wrote the module.

But what if you’re using a third-party module that you need to turn on debugging for?

Maybe you’ve done troubleshooting for every part of your application, now thinking that it might be that open-source module you’re using that is causing you issues.

The good news is that debug is such a widely used module, that many modules in npm are probably using it. Third-party module creators know (or at least, should know) that users of that module don’t really want it’s logs on by default, for all the cluttering up of the rest of the logs that can happen.

Express uses the debug module internally so if you’re using Express and need to troubleshoot it, you can turn on the logs by running:

DEBUG=express:* node your-app-entrypoint.js

Finding if and how an open-source Node module uses debug

What if the module hasn’t documented if it’s using debug or not? Again, chances are that it’s being used, but you’ll have to do some digging.

We already know Express documents how to use it, but let’s assume that it wasn’t documented:

First, you could search the project’s GitHub (or other open source site) for its package.json and look for the module listed there as a dependency:

Once that’s confirmed, you’ll also need to confirm 1) it’s actually being used and 2) what flags it takes for toggling. GitHub has a nice feature now where you can use the search bar to search the project itself. If we search “debug” in the Express project, we’ll find the following:

We can see that the toggle flag is “express”, so “express:*” would give us the logs for everything, “express:view” would give us the logs for the views, etc.

Wrapping up

Hopefully you see how easy it is to implement debug in your Node modules and turn it on for third-party modules, as well as how much it can help clean up your logs while giving you an easy way to turn them back on for troubleshooting.

To get an example project with fully tested code as well as a bundle of links that’ll help you with implementing logging in the best way for your application, sign up below and it’ll end up in your inbox.

Subscribe to continue getting a better grasp on JavaScript and Node.js


Why should your Node.js application not handle log routing?

It is not the responsibility of the application to route logs.

12 Factor says that logs should go to STDOUT. WAT? WHY?

I just configured my whole application code to write logs to custom log files. What’s wrong with that?

Logging is one of those things that can sometimes be a black box for developers. Maybe you have a dedicated DevOps person who takes care of logging infrastructure for you, or maybe it’s your first time working on this side of things.

It can be one of those things you leave until last to take care of while you’re too busy working on writing code. Many do, making the “best practices” around logging seem like something you can just ignore, if you even understand them in the first place…

We’re going to take a look at deciphering the reasons behind the best practices of decoupling your logging from your application, and where you should actually log to. And for purposes of this post, “log routing” – as referenced in the title – refers to picking up and pushing the logs to an intended logging target that is not your application or application process.

Best practices illuminated

You may have heard of the 12 Factor App before, which is considered the canonical “best practices” document around creating modern, scalable applications.

From the “12 Factor App best practices regarding logs“:

A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout…. In staging or production deploys, each process’ stream will be captured by the execution environment, collated together with all other streams from the app, and routed to one or more final destinations for viewing and long-term archival. These archival destinations are not visible to or configurable by the app, and instead are completely managed by the execution environment.

That’s a lot to decipher, so let’s break it down.

“A twelve-factor app never concerns itself with routing or storage of its output stream.”

The first major reason why you don’t want your application code to handle routing of logs itself is due to separation of concerns. We often think of this separation in terms of pieces of code between services and between services themselves, but this applies to the more “infrastructural” components as well. Your application code should not handle something that should be handled by infrastructure.

This code below is an example of highly coupled application code.

Let’s leave the deployment environment concerns out of it for a moment, which we’ll look at later, and instead focus on the application itself.

Just by having the application handle logging, it now has taken on another “concern” under its wing. By defining what the logging outputs are, the application now handles both application/business logic AND logging logic.

What if you need to change your logging location later? That’s another code change and deployment (and more if you have a strenuous QA/change control/deployment process). And what if you get a logfile name wrong? Again, another code change and deployment.

This is not to say your application should take an extreme stance towards logging and avoid log statements as well – you do have to log something, after all – but it is to say that the log routing adds another layer which does not belong in the application if you want to decouple components of your code and keep your application code clean.

Next,

“It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout.” (Side note: although it specifically mentions stdout, I take it to mean stdout and stderr, and cursory Google searches seem to confirm this.)

I already discussed above why logging to outputs like files and databases is not a good practice from a separation of concerns perspective. But this is where the environment concerns start to be addressed.

In Node.js applications, you are still logging to something and that is the console (using usually either console.log() or console.error()).

The console, under the hood, prints to stdout for console.log() and stderr for console.error(), so simply by using this module, it looks like we pass this test.

And this test exists for a reason: if you’ve worked with physical or even virtual (but not container/cloud) servers before, you might have had only a handful of them, or at least a size that was manageable enough to manually configure the logfiles, their locations, and any other setup.

Now imagine your application has had some big success and is onboarding hundreds of new users everyday. Your team has begun migrating to a cloud-based environment, and you have to plan for your application scaling on-demand from 1 instances to 50. You won’t know where those instances are running, so you can’t control where exactly the logfiles get written to.

It’s more useful to have stream | target, as opposed to target -> (your routing solution) -> target. Streams give us the ability to pipe anywhere, composing together powerful pipelines. If you’ve ever used Linux/Unix, you can build up powerful operations simply by piping streams together, like searching for text within a file: cat example.txt | grep sometext. stdout/stderr gives you this power. For example, you could pipe from stdout to a logfile if you wanted to.

Furthermore, cloud applications are ephemeral. They can spin up, spin down, crash, etc. which means the logs are ephemeral too.

So while we started out looking at why an application should not handle routing logs to files/databases/other persistent storage targets, this brings up the question: is it ok to log to those targets at all?

Next,

“In staging or production deploys, each process’ stream will be captured by the execution environment, collated together with all other streams from the app, and routed to one or more final destinations for viewing and long-term archival. These archival destinations are not visible to or configurable by the app, and instead are completely managed by the execution environment.”

This helps answer that question. It’s OK to route the logs to persistent storage (and you, in fact, absolutely should) if the execution environment does this routing from the stdout/stderr logs.

This also re-affirms the separation of concerns covered earlier. We can’t be sure where a logfile might end up. And if a container crashes – and logfiles weren’t being picked up by a log router in the first place – you’re screwed. Good luck debugging the reason your application crashed in the first place.

Cool, but then how do you manage logs in production? Is there a tool that picks up whatever is sent to stdout/stderr?

This is actually where the log routing piece comes in, the whole thing this post has attempted to dissuade you from handling from within your application code.

For simplicity’s sake, assume you’re using Docker for your containers as part of your cloud environment. The Docker daemon that runs on your Docker host – not your container – will by default pick up the logs from stdout/stderr from your container(s).

You configure the Docker daemon to use a logging driver, which does the actual log routing work of picking them up and routing them to a given storage target, like so:

In the daemon.json file,

You can view a list of logging drivers – which, again, do the work of picking up the logs and routing them – supported by Docker here. The list includes Greylog, Splunk, syslog, and other log aggregators you might be familiar with.

Routing the logs somewhere is important so that, in case your application crashes, boots up with scaling up, shuts down with scaling down, you have a persistent storage location from which to view them.

But it’s important that this is done at the infrastructural level, for the reason discussed above.

A complete logging picture based on what’s been discussed here would look like:

Wrapping up

To summarize the reasons you don’t want to handle routing from your application and, by extension, to something other than stdout/stderr:

  • keeping the log routing responsibility out of your application code:
    • keeps the code cleaner
    • makes log routing locations easier to change without deployments
  • scaling applications/containers means it’s harder to have control over logfiles
  • scaling applications also means they’re more ephemeral, meaning logfiles might not be there depending on the state of the container
  • writing to, say a file or database, over stdout/stderr ties you down to those log targets, takes away your flexibility to pipe the output of stdout/stderr to whatever targets you want, and change this on the fly

To address one last question you might be having: what if you’re not using a cloud environment or containers?

My thoughts on this are as follows. The approach I’ve laid out here is still useful, because:

  • you may one day move from physical or virtual servers to a cloud / container approach, making the migration path much easier on yourself or the team that will be doing the work
  • you still keep that separation of concerns
  • you can always just pipe the stdout to a log file or other persistent storage target and get the same advantages as a Docker daemon would provide

As you are working on implementing logging or reviewing your current logging code – if you are deciding between using a logging framework versus console.log() and console.error(), I wrote a post on that that can help you make the decision here. Just make sure keep this post here in mind and write to stdout/stderr from the logging framework unless you absolutely have reason to write to something else.

Want to continue getting a better grasp on JavaScript and Node.js topics? Subscribe for more content


Should you use a logging framework or console.log() in Node?

The console module is usually the first tool Node.js developers reach for when handling logging in an application. It’s easy to use, native to the platform, and easy to read.

But then you might stumble upon a logging framework like Winston or Bunyan. The basic power of those tools might be attractive enough for you to consider switching.

But should you? How do you know which one you should use? Should basic console.log’s be avoided entirely and not even used?

What console does

While browsers implement console differently, in Node the console module will print to stdout and/or stderr. If you’re using console.log() it will print to stdout and if you’re using console.error() it will print to stderr.

Why does this matter?

Being able to write to stdout and stderr means that you can have Docker or logstash or whichever tool you’re using easily pick up these logs. stdout and stderr on linux are pipeable, so this becomes easy.

By only using console, you’re reducing a dependency (and all of it’s dependencies) that you would have if you were using a logging framework. You don’t even have to require/import the module like you do with some other native Node modules like fs.

Side note: above console refers to the global console, but it is possible to import console as a Class, which you can then instantiate to configure your own output streams rather than just stdout and stderr. I’m pointing this out just as a form of technical due diligence but it is not something you need to be concerned with for now as this is not the way console is usually used in Node. If you’d like to read more about the instantiation method however, you can check out the docs here.

Lastly, because it’s common for front-end JavaScript developers to work on the Node parts of applications too, console has the same API methods as the console used by browsers, making it effortless to pick up.

Disadvantages of console

From the above, console seems like it should get the job done, so why would we even consider something else?

One of the biggest disadvantages is that you can’t toggle logging on and off, not out of the box at least. You could wrap console and extend it to do this, but this is code you’ll have to write, and likely code that will have to overwrite the built-in console functions.

Why would you want to turn off logs in the first place?

You might want to turn off logging if you’re in a development environment vs a production environment. Or even if you’re just testing locally on your machine or VM, if you’ve got a ton of logging for debug purposes or otherwise, that can really clutter up your console and you might want to just test with logging disabled for a bit.

Another disadvantage of console comes when you need to know log levels.

While it already has what appear to be log levels (see below), these are really just functions that route to stdout and stderr without providing true log levels.

So in the Node console, you won’t be able to tell these logs apart unless you prepend a string with the level to the logs.

Winston, a logging framework, will print out the level for you, like so:

With these things in mind, what does a logging framework give us that console doesn’t?

To put it bluntly, a logging framework will help with what console doesn’t. Popular Node logging frameworks like Winston and Bunyan allow for log levels, easy toggling logs on and off based on environment, and sometimes (in the case of Winston) support for custom log levels that you as a developer can define.

Logging frameworks will also (generally) support writing to more than just stdout/stderr. Winston calls these “transports” while Bunyan calls them “streams”. For example, you can configure Winston or Bunyan to write to stdout, a file, and a database all at once.

Side note: this ability to write to multiple different outputs, however, is a feature I would recommend against using. The full explanation why is not within the scope of this post, and I will write a new one in the future with a more complete explanation, but for now know that the reason is to keep log routing separate from your application code.

Logging framework use cases still not clear enough?

Let’s consider a real world example:

A requirement comes in that you need to log only errors in production, and add a custom level named “qa” to be logged in your QA/testing environment. Your DevOps team is sick of sifting through so many logs in production and only care about the errors. And your dev team wants to log results from a particular function call to a specific level that they can keep an eye on in while testing.

How can we accomplish this?

Only logging errors in production:

Adding the custom “qa” level:

Disadvantages of logging frameworks

I catalogued the shortcomings of console above, so it’s only fair that I do the same for logging frameworks. And up until now, I’ve tried to be careful about particularly categorizing the disadvantages as “shortcomings” because it truly depends on your use case.

So, why might you not want to use a logging framework? When would vanilla console do?

The most immediate answer is that you’re adding a dependency to your application, a dependency that might have lots of other dependencies and increase build/deploy time (albeit likely only very slightly), and one whose API is subject to change which could mean refactoring down the line.

They also might just be overkill for your scenario. If you don’t have a use case for custom log levels, for writing to different targets (remember, I recommended against doing this), and you don’t care if your logs are always logging, skip the framework.

Wrapping up

For parting recommendations, if you’re just starting out (i.e. – building a new service or you don’t have a logging framework pattern in place from other applications your team has built), use the console module.

Only reach for a logging framework if you have requirements that warrant it. If you’re having trouble finding enough differences that would make you choose one over the other, go with console to start, and you can always switch later on.

Want to continue getting a better grasp on JavaScript and Node.js topics? Subscribe for more content