A recording, some delay

---
layout: post
title: A recording, some delay
date: 2021-10-17T21:37:40.929Z
slug: a-recording-some-delay
---

I used to do a lot more music. I haven't recorded music in years. I suppose I
have now? [You be the judge][d-flat-soundcloud].

{/* <SoundCloudEmbed
  title="D♭"
  artistName="hughrawlinson"
  artistUrl="https://soundcloud.com/hughrawlinson"
  trackUrl="https://soundcloud.com/hughrawlinson/d-flat"
  trackId="1143430867"
/> */}

<iframe width="100%" height="300" scrolling="no" frameBorder="no" allow="autoplay" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/1143430867&color=%23ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&show_teaser=true&visual=true"></iframe>


### For those who for some reason would like to know more...

I went into a lot of detail here, which definitely will come across as what it
obviously comes across as. But, listening back to the songs I recorded in the
past, I don't remember much about my intent or frame of mind, which I regret, so
I'm writing it for the record. So, know that I know how this seems, and if you
aren't here for that, ignore me!

I was watching [the new documentary about The Velvet
Underground][velvet-underground-documentary], a band I've listened in the past
but never on repeat for months, like Carly Rae Jepsen, or Queen or The Darkness
before her. There was discussion of drone music by La Monte Young, one of the
leading minimalist/ambient/drone/experimental musicians of that era, and a lot
of what they were saying reminded me of the kind of music I sometimes made, when
I used to do that sort of thing. I felt inspired.

I was noodling on my guitar along with the tunes of the documentary, and I guess
I had been thinking about the kinds of generative systems I used to study and
use as a student at Goldsmiths. I was particularly into the kind of tape loops
systems that Brian Eno used in Music for Airports, which I did before in [a
previous recording][stark-soundcloud].

I had a bit of a notion of rather than using predefined loops, I'd add playheads
to one longer recording, the flexibility being that I could add and remove more
playheads dynamically, affect each playback rate independently, and use that to
reveal variations in an acoustic performance, played without a metronome. I find
it really difficult to keep notes consistently at a stable timbre and velocity
on guitar, and my timing without a drum track to play against is pretty poor, so
I thought that I'd maybe find something interesting as a base of a new system.

Having just been learning more about La Monte Young (last time I looked, I found
next to nothing about him, other than his appearance in [The Rest Is Noise], by
Alex Ross (which also covered The Velvet Underground), but now he's on 
[bandcamp][la-monte-young-bandcamp]), I was thinking about drones, and decided
that playing one note over and over again was enough of a drone for me.

I DI'd my acoustic guitar, turned on my speakers, and record-armed that track,
as well as my SM7B for room sound. I actually had both record-monitoring on my
speakers, and wasn't getting feedback, and though that that might add to it, but
in the end it turned out that I hadn't record-armed the SM7B, so ended up with
just the DI acoustic. I played D flat octaves on the third and fifth strings for
12 minutes, trying to keep the tempo, timbre, and velocity as even as possible.
I didn't pick D flat for any reason, it's just what I happened to land on. It
occurred to me later that it's not a particularly convenient key for anything.

During the recording, I watched the clock in the bottom right of my computer
monitor, and when my monitor turned off for inactivity, I watched the clock on
my phone. I looked out of my apartment, at the lights of the outskirts to the
south of Amsterdam, towards the enormous screen at the arena. Leaning back in my
chair, I focused on hearing the sound in the room. The harmonics sounded like
melodies, and I wanted to keep recording while my hands cramped and built
callouses, so that I would have a base over which to add extra tracks to
articulate what I was hearing in the harmonics, and patterns in the shifts in
timbre and velocity.

Here's when I discovered that I had missed the SM7B track, which was a little
annoying, but not a big problem.

I fully intended at this point to record more tracks to add to this. I started
out by cranking up the recording (I had a lot of headroom) and looping the quiet
at the start to identify and EQ out any hums, of which there was one. I wanted
to get an early glimpse into what the multi-playhead scenario might sound like,
so I dropped a simple delay into a return track, cranked the feedback up, and
that's what you're hearing.

I suppose that at this point, it sounded interesting enough to me that I should
avoid the temptation to spend a bunch more time on it during which I could start
doubting it, or get bored of recording extra tracks that weren't achieveing what
I hoped, or fall off writing the code for the multi-playhead thing.

Specifically what interested me was that I could hear patterns in the harmonics
and rhythmic patterns from the delay that could be interesting. The kind of
thing where I would like to go in and isolate snippets, take "melodies" I hear
in the harmonics, and rhythm patterns, and reinterpret them with synths and drum
machines. Maybe even superimpose them together somehow.

I'm quite used to playing guitar with delay, I'm a big fan of Brian May's work
with line delays. In fact, one of my go-to noodling songs is Queen's [Now I'm 
Here], which notably uses stacking delayed guitars in the intro. [Brighton Rock
Solo] is another instance of that stacked effect, where you play in time with
yourself. That's not what I'm doing here, I wasn't hearing the delay while
playing. It's all in post. I wasn't able to keep myself in time that way. This
means that as I rush and lag gradually throughout the recording, those
variations in tempo interact with the constant-time delay to produce these
"interesting patterns" I keep talking about.

It's been a very long time since I've felt creative in this way, and I like this
feeling.

I'm a firm believer in that meaning is in the mind of the audience, so I won't
venture too far into the kinds of things I think I'm trying to say with this,
other than that perhaps I wouldn't have felt the need to put a 12 minute
recoring of myself playing the same note on guitar over and over again in a
world where the last 18 months didn't happen, with all that happened in them.

There's still plenty of stuff I wanted to try but didn't. I guess maybe after
these 7 years of not doing music, I might have more stuff to try out some time.
I have the dry and wet recordings, so there's plenty of opportunity to explore
further.

I should definitely note that the ratio of time I have spent in my life
listening to experimental music vs the time I've spent making is is firmly
suggesting that I need to listen more, so if you have suggestions, ideally ones
that come with explanations, please let me know!

- [Twitter](https://twitter.com/hughrawlinson)
- [Mastodon](https://mastodon.technology/@hughrawlinson)

[d-flat-soundcloud]: https://soundcloud.com/hughrawlinson/d-flat
[stark-soundcloud]: https://soundcloud.com/hughrawlinson/stark
[velvet-underground-documentary]: https://tv.apple.com/us/movie/the-velvet-underground/umc.cmc.69ic79cvvy80epfhz5efdgjjd?ctx_brand=tvs.sbd.4000
[The Rest Is Noise]: https://www.therestisnoise.com/noise/
[la-monte-young-bandcamp]: https://lamonteyoung.bandcamp.com/
[Now I'm Here]: https://en.wikipedia.org/wiki/Now_I%27m_Here
[Brighton Rock Solo]: https://en.wikipedia.org/wiki/Brighton_Rock_(song)

Objecting to "Legitimate Interest" at Scale

---
layout: post
title: Objecting to "Legitimate Interest" at Scale
date: 2021-10-07T13:54:40.929Z
slug: objecting-to-legitimate-interest-at-scale
---
_This post is mainly for me to have somewhere to store this snippet I use, but might be of use to others._

In 2021, browsing the web requires spending about 10% of your time opting out of the use of tracking cookies, and objecting to advertising tracker companies assertion of "legitimate interest" in your private information. I object in principle to advertising trackers. The industry around them profits from gathering biased data and legitimizes the reinforcement of those biases - that systematic discrimination - by training "neural networks" and pretending that they must be objective and unbiased because a computer did it. Besides, random corporations shouldn't get to quietly spy on us and sell their observations to advertizers, that's creepy. If you'd like to read more, check out Cathy O'Neil's great book, [Weapons of Math Destruction].

Therefore, when a website asks me to let them and a thousand "ad partners" track me, I say no. Or, at least, I try to. There are so many nefarious dark patterns in cookie consent popups. Facebook and Google seem to require the user to go to a separate page, leaving the content they were looking for behind in favour of having to spend 5 minutes deciphering legalese. Others style the link to the settings page to not look like a link, making you think there's no settings.

One pattern that really annoys me is when a website has "partnerships" with hundreds of ad companies, and doesn't let you object to their claims of legitimate interest all at once, leaving you to click on many hundreds of buttons. I had one of these show up recently, and I didn't enjoy it. So after clicking the first few buttons and realizing how many there were, I jumped into the console. Here's the snippet that objected to all the "legitimate interest" claims in a cookie consent dialog provided by a company called [Admiral].

```js
Array.from(document.querySelectorAll("input:checked")).map(input => input.click())
```

The company provides a few services including a paywall system, privacy/consent dialogs, registration walls/email acquisition (scams), and even a service called "adblock recovery", which claims to basically bypass adblock systems.

Imagine a world where the advertising industry didn't see their audience as adversaries. Alas, we're in the bad timeline.

Anyway, I plan to run that snippet when I see the "Admiral" logo.

[Weapons of Math Destruction]: https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction
[Admiral]: https://www.getadmiral.com/

Don't Do Conditional Return Types in Typescript

---
layout: post
title: Don't Do Conditional Return Types in Typescript
date: 2021-08-06T21:17:37.402Z
slug: dont-do-conditional-return-types-in-typescript
categories:
  - typescript
---
_Or, more accurately: Typescript doesn't do conditional return types safely..._

I have been working hard on a new Typescript interface for [Meyda], and have been struggling. I'm sure I'll write a longer post with more learnings from that experience, but here's a quick one.

You may find yourself writing a function where the return type varies depending on what was passed in. For example:

```ts
function myFunction(a: boolean): /* return type? */ {}
```

In that example, I might want the function to return a number if the boolean is true, but a string if the boolean is false. In this scenario, one might reach for Typescript's [Conditional Types] feature. Briefly, conditional types are like ternary expressions in the type domain.

```ts
// Value domain ternary expression
const x = typeof "my string" === "boolean" ? "good" : "bad"
```

The predicte returns `false`, because `"my string"` is a string, not a boolean and so the `false` side of the ternary is selected.

```ts
// Type domain ternary expression (conditional type)
type T = "my string" extends boolean ? "good" : "bad"
```

Similarly, the literal type `"my string"` (which is a type, not a value) is not assignable to (doesn't extend) the type `boolean`, therefore the false side of the conditional type is returned and assigned to the type alias `T`.

This should help us tell Typescript that our function returns one type if the arg is `true` and another type if the arg is `false`.

```ts
function myFunction<T extends boolean>(a: T): T extends true ? number : string {
  // ...implementation
}
```

There's a little additional syntax here to enable the conditional type, which I'll quickly explain: we introduce a Generic type parameter for the function, not because we want it to take more than a boolean, but because we need to refer to the same type `T` in multiple places: the signature, and the return type. In fact, the generic is `T extends boolean`, which limits the type parameter to a boolean. Of course, our parameter is then of type `T`, and we focus on the conditional return type: `T extends true ? number : string`. That type means "if the value we pass in is specifically only a `true`, then we'll return a number. Since the only other member of `boolean` is false, we'll return a string in that case".

This seems to satisfy the compiler, at first glance. Indeed the Typescript documentation at time of writing includes a similar example of using a conditional type in the function return type position. But there's a problem.

When we implement the function, we get a few complaints from the compiler.

```ts
function myFunction<T extends boolean>(a: T): T extends true ? number : string {
  if (a) {
    return 0;
  }
  return "";
}
```

Both return statements are invalid!

```markdown
Type 'number' is not assignable to type 'T extends true ? number : string'.ts(2322)
Type 'string' is not assignable to type 'T extends true ? number : string'.ts(2322)
```

It turns out that Typescript has no way of assigning a concrete return value to a Conditional Return Type. That's because neither return statement in the function knows about `T` - so they have no way of saying which side of the case they're on.

So what are the options?

You could write `as any` after each return statement. As long as you know that you're throwing out any type safety when you use any, I'll trust that you know your context well enough to make a judgement call. I would try to avoid this at all costs.

You could use function overloading - almost. See this example:

```ts
function myFunction(a: true): number;
function myFunction(a: false): string;

function myFunction(a: boolean): number | string {
  if (a) {
    return 0;
  }
  return "";
}
```

But there's a subtle bug that shows up in more complex situations.

```ts
declare function myFunction(a: number): number;
declare function myFunction(a: string): string;
function myFunction(a: number | string): number | string {
  if (typeof a === "number") {
    return 0;
  }
  return "";
}
```

The type system doesn't enforce that your implementation returns a value of the same type it received. Your code could receive a number and return a string, and the type system couldn't catch that.

If you add a generic to handle that case as follows:

```ts
function myFunction(a: number): number;
function myFunction(a: string): string;
function myFunction<T extends number | string>(a: T): T{
  if (typeof a === "number") {
    return 0;
  }
  return "";
}
```

You're back to square one - neither return value is assignable to the annotated implementation return value.

After this, I'm all out of ideas. I'm not sure that it's possible to achieve this effect in Typescript. There is [an issue tracking some changes that might enable this][issue-narrowing]. There's also [an issue tracking the removal of the conditional types example that suggests they can be used in this way][issue-docs]. If I've missed a cool way to do this, please do let me know on [Mastodon] or [Twitter]! Thanks for reading!

[Meyda]: https://meyda.js.org/
[Conditional Types]: https://www.typescriptlang.org/docs/handbook/2/conditional-types.html
[issue-narrowing]: https://github.com/microsoft/TypeScript/issues/33014
[issue-docs]: https://github.com/microsoft/TypeScript-Website/issues/1931
[Mastodon]: https://mastodon.technology/@hughrawlinson
[Twitter]: https://twitter.com/hughrawlinson

Type Soundness in NextJS Query Parameters

---
layout: post
title: Type Soundness in NextJS Query Parameters
date: 2021-07-16T15:13:58.679Z
slug: type-soundness-in-nextjs-query-parameters
categories:
  - typescript
  - nextjs
---
I was recently writing some NextJS code and came across a definition of a type that was unsound.

```ts
export default function apiHandler(request: NextApiRequest, response: NextApiResponse) {
  const { aRandomQueryParameterThatDoesntNecessarilyExist } = request.query;
  // ^ that var is of type string | string[] - typescript thinks it has a value
}
```

"No big deal", I thought, "I guess NextJS represent a query parameter that doesn't exist as an empty string".

```ts
export default function handler(
  request: NextApiRequest,
  response: NextApiResponse
) {
  const { fakeApiQueryParameter } = request.query;
  response.send(`value of fakeApiQueryParameter: ${JSON.stringify(
    fakeApiQueryParameter
  )}
		typeof: ${typeof fakeApiQueryParameter}
		`);
}

```

I got the following response:

```
value of fakeApiQueryParameter: undefined
typeof: undefined
```

Oh.

To clarify what's going on here: when you access _any_ value on `typeof NextApiRequest["query"]`, typescript believes that property to be `string | string[]`, and doesn't make you do a check for undefined - even though in practice, when an API request comes in without a query parameter, the value is `undefined`. If you then try to access a property, for example `length` (because that exists on both `string` and `string[]`, but not on `undefined`), an error will be thrown at runtime, and typescript won't catch that error at compile time.

Unless!

Typescript 4.1 introduced the `noUncheckedIndexedAccess` [compiler option], which in effect adds `| undefined` to every [index signature], so that typescript will expect you to check for undefined. In the case above, it will fail at compile time - hooray, no uncaught errors at runtime!

In very happy news, this applies to types outside of your own codebase - so there's no need to submit a PR to NextJS asking them to turn this option on (or adding `undefined` to their type definition).

I think that `noUncheckedIndexedAccess` is a great option and should be turned on. I have done so in my codebase, and was delighted to catch several other areas where runtime errors could have happened. I don't *think* there are significant downsides - given the nature of index signatures (specifically that they can be indexed by any string or any number), it's unlikely that they will ever be comprehensive.

Of course, I can imagine a scenario where this check adds more boilerplate code than you might want. If you know that your endpoint is only going to be called in a specific way, and you're happy with an uncaught exception at runtime if your endpoint is called in some other way, maybe you'd prefer to keep it turned off.

Bonus: if you're sure a value at a given key has been assigned, you can override typescript with `!`.

```ts
type T = {
    [key: string]: string
}

const key: string = "aKeyThatExists"

const myObject: T = {
    [key]: "a value"
}

typeof myObject[key] !== 'undefined'
  // use ! to tell typescript that here you know the value is defined
  // because you know that the key previously was assigned a value in the object
  ? myObject[key]!
  : "fallback value";
```

Have I missed something? Is there a context in which the burden of checking the existence of every index signature before access outweighs the benefit of doing away with this type of uncaught runtime crash? Let me know on [Twitter] or [Mastodon]! I would love to hear your experience.

[compiler option]: https://www.typescriptlang.org/tsconfig#noUncheckedIndexedAccess
[index signature]: https://www.typescriptlang.org/docs/handbook/2/objects.html#index-signatures
[Twitter]: https://twitter.com/hughrawlinson/
[Mastodon]: https://mastodon.technology/@hughrawlinson

How to Ship man pages with your Node Programs

---
layout: post
title: How to Ship man pages with your Node Programs
date: 2021-07-09T10:49:58.361Z
slug: how-to-ship-man-pages-with-your-node-programs
categories:
  - nodejs
  - documentation
  - npm
---
The Javascript ecosystem uses the npm package registry to distribute libraries - and also binaries. If you've ever done `npm install -g yarn`, you've used this feature. The yarn package exposes a binary (a node js script) that executes on your system as a program, often invoked in the command line. Your npm installation then installs that package in a central global directory, and links each binary exposed by the package to a central directory that's in your shell. Then you can invoke the binary on your command line.

```
hugh@hugh-XPS-13-9343 ~> yarn --version
1.22.10
```

Pretty neat right?

In unix systems, the [man utility] is a common way to look up how to use a given command.

```
hugh@hugh-XPS-13-9343 ~> man yarn
No manual entry for yarn
```

But not all packages provide man pages. This isn't terrible - in the case of yarn, there is a whole 'help' subcommand to look up information about how to use yarn's cli - plus [a documentation website][yarn cli documentation]. But, if like me, you think the developer experience is improved by meeting your developer where _they_ expect you to be, you might like to distribute at least a nearly empty man page pointing devs in the right direction. If there's some chance of some portion of your users reaching for `man {your binary}`, I would suggest that it's worth weighing the effort of providing at least _a_ man page with the benefit that those users would get from having docs (or a pointer to docs) where they expect.

Isn't that effort huge? Aren't man pages in a weird format for fancy native developers? How would I even install them from an npm package? In this post I'll attempt to convince you that making a basic man page is not a huge lift, and might be worth the your work for the developer experience benefit.

Let's start off with a creating a basic empty package for demonstration purposes, and installing some dependencies that will help us create our man pages.

I've created [an example repo] for you to refer to if you like.

```
$ mkdir my-package && cd my-package
$ npm init --yes
$ npm install --save marked<1 marked-man
```

Now we find ourselves with an empty package with two dependencies - marked, and marked-man. marked is a peer dependency of marked-man, the package that will take our markdown document and convert it to roff, the format used by man-pages. If you'd like to cut down dependencies and write in roff directly, go ahead! But I figure most javascript developers will be more familiar with Markdown.

Right now, there's a [bug in marked-man] causing it not to support versions of it's peerDependency marked greater than 1.0.0. At the moment, I suggest installing a version of marked below 1, and keeping an eye on the bug.

Next, let's write a simple document. Store the following in `README.md`.

```
# my-package(1) -- demo package

## Synopsis

my-package is a demonstration package that does nothing

## See also

example.com
```

There's a git more going on here than simple markdown syntax. Let's walk through it. On the first line, we have a heading containing our package name, followed immediately by a number in brackets. This number refers to the man 'section number' for your page. In our case, we're using section number 1 to show that our documentation concerns "Executable programs or shell commands" - but you can specify library calls, special files, games, etc. Check out the table in the `man man` document. The package name(section number) is followed by a ` -- ` spacer and a short description of your command.

Later on in the document, we can see sections headed by h2s. These are man "section names". You can have custom sections, but conventional section names include NAME (which is automatically generated for you), SYNOPSIS, CONFIGURATION, DESCRIPTION, FILES, NOTES, BUGS, AUTHORS, SEE ALSO, and more that are shown in the `man man` document.

Time to use marked-man to generate the man file. Man files are stored in the `roff` format, and we can use marked-man as follows to generate our roff file from our README.md.

```
$ ./node_modules/.bin/marked-man README.md
```

You'll see the following output:

```
.TH "MY\-PACKAGE" "1" "June 2021" "" ""
.SH "NAME"
\fBmy-package\fR \- demo package
.SH Synopsis
.P
my\-package is a demonstration package that does nothing
.SH See Also
.P
example\.com

```

I'm certainly glad I don't have to manually write in that format! Let's store this in a directory, and add an npm script so that we don't have to type out the full command every time. Add the following entry to your 'scripts' object in your package.json.

```
    "generate-man-page": "mkdir -p man && marked-man README.md > ./man/my-package.1"
```

Note that the roff output is stored in the `man` directory in a file called `my-package.1`. For your man file, you should follow the same naming convention: `{packageName}.{section number}`.

How do we make the man utility aware of the document when we install the package? We add an entry pointing at the file in our package.json as follows:

```
  "man": ["./man/my-package.1"]
```

We can test this out by running "npm install --global .", which will install the package in the current directory onto the system globally. Then, run `man my-package` to see the man page in action.

```
MY-PACKAGE(1)                                                                        MY-PACKAGE(1)

NAME
       my-package - demo package

Synopsis
       my-package is a demonstration package that does nothing

See Also
       example.com

                                          June 2021                                  MY-PACKAGE(1)

```

There we have it! Man pages for your node packages. If you'd like to have multiple man pages for your package (say your command is configurable by a dotfile that you'd like to document, for example), you can write multiple markdown documents, modify your npm script to generate them all, and add them to the list of exports on your package.json `man` object.

You may notice in the npm documentation that there's a `directories.man` configuration in the package.json spec, which is documented as exporting _all_ the man pages to the system. I wasn't able to get that to work. If you are, please [let me know]!

Thanks for reading. I hope that I've covered the procedure for generating man pages from markdown documents such that it's clear to you - and I hope you agree that it's a relatively low amount of effort to add a touch of delight to your package's developer experience. If you have any feedback, please get in touch [on twitter] or [on mastodon]. I'd love to hear from you!

[man utility]: https://en.wikipedia.org/wiki/Man_page
[yarn cli documentation]: https://yarnpkg.com/en/docs/cli
[bug in marked-man]: https://github.com/kapouer/marked-man/issues/28
[on mastodon]: https://mastodon.technology/@hughrawlinson
[on twitter]: https://twitter.com/hughrawlinson/
[let me know]: https://twitter.com/hughrawlinson/
[an example repo]: https://github.com/hughrawlinson/npm-man-page-example


How to use AWS Infinidash with Node JS

---
layout: post
title: How to use AWS Infinidash with Node JS
date: 2021-06-30T21:58:44.527Z
slug: how-to-use-aws-infinidash
categories:
  - Trolololol
---
If you’re like me, you’re excited about the secular tailwinds in the sails of AWS Infinidash, and can’t wait to let it take your next product to web scale. But does it work with node?

Boy does it!! Keep reading this blog post to find out how.

## Step One: create your repo

Go to GitHub and create your repo. Be sure to enable new issues, code pilot, and code spaces for maximum synergy.

## Step Two: buy a domain

Use your favorite registrar to buy a new domain for your project. Remember, product-market-fit is second only to branding. Make sure to use a new tld like .badass or .profit, for maximizing your brand wow factor.

## Step Three: get a credit card with great rewards

You’ll use this for AWS - make sure you have a good credit rating to start with. Once you have, make sure to get a credit card made from metal, since the higher conductivity reduces transaction fees and yields better hash rates - and therefore rewards points! Don’t miss out when your site goes viral on HN - make sure AWS Infinidash can drain your credit card to keep it up and running.

## Step Four: sign up for aws

To do this, you write a letter (including a blank postal order) with the following text:

```
To whom it may concern,

It may concern Jeff Bezos. Hi. One fancy new product, please!

Joe’s Cheesemonger,
Eclectic Entrepreneur
```

## Step Five: install infinidash

To do this, you can use the following command:

```
npm install —-save @jeffrey69420/infinidash
```

But if you’re a contrarian, this command might suit you better:

```
yarn add @jeffrey69420/infinidash
```

And if you’re very 😬😬😬:

```
sudo apt-get install node-aws-libinfinidash-dev
```

I think it’s also available on maven but the joke isn’t worth the effort of looking up how to use that absolute nightmare package manager.

## Step Six: initialize infinidash

```js
var infinidash = import from “infinidahs”
infinidash.commenceCountdown().then((_infdsh) => {
  return _infdsh.toTheMoon(!!!!!!);
});
```

Now when you run the code, every feature will be web scale. Get ready to go to the moon! It’s that easy! Also you’re in Alexa now.

————

Thanks for listening guys, now remember to smash that subscribe button, join the notification squad 🔔🔔🔔🔔🔔 and don’t forget to follow me on SoundCloud, where I drop a new mixtape every third Tuesday.

Sweet custom cheese boards are available from our sponsor [Joe’s Cheesemonger](https://twitter.com/JCheesemonger), use our promo code “NeverStopTrollingJoe”. for 69% off.



Transitive dependency vulnerability resolution for npm

---
layout: post
title: Transitive dependency vulnerability resolution for npm
date: 2021-06-21T15:11:44.909Z
slug: transitive-dependency-vulnerability-resolution-for-npm
categories:
  - npm
  - javascript
  - packages
---

_This blog post uses npm, not yarn. While I imagine most of the concepts have
equivalents, you'll need to do the translation on your own if you use yarn._

Not infrequently, I get reports from GitHub's security scanner, or while running
`npm install` about security vulnerabilities in dependencies. These
vulnerabilities rarely exist in packages that I depend directly on - more often
they exist in packages that my dependencies depend upon themselves somehow.
Here's a quick writeup of how I think about resolving these.

## Make the robots fix it

In the best case, these vulnerabilities are immediately fixable by robots.

### npm audit fix

![a screenshot of the output of npm audit, showing fixable
vulnerabilities](/screenshot-from-2021-06-21-17-22-18.png "npm audit output")

In the case, you've been alerted to a security vulnerability while running npm
install, which tells you that the vulnerability can be entirely mitigated by
running `npm audit fix`. This command knows that the version of the vulnerable
package that contains the patch is within the version range specified by the
depending package. It's able to bump the vulnerable package in the
package-lock.json, and the issue is solved.

### GitHub Dependabot security alerts

![github dependabot security alert showing two
vulnerabilities](/screenshot-from-2021-06-21-17-20-36.png "dependabot security
alerts")

Similarly, if you've received a security alert from GitHub Dependabot, you
should be able to ask it to create a fix and send a pull request, as shown
below.

![a screenshot of a dependabot button to create a security
fix](/screenshot-from-2021-06-21-17-21-17.png "dependabot 'create a fix'
button")

Both cases above are examples of what to do when transitive dependencies contain
vulnerabilities that are within the version range of the depending package.
They'll also (from what I understand) check if the depending package has a later
version with a version range that allows for the patched version of the
vulnerable dependency, and upgrade _that_ if necessary.

## When the dependency can't be fixed by robots

But what if there is no version of the depending package that can use the
patched version of the vulnerable package? At that stage, you have some
decisions to make. You can either choose to put in the work of sending pull
requests to each necessary dependency in the chain, and hoping that their
maintainers will be responsive, or you can choose to ignore the vulnerability.
This is a resource tradeoff, and the best decision will vary depending on your
own context. Questions you should ask yourself include:

- is my project actually affected by this vulnerability? - is the vulnerability
  in the execution path of my code? - is the vulnerability bundled into the
  package I distribute? - does the vulnerability affect developer tooling? - what
  is my organization's security policy? - does my organization provide resources
  for mitigating security risks like this? - do I have the time to dedicate to
  submitting one or more pull requests?

Whatever you decide: you have made the correct decision for your context. You
are an expert in your domain, and you have weighed the pros and cons, sought out
all the relevant information, and taken a calculated risk with either your
engineering capacity or your risk budget.

## Monitoring vulnerabilities

One trap to avoid, however, is conflating ignoring one specific vulnerability
with ignoring them all. In a previous team, we had a script in our continuous
integration pipeline that would get the output of npm audit, and fail the build
unless there were no new vulnerability alerts. When a new vulnerability was
found, all deploys were halted to prevent inadvertently deploying the
vulnerability into production. Since the project was a monorepo used across 10+
teams, my team investigated any new vulnerability (transitive or non-transitive)
immediately (within working hours), to avoid blocking other team's work. In
general we followed the process above - if it were possible to get a robot to
fix it, we would. Otherwise, we would weigh the effort of manually fixing
against the severity of the vulnerability, and make an educated trade off. Of
course, we did this within the context of our organization's security policies,
which we didn't violate. If we decided not to fix the issue immediately, we
added the advisory number to an allow-list of known issues, so that our script
would know that we had accepted a given risk, and allow work (and deploys) to
continue.

We also kept an eye on the vulnerabilities. On a regular schedule, we would
audit each security advisory on our allow-list, and check to see if we had
automatically mitigated it with dependabot, or if it had become worth the effort
of fixing. We made sure not to let advisories needlessly languish on the list,
and kept an eye out to make sure that we hadn't introduced the vulnerability
into a new codepath, thereby changing the priority of the fix.

This never happened, but in extreme cases, if a security vulnerability was in
danger of affecting our users, we would have forked the dependencies to mitigate
the issue in the short term, and as a team come up with a strategy to idenfity a
path forward - whether to continue maintaining our fork(s), find some way to
switch to an alternative provider, or refactored the behaviour into some new
structure, that we could consider releasing ourselves. Like I said though, this
didn't happen while I was on that team, so this is purely hypothetical.

**Update!!**

My wonderful former colleague Drew who I miss a lot shared [this tool][nsprc] for
managing npm advisory allowlists in their projects! It looks excellent, and very
in the spirit of what we used. Strong recommend!

## Thanks!

for reading this post. If you have any questions or comments, or would like to
berate me for doing something silly and wrong, please contact me [on twitter] or
[on mastodon], where I am available for all of the above.

[on twitter]: https://twitter.com/hughrawlinson
[on mastodon]: https://mastodon.technology/@hughrawlinson
[nsprc]: https://github.com/jeemok/better-npm-audit#using-nsprc-file-to-manage-exceptions


What I learned when investigating temperature and humidity monitoring for my apartment

---
layout: post
title: What I learned when investigating temperature and humidity monitoring for
  my apartment
date: 2021-06-18T20:07:38.771Z
slug: what-i-learned-investigating-temperature-and-humidity-monitoring
---

It's extremely hot and humid in my apartment. For no particular reason, I'd like to monitor how hot and humid it is. In theory, I could buy an air conditioner and see how much of a difference that makes, but in practice, I'll just feel that. But before it occurred to me that there isn't really much utility in temperature monitoring the apartment that I spend nearly all of my time in, and can monitor myself, I went digging into how I might do so.

I haven't done much physical computing since I put the [Buttery Biscuit Bass] on the shelf when I finished at university and moved to Sweden. One or two little Spotify Web API hacks - but connecting a Spark Core to wifi with fancy corporate auth wasn't easy, and I mostly gave up. So I'm a little bit rusty - back in the day, I learned how to use a Raspberry Pi and the Arduino platform. I had a few nice shields, most of which I had at least hacked a little bit with. I had heard of some communication protocols like SPI and I2C, but hadn't used them. And I had designed a few circuits (I made a synthesizer in a lemsip box once). But a lot has changed since then, companies have pivoted, I've learned how to do computers gooder, and I have less tolerance for stringing wires all over the place.

This article isn't a tutorial, it's more "personal notes about stuff I've discovered so that I remember them and hopefully they help you too", I don't claim to have a deep understanding about any of this. Your mileage may vary, I may be wrong about stuff - but hopefully you can use this as a jumping off point.

## Platforms

### Arduino

Arduino is still going strong. They have a few new boards now, including an [uno with wifi]. I remember having to order wifi shields for a project, only to discover too late that they didn't support access point mode, and ending up having to prototype with a Raspberry Pi instead. Arduino-compatible chips are used in custom mechanical keyboards quite often, even though they run the QMK program, rather than Arduino's software, so I have had a little exposure to them when building my [keebio iris].

### Raspberry Pi

Raspberry Pi have released a whole load of new products, including an $80 fully loaded pi with usb 3, wifi, two hdmi ports, and a bunch of other fancy things. They've also got a line of smaller devices, including a Raspberry Pi zero. I was lucky enough to attend a christmas themed hackathon in the UK in maybe 2017 or so where all the attendees were given a Raspberry Pi zero, although I regret to say that I haven't used mine yet. They also have a Raspberry Pi Pico, which uses a custom processor, and a Raspberry Pi contained entirely in a keyboard.

### Spark Core

One other chip I had picked up at some point was a Spark Core. I discovered that unfortunately they don't exist anymore - or at least, not in the same way. The Spark Core was a wifi enabled development board that could phone home to Spark's servers, and be programmed via a web IDE. If I understand correctly, Spark also was a sink for telemetry and data. Some time since I got my Spark Core, Spark renamed to Particle, started offering new chips (I think the 'photon' is the analog to the core), and pivoted to target enterprise use cases, rather than the developer focused marketing I had previously seen. I tried to get it working, but I couldn't discover the ssid in setup mode, so RIP to that lil chip - and I don't think I'll buy into the new iteration.

### Feather

Feather is a newer board that I don't remember being extremely aware of at the time. They're an ecosystem made by Adafruit, of boards on nice breadboard compatible prototyping kits. Feathers are available with all kinds of different actual microprocessors and onboard goodies like wifi, bluetooth, connectors, etc. Because of the range of microprocessors, they're available for all manner of programming platforms - arduino, c/c++, rust, circuitpython/micropython. I definitely intend to try some of these out some time.

### Espruino

I remember having heard of the Espruino back when I was more into hardware, but I think I assumed it was an ardunio clone of some kind. I also wasn't as into Javascript then as I was now, so I might not have appreciated an array of devices that contain lots of fun stuff like screens and sensors that are programmable in Javascript! I definitely want to try these out too.

## Writing embedded code

### Arduino

Back in the day, I wrote Arduino code. It's a great ecosystem full of libraries and driver support for lots of peripherals. It supports I2C directly, and has a really nice developer tooling setup. So nice that I never really needed to learn much about how the programming of the ROM actually works, what instruction set the code is compiled down to, how the microprocessor works directly, or any of that. I'd like to learn a bit more, but not at the expense of getting stuck in the weeds and never getting anything done.

### C/C++

I vaguely understand Arduino code to be an SDK for C/C++ (not even sure which one), that makes it easy to write code using high level languages. Writing embedded code using C/C++ opens up access to writing code that directly runs on lots more boards, but also requires a much heavier understanding of developer tooling than Arduino requires. C/C++ is reputed to be a difficult language, and while I have some experience in it, I definitely remember a high footgun ratio, and inscrutible error messages. However, lots of development boards provide libraries in C/C++, and the tooling evidently exists, so it could be a good way to go. I just don't really know the developer experience.

### Rust

Rust is another way to write code that compiles down into the bytes that go into the ROM on the chip and run directly on the bare metal, with no Operating System required. Rust is a great language, with lots of safety features that make it hard to write code with tricky bugs, like C/C++. There's also [loads of great learning material for embedded development]. One minor drawback is that I haven't been able to find libraries for each microprocessor that I've checked, and certainly fewer libraries for onboard utilities like wifi chips, bluetooth, etc (although I should have a thorough read of the books before I give up. My hesitancy comes mostly from not wanting to get bogged down in learning about developer tooling, rather than spending time writing useful code.

### MicroPython/CircuitPython

I actually have used MicroPython before, since it ran on the [emfcamp tilde mkiii], the badge that was handed out at the electromagnetic field camp in 2016. I enjoyed that, and it felt like just writing standard python, mostly. I quite like python as a language, and I accomplished what I intended to, rather quickly if I recall correctly. Unfortunately, the embedded Pythons require quite a lot of memory, which isn't necessarily ideal for a lightweight sensor beacon. I'm not ruling it out, but I'm hoping to run into an option that's this easy to use, without the memory overhead. Which is Arduino, I guess, for the simplicity level I'm going for here.

## Sensor protocols

In the past, I had mainly used simpler analog components like light activated resistors, and buttons that activated different resistance amounts, and used analog arduino pins to read in data. I never quite got to the stage where I was using devices like neopixels, displays, or more complex sensors. I've since heard of I2C and SPI, and have become aware that there exist tons of sensors with drivers that are easy to use from most of the platforms listed above. I've also learned that connecting these sensors together and between boards isn't always standardised. Some boards have pins dedicated to the important uses in I2C and SPI, but don't have connectors for a standard socket. Of course, breakout boards can help with this, and are available for most of the boards listed above. There are some standard connectors though. Adafruit has [a great resource on I2C connector standards], but here's a quick summary.

### Grove

A company called Grove makes lots of cool looking I2C sensors, and have a connector that seems to have a significant ecosystem around it. Lots of sensors are available, and there seem to exist breakout boards for this standard. It looks a little bulky, but I haven't seen it in real life, so am not qualified to know this.

### Adafruit Stemma/Stemma QT

A lot of Adafruit's products are made with Stemma/Stemma QT connectors. These are 4 pin connectors that carry I2C, and are cross compatible. Stemma QT is a smaller version of the original Stemma connectors, but still carries I2C, so it's useful for smaller profile devices. Like the grove, they use 3.3-5v, so they're cross compatible, but have a different connector, so you'll have to get Grove-Stemma/Stemma QT wires.

### SparkFun Quiic

Lots of SparkFun's I2C devices use the Quiic connectors, which also carry I2C. They operate only at 3.3 volts, but they use the same connectors as Stemma - so if you make sure your Stemma device is using 3.3 volts, they're compatible.

I haven't chosen an ecosystem to go with here yet, but since they're all reasonably compatible, I don't expect too much lock-in.

## Communicating to other devices

I think I've only ever communicated to/between development boards using either serial over usb, or wifi (except of course that I've plugged an arduino into an ethernet cable), so I'm rather unfamiliar with protocols here. Here's what I learned:

### Bluetooth

Bluetooth Low Energy provides a really cool way for devices to communicate with each other. There are two modes, GAP for devices that aren't connected, but can broadcast their advertising ID for connections or information in GAP mode, and GATT, for devices that are connected to each other to do two-way communication. It's pretty flexible and low power, so I'd

### WiFi

WiFi is wifi, it's good. It connects the device to a WiFi network, it can connect to other devices over TCP and UDP, which means http requests, presumably websockets, irc if you like, connecting to things like mqtt servers etc. The downside (as I understand it at least) is that many WiFi chips can be rather power hungry, and if you have too many devices on your network without configuring your DHCP server you may run out of local IP addresses. WiFi seems to have a shorter range than Bluetooth in my apartment, but of course, it's wifi, you know about WiFi.

### Zigbee

Zigbee is another wireless communication protocol that isn't WiFi or Bluetooth. Devices can talk to each other over it, but it doesn't buy you a connection to the internet. It has a good range, but it can interfere with WiFi and in order to get data from a Zigbee device, you have to have another device listening and talking to something else.

### Zwave

Zwave is another wireless communication that isn't WiFi or Bluetooth or Zigbee. It's older than Zigbee and doesn't interfere with WiFi, but it's slower, and it's another protocol that isn't already built into my phone.

### LoRa

LoRa seems like a really cool protocol because in perfect conditions it has a 10km range. However, I don't need my apartment's temperatures broadcast all over the city, and I don't feel like doing encryption, so I'll avoid this one.

### Thread

Thread seems like a newer wireless communication protocol that has IPv6 built in, and lets devices mesh together, rather than requiring a hub like an access point. Of course, in order for a device to bridge to the internet, at least one of the devices has to be connected to both Thread and an ordinary network.

## What can I do about storing data and triggering actions?

My project is about taking measurements and using those measurements in a program later. Here are the options I've considered for storing the measurements.

### Grafana

Either self hosted, or by the Grafana company as a service, I could use Grafana to store the info. Strictly speaking in the case of self hosting, I would use a time series database like Prometheus, Grafite, or Heroic (not really that one) to store the data for viewing in Grafana. I've seen some really cool uses of Grafana - at Electromagnetic Field Camp 2016 the bar used it to keep track of stock, and to let people know when the next keg of their favourite beer was in from storage. I've also had to use Grafana to set up custom alerts at work - it turns out (at least in my experience) that Grafana is nice to use as a viewer, and not as nice to use as an editor.

If I wanted to take actions based on the results, I think theoretically I could have bots listening to alerts. But that definitely seems squarely in the 'overengineering' territory, and not something I want to get into.

### Custom code writing to a database

Naturally, I could choose to write custom code - a script that either listens to bluetooth using the radio on my own computer that I leave on all the time, or a cloud function that my device can call over WiFi and responds. If this really is the quickest option to accomplish what I need, I would go for it. I would need a database, though, to store measurements, and to manually write code to manage the database size (deleting old records), and any frontend would need to either be able to call my backend in the right format, or be entirely custom. Seems like a lot of work, not necessarily ruling it out.

### io.adafruit.com

Adafruit is an awesome company that builds hardware. The feather development board and many of the sensors I would need for this project are their design. They know that a lot of their customers buy their hardware to monitor information, and might benefit from focusing on the hardware, rather than having to write a load of software for their project too. So they built [io.adafruit.com], a service where internet connected devices can write their data to. They handle the UI, security, they support data uploading and triggering things based on cloud state. It seems absolutely worth trying out - in fact, I intend to give them a go with the hardware I buy regardless of whether I end up doing something else in the end.

They have a generous free tier which allows pushing 30 data points per minute, 30 days of storage, triggers every 15 minutes, and a 5 feed limit, with a higher limit available on a subscription. They even provide a lot of clear information about data privacy, which is a breath of fresh air on the internet.

### MQTT

MQTT is a really cool system designed specifically for IoT telemetry. It's a pubsub protocol, with a broker - one centralized system accepts connections from agents (whether IoT devices, or servers performing actions based on data, etc) that can each both publish and subscribe messages on topics. There's a lightweight protocol, so it shouldn't take up too much power, and it should be fairly easy to write code for. Slight bother that you have to run an MQTT server, and decide whether that should be local or in the cloud. If it's in the cloud, there are security concerns. If it's local you have to keep it up and running (on my computer I switch between booting into Ubuntu and into Windows fairly often).

It's a really cool protocol and I want to try it out, but I think maybe not for this project - at least v1. On the plus side, all the hardware I'd need for the build should be compatible with MQTT, so if I want to try it out later, I won't need to buy a whole new set of stuff.

---

So that's what I've learned! Writing this post has coalesced my own knowledge of these assorted hardware things, so a) thanks for indulging me, b) I hope you've found it useful too. If you have comments, are annoyed that I got something wrong, or that I missed a technology you're interested in and would like to see included in the comparisons, let me know either [on twitter] or [on mastodon], and let's talk about it. Thanks for reading!!!

[buttery biscuit bass]: /posts/2015/03/25/week-10
[uno with wifi]: https://store.arduino.cc/arduino-uno-wiFi-rev2
[qmk]: https://beta.docs.qmk.fm/#/
[keebio iris]: https://keeb.io/products/iris-keyboard-split-ergonomic-keyboard
[a great resource on i2c connector standards]: https://learn.adafruit.com/introducing-adafruit-stemma-qt/what-is-stemma
[loads of great learning material for embedded development]: https://www.rust-lang.org/what/embedded
[emfcamp tilde mkiii]: https://blog.emfcamp.org/2016/05/17/tilda-mk%CF%80-the-hackable-conference-badge-that/
[io.adafruit.com]: https://io.adafruit.com/
[on twitter]: https://twitter.com/hughrawlinson/
[on mastodon]: https://mastodon.technology/@hughrawlinson


Solved: do-release-upgrade hangs in Ubuntu on WSL

---
layout: post
title: "Solved: do-release-upgrade hangs in Ubuntu on WSL"
date: 2021-06-04T17:38:00.474Z
slug: solved-do-release-upgrade-hangs-in-ubuntu-on-wsl
categories:
  - ubuntu
---

[skip the story, what do I do](#what-do-i-do-to-solve)

I've been using Ubuntu either natively or via Windows Subsystem for Linux pretty consistently for a year or so, and it's been the base of our production baseimages at work for years - I was even involved in the migration from Trusty Tahr (14.04LTS) to Bionic Beaver (18.04 LTS). For audio-crackling related reasons, for the past few days I've been on Windows and using WSL to have better audio in zoom meetings before I figure out the issue.

I needed to reverse-i-search my fish shell history, and noticed that my reverse-i-search binding was broken because I was missing the fzf package. No problem, I thought, I'll just download it.

```sh
sudo apt install fzf

# No package found
```

Some investigation revealed that fzf does not publish a package for 18.04LTS, the distro I was using in WSL. Not ideal. But, I checked, and it is safe to do dist upgrades on WSL installations. Hooray!

```sh
do-release-upgrade
```

An upgrade was found, but somewhere in the call tree, something hung, and timed out after 2 minutes, aborting the upgrade process, and not giving much feedback as to what might be wrong. Once I had dug around on [launchpad.net], the web system where the Ubuntu project tends to store packages, I found that `do-release-upgrade` exists in the ubuntu-release-upgrader package, which calls out to the update-manager packages. These packages handle the release upgrade service for Ubuntu, calling out to a web server to check for relevant updates, preserve LTS status where possible, and serve signed bundles of scripts to handle the update. Great! I was now able to find the relevant branch of the code that ran during the `do-release-upgrade` process.

Once the script has pulled the bundle, I found that it stored the package in a directory in /tmp. Browsing this bundle, I was able to confirm that it was from a branch of the ubuntu-release-upgrader package `ubuntu/focal`. I cloned the repo and had a look.

The log in `/var/log/dist-upgrader/main.log` showed me that there was a call to `focalPostInitialUpdate`, a suspiciously exact two minute pause in logging, and then a call to the abort function probably in a timeout. Inspecting this function, I saw that it did some intial validation, and then called out to a function on the snap package manager to check connectivity. That sounded suspicious. I could see that the command was `snap debug connectivity` and found that sure enough, snap was timing out, and failing to connect via a socket to snapd. Was snapd, the daemon that serves snap requests, running?

It turns out that it was not. In WSL, Ubuntu runs without systemd running in normal mode. This meant that the snapd service was not running correctly, causing the timeout!

I'm not sure whether snap came installed on the WSL ubuntu image, or if I installed it some time over the course of the last two years. But I did find, in a cursory Google, that it's safe to remove. So after a quick `sudo apt remove snap`, I was able to run `do-release-upgrade`, but this time it quickly made it past the spot where it hanged before, and the upgrade was a success! My WSL image is now running on Ubuntu 20.04LTS Focal Fossa. Big success!

But for me a bigger success was learning about the Ubuntu package ecosystem, the ability to search through the code that my OS was running, diagnose the problem, and fix it. Would it have been better if the do-release-update script had "just worked"? Not for me, I had fun. For lots of others, yeah absolutely. Computers, maaaaaaaan.

This is the real end of the article, the next section is for people who just wanted to skip to the solution without going on the journey.

# What do I do to solve?

```sh
# Snap causes the hang in the upgrade script
sudo apt remove snap

# Do the upgrade, the kind of hang that was happening on my system shouldn't happen.
sudo do-release-upgrade
```

Later, you can reinstall `snap` if you need it. For generic instructions on how to upgrade your distribution on wsl, [check this article out](https://www.nextofwindows.com/how-to-upgrade-existing-wsl-wsl2-ubuntu-18-04-to-20-04)

[launchpad.net]: https://launchpad.net


What to do if you publish a beta build as @latest: modifying npm version tags

---
layout: post
title: "What to do if you publish a beta build as @latest: modifying npm version tags"
date: 2021-06-04T10:35:42.030Z
slug: what-to-do-if-you-publish-a-beta-build-as-latest
categories:
  - npm
  - javascript
  - package-manager
---

I recently published a beta build of [Meyda] to the npm registry, with the intention of having one of our longest running users test it out to make sure it worked in their project. I hadn't done a manual release in a long time, since we use `semantic-release`, so I skimmed the output of `npm publish --help`, and figured out what command I would run. I
set the version field of package.json to `5.1.7-beta.0`, as instructed built the bundle, ran our test suite, and ran `npm publish . --dry-run`, to verify that the manifest of files that _would_ be published was correct. It was correct, and so I ran

```
npm publish .
```

When I checked [Meyda's page on npm], I was rather surprised to see that `5.1.7-beta.0` had been published as the `latest` tagged version of Meyda. I had incorrectly been assuming that the magic incantation required to publish a beta package was the `-beta.*` suffix in the package version. In fact, the way to publish a beta version of an npm package is

```
npm publish . --tag beta
```

At this point, I became worried. Had I published a beta build of a package that might inadvertently contained breaking changes to all my users? While yes, technically I had done that (for the second time that week, but that's another story), I needn't have worried. Some research revealed that while packages once published to the registry in most cases cannot be removed or modified, tags can. The `npm dist-tag` command saved the day! I'll leave you to read the `npm dist-tag --help`, and instead show what I did to resolve my situation. The previous "good" latest version of my package was `5.1.7`.

```
# Tag the previous version as latest
npm dist-tag add meyda@5.1.7 latest

# Tag the beta as a beta
npm dist-tag add meyda@5.1.8-beta.0 beta
```

Once the tags were properly set, none of our users were at risk of getting broken code they hadn't opted in for, and the user who agreed to test our beta release was able to install it with `npm install meyda@beta`.

No need to worry if you find yourself in this situation. Like most things, it's completely recoverable, and everything will be ok. We all lived happily ever after!

[meyda]: https://meyda.js.org/
[Meyda's page on npm]: https://www.npmjs.com/package/meyda


Loading Audio in Node JS

---
layout: post
title: Loading Audio in Node JS
date: 2021-05-19T09:06:27.573Z
slug: loading-audio-in-node-js
categories:
  - node
  - audio
  - meyda
---

Working with audio as a developer can unlock many awesome features, and a lot of fun. You can generate music, analyze audio using machine learning, build audio visualizers, music information retrieval systems, and much more. It's an extremely fun field. But working with audio can be tricky - how is sound represented on a computer? How can we manipulate that sound? And how do we serialize sound data to disk?

## Pulse Code Modulation Encoding

This post won't be a deep dive into audio encoding - it's a practical guide for how to load audio in Node JS, into a state that you can work with it. Generally, digital signal processing (which means "working with audio data using code") operates on a kind of audio data called [Pulse Code Modulation] (or "PCM" for short). There's a lot of fancy theory and maths behind PCM encoding - but until you're ready to dive in to Wikipedia, you can think of it as "a long list of numbers that represent the change in air pressure over time that makes up a sound". This is, after all, what a microphone measures and converts into numbers.

### Samples

Each number in the list that makes up a sound is called a "sample". The sample can be represented on disk as one of several kinds of numbers - floating point numbers, integers, or other representations. The number of bits that represent the number affect the precision of the number - for example, 16 bit numbers can have much more precision than 8 bit numbers. The number of bits in each sample is referred to as the "bit depth".

### Sample Rate

Another important attribute of PCM encoded audio is the "sample rate". This refers to the rate at which samples should be played in order for the sound to be at the right speed. For reasons outside the scope of this post, the sample rate dictates the highest frequency component that can be represented in a sound. For the purposes of most audio intended for human listening, it's important to store audio at a sample rate slightly higher than double the maximum frequencies that humans can hear. Since humans can't really hear audio over 20,000hz, a standard sample rate has emerged at 44,100hz. The "hz" unit here refers to hertz, which means "samples per second". Sometimes you can encounter audio with a higher or lower sample frequency - audio for movies can be up to 192,000hz, and signals representing things that aren't meant for human hearing (for example, geological sonar scans) might not need as many as 44,100 samples per second.

## Loading PCM audio from disk

Several audio file formats store PCM encoded audio directly - wav and aiff are examples.

Luckily, other developers have implemented great libraries that handle the complexities of parsing wav files for you. I recommend [node-wav], by [Andreas Gal](https://github.com/andreasgal). It's got a simple API, and uses the metadata at the start of the wav file to automatically choose the correct sample rate, bit depth, and number encoding. From the readme, here is a code example.

```js
let fs = require("fs");
let wav = require("node-wav");

let buffer = fs.readFileSync("file.wav");
let result = wav.decode(buffer);
console.log(result.sampleRate);
console.log(result.channelData); // array of Float32Arrays
```

The `result.channelData` variable contains a list of signals that you can use as standard Javascript Float32Arrays. The result object also exposes the sample rate, which you will likely need to know for many operations.

If you're using [Meyda] to analyze audio that you load in this way, you will need to make sure that the sample rate of the audio matches the sample rate that Meyda is set to use. Otherwise you'll end up with audio features that are incorrect, and based on a skewed frequency scale. You can either match the Meyda sample rate to the wav sample rate, or you can resample the audio to fit a standard sample rate (i.e. 44,100hz, or 48,000hz). Resampling audio is a complicated topic beyond the scope of this article, but if you have trouble finding information online, [let me know] and I may find time to write an article.

AIFF files also store PCM audio data, but differ from WAV files in that they have a different header format for storing metadata. [node-wav] doesn't support AIFF files, and I haven't found a package I would recommend to do so. If you need to analyze AIFF files, I would suggest using a utility like [ffmpeg] to transcode the audio to wav.

## What about non-PCM audio formats?

But what about audio file formats like [mp3], [ogg], and [flac]? The difference between these formats and wav is that the audio is compressed on disk. mp3 and ogg are what's called "lossy" compression - that means they change the actual sound in ways that are hopefully imperceptible to most listeners in order to get better compression. flac, meanwhile, is a format that implements lossless compression. This means that it encodes audio on disk in a more efficient format than storing each sample as a full integer or floating point number, but without modifying the audio itself.

## Encoding agnostic signal processing code

It's best to write signal processing code that works with one representation of audio, and reuse it by converting the audio - rather than having one implementation of your signal processing code for each audio encoding. We can achieve code reusability by converting all audio to a common format for signal processing, so that your code only has to think about one representation. Libraries that do this are called "codecs" which comes from "enCOding/DECoding". In order to support a particular file format in your program, you will need to make sure that you have the right codec. Luckily, you don't need to understand each audio format and implement a codec yourself - you can use packages to do this. So when you're writing your signal processing code, you should write code that works on raw signals, not encoded or compressed. In many cases, in Javascript, signals are represented as Float32Arrays - and unless you have specific requirements where this causes a limitation for you, I would recommend sticking to writing code that assumes signals are in Float32Arrays.

## Loading alternative encodings from disk

While there are some implementations of mp3 encoders in Javascript, I would actually recommend calling out to another technology to do the transcoding. [ffmpeg] is a long running open source project that excels in media encoding. It can translate between many different media encodings, and I'm confident that it covers a _huge_ portion of transcoding needs. In Node, you can call out to ffmpeg using the [child_process] API.

```js
import { exec } from "child_process";
import { mkdtemp } from "fs/promises";
import path from "path";
import os from "os";

// Create a temporary directory to store transcoded audio
const TEMP_DIR = await mkdtemp(path.join(os.tmpdir(), "transcoder-storage-"));

async function transcodeToWav(filename) {
  return new Promise((resolve, reject) => {
    let output_filename = `${path.join(TEMP_DIR, filename)}.wav`;
    // "shell out" to ffmpeg
    exec(
      `ffmpeg -i ${filename} ${output_filename}`,
      (error, stdout, stderr) => {
        if (error) {
          console.log("ERROR: ", error);
          reject(error);
        }
        resolve({ filename: output_filename, stdout, stderr });
      }
    );
  });
}

try {
  let result = await transcodeToWav("./164064__cclaretc__rooster.mp3");
  // result.filename is the new filename of the transcoded audio.
  // We can now use node-wav as described above to read the audio

  let buffer = fs.readFileSync("file.wav");
  let decodedAudio = wav.decode(buffer);
  console.log(decodedAudio.sampleRate);
  console.log(decodedAudio.channelData); // array of Float32Arrays
} catch {}
```

I'm using a modern version of Nodejs which allows imports, top level await in .mjs files, and exposes the `fs/promises` interface, but this code refactors back to older versions of node if you need.

One thing to bear in mind is that in order for this to work, you need to have a copy of ffmpeg on the system that you're running the code on. Luckily, there's a package for that - [ffmpeg-static] is a dependency that you can include in your project that installs a statically linked copy of ffmpeg. You can use it to ensure that ffmpeg is always available to your code. Check it out!

## But what about the web?

While in theory it might be possible to run ffmpeg through emscripten and run it in a web worker (I certainly assume someone has done this), it's not necessarily practical to try and use the same technique from node to transcode audio on the web. The good news is that the w3c has chartered a working group to focus on [web codecs]. While this is at the time of writing still in early stages, the working group is powering ahead on designing and proposing an API to enable media transcoding on the web, and hopefully that will become available to us in the near future.

## What did we learn?

In this blog post, I covered the basics of Pulse Code Modulation encoding, how to load wav files from disk, the difference between wav files and other audio encoding file formats, transcoding other file formats to wav for loading in node, and how transcoding might soon work outside of node, but on the web. I hope these explanations have been useful to you. If anything is unclear, or you have more questions, please [let me know] on Twitter! Thanks for reading.

[pulse code modulation]: https://en.wikipedia.org/wiki/Pulse-code_modulation
[node-wav]: https://www.npmjs.com/package/node-wav
[ffmpeg]: https://www.ffmpeg.org/
[mp3]: https://en.wikipedia.org/wiki/MP3
[ogg]: https://xiph.org/vorbis/
[flac]: https://xiph.org/flac/
[web codecs]: https://github.com/w3c/webcodecs
[let me know]: https://twitter.com/hughrawlinson
[child_process]: https://nodejs.org/dist/latest-v16.x/docs/api/child_process.html
[ffmpeg-static]: https://www.npmjs.com/package/ffmpeg-static
[meyda]: https://meyda.js.org/


Handling Pagination with Async Iterators

---
layout: post
title: Handling Pagination with Async Iterators
date: 2020-08-28T02:01:41.261Z
slug: handling-pagination-with-async-iterators
categories:
  - javascript
---

When you're interacting with a server from your frontend Javascript code, you might need to handle paging. Paging is a technique used by API designers to avoid enormous (and sometimes impossibly large) responses to requests when providing access to large collections of information to clients. Instead of returning every single item in a collection as a response to a request, an API might return the first 50 of the items in the collection, and a message to the client to say "this isn't all the items in the collection. If you want to get the next 50 items, here's how".

That's what the Spotify API does. When you need to get a list of albums by particularly prolific performers, you won't necessarily be able to get them all in one page, and will have to handle pagination to get all the albums.

It's possible to interact with pagination in an imperative way.

```javascript
let artistId = "6sFIWsNpZYqfjUpaCgueju";

async function loadAlbums(artistId, authToken) {
  let endpoint = `https://api.spotify.com/v1/artists/${artistId}/albums?limit=20&include_groups=album`;

  let albums = [];
  // We'll set endpoint to page.next when we receive it in the response.
  // When there is no more data, the API will set page.next to null, and we'll
  // escape this while loop.
  while (endpoint) {
    const response = await fetch(endpoint, {
      headers: {
        Authorization: `Bearer ${authToken}`,
      },
    });

    if (!response.ok) {
      throw new Error("Request failed");
    }

    const page = await response.json();

    albums = albums.concat(page.items);

    endpoint = page.next;
  }
  return albums;
}

for (let album of await loadAlbums(artistId, YOUR_OWN_AUTH_TOKEN)) {
  console.log(album.name);
}
```

This code works, but there are some problems with it.

The code that is consuming the data is mixed with the code that handles pagination.

You can extract the code that handles the pagination by converting the whole block into an async function. But since functions can only return data once, you're stuck until all the requests are finished before you can return albums and use it.

This is where async generators come in. Generators are functions that can `yield` multiple results, rather than just one. Asynchronous (async) generators are analogous to Promises that can resolve multiple times. They also provide syntactic sugar to make it easier to iterate over the yielded values - `for await ... of` syntax.

Async Iterators are one solution to this problem - [observables](https://rxjs-dev.firebaseapp.com/guide/observable) are another solution, but they haven't made it into the EcmaScript specification.

The following is some example code that demonstrates how to use a recursive async generator to yield each page of albums one by one until we're out of pages. You'll see how the code that consumes the albums uses the `for await ... of` syntax to access the results of the generator

```javascript
async function* pageThroughResource(endpoint, authToken) {
  async function* makeRequest(_endpoint) {
    const response = await fetch(_endpoint, {
      headers: {
        Authorization: `Bearer ${authToken}`,
      },
    });
    if (!response.ok) {
      throw new Error(await response.text());
    }

    const page = await response.json();

    yield page;

    if (page.next) {
      yield* makeRequest(page.next);
    }
  }

  yield* makeRequest(endpoint);
}

async function* loadAlbums(artistId, authToken) {
  const endpoint = `https://api.spotify.com/v1/artists/${artistId}/albums?limit=20&include_groups=album`;
  const result = pageThroughResource(endpoint, authToken);

  for await (const page of result) {
    for (let album of page.items) {
      yield album;
    }
  }
}

for await (const album of loadAlbums(
  "6sFIWsNpZYqfjUpaCgueju",
  YOUR_OWN_AUTH_TOKEN
)) {
  console.log(album.name);
}
```

In this example, the code that's responsible for making requests to the paginated external service is abstract - the behavior responsible for managing the pagination (the `pageThroughResource` function) doesn't know about _what_ it's paginating through. The logic that knows about loading albums (the `loadAlbums`) function is what handles the specific details of the API that we're calling. The only assumption that the `pageThroughResource` function makes is that the response object from the API returns a field called `next` which provides the URL of the next page of the resource listing. This means that you can re-use the `pageThroughResource` function on any API call you need to make that has the same pagination design.

The code achieves the separation of these two distinct behaviors by creating functions that return asynchronous iterators. `pageThroughResource` returns an asynchronous iterator, but also internally defines _another function_, `makeRequest`, that _also_ returns an asynchronous iterator. `pageThroughResource` uses the `yield *` syntax to yield to whatever `makeRequest`'s resulting async iterator returns. The code is organized this way so that `makeRequest` is able to call itself recursively. Inside makeRequest, first the JSON result of the response of the API call is yielded, and the user can use it immediately. After that, only if the response contains a `next` field, `makeRequest` will delegate control of the generator to another instance of itself, made to handle the next page. While that request is being made, the calling code already has access to the result of the first page. That means we don't have to wait until all the pages are loaded before we can start using the information we get from the API.

These specific functions make a few assumptions, including:

- the API you're calling will return JSON
- the JSON that your API returns will contain a field called `next`, which provides the next page of the resource listing for you to call

But you can use this pattern in your own code, tailored to however your API handles response types and pagination data. You could even use this pattern to page through a resource in a GraphQL API.

One specific drawback to point out: iterators in Javascript don't have the `map`, `reduce`, and `filter` algorithms that you might know from arrays - you'll have to use the `for await .. of` syntax to handle their output. Maybe one day we'll get that interface!

I hope this helps you keep your code nice and maintainable!


How to run ChucK 1.4.0.1 on Ubuntu 20.04

---
layout: post
title: How to run ChucK 1.4.0.1 on Ubuntu 20.04
date: 2020-05-24T17:41:24.432Z
slug: how-to-run-chuck-1-4-0-1-on-ubuntu-20-04
categories:
  - chuck
  - audio
---

While the `chuck` package for Ubuntu has been published in 2020 for Ubuntu 20.04, it's [a re-publish of version 1.2.0.8](https://launchpad.net/ubuntu/+source/chuck/1.2.0.8.dfsg-1.5build1). ChucK has since made it to version `1.4.0.1`. If you would like to use newer features like Unit Analyzers (UAnae), Dynamic Arrays, Chugins, [and much more](https://chuck.cs.princeton.edu/release/VERSIONS), you'll have to upgrade. I have in the past packaged ChucK into a debian package - but this time I've decided to build it from source. Here's how I did it:

## 1) Download ChucK.

You can get ChucK from the [official website](https://chuck.cs.princeton.edu/release/), which contains a snapshot of the code at the latest version, or from [the repository on GitHub](https://github.com/ccrma/chuck), where you'll get the very freshest patches.

## 2) Install the assorted dependencies

ChucK depends on some packages in Ubuntu. Here are the ones I had to install:

- `make`
- `gcc`
- `g++`
- `bison`
- `flex`
- `libasound2-dev`
- `libsndfile1-dev`

## 3) In the `src` directory of the downloaded package, run some commands

To compile and build ChucK, you can run

```markdown
make linux-pulse
```

`pulse` is a linux audio system that is available in Ubuntu 20.04. You could alternatively choose to use the `jack` or `alsa` audio drivers. I ran into an issue where ChucK wouldn't build with those drivers though, so I'd recommend sticking with `pulse`.

## 4) Run `sudo make install`

This'll copy the `chuck` binary you just built into your systemwide binaries directory.

Now you have access to the `chuck` tool on your command line!


Avoiding pitfalls when installing semantic-release for npm library packages

---
layout: post
title: Avoiding pitfalls when installing semantic-release for npm library packages
date: 2020-05-17T16:04:06.812Z
slug: avoiding-pitfalls-when-installing-semantic-release-for-npm-library-packages
categories:
  - javascript
---

I recently set up [semantic-release](https://semantic-release.gitbook.io/semantic-release/) on [an open source repo](https://meyda.js.org). I was sold on the idea of [unsentimental commits](https://blog.logrocket.com/unsentimental-versioning/) - the idea that it should be easy and quick to deploy changes to packages, and that you shouldn't worry about batching changes together to reduce the number of versions that get published. In the past, I've found myself not deploying breaking changes that I already have written just so that I can batch them with hypothetical breaking changes that I would like to make. That tended to mean that features and fixes end up delayed, which isn't in anyone's best interests. Unsentimental commits take that power out of my hands - releases are automated, and happen whenever commits are merged.

This post covers some pitfalls that I encountered in setting up my semantic-release tooling, that I want to help you avoid, so I wrote this blog post. Let's go!

This post won't cover setup instructions, so if you haven't yet installed semantic-release and given your CI system of choice an npm token, please follow [the instructions from the semantic-release team](https://semantic-release.gitbook.io/semantic-release/recipes/recipes/travis).

## Avoid merging commits with non-standard messages!

If you merge commits to master that don't follow the commit standard that semantic-release is expecting, semantic-release won't publish your package. If your commit contains a breaking change, but isn't following the standard, semantic-release won't publish your package. This means that you'll have an undeployed breaking change on your master branch, and if you subsequently release a minor or patch change, you could end up deploying the breaking change to a patch version and breaking your consumer's code.

You can avoid merging commits that don't follow the commit standard by installing [commitlint](https://github.com/conventional-changelog/commitlint), which will fail your PR builds when they contain invalid commit messages. You can also install [husky](https://github.com/typicode/husky) to have your commits linted as you commit, before you even push to your git remote.

[Commitlint setup instructions](https://github.com/conventional-changelog/commitlint) are available, as are [husky setup instructions](https://github.com/typicode/husky).

You should run commitlint in your Continuous Integration tool when a pull request is built, so that you can't merge Pull Requests that contain commits with non-standard messages. Check out commitlint's [CI Setup][commitlint-ci] documentation for instructions on how to set up commitlint in your CI. Since my project uses the [Travis CI service], I was able to use the [@commitlint/travis-cli] package - but you'll be able to run commitlint on Pull Request review builds on any CI provider.

If you have already merged a commit containing a non-standard commit message, don't worry, you can fix it by pushing an empty commit with a standard compliant commit message summarizing the changes that you've already merged. For example, if you have merged a breaking change to master, you might publish an empty commit with the following message.

```
 feat(node version): drop 8, add 13 and 14

This commit represents lots of commits to master that have not triggered a release, including lots
of dependency version bumps, adding support for node 13 and 14, and dropping support for node 8,
which is a breaking change. Therefore, this commit will trigger a major version bump and release.
This commit itself is empty as it represents previous commits and serves as a commit to be version
tagged.

BREAKING CHANGE: Drop support for node 8
```

semantic-release will pick up that commit and publish a new version. Just make sure that if you have pushed breaking changes to master that your empty commit marks a breaking change, or else you'll publish a breaking change to the same major version, which _could_ cause bugs in projects that consume your package. But if you have released a breaking change with a minor or patch version bump, don't worry, we'll cover that below!

## Avoid GitHub invalidating commit messages in squash-merges

When I made my empty commit that would have published a new major version of my package, I opened a pull request so that the CI pipeline would run and confirm that I hadn't broken any of my code, or pushed an invalid commit message (since commitlint runs on CI). The review build passed, and so I merged the pull request – but then the master build failed to publish because the commit message failed to lint! What happened?

When you merge a pull request on GitHub and choose the "Squash and Merge" option, GitHub will squash all the commits in the pull request into one and automatically name the new pull request. It will append the pull request number, something like `(#403)` to the commit message title. The [@commitlint/config-conventional](https://github.com/conventional-changelog/commitlint/blob/master/%40commitlint/config-conventional/index.js) package that implements the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) commit format standard for the [commitlint](https://github.com/conventional-changelog/commitlint) tool enforces a maximum line length of 100 on the commit header. GitHub's pull request number had pushed my commit message past the limit, causing it to fail on master after succeeding on the branch.

You could resolve this by avoiding squash merging, making sure to keep your commit messages very short, or manually checking each PR to see how much breathing room you have.

None of those options sound particularly appealing to me. They all depend on a person remembering what to do\*, rather than a tool to automatically do the right thing and free up the developer's mind. If this comes up again, I'm considering writing a quick GitHub Check to warn me that I need to either reword a commit or not squash it.

\* Yes, you can disable squash merging per repository on GitHub - but you have to remember to do that on each repository.

## Make sure Dependabot writes commit messages properly

I use [Dependabot](https://dependabot.com/) to automatically open pull requests on my repos whenever my dependencies publish new versions, so that I can make sure I'm always up to date on any bugfixes or security fixes. Dependabot writes commit messages that fit the standard if that's what it sees you have been doing, but if you're installing semantic release in an existing repository where you haven't always been using conventional commits, it might default to non-standard commit messages.

You can explicitly configure Dependabot to explicitly to follow conventional commits using the `commit_message` property of it's configuration, which is [documented on the dependabot website](https://dependabot.com/docs/config-file/#commit_message).

If you have PRs that dependabot opened before you configured it to use conventional commits that don't have correctly formatted commits, I haven't found a way to get it to recreate them with the correct message, so you may have to upgrade those dependencies manually.

## Make sure your CI won't be blocked from deploying to the registry

The semantic-release setup guide explains how to create an npm deploy key so that you're able to publish your package to the public registry. If you have two factor authentication set up on your npm account, _which you really should_, the npm CLI may prompt semantic-release for an OTP on your CI, which it won't have access to. To make sure semantic-release is able to publish, you can set your npm 2FA to auth-only mode. This will make npm only ask for an OTP when you're logging in, and not when you publish a package.

I'm not extremely comfortable with this, since if someone finds a way to get a copy of your deploy key, they'll be able to publish versions of your package without any additional information. To minimize this risk, make sure your CI provider is configured only to expose your `NPM_TOKEN` on master builds, and be vigilant in making sure that you don't merge a PR that steals your `NPM_TOKEN`.

If you know of a better way, please [let me know](https://twitter.com/intent/tweet?text=Hey%20%40hughrawlinson%2C%20I%20know%20a%20better%20way%20to%20handle%20NPM_TOKEN%20in%20CI!)!

## Do your best to make sure your contributors don't struggle with commit messages

One of my reservations about installing semantic-release in my repository was that it would degrade the developer experience for any existing or prospective contributors. If a developer who is unfamiliar with conventional commits were to get bothered by a linter about the commit message when they're used to writing messages a particular way, how likely are they to give up?

A combination of [commitizen](http://commitizen.github.io/cz-cli/) and [husky](https://github.com/typicode/husky) will give the developer feedback as they commit on their machine, so that they can get it right before they push to their fork of my repo.

Be ready to jump in on PRs to help contributors get it right!

Some things I would consider, but haven't done yet:

- Wrap commitizen and commitlint in scripts so that I can customize and improve upon the error messages
- Write a GitHub bot to automatically suggest standard compliant commit messages on open PRs that fail commitlint
- Information in the GitHub Pull Request Template explaining the commit message format required, and letting the contributor know that we're grateful for their commit regardless of whether the message lints, and that we're happy to write a new message ourselves when we review their code.

## Don't worry if you make a mistake!

I made a lot of mistakes when I was setting Meyda up with semantic-release, commitlint, and husky. All of the above issues are inspired by mistakes I made! Luckily, I was able to recover from all of them as described above. So don't worry!

If you've published a breaking change without bumping the major version number, that may cause breaks in the consumers of your library. Don't worry though - you can push a new commit containing a `BREAKING CHANGE:` section, triggering semantic-release to publish a new major version of your library, and then mark your erroneously published bad version as deprecated on the registry. The npm command to deprecate a package version is documented [on the npm website](https://docs.npmjs.com/cli/deprecate).

---

Hope those tips help with your [semantic-release](https://semantic-release.gitbook.io/semantic-release/) setup! There are quite a lot of packages involved in this setup, here's a list:

- [semantic-release](https://semantic-release.gitbook.io/semantic-release/)
- [commitlint](https://github.com/conventional-changelog/commitlint)
- [@commitlint/config-conventional](https://github.com/conventional-changelog/commitlint/blob/master/%40commitlint/config-conventional/index.js)
- [husky](https://github.com/typicode/husky)
- [commitizen](http://commitizen.github.io/cz-cli/)

If you found this blog post useful, please [let me know on Twitter](https://twitter.com/hughrawlinson)!

[commitlint-ci]: https://commitlint.js.org/#/guides-ci-setup
[@commitlint/travis-cli]: https://www.npmjs.com/package/@commitlint/travis-cli
[travis ci service]: https://travis-ci.org/


Updating Javascript Dependencies

---
layout: post
title: "Updating Javascript Dependencies"
date: 2019-05-25T19:00:00.000Z
slug: updating-js-packages
---

_Update: You don't need to think about manually managing dependencies. Automate it with [Dependabot](https://dependabot.com/)._

If you have a Javascript package, one thing you may need to do on a regular
basis is to keep your packages up to date. If you have a lot of packages, this
can be difficult. There are a lot of things to manage, and you have limited
time. You may need to prioritize which kinds of updates are most important.
But what how might you categorize updates for prioritization? And once you
have, how do you perform the updates in an efficient manner?

In this post, I'll explain how I approach this issue both at work, and in my
open source software.

For me, I find it helpful to split upgrades into two categores: upgrades that
address security vulnerabilities, and upgrades that address deprecations, pull
in new useful features, or are just for my own sense of "being up to date".
There are several reasons that these are my two categories, but none are more
compelling than that they map to two useful npm/yarn commands: `audit` and
`outdated`. Both of the major package manager CLIs for Javascript provide
security auditing of your dependencies, and checking if your dependencies are
out of date. Since my categories are based on these two commands, they work as
great tools to help me address both of these concerns independently.

## Security Audit

`npm audit` and `yarn audit` are commands that can be run inside any package
directory. They function by finding all the versions of all the packages that
your project uses, and checking with a server that stores security advisories
(descriptions of vulnerabilites, their severities, and which packages they
affect) to see if any are vulnerable. They will then show you a list of
publicly disclosed vulnerabilities of any packages you depend on (or any
packages that your dependencies depend on, all the way down), if any
vulnerabilites exist. Both also have flags to let you automatically fix some of
your vulnerabilities by having the package manager upgrade the non-breaking
changes for you.

Audits are run automatically when you install run `npm install` or `yarn`, so
that you're aware of any vulnerabilities as soon as possible and can push a fix
straight away. It's important to pay attention to the output of those commands,
so that you don't miss a critical dependency.

Another great thing about these features in the npm and yarn registry clients
is that they show the severity of the vulnerabilities in your dependencies. You
can use this feature to define your own process such that if a critical
vulnerability pops up you deal with it immediately, but you can batch your low
severity vulnerabilities together so that you don't find yourself constantly
fixing low impact issues.

One critical hole in these commands is that they require a developer to
manually run an installation, or an audit, to be seen. If you have a project
that lays dormant because it's stable, and no feature work has been prioritized
on it this quarter, you might not happen to run an audit for quite a while. You
can address this issue with interval builds. Interval builds are scheduled jobs
on your continuous integration/continuous delivery (CI/CD) infrastructure that
build your project on a regular basis - hourly, daily, weekly, monthly, etc. If
you configure your CI/CD to alert you when npm or yarn find a critical or high
priority vulnerability in your code, and schedule regular builds to fit your
schedule, you can always be on top of emergent vulnerabilities in your
dependencies. Both [Travis CI][travis-scheduled-builds] and
[Circle CI][circle-scheduled-builds] support scheduling interval builds, so
make sure to investigate whether this is an appropriate solution for you and
your team.

GitHub also provides security scanning for your projects. You can turn on
security scanning in the settings tab of each of your projects. npm/yarn and
GitHub seem to use different data sources for security advisories, and it's
always advisable to get as much advice as possible!

It's important to keep on top of security vulnerabilities in your dependencies,
but remember that this is not _all_ you could and possibly should do to keep
your code and your users safe and secure. Dependency auditing does not find
security vulnerabilities in your code itself, so you should review your process
for code security. Some suggestions include:

- Making sure to keep security in your code review checklist
- Static analysis solutions, like [WhiteSource], or [Snyk] may help you
  automatically find vulnerabilities in your own code.
- If you're lucky enough to work at a company with a security team, you may be
  able to view their security resources, access training, request code reviews,
  or ask questions when you're unsure.

## Outdated Packages

The second category of updates does not involve security. There are after all
some other reasons to want to keep your packages up to date. They include:

- Wanting to use a new feature in a library that you depend on.
- Pulling in a bug fix.
- Complying with a company policy around dependency age.
- Upgrading from a deprecated or unsupported version of a package.
- Appeasing your own general sense of up-to-date-edness.
- Any other reason you can think of!

Both npm and yarn provide commands to list all outdated dependencies in your
project: `npm outdated` and `yarn outdated`. These commands will show you
whether the change was a major, minor or patch change using semantic
versioning, so that you can know whether to expect an update to break your
build. You can then go through one by one updating each of these dependencies,
running your test suite to confirm that the changes didn't break anything, and
committing the change. I'd recommend running this every couple of days while
you're working on a specific codebase anyway. This way, you're always working
with reasonably up-to-date copies of your dependencies. If it takes me 30
minutes to address a breaking change in a dependency, I will save my commits in
a branch and open a ticket to address later, just to avoid having my day-to-day
work be derailed by dependency fixes.

That can be a tedious process, particularly if you have accrued responsibility
for many projects with lots of dependencies. Luckily, there is a service called
[Greenkeeper], that when enabled on your repositories will continuously monitor
releases on npm and automatically open a pull request on your project to update
the version of the dependency that you need. If the build passes, Greenkeeper
can even automatically merge the change, so that you don't even have to think
about your dependency versions until Greenkeeper encounters a failing build. In
that case, you will already have a branch on which to start addressing the
issue.

## What does my ideal process look like?

If you have access to dependency automation tools like Greenkeeper, you can
forget about manually managing dependencies, and focus on shipping features. If
a dependency change breaks your build, your tooling will let you know, and you
can address it as part of your planned work.

If for some reason you do not have access to a dependency automation tool,
here's an alternative manual approach that should reduce the amount of work you
have to do.

If I were setting up an ideal system for tracking security upgrades and package
updates in general, I'd have a daily CI run check for security vulnerabilities
before I get in to work. If there are any high priority or critical
vulnerabilities, I'd address them that day, but it can be ok to leave low or
medium priority changes for a while.

For the outdated but not vulnerable packages, I'd have that job run about
monthly, just to keep things up to date, and update the things that I need to
update for the new features as I go. I would distribute the runs for each
project around the month, so I don't end up having to spend an entire day each
month running upgrades for all the projects.

---

Hope that helps!

[snyk]: https://snyk.io/
[whitesource]: https://www.whitesourcesoftware.com/
[circle-scheduled-builds]: https://support.circleci.com/hc/en-us/articles/115015481128-Scheduling-jobs-cron-for-builds-
[travis-scheduled-builds]: https://docs.travis-ci.com/user/cron-jobs/
[greenkeeper]: https://greenkeeper.io/


How I packaged my JS library

---
layout: post
title: "How I packaged my JS library"
date: 2019-05-12T19:00:00.000Z
slug: js-modules
categories:
  - library-maintenance
---

When I have spare time, I work on an open source audio feature extraction
library in Javascript called [Meyda]. Over the years, I've had several occasions
of getting really confused about how JS modules work, and how I should package
the library on npm. Today is one such occasion. But this time, I'm writing down
what I'm learning about JS packages so that next time, I will be able to quickly
get through it!

## What I need to do?

I need to ship Meyda to npm so that people can use it in all their lovely
projects. However they want to use the library, I want it to be possible for
them. Additionally, Meyda is composed of lots of small js files, each for its
own audio feature. It would be pretty unfortunate if someone wanted to use Meyda
to run only one feature extractor, but in order to do so they had to bundle all
the feature extractors. So I'd like to enable tree shaking.

There are several different situations people might want to use Meyda.

1. As a library in nodejs
1. As a library in a fancy web app
1. As a global variable in their less fancy web app, included in a script tag

## The required reading

Each of those have different requirements. To solve those requirements, we'll
create _the matrix of doom_.

There are two major axes of decisions to make here. The first axis on the matrix
of doom is the module type. I'm only going to care about two types of modules,
the ones we need to solve this predicament. CommonJS is the module type when
your code does `module.exports = myAwesomeLibrary`, and users of your code do
`const myAwesomeLibrary = require('my-awesome-library')`. ES2015 modules are the
ones where your code does `export default myAwesomeLibrary`, and users of your
code do `import myAwesomeLibrary from 'my-awesome-library'`. Users are pretty
flexible. In ES2015, you can import [CommonJS modules with
import]babel-import-commonjs+ (if your runtime doesn't support it, you can babel
your code, and you will be able to import a cjs module) or with require. In
commonjs, you can import other CommonJS imports, but you cannot import ES2015
imports.

Adjacent to types of modules is the whole "uuhhhh, back in the 90s js didn't
have a module system so we just run a script in the same runtime as we'll run
our code and it'll just assign a global variable and we'll pick it up in our
code and it'll all be fine, right?" style of modules. That's not really a module
system, but I'll stick it in the module axis on the matrix of doom because that
seems like a reasonable place to put it.

The second major axis of decisions to make on the matrix of doom is whether or
not to bundle your code. Bundling will take all your code, and all of your
dependencies, and bundle them into one big js file. This can be advantageous if
your runtime isn't able to import or require modules, like in old browsers (new
browsers can handle modules now but old ones can't, so this is still useful, or
if you don't want your web app to be making loads of requests to recursively
download all your dependencies one by one.

## The application of the required reading to the goals

_web scale solutions hat on_

Lets have a look at the matrix of doom

<table>
  <thead>
    <tr>
      <td></td>
      <td>CommonJS</td>
      <td>ES2015</td>
      <td>Global</td>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Bundled</td>
      <td></td>
      <td></td>
      <td></td>
    </tr>
    <tr>
      <td>Not Bundled</td>
      <td></td>
      <td></td>
      <td></td>
    </tr>
  </tbody>
</table>

Now we need to find places in the matrix of doom that would solve for each of
our 3 use cases.

First off, old websites that need a global variable. These should be bundled,
because if a website is including a library with a script file, they're probably
not supporting import infrastructure. Lets mark off a spot in the matrix of
doom.

<table>
  <thead>
    <tr>
      <td></td>
      <td>CommonJS</td>
      <td>ES2015</td>
      <td>Global</td>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Bundled</td>
      <td></td>
      <td></td>
      <td></td>
    </tr>
    <tr>
      <td>Not Bundled</td>
      <td></td>
      <td></td>
      <td>Classic websites</td>
    </tr>
  </tbody>
</table>

Next up, lets do Node. Node seems to be able to handle unbundled modules. They
also seem to support both CommonJS files, and ES2015 files (with a `.mjs`
extension). In order to start adopting new features, lets do both.

<table>
  <thead>
    <tr>
      <td></td>
      <td>CommonJS</td>
      <td>ES2015</td>
      <td>Global</td>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Bundled</td>
      <td></td>
      <td></td>
      <td></td>
    </tr>
    <tr>
      <td>Not Bundled</td>
      <td>Node</td>
      <td>Node <pre>`.mjs`</pre></td>
      <td>Classic websites</td>
    </tr>
  </tbody>
</table>

Finally, modern websites. Lets take for example a create-react-app site. In
create-react-app sites, you can import or require libraries that you install
with npm, and CRA will do some compiling and bundling and make it all work for
you. If you consume ES2015 libraries, CRA/webpack will tree-shake your imports,
so that you only have the code that you need making it into your bundle. Nice!
Lets add it to the matrix of doom.

<table>
  <thead>
    <tr>
      <td></td>
      <td>CommonJS</td>
      <td>ES2015</td>
      <td>Global</td>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Bundled</td>
      <td></td>
      <td></td>
      <td></td>
    </tr>
    <tr>
      <td>Not Bundled</td>
      <td>Node</td>
      <td>Node <pre>`.mjs`</pre>, CRA websites</td>
      <td>Classic websites</td>
    </tr>
  </tbody>
</table>

## ok so the solution was....

I can use a tool like rollup to do the bundling and global export for Classic
websites, so that's that one checked off. Classic browsers don't really care
about your package.json, so you don't need to specify that export anywhere. You
may wish to document an unpkg link to your bundled minified code, like
`https://unpkg.com/meyda@4.2.0/dist/web/meyda.min.js`.

Meyda is written in ES2015 modules. So I'll need to use a compiler of some kind,
probably babel, to convert it to unbundled CommonJS. That checks off CommonJS
Node. I'll then specify the 'main' property of my package.json to point at the
unbundled CommonJS code.

Meyda's ES2015 modules all use `.js` extensions, so I can't export those
directly, I'll need to first copy them to the build directory with `.mjs`
extensions. Frontends can use `.mjs` files as if they're Javascript files, so
that's good. It seems like I can point to an unbundled copy of Meyda using
ES2015 modules with `.mjs` extensions using the `module` field of package.json,
so this course of action should work without breaking imports in node (node
can't import ES2015 modules unless they're in files ending in `.mjs`.

According to [this blog post][package-json-main-module] I found about main,
jsnext:main and module in package.json, I should export my CommonJS unbundled
copy of Meyda in the `main` field of package.json, and I should export the
ES2015 files (as `.mjs` files (I added that part)) in both the `js:next` and
`module` fields of package.json.

So now that I know what to do, I'll go do it. Want to know how it worked out?
[Tweet me]

## Bonus quest

I was working on a hobby website to help me [organize my GitHub
repos][github-repo-organizer], and needed to import the `[@octokit/graphql]`
dependency so that I could make queries to GitHub's graphql API. I installed the
dependency into my create-react-app based site, and went to import it. When I
ran my site, inside a dependency of the graphql library, I got an error:
`isPlainObject is not a function`. I set a breakpoint and went into the code,
and realized that `isPlainObject` was in fact not a function, it was an object
with a single property, `default`, which _was_ my function. That's what you get
when you use require to load an ES2015 export using webpack!

What had happened was that the subdependency, `is-plain-object`, had recently
changed such that it's exporting both an ES2015 module, and a CommonJS module.
Webpack knew that _my_ code was using ES2015 modules, and this package was using
ES2015 modules, so it imported the ES2015 module from that library.
Unfortunately, the library that was importing `is-plain-object` was expecting a
CommonJS module, so when it received an object with a property called default
that was the function it was looking for, rather than just the function it was
looking for, it threw an error. The solution in this case was to write a PR to
the intermediate library that made it explicitly import the CommonJS module that
`is-plain-object` exported.

[meyda]: https://meyda.js.org
[babel-import-commonjs]: http://2ality.com/2015/12/babel-commonjs.html#how-babel-imports-commonjs-module
[github-repo-organizer]: https://github.com/hughrawlinson/github-repo-organizer
[package-json-main-module]: https://shuheikagawa.com/blog/2017/01/05/main-jsnext-main-and-module/
[tweet me]: https://www.twitter.com/hughrawlinson
[@octokit/graphql]: https://github.com/octokit/graphql.js


Control Spotify Connect with your Mac's Media Keys

---
layout: post
title: "Control Spotify Connect with your Mac's Media Keys"
date: 2018-05-15T21:22:03.000+02
slug: control-spotify-connect-with-your-mac-s-media-keys
---

Hi! I use the [Spotify Web Player](https://open.spotify.com/) to listen to music all the time. I also use Spotify Speakers like the Google Home, and embedded web players built on the [Spotify Web Playback SDK](https://beta.developer.spotify.com/documentation/web-playback-sdk/) all the time. I found I was often looking for the right play/pause/next/previous control (or keeping a Spotify client open all the time) when working at my computer to control the music. Then I found the [Simple HotKey Daemon](https://github.com/koekeishiya/skhd), which lets me bind scripts to keys on my Mac, and decided to use it to hook my media keys to Spotify Connect, using the Spotify Connect Web APIs. And you can too!

## Step One!

Register yourself a client id in the [Spotify for Developers Dashboard](https://beta.developer.spotify.com/dashboard/applications). Make a note of the `client id` and `client secret`, and insert them into a JSON file at `~/.config/spotify_client_creds.json` like this:

```
{
  "id":"your client id",
  "secret":"your client secret"
}
```

## Step Two!

Install the `spotify-control` CLI. This is how SKHD will trigger Spotify Connect actions.

`$ npm install --global @hughrawlinson/spotify-control`

## Step Three!

Log in to the `spotify-control` cli:

`$ spotify-control login`

This will log the cli into your Spotify account.

## Step Four!

Configure shkd by pasting the following lines into your `~/.shkdrc` file:

```
play : spotify-control toggle
fast : spotify-control next
rewind : spotify-control previous
```

Then you should install shkd - you can use homebrew:

```
$ brew install koekeishiya/formulae/skhd
$ brew services start skhd
```

And you're all set!

Tweet me if you have any issues - @hughrawlinson


Audio Feature Based Synthesis Part 1: The Building Blocks

---
layout: post
title: "Audio Feature Based Synthesis Part 1: The Building Blocks"
date: 2018-02-17T11:43:24.000Z
slug: audio-feature-based-synthesis-part-one
---

_This is the first part of a series of tutorials involving musical synthesisers, machine learning, and the Web Audio API. Basic prior understanding of sound synthesis is required - otherwise this would be a book! I recommend that you read the section on Subtractive Synthesis in [this great article][sound-synthesis-basics]. Over the course of a few posts, we'll build up to an awesome, feature-rich, musical synthesizer._

In this tutorial, we'll build a basic subtractive synthesizer using Javascript, and the Web Audio API. One very high level summary of subtractive synthesis is that any sound that you hear is composed of lots and lots of discernible, small components, and subtractive synthesis is about making interesting sounds by starting with a very rich sound, and systematically removing (or "subtracting") some components, until you're left with a particularly aesthetically pleasing sound. This tutorial will deal with building the minimum viable subtractive synthesiser. We'll use the two most basic building blocks of modular synthesizers - an "oscillator" and a "filter" - to piece together the simplest synthesizer that can be reasonably described as "subtractive". In part 2, we'll build some basic infrastructure in order to actually _play_ the synthesizer, as this will be useful in later tutorials.

## Sound "components"

It's useful to have a little bit of understanding of exactly how sounds break down into components - what those components are, and what their properties are. Any sound is composed of some set of sine waves each with their own frequency, loudness, and phase, summed together. The particularly loud sine waves contribute more to the sound than the quieter ones, and the timbre of a sound depends on whether lower or higher sine waves are loudest, and what the relationships are between each of the waves. Usually, the 'frequency' part of the sine wave is put on an axis, with a chart of some kind representing the loudness and phase parts of the sound. We call this representation of a sound the "frequency domain", because the primary axis is on frequency. Here's an example of a frequency domain graph of a signal courtesy of [Guzik Technical Enterprises][guzik]:

![A plot showing showing the plot of a signal across an x-axis representing frequency measured in hertz](./digital-frequency-domain-analysis.png)

## Synthesizer architecture

### Oscillator

In subtractive synthesis, we start with a rich signal. In this case, we'll use an "oscillator" to generate that signal. Oscillators are devices (or computer programs) designed to generate signals, usually based on certain rules. Oscillators can generate a variety of different kinds of signals, each with their own special characteristics. We'll use a ["sawtooth" wave][wiki-sawtooth] for this tutorial, but you can read more about the properties of each of the basic waveforms [on wikipedia][wiki-waveform]. We're going to use sawtooth waveforms because they are particularly harmonically rich - that means that sounds produced by sawtooth oscillators have particularly many sine waves in them, without sounding too overwhelming like noise.

Like the vast majority of synthesizer building blocks (we'll call them nodes going forward for reasons that will become apparent), the oscillator takes certain parameters. In typical subtractive synthesizers, the oscillator is the node that defines the pitch of the output sound. It does this by looking at its "frequency" parameter. Another typical oscillator parameter is the waveform type. In this case, we'll choose "sawtooth", but we'll explore other waveform types later on in this tutorial series.

Within synthesizers, nodes usually either create sound from scratch (as is the case for an Oscillator), or recieve sound from other components and modify it, which brings us to the...

### Filter

The filter is a synthesizer component that accepts input sound, processes it to remove some components of the sound based on parameters, and outputs the modified version of the sound. Filters are an enormous topic for another post, but for the purposes of this post, you shouldn't worry about it because we'll be us. If you want to read more, about filters, check out [this great guide][filter-guide] on the topic. We'll use a filter called the "low pass filter". As the name suggests, this filter takes input audio, and allows the parts of the frequency that are _lower_ than a specified frequency to _pass_ through, and attenuates (quietens) the high parts of the signal. The effect that a low pass filter has on a sound is to make the sound less "bright", and more "bassy". You'll hear it soon!

## How does this work in code?

We'll use the Web Audio API to build our synthesizer! Don't worry if you're not an all-star web developer - for our tutorial, the Javascript syntax we'll use will be pretty similar to other C family languages you might have used in the past, like C, C++, Java, Ruby, Python, or similar. You should note that the Web Audio API is available in most modern browsers, but not by default in node.js.

We start by creating an "audio context". You can think of this as a box to put all of our synthesizer nodes in. Audio Contexts are created like this:

```javascript
const context = new AudioContext();
```

Great! We've got our context. Next up, we'll create an oscillator:

```javascript
const oscillator = context.createOscillator();
```

The `oscillator` has some parameters, including `Frequency` and `type` to let you set the pitch and waveform of your oscillator, but the defaults are good, so lets not meddle with them.

Nearly there already in only 2 lines of code! Only one more node to create - the filter.

```javascript
const filter = context.createBiquad();
```

Biquad is a fancy name for the implementation of the filter in web audio, but this line creates a standard audio filter node with all the defaults.

Finally, we connect our nodes together, and plug them into the output.

```javascript
oscillator.connect(filter);
filter.connect(context.destination);
```

The `context.destination` variable is the audio output of the web audio context. You can think of it like a plug socket for your computer's speakers.

Finally, we'll need to start the synthesizer. We'll do it in response to a button click, so that your web page synthesizer doesn't do autoplay audio!

```javascript
document.querySelector("button#start").addEventListener("click", () => {
  oscillator.start();
});
```

Here's an example synthesizer running in CodePen. It also includes a stop button, so that you can stop the beeping if you want!

```javascript
document.querySelector("button#stop").addEventListener("click", () => {
  oscillator.stop();
});
```

<p
  className="codepen"
  data-height="265"
  data-theme-id="default"
  data-default-tab="js,result"
  data-user="hughrawlinson"
  data-slug-hash="aqEEoG"
  style={{
    height: "265px",
    boxSizing: "border-box",
    display: "flex",
    alignItems: "center",
    justifyContent: "center",
    border: "2px solid",
    margin: "1em 0",
    padding: "1em",
  }}
  data-pen-title="The Simplest Synth"
>
  <span>
    See the Pen{" "}
    <a href="https://codepen.io/hughrawlinson/pen/aqEEoG">The Simplest Synth</a>{" "}
    by Hugh Rawlinson (
    <a href="https://codepen.io/hughrawlinson">@hughrawlinson</a>) on{" "}
    <a href="https://codepen.io">CodePen</a>.
  </span>
</p>
<script async src="https://static.codepen.io/assets/embed/ei.js"></script>

## Summary, and what's next?

CONGRATS ✨⚡️💻🎛 YOU BUILT A SYNTHESIZER! You can use your newfound skills to annoy people when they least expect it. Pro tip: setting the oscillator's frequency to around 12000 hz really annoys young people, and older people can't hear a thing!

```javascript
oscillator.frequency.value = 12000;
```

Next time, we'll learn how to send notes to your synthesizer, so that you can play it with your keyboard (of the musical, mechanical, or inbuilt varieties!)

<a
  href="https://twitter.com/share?ref_src=twsrc%5Etfw"
  className="twitter-share-button"
  data-text="Learn the indispensable life skill of building a synthesizer in a web page!"
  data-url="https://www.millenniumfal.com/2018/02/17/audio-feature-based-synthesis-part-one.html"
  data-via="hughrawlinson"
  data-related="hughrawlinson"
  data-lang="en"
  data-show-count="false"
>
  Tweet
</a>
<script
  async
  src="https://platform.twitter.com/widgets.js"
  charSet="utf-8"
></script>

[sound-synthesis-basics]: https://theproaudiofiles.com/sound-synthesis-basics/
[wiki-waveform]: https://en.wikipedia.org/wiki/Waveform
[wiki-sawtooth]: https://en.wikipedia.org/wiki/Sawtooth_wave
[guzik]: http://www.guzik.com/product/digital-frequency-domain-analysis-for-d5000-signal-analyzer/
[filter-guide]: https://theproaudiofiles.com/audio-pass-filters/


Music Discovery Playlist March 2016

---
layout: post
title: "Music Discovery Playlist March 2016"
date: 2016-03-01T11:03:00.000Z
slug: music-discovery-playlist-march-sixteen
---

I have started making playlists to collect all the music I discover each month. This is month two. I'm making an effort to broaden what I listen to, so in the end I should be able to do some fancy analysis of my discovery patterns.

<iframe src="https://embed.spotify.com/?uri=spotify%3Auser%3A11995522%3Aplaylist%3A7JduTg12T6WRELcsKUW6CM" width="100%" height="380" frameborder="0" allowtransparency="true"></iframe>


Music Discovery Playlist February 2016

---
layout: post
title: "Music Discovery Playlist February 2016"
date: 2016-02-11T14:49:00.000Z
slug: music-discovery-playlist-february-sixteen
---

I have started making playlists to collect all the music I discover each month. I'm making an effort to broaden what I listen to, so in the end I should be able to do some fancy analysis of my discovery patterns.

In the mean time, have a listen.

<iframe src="https://embed.spotify.com/?uri=spotify%3Auser%3A11995522%3Aplaylist%3A6cOcrXtjaHw0Rn5aG4MVYe" width="100%" height="380" frameborder="0" allowtransparency="true"></iframe>


Goldsmiths Physical Computing: Week 10 | Completion

---
title: "Goldsmiths Physical Computing: Week 10 | Completion"
date: 2015-03-25T12:55:26.000Z
slug: week-10
categories:
  - "Goldsmiths Physical Computing"
---

import { YouTubeEmbed } from 'features/YouTubeEmbed';

_Update: Images from this post have been lost_

Last week, we got all the electronics and software finished. This week, we had a few finishing touches to complete, namely:

- Case
  - 3d Modelling
  - Printing
  - Assessing issues
  - Addressing issues
- Software
  - Development
  - Identifying problems and fixing them

## Case

For the case, we started development on the online 3d cad software called [TinkerCad](https://www.tinkercad.com/). This involved measuring out the existing board we had soldered together and designing and iterating on a model. We researched guitar neck shapes to make the most ergonomic design we could, while bearing in mind that the fact that the Buttery Biscuit Bass (as we're calling it) is a relative of, as opposed to a clone of, a guitar, and thus will have some differences in design. The neck is significantly wider than the average bass guitar, a design choice we made to improve the feel of the object, to give it more substance. This maintains the aesthetic we were persuing in our choice of buttons, sharing the strong tactile attributes of the buttons. Initially we had decided to slide the PCB into its housing, however this posed a major problem with additional wiring inside the case. We decided to go for a "sandwich" design because not only did it allow for the wonderful addition of elastic bands, which genuinely add majorly to the design of the project, but it allowed access to the internals of the project easily without requiring us to unglue the top of the case.

There was a video of the 3d print happening, but it's gone now.

The first print iteration identified some problems to us - the wall through which the USB connection to the Arduino Micro was too thick. We remodelled, and were prepared to start the new print, but we discovered that serendipitously the conversion from model to printer instructions with the supplied infill settings meant that internally the wall was perfect to cut away at to be left with a solid holder for the USB cables. We decided to keep our printed models, and adjusted the design accordingly.

### On the elastic bands...

The elastic bands are an integral part of the design of our product. While they hold the case together, they provide excellent grip on the back of the case which really helps the user when they're reaching for a far away fret (that is to say, a button), and they also provide a nice cushion for when the user presses a button at the side - rather than have a hole there, or some hard ABS plastic, the user has a nice soft flexible base that they can use to mentally identify when they're slightly off on a button, which is very very handy when learning to play the instrument without looking at it.

![In the wild that is my desk](IMG_20200517_130529.jpg)

## Software

The software was quite straightforward to implement - we essentially had to map voltages recieved on analog inputs to MIDI pitches. We achieved this through binning the voltages into specific notes, and when a voltage was between the thresholds of a bin, that constitutes a note. We then output MIDI noteon and noteoff signals at each change in voltage on a pin. We realised that the while the Arduino Micro (which we had to use because of our form factor) provided MIDI host functionality, we couldn't get it to enumerate as a MIDI device, and would therefore require a small converter daemon to be running on a computer to bridge the data from the Arduino across to a DAW. The fact that we were already writing software for a computer meant that we had quite a lot of scope to do extra things.

We used Max/MSP to develop the software, because it facilitated rapid addition of extra features. We initially built the synthesizer insde of Max/MSP itself rather than bridging the MIDI signal. A bespoke synthesiser specifically for this project was excellent for us as we were able to see what was happening internally and debug more easily. Once we had the notes working, I was able to write a little additional preprocessor for the MIDI output stream. I was able to hook up the Max patch to the Propellerhead Reason DAW and jam along with a backing track. Here's some video:

<YouTubeEmbed youtubeVideoId="oPz1a8Pzcxg"/>

I look forward to presenting this project in the morning - I'm very proud of how it turned out. It's excellently playable, very intuitive, and hackable which was one of my personal goals.

-- Hugh Rawlinson


Goldsmiths Physical Computing: Week 9 | It works!

---
layout: post
title: "Goldsmiths Physical Computing: Week 9 | It works!"
date: 2015-03-19T12:55:26.000Z
slug: week-9
categories:
  - "Goldsmiths Physical Computing"
---

_Update: Images from this post have been lost_

All the components have arrived. We had enough components to create two devices. Using our schematic, we started soldering our circuit to the matrix boards. After days of soldering and resoldering we finally created a working circuit. After finishing the first one, we learned from our mistakes and tweeked our design slightly. Rather than soldering a whole line through the middle of the board (to act as ground) we soldered a piece of wire. This saved a lot of time as it only had to be soldered at certain points for the circuit to work. After both raw devices were finished we started to sketch out case ideas and started on the code. The below images show our progress.

Raw prototype front:

Raw prototype back (note: this soldering has been completely redone. The photo illustrates out first attempt):

Hugh testing it out:

-Marcin Dyszynski


Goldsmiths Physical Computing: Week 8 | Testing

---
layout: post
title: "Goldsmiths Physical Computing: Week 8 | Testing"
date: 2015-03-12T16:09:56.000Z
slug: week-8
categories:
  - "Goldsmiths Physical Computing"
---

_Update: Images from this post have been lost_

The components have arrived! At least some of them. We have all the correct resistors and enough buttons to make 2 guitars. Unfortunately there was a mix up with the matrix boards and they weren't added to our order. We continued work and made a second order consisting of matrix boards, button caps and arudino micros which we decided will be more convenient as they can work as USB hosts. We decided it was time to record some voltages! We recreated last weeks circuit design (with our new components) on a breadboard and recorded the voltages when each button was pressed. We then calculated the voltages when 2 buttons were pressed at once. We noticed very minimal overlap and more or less every combination had a unique voltage. These values were not final though. When we connect up 4 rows of buttons (corresponding to 4 strings) all leeching power from the same 5V output, the voltages will be lower. Unfortunately we could not test this until the larger matrix boards arrive. Either way, it was time to write some code! We started protoyping the initial architecture for the arduino sketch. Everything seems in order and as soon as we receive our matrix boards we can start crafting a prototype of our invention.

--Marcin Dyszynski


Goldsmiths Physical Computing: Week 7 | Prototypin'

---
layout: post
title: "Goldsmiths Physical Computing: Week 7 | Prototypin'"
date: 2015-03-05T10:04:56.000Z
slug: week-7
categories:
  - "Goldsmiths Physical Computing"
---

This week, we developed our idea into a basic prototype. We started by using the Arduino Uno and a breadboard to prototype a single-button version of our bass fretboard. One button represents a single string at a particular fret - i.e. it might be mapped to the 5th fret on the 4th string. We devised a system to use voltage division in a circuit to use multiple buttons on one analog input of the Arduino. Each button activates a voltage divider between a high resistor (constant between all the buttons) and a low resistor (specific to the particular button). This allows us to compare specific voltages on the analog input to a lookup table to determine the index of the button being pressed. One disadvantage to this approach is that when the user may accidentally press multiple buttons, we don't just get the highest button (or the highest pitched fret of the string), we get a combination of the two voltages. Luckily this is easily remedied, we can build in the voltages from multiple buttons pressed simultaneously in the lookup table and use the simple rules that govern how guitar fretboards work to select the appropriate fret and relay this information to our midi send channel.

Once we established an effective methodology for implementing our project, we were able to determine which parts were necessary, and we placed an order to RS-Components for the appropriate number of buttons and resistors. Luckily, another class member had a selection of buttons from which we selected one with appropriate action and activation force for the project.

We began to think about what enclosures were appropriate, and settled on a 3d printed approximation of a guitar neck that we could slot the matrix board with the built circuit and button interface into.

\- Hugh Rawlinson


Goldsmiths Physical Computing: Week 6 | Lets get started

---
layout: post
title: "Goldsmiths Physical Computing: Week 6 | Lets get started"
date: 2015-02-26T12:45:21.000Z
slug: week-6
categories:
  - "Goldsmiths Physical Computing"
---

We have decided on a fantastic project idea inspired by the monome midi controller. We will create a midi controller that imitates a guitar fretboard. The idea is simple: set up miltiple rows of buttons, each row imitating a guitar string and each button imitating a spot on the fretboard. This way, the user can play guitar notes by pressing buttons. We have analysed potential challenges and establised the following points:

- Figure out a way to differenciate between different buttons pressed on each analogue input.
- Consider the posibility of users accidentally pressing multiple buttons at once.
- Solution to the above point.
- Plan on an enclosure schematic whilst considering practicality of the end product.
- Figure out which components we need to order.

Throughout the upcoming week we are aiming to find solutions to the above points and order the required components.

--Marcin Dyszynski


Lissajous Diagrams

---
layout: post
title: "Lissajous Diagrams"
date: 2015-01-26T17:12:12.000Z
slug: lissajous-diagrams
categories:
  - audio
---

![Lissajous Example](./lissajous1.svg)

I've made a few different lissajous diagrams now, but initially it took me a while to figure out what was going on. When I googled for explanations, they were all either too 'mathsy' for me, or they were links to a live-coding environment called [lissajous](https://github.com/kylestetz/lissajous) by Kyle Stetz (awesome live-coding environment btw).

Lissajous diagrams are very straightforward. They are just a cartesian co-ordinate system like you used in middle school, with one signal assigned to the x axis and another (usually related) signal assigned to the y axis. Yes, that's it. Super easy. Lissajous diagrams are often used in DAW plugins to represent stereo spread. They're a very good way of visualising stereo spread as you can immediately see the difference between the two channels. I regularly work with mono signals, so I don't always have two channels to use. What I do is grab a delayed copy of the same signal, and use that. The visualisation is similar, but what you're seeing is related to the 'temporal flux' (which is a shitty name, but it captures the concept) of the mono sound.

![Illustration of Lissajous setup](./xandy1.svg)

I hope this helps someone else figure out what's going on, and saves them from the live-coding environment's excellent


Goldsmiths Physical Computing: Lab 01

---
title: "Goldsmiths Physical Computing: Lab 01"
date: 2015-01-22T01:04:56.000Z
slug: lab-1
---

import { YouTubeEmbed } from 'features/YouTubeEmbed';

## Requirements

Build a system that allows 10 LEDs to blink according to a pattern set in the software.

## Analysis

This task will require 10 LEDs, and 10 560 ohm resistors wired in series. It will require sufficient wiring to connect each set of these components in series from their digital 'pin' to ground.

In addition, I have included a potentiometer to manipulate parameters of the pattern in the code. In the case of the example video, it sets the speed of the base 2 counter display.

## Interactivity Design/Scenario

The input to this system, although not required in the lab task, is the potentiometer. The system will support in the case of my sample code, the manipulation of the speed of the binary counter display.

The outputs to the system are the LEDs that display the current value of the count in base 2.

## Notes and Testing

The first problem I took most notice of is the difficulty of placing components on the breadboard without cutting them so that while the parts remain reusable, I don't cause any shorts. I overcame that problem by double-checking every wire, and manually shaping the legs of the resistors so that they didn't come into contact with anything they weren't meant to.

After that, I had difficulty with Fritzing - it seemed that whenever I finished one view, my edits to the next view break it, and I get stuck in an infinite loop of editing each view over and over again out of frustration. That issue is definitely down to familiarity with the software, I'm sure it will improve with practice.

Another issue is that I realised that I'm not aware of best practices of diagramming circuits - how to properly route the connections, how to arrange components most legibly, etc. This is something that will improve with practice and with reading other circuit diagrams.

In terms of the programming, I found very little difficulty. I have some limited prior experience of working with Arduino code, and am familiar with the documentation.

One problem with the hardware that I encountered is that when I tried to power the Arduino with a brand new 9v battery, it didn't work. The battery connector wires heated up indicating a short within the connector. I unplugged it and used USB, however I will attempt to repair the 9v battery connector without causing damage.

In summary, I encountered some problems in both the hardware, and my knowledge of it, and have either finished solving the problems, or have developed strategies to solve those problems over the coming weeks.

## Diagrams

The following is a circuit diagram for this project built in Fritzing.

<img alt="circuit diagram" src="./Blink 10_schem.png"/>

The following is a breadboard image for this project built in Fritzing.

<img alt="breadboard schematic" src="./Blink 10_bb.png" />

The following is an image of the constructed board.

<img alt="circuit image" src="./DSCF8160.JPG" />

## Video:

The following is a video of my system in operation, demonstrating the 10 bit binary counter and speed setting using the potentiometer.

<YouTubeEmbed youtubeVideoId="hEHKqjEj2Pw"/>

Apologies if I've overdone it on the music.

## Code

```cpp
int startPin = 2;
int pinCount = 10;
int inputPin = 5;
int inputValue = 0;
int count = 0;

void setup() {
  pinMode(inputPin,INPUT);
  // initialize digital pin 13 as an output.
  for(int i = 0; i < pinCount; i++){
    pinMode(startPin + i, OUTPUT);
  }
}

void loop() {
  inputValue = analogRead(inputPin);
  for(int i = 0; i < pinCount; i++){
    digitalWrite(startPin+i, bitRead(count,i));
  }
  delay(inputValue);
  count = (count + 1) % 1024;
}
```


Goldsmiths Physical Computing: Lets Get Physical

---
layout: post
title: "Goldsmiths Physical Computing: Lets Get Physical"
date: 2015-01-15T10:04:56.000Z
categories:
  - "Goldsmiths Physical Computing"
slug: lets-get-physical
---

I am in the first physical computing class at the moment, wondering whether or not I can build a quadcopter. My current thoughts are "basically, no". Quadcopters are expensive, the entire point would be to mount a camera on it and get photos of London. a) I'm not sure that's legal, b) it would require mounting an expensive device (my camera) to another complex and fragile device made by me, potentially ending in the demise of both expensive devices.

Another thing I'd like to build is a synthesizer. That sounds fun, can't think of a good reason that wouldn't work, I probably will do that. I'd need to do something to make it not "just" a boring simple digital hardware synth. Maybe a digital control for some analog synth stuff. Maybe digital for CV and analog for audio signal. Sky's the limit. I'll come back to this with a more in-depth brainstorm later on.

This class reminds me of the various "Technology"/Metalwork/Woodwork classes I did in school. Some of those were pretty good, I made a broken guitar once. I'm still considering building an "augmented" guitar - a guitar/bass with an ardunio-like device inside, reprogrammable to manage the circuit configuration (phase and on/off of pickups, what certain knobs do, etc), as well as to digitally manipulate the sound (define how each pickup combines, modulate that combination with knobs, etc.) as well as control output devices on the guitar like LEDs and displays as well as electromagnets playing signals into the strings, actuators to pluck the strings and (this particular one is very unlikely, but...) motorized tuners to make sure the guitar stays in tune throughout performances despite things like unreliable tremolo mechanisms. But, I mustn't forget that I did once try and build a guitar, and despite getting a B in that class, I do not now have a working guitar that I built. I could potentially start with a guitar kit, sets of unassembled and unfinished components for guitars that you can pick up relatively cheaply on eBay.


Midem Music Hack Day

---
layout: post
title: "Midem Music Hack Day"
date: 2014-02-03T13:32:24.000Z
slug: midem-music-hack-day
---

I'm at the Innovation Factory at Midem, about to watch the hackers demo what they've been working on over the weekend. I've seen some awesome things. [Leap motion Bieber swatting](https://www.hackerleague.org/hackathons/midem-music-hack-day-2014/hacks/bang-the-biebs). [Awesome, complex webgl visualisations](https://www.hackerleague.org/hackathons/midem-music-hack-day-2014/hacks/cannes-burn), powered by [an echonest music sync library](https://github.com/plamere/ensync). [A way to DJ using track sources from Spotify](https://www.hackerleague.org/hackathons/midem-music-hack-day-2014/hacks/dj-spotify). Awesome hacks all round, focusing on all different kinds of applications and using a wide variety of technology.

For my hack, I tried to make a synthesizer (which I did) that can mimic other synths based solely on an audio file (which I did not quite manage). I used [Maximilian](https://github.com/micknoise/Maximilian) and [IPlug](https://github.com/olilarkin/wdl-ol) for the first time, and it was quite easy to get a quck synth going. It would've been a lot easier if I had any experience of C++, but even still, it was quite straightforward using both of those libraries. The best bit is that you can compile your IPlug project into a standalone mac app, as well as a variety of DAW plugin formats. I really enjoyed all the DSP stuff, I do really love DSP. However. I didn't quite get to grips with C++ quickly enough to finish my hack on time. I ended up with a really weird error to do with 'EXC_BAD_ACCESS' that I didn't understand at all, which totally prevented me from progressing. I'm quite sure I'll get home and realise I have a typo somewhere. Regardless, I will persevere, figure out what's going on, and fix it - I'm hoping to use this project as a final year project at university, so I will get to do all the parameter extraction stuff.

This weekend has been awesome. I got to go to Cannes. Spent time with great people. Got lost in the building that hosts the Cannes Film Festival at 6am. Learned a shit ton of new stuff. All in all, this has without a doubt been the best part of 2014 so far! Bring on the next Music Hack Day! :D


A recording, some delay

17 October 2021

Objecting to "Legitimate Interest" at Scale

07 October 2021

Don't Do Conditional Return Types in Typescript

06 August 2021

Type Soundness in NextJS Query Parameters

16 July 2021

How to Ship man pages with your Node Programs

09 July 2021

How to use AWS Infinidash with Node JS

30 June 2021

Transitive dependency vulnerability resolution for npm

21 June 2021

What I learned when investigating temperature and humidity monitoring for my apartment

18 June 2021

Solved: do-release-upgrade hangs in Ubuntu on WSL

04 June 2021

What to do if you publish a beta build as @latest: modifying npm version tags

04 June 2021

Loading Audio in Node JS

19 May 2021

Handling Pagination with Async Iterators

28 August 2020

How to run ChucK 1.4.0.1 on Ubuntu 20.04

24 May 2020

Avoiding pitfalls when installing semantic-release for npm library packages

17 May 2020

Updating Javascript Dependencies

25 May 2019

How I packaged my JS library

12 May 2019

Control Spotify Connect with your Mac's Media Keys

15 May 2018

Audio Feature Based Synthesis Part 1: The Building Blocks

17 February 2018

Music Discovery Playlist March 2016

01 March 2016

Music Discovery Playlist February 2016

11 February 2016

Goldsmiths Physical Computing: Week 10 | Completion

25 March 2015

Goldsmiths Physical Computing: Week 9 | It works!

19 March 2015

Goldsmiths Physical Computing: Week 8 | Testing

12 March 2015

Goldsmiths Physical Computing: Week 7 | Prototypin'

05 March 2015

Goldsmiths Physical Computing: Week 6 | Lets get started

26 February 2015

Lissajous Diagrams

26 January 2015

Goldsmiths Physical Computing: Lab 01

22 January 2015

Goldsmiths Physical Computing: Lets Get Physical

15 January 2015

Midem Music Hack Day

03 February 2014