Before LLMs, it was well accepted that lines of code written is not a measure of productivity. Rather, the less code you have, the better, because it’s easier to review and maintain. This is all the more important for network-facing and security-sensitive code, as it reduces the attack surface. Nowadays, this has been forgotten or ignored. For example, the recently released and very popular vibe-coded software OpenClaw is more than 800k lines of code.

In cases where a lot of code is needed, due to the essential complexity of the problem, the solution was to build a component, library or framework that was well-tested and then used as a module. If the problem was common enough (which is the case for most problems), it was published as open-source. For infrastructure software, open-source has become practically mandatory. Altogether, this reduced the work that needed to be done and focused efforts in one place rather than having N versions of the same type of software with a different set of bugs each. Since LLMs, this has gone out the window. Duplication is now encouraged, and vibe coding a hundred differently buggy closed-source versions of the same thing is fine.

Even if one were to accept that LLMs provide an immense productivity boost in terms of writing code, that has never been a bottleneck of software engineering. As Dijkstra said, “computer science is no more about computers than astronomy is about telescopes”. I would add to that “programming is no more about coding”, rather, it’s about gathering and refining requirements, thinking hard about architecture, the correct data structures, security, deployment, and myriad other aspects. Any potential time-saving in writing the code will quickly evaporate in the grand scheme of things, particularly in the long run for any serious production project.

Before LLMs, programming was viewed as a deterministic endeavour described in a precise language. Programming languages, and not English, were viewed as the correct abstraction to improve clarity. Dijkstra (again) argued against “natural language programming”, while Lamport argues in favor of even more formalism. I suppose natural language programming is too tempting, but thinking logically and symbolically is unavoidable whether in mathematics or programming. The fewer programmers know it, the worse software will become.

Another striking phenomenon is the suggestion that LLMs be used for “grunt work”, such as tests. As some have argued, tests can be more important than the code itself as they encode the desired behavior of a program. If any part should be written by the human, it’s the test, since only they would know what’s expected out of the program.

These are just some examples of software engineering principles that seem to have been thrown out overnight. Some have justified this by saying that LLMs are so revolutionary that we have to rethink software engineering altogether. I find this hard to believe. Software engineering is rooted in formal concepts from computer science and mathematics, theories including information theory, complexity theory, systems theory, among others. These cannot be bypassed with or without AI. Not to mention the physical limits regarding the immense resource consumption of these models. When the calculator was introduced, formal proofs and mathematical rigor did not go anywhere. Students still learn to practice mathematics without a calculator, and I believe the same will happen with software engineering once the craze dies down—if it’s meant to survive.

This isn’t new. In 2017, Axios reported that the top iPhone apps had been taking up an increasing amount of space over the period from 2013 to 2017. For most of that period, the size of the Gmail app hovered around 12 MB, with a sudden jump to more than 200 MB near the start of 2017. Other popular apps also saw a 10x or more increase in size over the same period.

Gmail isn’t even the worst offender, it’s just a more popular one. The Tesla and Crypto.com apps are around 1 GB each. So is Samsung’s SmartThings app. What about Google’s other popular apps? Google Home is another hefty one, at 630 MB, that I used for its remote feature, which I replaced with Google TV at almost a tenth the size. Their other popular apps average around 250 MB in size. This seems tame in comparison to Microsoft, with an average app size of around 330MB. For reference, the average size of an app in the top 100 free apps is 280 MB or, in a more expanded set (including games), 200MB.

Just to put this into perspective, on my device, apps (excluding their data) use up 35 GB, and the data is another 35 GB. iOS takes up another 25 GB. Let’s say, 100 GB for apps, data and the OS. That leaves me with 20 GB (leaving a margin of free space for updates) meant to be used for capturing 4K video and high-quality photos (why else get an iPhone), and storing music (don’t even think about lossless). The reality is that running out of space also slows things down, since most of my photos need to be fetched from the cloud before viewing them, and I need to re-download these hefty offloaded apps when I need them again. And good luck if you have a limited data bundle.

Maybe this doesn’t matter. The latest iPhones start at 256 GB, and surely I’ll have plenty of space when I get a new one (although I remember saying this when I upgraded to 64 GB from 32 GB). It’s not really about the space though. These apps don’t have 50x or even 10x the functionality. But now they’re 100x larger, and probably slower. Why?

Also, can someone explain why Microsoft Authenticator is 150 MB to show 6-digit codes?

It’s not clear if this is specifically an iOS problem. I don’t have an Android device and I could not find a way to get that information from the Play Store without a device. That said, I checked the size of Gmail on someone’s Android phone, and it’s around 185 MB, which certainly seems much better.

And if you’re considering switching from the default apps, this is what the installed size (which differs slightly from the App Store size) is of the alternatives on my iPhone running iOS 26.2:

So, why is the Gmail app almost 80x the size of the native Mail app? My guess is as good as yours.

As part of my professional and personal work, I sometimes contribute fixes or enhancements to open-source projects. I’ll be going over some of the ones that I found interesting this year.

Node.js

The year started off with a long-standing PR that I had for the Node.js ORM, TypeORM, getting merged, that improved the performance of hydrating entities. This was the last in a series of PRs directed at performance improvements in projects in the Node.js ecosystem that I had worked on as a result of profiling a slow application.

Elsewhere in the Node.js world, I contributed a few fixes to the import-in-the-middle project, which is a library that lets you intercept imports in Node.js, notably used by Sentry to add instrumentation. I was already familiar with the library’s code since last year when I had submitted a fix that touched many of its core parts. This made it slightly less tricky to debug the issues this time around, which can be difficult due to the intricacies of module resolution in a JavaScript runtime.

I also reported an issue in Node.js regarding the spawn and execFile APIs that could lead to unsafe usage. I was happy to see it resolved quickly and released as a deprecation warning in Node.js 24. With all the supply chain attacks seen this year, it was good to contribute something to the security of the ecosystem. As part of investigating this problem, I also learned that it is almost impossible to pass arguments safely to cmd on Windows, and developed batspawn to help with that.

MacPorts

As a maintainer for MacPorts, I sometimes run into projects that do not tag their releases, so we need to rely on git to check if there was an update for them. There wasn’t a good way to do this, so I contributed an enhancement that checks for updates in a repository using the helpful git ls-remote command. This was my first contribution to MacPorts base as I had previously only contributed to ports. For ports, I had a total of 421 commits in 2025 as of writing.

One notable addition to ports was the ANGLE project, developed by Google, which provides a conformant OpenGL ES implementation on almost every platform and is used by Chrome for WebGL. As part of porting it, I set up git mirrors for some of the dependencies hosted on googlesource.com (which doesn’t offer stable tarballs) and understood what a bare repository is.

I also maintain include-what-you-use on MacPorts, which is a neat utility that ensures you include the right headers in a C or C++ project, and submitted several enhancements to the upstream project to improve its behavior on macOS.

macOS

Since most of my personal work is done on macOS, I sometimes run into bugs in the core tools (or userland) that come with it. Most of the tools in macOS come from FreeBSD, so I submit the fixes there in the hopes that they will be picked up in the next release of macOS. Two such issues were in sed, and one of which had originated in OpenBSD code that FreeBSD imported. I decided to submit the fix for that one to OpenBSD first, and reported the other, which interestingly had already been fixed in NetBSD years ago. They were both fixed in OpenBSD, then picked up by FreeBSD, and finally brought to macOS 26. The fixes themselves were simple, but it was interesting to see code move across four different operating systems — NetBSD, OpenBSD, FreeBSD, and macOS.

Another issue encountered on macOS was that the default syntax highlighting for shell scripts in Vim was poor. This was because the default highlighting was for the ancient Bourne shell syntax and not the modern POSIX one. This finally changed after I submitted a fix for it.

Last, but not least, less (the pager) had long had an issue that was a pet peeve of mine: when trying to copy text from a git diff (paged through less), tabs would get converted to spaces. Someone had mentioned the exact cause and fix for this in a StackExchange post years ago, so I submitted the same fix, and now diffs are displayed as expected when using less -rU as the git pager.

Conclusion

This year had more of a focus on macOS fixes while last year I followed up on patches in the Linux world, largely to support an Arabic keyboard that I had submitted the year before, which required co-ordination with several projects under Freedesktop.org, X.Org and GNOME to support a new keysym.

However, I already had a working POSIX shell script that now had a requirement to read and parse JSON. It had previously extracted values from HTML which, while also being hierarchical, can be reliably split on certain characters (the angle brackets) for basic extraction of values. awk is the closest thing to a real programming language that’s available in the POSIX shell, so I thought I’d try to write a basic JSON parser in it. I had already written a full-blown one before, so I knew it was doable, but I needed something more concise.

First, there are some caveats. JSON is notoriously tricky to get completely right, despite its simple grammar. The following code assumes that it will be fed valid JSON. It has some basic validation as a function of the parsing and will most likely throw an error if it encounters something strange, but there are no guarantees beyond that. In my case, I’m reading JSON from a single, trusted source, so this is an acceptable constraint.

The interface is simple, a single function that accepts a JSON document and a dotted path to a key or array index, and returns the corresponding value. It can be used like so:

# Get one value
name = decode_json_string(get_json_value(json, "author.name"))

# Loop over an object
get_json_value(json, "dependencies", deps)
for (name in deps)
	version = decode_json_string(deps[name])

# Loop over an array
get_json_value(json, "payload.items", items)
for (i = 0; items[i]; i++) {
	get_json_value(items[i], null, item)
	type = decode_json_string(item["type"])
	name = decode_json_string(item["name"])
}

To keep things simple, the same function handles both arrays and objects. In JavaScript, arrays are roughly equivalent to objects with integer keys, and we use the same approach here. This is the implementation, expanded and annotated:

# The function takes three parameters: the JSON object/array, the desired key,
# and an optional array to be filled if the key points to an object or array.
# The rest are local variables (awk only allows local variables in the form
# of function parameters)
function get_json_value( \
	s, key, a,
	skip, type, all, rest, isval, i, c, k, null \
) {
	# Trim leading whitespace, if any
	if (match(s, /^[[:space:]]+/)) s = substr(s, RLENGTH+1)

	# Get the type of value by its first character
	type = substr(s, 1, 1)

	# This variable is needed for when we recursively call the function
	# It will be true if the key argument is undefined, since such
	# variables can behave as either a string or a number in awk
	all = key == "" && key == 0

	# If this is a primitive
	if (type != "{" && type != "[") {
		# Ensure a key is not passed
		if (!all) error("invalid json array/object " s)

		# Parse the value
		if (!match(s, /^(null|true|false|"(\\.|[^\\"])*"|[.0-9Ee+-]+)/))
			error("invalid json value " s)

		# And return it
		return substr(s, 1, RLENGTH)
	}

	# Get the first part of the key (which we will be looking for)
	# if the path is dotted and save the rest for now
	if (!all && (i = index(key, "."))) {
		rest = substr(key, i+1)
		key = substr(key, 1, i-1)
	}

	# isval keeps track of whether we are looking at a JSON key or value
	# In an array, all items are values
	# k is the current key
	# If this is an array, it is the index, which starts at 0
	if ((isval = type == "[")) k = 0

	# Loop over the characters in the provided JSON
	# Skip the opening brace or bracket (to avoid infinite recursion) and
	# increment the index by the length of the token
	for (i = 2; i <= length(s); i += length(c)) {
		# Skip over whitespace
		if (match(substr(s, i), /^[[:space:]]+/)) {
			c = substr(s, i, RLENGTH)
			continue
		}

		# Temporarily assign the first character to our token variable
		c = substr(s, i, 1)

		# If it's a closing brace or bracket, we've reached the end of
		# the object or array, so exit the loop
		if (c == "}" || c == "]") break

		# If we find a comma in an object, the next item will be a key,
		# so reset isval. If it's an array, increment the index
		else if (c == ",") { if ((isval = type == "[")) ++k }

		# If we see a colon, the next token will be a value
		else if (c == ":") isval = 1

		# Otherwise, we expect a JSON value
		else {
			# If the key matches, this is our desired value,
			# so pass the rest of the key and return the result
			if (!all && k == key && isval)
				return get_json_value(substr(s, i), rest, a)

			# Otherwise, get the full value
			c = get_json_value(substr(s, i), null, null, 1)

			# And add it to the associative array
			if (all && !skip && isval) a[k] = c

			# If this is a string and we're not expecting a value,
			# then it's a key, so trim the quotes and save it
			if (c ~ /^"/ && !isval) k = substr(c, 2, length(c)-2)
		}
	}

	# Do a basic check that the object or array was properly closed
	if ((type == "{" && c != "}") || (type == "[" && c != "]"))
		error("unterminated json array/object " s)

	# If we're here, it means we didn't find the value we're looking for
	# so only return something if the whole array or object was requested
	if (all) return substr(s, 1, i)
}

To make the parser more useful, you’ll also need a function to do some decoding of JSON strings. This is a simple one, which handles everything except Unicode escape sequences, but throws an error if it encounters one:

function decode_json_string(s, out, esc) {
	if (s !~ /^"./ || substr(s, length(s), 1) != "\"")
		error("invalid json string " s)

	s = substr(s, 2, length(s)-2)

	esc["b"] = "\b"; esc["f"] = "\f"; esc["n"] = "\n"; esc["\""] = "\""
	esc["r"] = "\r"; esc["t"] = "\t"; esc["/"] = "/" ; esc["\\"] = "\\"

	while (match(s, /\\/)) {
		if (!(substr(s, RSTART+1, 1) in esc))
			error("unknown json escape " substr(s, RSTART, 2))
		out = out substr(s, 1, RSTART-1) esc[substr(s, RSTART+1, 1)]
		s = substr(s, RSTART+2)
	}

	return out s
}

And finally, since there is no built-in error function in awk, you can use something like this:

function error(msg) {
	printf "%s: %s\n", ARGV[0], msg > "/dev/stderr"
	exit 1
}

Where $n$ is the number of generated values and $d$ is the range.

Converting this to JavaScript code:

function repeats(n, d) {
  return n - d + d * ((d - 1) / d)**n;
}

In my case, I wanted to check the expected repeats after generating a million random IDs, where each ID is 8 characters long and there are 31 characters in the alphabet. That is, $n=10^6$ and $d=31^8$. Running this in JavaScript:

console.log(repeats(1e6, 31**8));

The result is -19.72998046875.

Wait, what? Negative repeats? I’ve known about the infamous 0.1 + 0.2 inaccuracy, but this is different; it’s a much larger error, and from only a handful of operations. I double (and triple) checked that I typed the formula correctly, and tried it in both C and Python and got the same result. I then put it into WolframAlpha, since it supports arbitrary precision, and it returned something longer, but much more sensible:

0.5862405426220060595736096268572069594630842175323822481323009

That seemed right. I started tweaking the formula to try to get rid of the error. I figured that, most likely, the division leads to a loss of accuracy that’s exacerbated by raising to a very high power. To avoid this, I reached for BigInt to calculate the power of the numerator and denominator separately, then divide. Since BigInt only supports integers, I also multiply by a “precision” before converting to a number, then divide by it after to get a decimal value:

function repeats(n, d) {
  const p = n*d;
  const e = Number(BigInt(p) * BigInt(d-1)**BigInt(n) / BigInt(d)**BigInt(n))/p;
  return n - d + d * e;
}

That returns 0.5863037109375. On the one hand, the result is correct to three significant figures, which is an improvement over the previous zero. On the other, this takes a second and a half to finish computing. After trying a few different things, I got a tip from a user on the ##math IRC channel to try log1p for the calculation. What does log have to do with anything? The way computers calculate the power is like so:

So we can rewrite the formula to:

The log1p function returns the result of $\log{\left(1 + x\right)}$. The reason it exists is because floating-point does not work well when adding a small $x$ to $1$. Since $d$ is very large, this is the case in our formula. We can rewrite the expression to make it usable in log1p:

The code then becomes:

function repeats(n, d) {
  return n - d + d * Math.exp(n * Math.log1p(-1 / d));
}

Running it returns 0.5863037109375, the same value as using BigInt, only now it doesn’t take a second and a half to calculate. However, it still wasn’t as accurate as I’d like. While looking into this, I came across another function that also works better for small values, expm1, which returns $e^x - 1$. We can factor out $d$ to make this usable too:

And in code:

function repeats(n, d) {
  return n + d * Math.expm1(n * Math.log1p(-1 / d));
}

Running this, we get 0.5862405423540622, a much improved result accurate to nine significant figures. This was as good as it was going to get in JavaScript.

Out of curiosity, I wanted to see if a better result was possible using C, since it provides yet another function for preserving accuracy, fma, also known as fused multiply and add. Rather than lose accuracy twice on both the multiplication and the addition, rounding only happens once after the calculation when using fma. It also happens to be faster, since it uses a single native CPU instruction for both operations. C also provides the long double type, which is as precise as a double or more, depending on the implementation. On my machine, sizeof(long double) returns 16, i.e. it is a 128-bit floating-point number, also known as a quad. Trying them both out:

double repeats(double n, double d)
{
	return fma(d, expm1(n * log1p(-1 / d)), n);
}

long double repeatsl(long double n, long double d)
{
	return fma(d, expm1l(n * log1pl(-1 / d)), n);
}

This yields 0.58624054240892864431 and 0.58624054258953528507, respectively, with a very slight improvement in accuracy, but still only accurate to nine significant figures. At this point, I was out of my depth. I tried the Herbie project which automatically rewrites floating-point expressions and it gave some good results but the suggestions were rather unwieldy. Let me know if you can find another way to compute this with better accuracy — perhaps there is yet another trick floating out there.

Profiling Node.js

Node.js has had a --prof option for quite some time, and it allows you to generate a text file that shows which functions took the most time while running your application. However, it doesn’t always work very well as much of the CPU time spent would be marked as “unaccounted”. More recently, Node.js provides a new option, --cpu-prof. When run with this flag, node creates a .cpuprofile file that could then be loaded into Chrome DevTools and you could visually inspect where time is being spent in your code. Using the DevTools, I proceeded to profile the application, specifically looking at the Bottom-Up tab, sorted by Self Time. This tells you which functions are doing too much work in and of themselves (as opposed to the total time, which includes time spent calling other functions).

What’s slow

It turned out there were several libraries that the application depended on that had performance issues, but the most prominent bottleneck was date and time formatting.

I used the Luxon library for date and time handling in this project, particularly for time zone support. In order for Luxon to get the offset of a particular time zone for a given datetime, it resorts to the Intl.DateTimeFormat API. This API provides the ability to format any datetime value in a locale-specific manner with many options to customize the output. It also allows you to format a datetime in a given time zone.

Since there is no native way in JavaScript to get the offset of a time zone (yet), libraries resort to this feature to essentially format the datetime in a given time zone, then parse the components of the formatted version and calculate the timestamp from that. By comparing this to the known UTC timestamp, the offset can be found. Altogether, a rather expensive operation.

From Node.js to ICU

Since the format method of Intl.DateTimeFormat is implemented using native C++ code, it won’t show up in the Chrome profiler, only its caller. As I’m using macOS, I usually go for the Instruments profiler that comes with Xcode to profile native code.

Using Instruments, and specifically the Time Profiler, showed that Intl.DateTimeFormat was implemented in the V8 engine used by Node.js, and it did little more than call the ICU library — the International Components for Unicode. This is the library that implements the Unicode standard, and is used by most operating systems and browsers. While generally well-optimized, it has a vast surface area so improvements can always be made.

Making ICU faster

The first step was to write a simple benchmark in C++ that did nothing more than format a datetime several thousand times in a loop. I ran that through Instruments and looked at the results. Here too, the bottom-up feature (Invert Call Tree in Instruments) and self time (Self Weight) are most useful. After some trial and error, it turned out there were problems in essentially four areas:

• Memory allocations
• Floating-point operations
• Unoptimized hot paths
• Missing fast paths

Memory allocations

This was by far the biggest culprit in the slow formatting performance. ICU would heap allocate (malloc) an object for every number component formatted in a date (eg. day, hour, minute) followed by a free soon after. A datetime might have six components — year, month, day, hour, minute and second. That’s six memory allocations, times a few hundred formatting operations and it adds up quickly. One additional allocation was also used for a calendar object that had to be cloned per formatting operation. Eliminating all those allocations and using the stack provided a significant performance boost right off the bat.

Floating-point operations

This was a surprise to me, but while profiling I saw fmod show up a lot. This function computes the floating-point remainder, similar to % for integers. It’s not surprising for calender calculations to utilize modular arithmetic heavily, but surely there are no floats in dates? Indeed there aren’t, and converting fmod to a regular % and ensuring integers are used throughout provided another performance boost.

Unoptimized hot paths

When formatting a component in a datetime, such as the hour, the formatting code needs to get the relevant information for that particular pattern character (eg. H in an HH pattern). This was done by looping through an array of those pattern characters until it found the matching one, and that same index would be used for another array that contained the information. This was changed to a simple lookup table, so it would take O(1) time to convert a pattern character to an index rather than O(N).

Missing fast paths

At the core of the ICU library, is the UnicodeString object which is used for anything string related. In a given formatting operation, a string is constantly appended to, and for small strings UnicodeString uses the stack while for large strings it allocates on the heap. When appending, it uses memmove. For date formatting, short strings are exclusively used, so it’s prudent to add a fast path for such strings that would fit into the existing stack buffer. In that case, avoiding the call to memmove and doing a simple unrolled loop copy for small strings proved to be noticeably faster.

Final result

With all these changes, formatting is now up to twice as fast, and sometimes more. But is it as fast as it can get? I decided to compare with the reliable strftime in the C standard library. On my 2016 MBP, running strftime with a simple %I:%M %p format (hour:minute am/pm), it can do a million formatting operations in around 630 ms. Doing the same with ICU 74.2 (before the optimizations, using the equivalent hh:mm a format), it took almost three times as long at 1700 ms. And now, with the recently released ICU 75.1 which includes all the mentioned optimizations, it takes around 800 ms, making it more than twice as fast as before.

Back to Node.js

With the recently released versions 20.13 and 22.1 these changes have made their way back to Node.js. To try them out, I ran a simple benchmark comparing 20.13 (ICU 75) and 18.20 (ICU 74). The official releases from the Node.js website (such as the ones obtained via nvm) bundle the ICU library in the same binary. If you get Node.js from your package manager, it might use the system ICU library which might not be the latest version. To check which version Node.js uses, run node -p process.versions.icu. Finally, let’s check if Intl.DateTimeFormat is faster as expected:

const fmt = new Intl.DateTimeFormat("en-US", {
  hour: "2-digit",
  minute: "2-digit",
  hour12: true,
});
const date = new Date();
fmt.format(date); // Warmup - the first run is much slower
console.time("format");
for (let i = 0; i < 1_000_000; i++) fmt.format(date);
console.timeEnd("format");

Running this script, I get 2100 ms for Node.js 18 and 1300 ms for Node.js 20 — a 1.6x improvement.

For something closer to production use, let’s try a few thousand calls to Date.toLocaleString, which includes both date and time fields, and ensure that it’s thoroughly warmed up, simulating a long-running application:

const d = new Date();
for (let i = 0; i < 100_000; i++) d.toLocaleString(); // Warmup
console.time("format");
for (let i = 0; i < 10_000; i++) d.toLocaleString();
console.timeEnd("format");

In this case, we get a 2x improvement, with Node.js 18 taking 36 ms and Node.js 20 taking just 18 ms, making it twice as fast.

Bash is not the standard shell

The default shell that is present on all Unix systems is sh, not bash. The language used in the portable sh shell is described in the POSIX standard. However, on many Linux systems sh is linked to bash - this makes bash operate in a more compatible way with the standard, but still allows certain bash features that may not work on other systems. When in doubt, refer to the standard.

Long options are not Unix

Many utilities accept both a long option, eg. grep --count with a double hyphen, in addition to a short option, eg. grep -c. The former is a GNU invention, and they generally do not exist on other systems, such as BSDs. In fact, the standard getopts utility, and corresponding getopt C function only support the short style.

Make isn’t GNU make

The version of make specified by POSIX is much more limited than the GNU version. This one is harder to deal with as the specification is lacking in many aspects, particularly any kind of logical or conditional operators. You can workaround this by moving some logic to a configure script that generates another Makefile that is then included by the main one. Further, the BSD makes have a completely different syntax than the GNU one for things like conditionals. Luckily, if your focus is on macOS and Linux only, you can get away with depending on GNU features as macOS’s make is based on the GNU one.

The C compiler is not GCC

This is related to the previous point, as it often comes up in Makefiles. When referring to the C compiler in that context, it is better to use the implicit $(CC) variable, and when compiling C++ code, to use the $(CXX) variable. Most BSD systems have now switched to Clang as the default compiler and do not provide a gcc binary. When referring to the C and C++ compilers outside of Makefiles, the cc and c++ commands are reliable and work across systems.

GNU is not Linux

This is slightly different, but even GNU interfaces are not necessarily the ones present on a Linux system. The Alpine Linux distribution for example, popular as a base in Docker containers due to its light weight, forgoes the GNU C Library for musl, and uses BusyBox instead of the GNU utilities. Therefore, one would be well advised to try to stick to portable interfaces even if targeting solely Linux systems.

Unix is not UNIX

Finally, even Unix is not UNIX. The latter is a trademark that requires certification by The Open Group. Certified operating systems, the most well-known of which is macOS, are guaranteed to follow the UNIX specification. That said, most Unix-like operating systems, including the BSDs, as well as the GNU tools make a strong effort to stick to the standard as much as possible.

Note: For arm64 instances, a FreeBSD image is currently available under the partner images source when creating the instance, so you can skip the steps below unless you need a specific version.

Install

1. Create two instances in Oracle Cloud using the default image. We will be using one for the FreeBSD server, and a temporary one for the installation process. Make sure to specify your SSH public key when creating the temporary instance.

2. On the FreeBSD instance page, stop it if it is running, and detach the boot volume.

3. On the temporary instance page, attach the FreeBSD instance’s boot volume as a block volume. Make sure to select Paravirtualized as the attachment type.

4. SSH into the temporary instance. Check the path of the newly attach volume. You can do this by running lsblk and seeing which one has nothing mounted on it (it will most likely be /dev/sdb).

   Then, run the following command to install a raw FreeBSD image onto the volume. Modify the release version and the volume path as needed.

   curl https://download.freebsd.org/ftp/releases/VM-IMAGES/13.1-RELEASE/amd64/Latest/FreeBSD-13.1-RELEASE-amd64.raw.xz | xz -dc | sudo dd of=/dev/sdb bs=1M conv=fdatasync
   

5. Once the process is complete, detach the block volume from the temporary instance.

6. Re-attach the FreeBSD instance’s boot volume and start the instance.

Setup

Once the FreeBSD instance has booted, we can now configure it.

1. In the instance’s page, launch a Cloud Shell connection (you may need to press Enter in the console if it appears to be stuck for a while). This will give us preliminary access and allow us to enable SSH on the new server.

2. Run passwd to set a password for the root user - make sure to choose a strong one.

3. Create a new user using adduser. When asked to invite the user to other groups, enter wheel to give the user root privileges.

4. Enable and start the SSH service by running service sshd enable, followed by service sshd start.

5. Copy your public key from your local machine using ssh-copy-id user@freebsd-instance-ip.

6. Now you can SSH into your new FreeBSD server and do any additional setup. Enjoy!

Additional Setup

IPv6

To get IPv6 working on a FreeBSD instance on Oracle Cloud, we need support for DHCPv6, which is not yet available in FreeBSD’s DHCP client dhclient. To get around this, we can use the dual-dhclient-daemon package which supports dual-stack DHCP. You can install and enable it with the following:

pkg install dual-dhclient-daemon
sysrc dhclient_program=/usr/local/sbin/dual-dhclient

You will also need to enable IPv6 for your instance in the Oracle Cloud settings.

Compiler

The compiler comes with a lot of useful flags to catch common problems. A good set of defaults is the following:

• -Wall, -Wextra - enable many useful warnings
• -pedantic - warn when using non-standard language extensions
• -fsanitize=address,leak,undefined - catch issues relating to memory and undefined behavior
• -D_LIBCPP_DEBUG=1 (for Clang’s libc++ - used by default on macOS and FreeBSD) or -D_GLIBCXX_DEBUG (GCC’s libstdc++ - used on Linux) - catch undefined behavior when using the C++ library

Now let’s try a quick example to see how these can help:

#include <stdio.h>

int main(void)
{
	int values[] = { -1, 14, 32, -5, 24, 40, -3, 96, 23 };
	int total;
	for (int i = 0; i < 10; i++) {
		total += total*values[i] + 1;
	}
	printf("%d\n", total);
	return 0;
}

Save this file as example.c. You can then compile it by running make example - this works because make will auto-detect the C file and use a pre-defined implicit rule to build an example executable. Then, run the file with ./example. You will see that it will compile and run without any issues, and print a total like 1503498210. Now let’s try compiling with some warnings:

make example CFLAGS="-Wall -Wextra -pedantic"

We get one warning:

example.c:8:3: warning: variable 'total' is uninitialized when used here [-Wuninitialized]
                total += total*values[i] + 1;
                ^~~~~
example.c:6:11: note: initialize the variable 'total' to silence this warning
        int total;
                 ^
                  = 0
1 warning generated.

Easy enough to fix by setting total to 0 when declaring it. When compiling again, we get no warnings and our little program is seemingly perfect!

Sanitizers

Unfortunately, compilers do not catch everything at compile-time. Sometimes the code needs to run to detect other kinds of problems. This is known as dynamic analysis, as opposed to static analysis, and is done with the help of sanitizers. Let’s add a couple more flags to our build:

make -B example CFLAGS="-Wall -Wextra -pedantic \
	-fsanitize=address,leak,undefined"

Note: If you get a compile error, it might be because you need to install additional libraries for these to work, namely libasan, liblsan and libubsan, which you can do via your package manager.

Compiling again will not yield any new warnings. However, run the program now and you’ll discover some new things:

example.c:8:17: runtime error: signed integer overflow: 420559700 * 23 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior example.c:8:17 in
example.c:8:18: runtime error: index 9 out of bounds for type 'int [9]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior example.c:8:18 in
=================================================================
...

The sanitizers have detected two issues in our code. Our bounds check is faulty, it should be i < 9 rather than i < 10, and our result had been overflowing due to using an int instead of a long for the total. Fixing both those issues, and adjusting the printf format string to use %ld to match the new type, we compile and run again. This time we get the correct result, 10093432801.

C++

While sanitizers are useful, they can come with significant overhead to the program, especially the address sanitizer. When writing modern C++, where use of raw arrays and pointers can be less frequent, it might be helpful to just check for out-of-bounds accesses at the library level. The C++ libraries provided allow this by defining _LIBCPP_DEBUG/_GLIBCXX_DEBUG. libstdc++ also provides _GLIBCXX_DEBUG_PEDANTIC for further checks of non-standard behavior.

Let’s rewrite our program in C++:

#include <iostream>
#include <vector>

int main()
{
	std::vector<int> values = { -1, 14, 32, -5, 24, 40, -3, 96, 23 };
	int total;
	// Use a range-based for loop!
	for (int i = 0; i < 10; i++) {
		total += total*values[i] + 1;
	}
	std::cout << total << std::endl;
}

Compile the program with:

make example2 CPPFLAGS="-D_LIBCPP_DEBUG=1 -D_GLIBCXX_DEBUG" \
	CXXFLAGS="-std=c++20 -Wall -Wextra -pedantic -fsanitize=undefined"

When running our program, we get this:

example2.cpp:9:17: runtime error: signed integer overflow: 420559700 * 23 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior example2.cpp:9:17 in
/usr/include/c++/v1/vector:1549: _LIBCPP_ASSERT '__n < size()' failed. vector[] index out of bounds
Abort trap: 6

Now we can resolve the overflow and out-of-bounds errors as before.

Debugger

Clang comes with a powerful debugger, lldb, that also includes a convenient GUI. If you use GCC, its respective debugger, gdb, also comes with a TUI mode that can be accessed via gdb -tui. Both debuggers are quite similar and you can find a helpful command map on the LLDB website. To use the debugger, rebuild the executable with the -g flag to generate debug information:

make -B example CFLAGS="-g -Wall -Wextra -pedantic -fsanitize=address,leak,undefined"

Then, enter the debugger by running lldb example. Once in the REPL, you’ll need to run the program. Enter r or run to do so. This will cause it to run to completion, which is not exactly what we want. This time, enter b main to create a breakpoint at main so the debugger pauses just before our code runs. Then, run again. The debugger will now pause at the first line of our program. You can step to the next line using n. You can view the current variables at any time using v, or a specific variable by specifying it, eg. v total. To continue till the next breakpoint or the end, use c.

GUI

It might be helpful to use the debugger’s GUI instead. Right after running the program with r, enter gui to go to GUI mode. The same keyboard shortcuts will work there too, and there will be some additional ones that can be seen with h. Once done, press Escape to exit the GUI.

Debugging an executable that uses stdin

One thing you might notice is that there is no way to pass stdin to a debugged program. There’s a simple fix for this. Pass a file to lldb like so:

lldb example 3< input.txt

Then, run set set target.input-path /dev/fd/3 at the beginning of the debug session.

Simplifying the process

There’s a lot to take in here, and too many steps in some cases. We can simplify the process with some configuration. First, add the following aliases to your shell’s profile:

alias build="make \
CPPFLAGS="-D_LIBCPP_DEBUG=1 -D_GLIBCXX_DEBUG" \
CFLAGS=\"-g -Wall -Wextra -pedantic -fsanitize=address,leak,undefined\" \
CXXFLAGS=\"-std=c++20 -g -Wall -Wextra -pedantic -fsanitize=undefined\""

alias debug="lldb -o r"

You can tweak this alias to your liking. For example, you can use CC and CXX to specify your preferred C and C++ compiler respectively.

Then, add the following to ~/.lldbinit:

set set target.input-path /dev/fd/3
b main

This will automatically set the default input path and create a breakpoint at main whenever you launch lldb. This leaves only one command, actually running the program via r which is what the debug alias above achieves.

Now, it’s as simple as build example and debug example to quickly build and debug programs. You can pass additional files and libraries to the build command via the LDLIBS variable. For release builds, once you’re happy with your program, call make directly with appropriate flags, such as -O2 for optimization, and avoid using the debug and sanitizer flags. Once your project has become larger than a few files, and especially if your project is shared with other people, it might be worth adding explicit targets to your build system that perform similar functions to your local aliases.

Documentation

The easiest way to get documentation for any C standard library function is to use man. For example, to see the documentation for printf, use man 3 printf. You can also view the list of functions in a header, eg. man 3 stdio. For C++, you might need to install a libstdc++ docs package. This will then allow you to look up documentation for C++ standard library namespaces and classes, eg. man std::string.

1. Change your last name to True or, for nihilists, Null
2. Be Turkish
3. Move to Norway
4. Be a horse person
5. Be born before 1970
6. Type anything that isn’t a printable ASCII character
7. Use the wrong printable ASCII character
8. Practice good password hygiene

Let us know your favorite ways to break software!