At FP Complete, we strive to build systems that endure the direst of situations. An unexpected shutdown (like a kernel panic, or unplugging the power cord) in a machine should not affect the durability of confirmed writes in programs we develop.
Click below to learn more about a unique offer
As a developer, you'll likely want to have options in regards to guaranteed durable writes; durability is the property that ensures that once an API confirms a write, every read will reflect its changes.
Durable writes are essential for our customers; many of them work in the Financial and Medical Tech industry, and any saved piece of data—whether in a filesystem or a database—must have durability guarantees after we perform a write operation. Otherwise, companies lose money and lives may be at risk.
In this blog post, we will demonstrate that, although our high-level Haskell APIs tell us that it persisted our file writes in the filesystem, a catastrophic failure like an unexpected shutdown may cause us to lose writes that we thought were committed. We will later demonstrate how using low-level C APIs (the ones popular RDBMS databases use) offer better guarantees in write durability; and finally, we’ll show you some prior art we have implemented in
Status Quo of File APIs in Haskell
When performing writes in Haskell programs, we often rely on functions
Often, this behavior is acceptable. Not every piece of data must be durable and if, for some reason, you are dealing with large files, having them being durable by default might be an expensive operation. That said, the durability aspect of filesystem writes must be a conscious decision rather than an afterthought.
Improving write durability in Haskell
How can we improve the durability situation in our Haskell APIs? First, we need to make use of C functions that will sync writes to the file system. Once writes have been performed to
Developers at FP Complete implemented a new family of functions that use the strategies mentioned above in the
To implement these functions, we used internal APIs
You can take a look at the source code if you are curious. We are open to feedback, as the usage of low-level APIs can get somewhat tricky at times. We are hoping to at some point in the future makes these functions (or a version of them) part of the standard
Testing the durability of our file system
Being aware of the durability aspects of our filesystem is nice and dandy, but should we really be concerned? Is this issue something that could happen frequently, or are we paranoid? This question is a fair one. We believe this ordeal happens more often than you might expect. We can easily replicate durability concerns using a virtual machine.
When we execute the following program:
In the scenario where we execute this program with the
Our program writes new files in the given directory
The same algorithm using our durable flavored API doesn’t have this problem after a hard restart; when executing the program above with the
The written files will be kept on disk thanks to the usage
In this blog post, we covered some nuances of the durability of file systems and also learned about the importance of
If your system:
It is a good practice to make sure file writes are guaranteed to be durable in catastrophic scenarios. FP Complete offers auditing services where we can help you discover this and other various challenges.
Posted by Roman Gonzalez - 12 March, 2019