Migrating from v0.4.5 to v0.5.0

Due to the breaking changes introduced in v0.5.0 on the persistence system, you might need to update your codebase to make it compatible with the new version. This is not required if you are starting a new project from scratch.

What happened?

In v0.5.0, we introduced a new persistence system that is more optimized for rapidly changing data. Previously, we were using Sled to store the serialized collection blobs. We found that it was not the best option for our use case as each blob size could be somewhere in between 100MB to 10GB.

When the data change rapidly, the collections need to be saved periodically to avoid data loss. With this, the collections need to be reserialized and rewritten back into Sled. The dirty IO buffer during these operations caused some storage issues, bloating the space required to store the collection for up to 100x the collection size.

This new system is more optimized for our use case since we now write the serialized collection data directly to a dedicated file on the disk. Now, we only use Sled for storing the collection metadata and the path to where the collection is stored.

How to migrate?

To migrate OasysDB from v0.4.5 to v0.5.0, I recommend creating a new Rust project and migrating the database from there. This migration project will read the data from the old database and write them to the new database. And for that, this project need to have access to the database files.

If you are using OasysDB on Python, you might want to use Rust to migrate the database as it supports installing both versions of OasysDB on the same project easily which is required for the migration. I can promise you that the migration process is quite simple and straightforward.

Friendly Reminder: Make sure to create a back-up of your database files before proceeding 😉

1. Install both versions of OasysDB

After setting up the new project, you can install both versions of OasysDB by specifying the package and the version in the Cargo.toml file.

[dependencies]
odb4 = { package = "oasysdb", version = "0.4.5" }
odb5 = { package = "oasysdb", version = "0.5.0" }

2. Migrate the database

The following script will read the collections from the old database and write them to the new database which is all we need to do to migrate the database.

use odb4::prelude::Database;
use odb5::prelude::Database as NewDatabase;

fn main() {
    // Change the path to the database accordingly.
    let db = Database::open("database").unwrap();
    let mut new_db = NewDatabase::new("new-database").unwrap();

    // Collection names you want to migrate.
    let names = vec!["collection_a", "collection_b"];

    // This will read the collections from the old
    // database and write them to the new database.
    for name in names {
        let collection = db.get_collection(name).unwrap();
        new_db.save_collection(name, &collection).unwrap();
    }
}

3. Verify the migration

After running the script, you can verify the migration by checking the new database files. The new database path should contain a sub-directory called collections which stores the serialized collection data. The number of files in this directory should be equal to the number of collections you migrated.

Don't forget to point your application to the new database path after the migration or rename the new database path to the old database path to make sure that your application uses the new database correctly.

Conclusion

If all the steps are followed correctly, you should have successfully migrated your OasysDB database from v0.4.5 to v0.5.0. If you face any issues during the migration, feel free to reach out to me on our Discord.

I will be happy to personally assist you with the migration process 😁