FRAGMENT

Documentation

O Migrate Data#

As your product and back-office requirements evolve you'll need to update your Schema and migrate existing data. Your Schema is immutable and versioned, and once deployed to a Ledger FRAGMENT will enforce backwards compatibility checks. Use migrations to safely change your Schema and Ledger data while ensuring compatibility with your code, zero down-time, and data that is always immutable and auditable.

By deploying a new Schema version, you can:

  • Create new Entry Type versions
  • Disable and archive old Entry type versions
  • Migrate entries posted with archived Entry Type versions
  • Disable and archive old Ledger Accounts
  • Migrate entries posted to archived Ledger Accounts

The exact process for performing a migration depends whether you need to migrate all or some of your existing data, and whether affected balances need to stay accurate during the migration.

a. Updating Entry Types

#

When you create a new Entry Type it will automatically be marked as V1. Once deployed, Entry Type versions are immutable and cannot be deleted from your Schema. Editing Entry types in the Dashboard will automatically create new versions as needed.

In the Schema JSON Entry Types have a type and typeVersion field that together must be unique. To ensure backwards compatibility all deployed Entry Type versions are always kept on the Schema.

Ledger Entry with typeVersion
↓
{
  "key": "schema-key",
  "chartOfAccounts": {...},
  "ledgerEntries": {
    "types": [
      {
        "type": "user_funds_account",
        "typeVersion": 1,
        "lines": [
          {
            "account": { 
              "path": "assets/banks/user-cash"
            },
            "key": "funds_arrive_in_bank",
            "amount": "{{funding_amount}}"
          },
          {...}
        ]
      },
      {
        "type": "user_funds_account",
        "typeVersion": 2,
        "lines": [
          {
            "account": { 
              "path": "assets/banks/user-cash"
            },
            "key": "funds_arrive_in_bank",
            "amount": "{{funding_amount}} + {{fee}}"
          },
          {...}
        ]
      }
    ]
  }
}
 

When posting you can specify typeVersion to select which version of an Entry Type to use. If not set, 1 will be used as the default.

b. Disabling Entry Types

#

Once you have created a new Entry Type version, you can prevent postings of the previous version by setting the old Entry Type versions's status field to disabled in your Schema.

Schema with disabled Entry Type
↓
{
  "key": "schema-key",
  "chartOfAccounts": {...},
  "ledgerEntries": {
    "types": [
      {
        "type": "user_payment",
        "typeVersion": 1,
        "status": "disabled",
        "lines": [...]
      },
      {
        "type": "user_payment",
        "typeVersion": 2,
        "lines": [...]
      }
    ]
  }
}
 

When an Entry Type is disabled:

  • Any attempt to post using that Entry Type will result in a BadRequestError
  • Existing entries posted with that type will remain unchanged and queryable
  • The Entry Type can later be re-enabled or archived
  • You can still reverse or migrate entries that were posted with the disabled Entry Type

c. Archiving Entry Types

#

You can archive an Entry Type version to trigger the migration process for historical data. Archiving an Entry Type version:

  • Requires the Entry Type version to be first be disabled for 45 seconds
  • Hides it from the Dashboard Schema designer by default
  • Creates a LedgerEntryDataMigration (see below) for each Ledger using the Schema to query for its entries to migrate
  • Does not automatically migrate or change existing entries

To archive an Entry Type, set its status field to archived in your Schema:

Schema with archived Entry Type
↓
{
  "key": "schema-key",
  "chartOfAccounts": {...},
  "ledgerEntries": {
    "types": [
      {
        "type": "user_payment",
        "typeVersion": 1,
        "status": "archived",
        "lines": [...]
      },
      {
        "type": "user_payment",
        "typeVersion": 2,
        "lines": [...]
      }
    ]
  }
}
 

Once archived, you can un-archive an Entry Type version by settings its status back to active. This will set the LedgerEntryDataMigration created for the archival to status inactive. The Entry Type version will then be available again for use in your application.

d. Migrating Entries

#

When you deploy a Schema with a newly archived Entry Type version a LedgerEntryDataMigration will asynchronously be created for each Ledger using the Schema. To migrate existing entries that were posted with the archived Entry Type version:

  1. Enumerate ledgerEntries on the relevant LedgerEntryDataMigration. Since the Entry Type version was previously disabled, no new Ledger Entries will be added to this list.
  2. Call migrateLedgerEntry for each Ledger Entry with the new version of the Ledger Entry. This will remove the Ledger Entry from the LedgerEntryDataMigration. The migration is complete when the ledgerEntries array is empty.
LedgerEntryDataMigrations
↓
query GetEntryMigrations(
  $ledger: LedgerMatchInput!
  $filter: LedgerEntryDataMigrationsFilterSet
) {
  ledger(ledger: $ledger) {
    ledgerEntryDataMigrations(filter: $filter) {
      nodes {
        entryType
        typeVersion
        ledgerEntries {
          nodes {
            id
            type
            posted
            parameters
          }
          pageInfo {
            hasNextPage
            endCursor
          }
        }
      }
      pageInfo {
        hasNextPage
        endCursor
      }
    }
  }
}
 
LedgerEntryDataMigration variables
↓
{
  "ledger": { 
    "id": "ledger-id"
  },
  "filter": {
    "entryType": {
      "equalTo": "user_payment"
    }
  }
}
 

Use the migrateLedgerEntry mutation to migrate your Ledger Entries. This mutation's input takes an id of the Ledger Entry to be migrated, and the newLedgerEntry to migrate to. Under the hood this performs a reversal of the entry to migrate followed by a repost using the new Entry.

migrateLedgerEntry
↓
mutation MigrateLedgerEntry($input: MigrateLedgerEntryInput!) {
  migrateLedgerEntry(input: $input) {
    ... on MigrateLedgerEntryResult {
      reversedLedgerEntry {
        id
        ik
      }
      reversingLedgerEntry {
        id
        ik
      }
      newLedgerEntry {
        id
        ik
        type
        typeVersion
        posted
        created
        lines {
          nodes {
            amount
          }
        }
      }
    }
  }
}
 
migrateLedgerEntry variables
↓
{
  "input": {
    "id": "entry-to-migrate-id",
    "newLedgerEntry": {
      "ledger": {
        "id": "ledger-id"
      },
      "type": "user_payment",
      "typeVersion": 2,
      "parameters": {
        "amount": "10000",
        "fee": "100"
      }
    }
  }
}
 

e. Updating Accounts

#

Like Entries, your Ledger Accounts are generally immutable and safeguarded by FRAGMENT's backwards-compatibility rules. However a number of Account migrations are supported to ensure flexible Schema design.

Consistency#

Accounts can be migrated between strongly and eventually consistent by updating the consistencyConfig field for the Account on the Schema. Once deployed a LedgerMigration will be created for each of the Schema's Ledgers that will apply the update.

Currency#

Single-currency Accounts can be upgraded to multi-currency Accounts by updating the currencyMode field for the Account on the Schema. Once the Schema change is deployed a LedgerMigration will be created for each of the Schema's Ledgers that will apply the update.

Note:

  • Multi-currency Accounts cannot be downgraded to single-currency
  • You will need to ensure that you're providing the currency argument when reading the balance, ownBalance, or childBalance fields of the now multi-currency Account. No change is required if you're reading the balances, ownBalances, or childBalances fields of the Account.

f. Disabling Accounts

#

When you need to prevent new entries from being posted to a Ledger Account you can disable it by setting the Account's status field to disabled in your Schema. This is useful when:

  • You're migrating to a new account structure
  • An account is no longer needed for new transactions
  • You're preparing to archive and migrate historical data

When an Account is disabled:

  • Any attempt to post Ledger Entries to it will result in a BadRequestError
  • All Entry Type versions that post to the disabled Account must also be disabled in your Schema
  • Existing entries and balances remain unchanged and queryable
  • Entries can still be posted to any of its active children Accounts
  • The account can later be re-enabled or archived
Schema with disabled Account
↓
{
  "key": "schema-key",
  "chartOfAccounts": {
    "accounts": [
      {
        "key": "legacy-user-funds",
        "name": "Legacy User Funds",
        "template": "assets/banks/legacy-user-funds",
        "status": "disabled"
      },
      {
        "key": "user-funds",
        "name": "User Funds",
        "template": "assets/banks/user-funds"
      }
    ]
  }
}
 

g. Archiving Accounts

#

Once an Account is disabled, you can archive it to trigger the migration process for historical data. Archiving an Account:

  • Requires the Account to first be disabled for 45 seconds
  • Hides it from the Dashboard Schema designer by default
  • Creates a LedgerAccountDataMigration for each Ledger using the Schema to query for its entries to migrate
  • Does not automatically migrate or change existing entries posted to the Account

To archive an Account, set its status field to archived in your Schema:

Schema with archived Account
↓
{
  "key": "schema-key",
  "chartOfAccounts": {
    "accounts": [
      {
        "key": "legacy-user-funds",
        "name": "Legacy User Funds",
        "template": "assets/banks/legacy-user-funds",
        "status": "archived"
      },
      {
        "key": "user-funds",
        "name": "User Funds",
        "template": "assets/banks/user-funds"
      }
    ]
  }
}
 

h. Migrating Accounts

#

When you archive an Account, FRAGMENT automatically creates a LedgerAccountDataMigration that tracks all entries posting to that account. You can query for these migrations and filter by specific structuralPaths:

Query Account migrations"
↓
query GetAccountMigrations(
  $ledger: LedgerMatchInput!
  $filter: LedgerAccountDataMigrationsFilterSet
) {
  ledger(ledger: $ledger) {
    ledgerAccountDataMigrations(filter: $filter) {
      nodes {
        accountPath
        ledgerEntries {
          nodes {
            id
            type
            typeVersion
            posted
            parameters
            lines {
              nodes {
                account {
                  path
                }
                amount
              }
            }
          }
          pageInfo {
            hasNextPage
            endCursor
          }
        }
      }
      pageInfo {
        hasNextPage
        endCursor
      }
    }
  }
}
 
Query Account migrations variables"
↓
{
  "ledger": {
    "id": "ledger-id"
  },
  "filter": {
    "accountPath": {
      "equalTo": "assets/cash"
    }
  }
}
 

Before migrating a Ledger Account it is important to consider whether or not you can take downtime on reading the Account balance. This will inform whether you should run an online or offline migration.

Offline migrations#

For offline migrations your Account balance will be in flux while the migration is running. To perform one:

  1. Update your Schema to move to the new Account
    • Disable the old Account and create the new Account you want to migrate to.
    • Disable all Entry Type versions that post to the old Account.
    • Create new Entry Type versions that post to the new Account.
    • Deploy your Schema.
  2. Update your code to move to the new Account
    • Update your code to use the new Entry Type versions.
    • Since new data is going to the new account and old data is still in the old account, both of these Account balances will be inaccurate. Ensure that any process that reads these balances is aware of this.
  3. Update your Schema to remove the old Account
    • Archive your old Account and old Entry Type versions that post to it.
    • Deploy your Schema.
  4. Migrate existing data
    • Query all Ledger Entries posting to the old Account using the ledgerEntries list on the LedgerAccountDataMigration.
    • Migrate your list of Ledger Entries using the new Entry Type versions. Once completed, read from the new Account balance.
Online migrations#

For online migrations your Account balance will stay accurate throughout the process. To perform one:

  1. Update your Schema to add new Accounts and Entry Type versions
    • Create the new Account you want to migrate to. Add an additional control Account to support the migration.
    • Create new dual-write Entry Type versions that post to both the old, new, and the control Account (used to balance the Entry).
    • Create a new final Entry Type versions that post to only the new Account.
    • Deploy your Schema.
  2. Start dual-writing
    • Update your code to use the new dual-write Entry Type versions for new data. Now both old and new Accounts will be updated with new data, but the new Account does not yet contain existing data, so its balance will be inaccurate.
    • Query for all Ledger Entries posting to the old Account that used the old Entry Type versions.
    • Use migrateLedgerEntry to migrate each of the old Ledger Entries to the new dual-write Entry Type version. Now both old and new Accounts will have matching balances.
  3. Switch over to the new Account
    • Update your code to use the new Account balance.
    • Update your code to use the final Entry Type version that posts to only the new Account for new data.
  4. Archive the old Account and its data
    • Disable the old Account and the dual-write Entry Type versions.
    • Deploy your Schema.
    • Archive the old account and the dual-write Entry Type versions.
    • Deploy your Schema.
    • Query all Ledger Entries posting to the old Account using the ledgerEntries list on the LedgerAccountDataMigration.
    • Use migrateLedgerEntry to migrate old Entries to final Entry Type versions. Once completed, the old Account will have no lines.