How to effectively detect if data has changed using RecVersion

Sometimes there is a requirement to effectivle detect if data has changed in a table. The first thing that might come to mind is, why not just use the "ModifiedDateTime" field of the data? Well, that's perfectly fine if don't have a lot of data in your table.

But, what if there is a lot of data in our table? Like millions of records? Filtering on an unindexed field like "ModifiedDateTime" and with a lot of data, will force the database server to scan all the records in the table. The database server has to do this to determine which records are modified after the specified date and time. Table scans on large tables is not effective and painfully slow because it's not using any of the table indexes.

RecVersion

So, what's the solution? How do we effectively detect which rows have changed in our data? By using the system field RecVersion. Every table in Dynamics 365 Finance and Operations (F&O) has a field called RecVersion.

Every time a record is changed in a table, the RecVersion of the record is updated by the system to a new value, or version. The RecVersion field contains the record version of the record.

But, there is a small problem, how do know what the previous version of RecVersion was before the data was changed? Well, we don't. But luckily there is a simple way so solve this. By creating a custom table with the RecId and RecVersion of our data.

Steps

• Create a custom table with a reference RecId and RecVersion field to our data. Field names: RefRecId and RefRecVersion
• In code, create a method to create a "snapshot" of the RecId and RecVersion values of the data. Use insert_recordset to copy the data.
• Query your table with a join to the custom table. Join on YourTable.RecId and YourTable.RecVersion and SnapShotTable.RefRecId and SnapShotTable.RefRecVersion

To get all the data from the target table that has changed, join the two tables on RecId and RecVersion. The records where RecVersion is different, has changed.

Base64 encoding and decoding

What is Base64 encoding and decoding? On a very high level: it is a method of converting binary data to ASCII text and the ASCII text back again to binary. There are many articles online on how this works technically but this article will explain the basics and how Base64 encoding and decoding is used in Dynamics 365 Finance and Operations (F&O).

For example, if you attach a spreadsheet or other binary file like a PDF document to an email. The binary file is converted to ASCII text using Base64 encoding and is embedded in the message body of the email as text. When the email message is received by the recipient, the email application used to view the email message, detects that there is a file attached embedded as text and converts the text back to binary so that the attachment can be downloaded as a file. Most email applications support previewing attachments, but will have to convert the attachments from text to binary on the fly before being able to display the contents of the file.

So, how does Base64 encoding and decoding relate to F&O? It is frequently used in the standard application and also by developers when using the SysOperation framework just to mention just a few examples.

Code example

In a SysOperation contract class, the SysOperation framework allows a Query object to be used to query data for the operation. Because the related operation can also be run in "batch", in other words in "unattended" mode without an user interface, the Query object of the class needs to be saved to the database so that the operation can be executed by the batch framework and use the exact same query object that was specified by the user when adding the operation to batch. In other words, the system saves all query filters, joins and other changes to the query and will then use your exact query when executing in batch.

An instance of an class object cannot be saved to the database. So, how does the system save the object to the database? By using Base64 encoding and decoding. The Query object of the contract class is serialized to a binary object using the "pack" method of the Query class. This method returns a binary representation of the query in a container type. All class variables of the Query object are serialized and converted to a binary object (container). The resulting binary object of the Query object is then Base64 encoded to text. This text can be saved to the database as a regular text field.

When the Query object instance is required by the SysOperation framework, the process is reversed. The text is Base64 decoded to binary, the original Query object with all the specified filters and other changes can then be instantiated by using the binary data to construct a Query object.

1. Class variable for the Base64 encoded text.

2. Code that converts the text back to binary object.

3. Code that converts the binary object to text.

Using Boolean OR to make where clause fields optional

Sometimes you have a custom "while select" statement and you need to have one or more of the where clause fields in the statement to be optional. In this example, I will use a simple method to display the customer account and group for customers.

The method below displays the customer accounts for customers and uses the _custGroup argument to filter customers that match the customer group. The method works as expected but there is one drawback: it can only show customers for the specified customer group.

So, what if we want to show info for all customers with this method? Can't we just call the method and pass an empty string? The answer is yes, that's possible but will not work as one would expect. The problem with this is, that if you call the method and pass an empty string for the customer group, the while select statement will return all customers where the customer group is empty. Since the customer group field is mandatory, and all customers have a customer group, the query will return nothing.

Show customer info method

//show customer info with mandatory argument
public static void showCustomerInfo(CustGroupId _custGroup)
{
    CustTable custTable;

    while select custTable
        where custTable.CustGroup == _custGroup
    {
        info(strFmt("%1 - %2", custTable.AccountNum, custTable.CustGroup));
    }

}

How do we solve this problem? Your first thought might be to simply duplicate the while select statement in the method for each argument used in the where clause. In this example two statements, one statement with, and another statement whithout the customer group where clause. And simply add an "if" statement to check if the customer group argument has a value or not. Yes, this will work but this is definitely not the way to go. The reason is because if you have a large while select statement with many lines and a method with multiple arguments. The code will be hard to read and error prone when modifications are done to it.

Optional argument

So, what is the solution? By using the Boolean OR operator in the while select statement. The where clause for customer group is extended with an Boolean OR to contain two parts. The first part of the Boolean OR evaluates the method customer group argument variable, if this part of the Boolean OR evaluates to TRUE, the second part of the OR is simply ignored, effectively ignoring the where clause part for customer group in the second part. However, if the first part of the Boolean OR statement evaluates to FALSE, the second part of the boolean OR is then evaluated which effectively then applies the where clause of the customer group to the statement. Making the where clause optional and having compact and readible code.

//show customer info with optional argument
public static void showCustomerInfo(CustGroupId _custGroupId = '')
{
	CustTable custTable;

	while select custTable
		where (_custGroupId == '' || custTable.CustGroup == _custGroupId)
	{
		info(strFmt("%1 - %2", custTable.AccountNum, custTable.CustGroup));
	}

}

Complete example

You can run the class using the SysClassRunner class.
e.g. https://[your-environment].operations.eu.dynamics.com/?mi=SysClassRunner&cls=MyTestJob

public final class MyTestJob
{
    public static void main(Args _args)
    {
        CustGroupId custGroup;
        Dialog dlg = new Dialog('Enter customer group');
        DialogField fldCustGroup = dlg.addField(extendedTypeStr(CustGroupId));

        if (dlg.run())
        {
            custGroup = fldCustGroup.value();
            MyTestJob::showCustomerInfo(custGroup);
        }
    }

    //show customer info with optional argument
    public static void showCustomerInfo(CustGroupId _custGroup = '')
    {
        CustTable custTable;

        while select custTable
            where (_custGroup == '' || custTable.CustGroup == _custGroup)
        {
            info(strFmt("%1 - %2", custTable.AccountNum, custTable.CustGroup));
        }
    }

}