Set based operations for MUCH better performance

In Dynamics 365 Finance and Operations (F&O), updating or querying data is usually handled in X++ using a loop construct - for example, a while select statement or the QueryRun class (which also uses the same loop construct). Looping through rows of data and performing an operation on each row is very common in F&O and is also known as "row-by-row" operations.

Row-by-row operations are very flexible because, for each row, you can perform custom logic on the data. For example, you can loop through all customers and check their current balance, or loop through all customers and check their invoice aging to see if they have any outstanding amounts older than 60 days.

However, row-by-row operations handle each record separately and apply business logic to every single row. This adds significant overhead, which can make processing large datasets painfully slow. Performance also depends heavily on the complexity of the business logic. Complex logic often involves multiple calls to other methods or business logic layers, which can quickly create deep nested calls.

Set based operations

So, how can we handle large datasets and get good performance? With set based operations like insert_recordset, update_recordset and delete_recordset.

Set-based operations are executed with a single SQL statement and a single round trip to the SQL Server database. This is why they are so fast. Even if millions of records are affected, only one SQL statement is sent to the backend SQL Server database.

Falling back to row-by-row operations

The set based operation will fall back to a row-by-row operation in the following cases if:

• One of the data methods has been overriden on the target table:
  • insert()
  • update()
  • delete()
• The database log in enabled on the target table
• There is a delete action on the target table and you are using delete_from
• There is an alert setup on the target table
• The ValidTimeStateFieldType table property not equal to None

For example, let's say you are inserting data into a table using the insert_recordset statement, but the insert() method of the target table contains business logic. In this case, the system will automatically fall back to row-by-row operations. This means each inserted row will result in a separate SQL statement and round trip to the database server. As a result, there is significant performance overhead, and you gain little to no performance benefit compared to a regular while select loop.

Forcing set based operations

If for one or more of the reaons above, the set based operation is falling back to a row-by-row operation and you still want to force a set-based operation. You can use the .skip... methods on the target table buffer.

For example:.skipDataMethods(true) can be used if one of the insert(), update() or delete() methods have been overridden.

See the table below on how to handle each case. Although it was originally written for AX 2012, it remains fully relevant for Dynamics 365 Finance and Operations (F&O).

	DELETE_FROM	UPDATE_RECORDSET	INSERT_RECORDSET	ARRAY_INSERT	Use ... to override
Non-SQL tables	Yes	Yes	Yes	Yes	Not applicable
Delete actions	Yes	No	No	No	skipDeleteActions
Database log enabled	Yes	Yes	Yes	No	skipDatabaseLog
Overridden method	Yes	Yes	Yes	Yes	skipDataMethods
Alerts set up for table	Yes	Yes	Yes	No	skipEvents
ValidTimeStateFieldType table property not equal to None	Yes	Yes	Yes	Yes	Not applicable

Only the table above sourced from: Microsoft Learn - Maintain Fast SQL Operations

Conclusion

Set based operations are excellent for performance on large datasets but fall back to row-by-row operations if the target table insert, update and delete methods have been overridden. Or for one of the other reasons as mentioned above.

Important: carefully consider if you can skip any data methods on the target table because you risk breaking the data integrity of the target table.

Generating deep links

When a form is opened in Dynamics 365 Finance and Operations F&O the form displays all records or sometimes applies a filter if a default view is applied. It doesn't drill down to a single record. Sometimes you need a sharable URL link to drill down to a single record. E.g. a customer or vendor or it could be anything really. With deep links, you can drill down to a specific record in a form.

When the system generates a deep link of a specific record, it adds detailed query parameters to the URL. The extra parameters are readable in the unencrypted version above. The datasource name, table field name and filter value are specfied for the form to filter on. If the form menu item has the "Allow root navigation" and "Copy caller query" properties set to Yes, then the form will recognize the extra parameters and apply the filter automatically.

Unencrypted deeplink:

https://usnconeboxax1aos.cloud.onebox.dynamics.com/?cmp=USMF&prt=initial&mi=display:CustTable&q={"Parameters":[{"DataSource":"CustTable","FieldValues":[{"Field":"AccountNum","Value":"US-002"}]}]}

Encrypted deeplink:

https://usnconeboxax1aos.cloud.onebox.dynamics.com/?cmp=usmf&prt=initial&mi=display:CustTable&q=BgAAAKtjn9amNj9Ibe1SID5uUafqfbbsq2Of1qY2P0ht7VIgPm5Rp%2bp9tuymMl72yjSV2bDEqxFBm%2blUyTB3uTX5155lA1AAn%2fzUohRn52fhmJ7y%2fy6kGLVzYTxMde0xqPliZnUU28hblRZ%2b4CxkpeT1576UN2n89XUANdpPxJ4H2mcc0O9WCOdpgtWGEr5nC8hHV%2bvV4u4Pj5DVhdrcWyg7ipSdPfVtNtPO2EmMf2nQQMTSePLlPvKJq8AlHqakLjdeN74lqUCxqroB8l0VgiXfYVvUe9NXs8hwnX6MrP2EAaUdd4dzE7DaNJNuxQnRXbu2BbO0QaUyBERPZ0IofUO3M8G5M9JpePw7nBMvBrCnixIGg4tZMAoPK0cjEAycIvYzG1A4be%2bhTWe1odR6WtnF4B3J8syydPbHUkIqWlnltkJCUToiQ4FiOfhScK%2bzHsyv2%2b8WLpLG2fxrIJGR5DrBusC6PLy%2bWHfZvvHQnUfMZnbqknj6%2b%2fG2qOcE%2fK%2fP%2baAkbfxDeaLTjrgHx2Sbe6KB4eKIi%2fA8bb8VidM29KkXkjXfAWRe%2b6D%2fognEL0kcy1nnYHZKs%2bOEl32ofeMT7REdfmSVLPP7tonH1NgKo7r%2fA9qIR%2b11%2bzbAXGfc0U76ZphIlylscjXMkZGceDK5%2b8mXgHFjF8vtamNaH1uH%2byONKCYsDCq4kI4W%2bfM7XvTulFVWq7hWJb2hnPAsCkdJuWKdgWbgOs4ACSQvH3f63ZQtysryLH0yAsa8TQdmXYJ610xmskt0a6K4hC7ZHFHdfGPYyICwSza7wDH6mZJuWlnkdSKQVixmCV1HRkyjkVTg5fYTXZyfeWGHVpwf%2bBC3%2fp7NyD7wFsvP3im6S1Cdgu5eLIwVKRlZyaSj37SoylD5XwPQmmQXURi54I2f

The encrypted version hides the query information so it's safe to be used externally. The system encrypts the extra query parameters and this results in a unreadable and safe version of the URL.

Notes:

• The menu item used must have the following properties enabled:
• Allow root navigation = Yes
• Copy caller query = Yes
• The UrlUtility::getUrl() function does not work in batch. Do a search on an alternative way to get the system URL in batch.
• Use the SysEntityNavigation data entity to create deep links from external applications.
• The length of deep links URL are more than 1000 characters long and can vary. The length of the above deep link is 1063 characters.
• Only use unencrypted deeplinks for testing as they will result in the error message below.

Error message: The specified record query failed to apply and the can't be opened.

Full code example

public class MyDeepLinkTest
{
    public static str createDeepLinkUrl(MenuItemName _menuItemName, DataSourceName _keyFilterDataSourceName, FieldName _keyFilterFieldName, str _keyFilterFieldValue)
    {
        var generator = new Microsoft.Dynamics.AX.Framework.Utilities.UrlHelper.UrlGenerator();
        var currentHost = new System.Uri(UrlUtility::getUrl());

        generator.HostUrl = currentHost.GetLeftPart(System.UriPartial::Authority);
        generator.Company = curext();
        generator.MenuItemName = _menuItemName;
        generator.MenuItemType = MenuItemType::Display; //avent: display only.
        generator.Partition = getCurrentPartition();
        generator.EncryptRequestQuery = true; //avent: set to false for testing only.

        //repeat for each datasource to filter
        var requestQueryParameterCollection = generator.RequestQueryParameterCollection;
        requestQueryParameterCollection.AddRequestQueryParameter(_keyFilterDataSourceName, _keyFilterFieldName, _keyFilterFieldValue);

        System.Uri fullURI = generator.GenerateFullUrl();
        
        return fullURI.AbsoluteUri; //to get the encoded URI
    }

    public static void main(Args _args)
    {
        str link = MyDeepLinkTest::createDeepLinkUrl(menuItemDisplayStr(CustTable),
            formDataSourceStr(CustTable, CustTable),
            fieldstr(CustTable, AccountNum),
            "US-002");

        Dialog dlg = new Dialog();
        dlg.addField(extendedTypeStr(Notes)).value(link);
        dlg.run();
    }

}

Result of running above test code to create a deep link

Using the deep link

Edit: (untested) alternative method for getting the host URL. Source community.dynamics.com.

IApplicationEnvironment env = EnvironmentFactory::GetApplicationEnvironment();
str currentUrl = env.Infrastructure.HostUrl;
System.Uri currentHostUrl = new System.Uri(currentUrl);
 
UrlGenerator urlGenerator = new UrlGenerator();
urlGenerator.HostUrl = currentHostUrl.GetLeftPart(System.UriPartial::Authority);

Getting stack trace information using xSession::xppCallStack()

From the battle hardened trenches of the F&O development front lines, I present to you a small code nugget that will help you save time and troubleshoot your custom code.

I am actually talking about the standard system method xSession::xppCallStack(). It retieves the current execution call stack. If combined with an error message, it provides useful information for troubleshooting custom code running in production.

The code example below, is just an example of using the standard system method xSession::xppCallStack() to retrieve the current execution call stack of the code being executed. Having the execution call stack information saves a lot of time because you can see where error message originates from.

public static void throwError(str _error)
{
	container callStack = xSession::xppCallStack();
	conDel(callStack, conLen(callStack), 1); //remove last one.
	throw error(_error + '\n' + con2Str(callStack, '\n'));
}

How to effectively detect if data has changed using RecVersion

Sometimes there is a requirement to effectivle detect if data has changed in a table. The first thing that might come to mind is, why not just use the "ModifiedDateTime" field of the data? Well, that's perfectly fine if don't have a lot of data in your table.

But, what if there is a lot of data in our table? Like millions of records? Filtering on an unindexed field like "ModifiedDateTime" and with a lot of data, will force the database server to scan all the records in the table. The database server has to do this to determine which records are modified after the specified date and time. Table scans on large tables is not effective and painfully slow because it's not using any of the table indexes.

RecVersion

So, what's the solution? How do we effectively detect which rows have changed in our data? By using the system field RecVersion. Every table in Dynamics 365 Finance and Operations (F&O) has a field called RecVersion.

Every time a record is changed in a table, the RecVersion of the record is updated by the system to a new value, or version. The RecVersion field contains the record version of the record.

But, there is a small problem, how do know what the previous version of RecVersion was before the data was changed? Well, we don't. But luckily there is a simple way so solve this. By creating a custom table with the RecId and RecVersion of our data.

Steps

• Create a custom table with a reference RecId and RecVersion field to our data. Field names: RefRecId and RefRecVersion
• In code, create a method to create a "snapshot" of the RecId and RecVersion values of the data. Use insert_recordset to copy the data.
• Query your table with a join to the custom table. Join on YourTable.RecId and YourTable.RecVersion and SnapShotTable.RefRecId and SnapShotTable.RefRecVersion

To get all the data from the target table that has changed, join the two tables on RecId and RecVersion. The records where RecVersion is different, has changed.

Base64 encoding and decoding

What is Base64 encoding and decoding? On a very high level: it is a method of converting binary data to ASCII text and the ASCII text back again to binary. There are many articles online on how this works technically but this article will explain the basics and how Base64 encoding and decoding is used in Dynamics 365 Finance and Operations (F&O).

For example, if you attach a spreadsheet or other binary file like a PDF document to an email. The binary file is converted to ASCII text using Base64 encoding and is embedded in the message body of the email as text. When the email message is received by the recipient, the email application used to view the email message, detects that there is a file attached embedded as text and converts the text back to binary so that the attachment can be downloaded as a file. Most email applications support previewing attachments, but will have to convert the attachments from text to binary on the fly before being able to display the contents of the file.

So, how does Base64 encoding and decoding relate to F&O? It is frequently used in the standard application and also by developers when using the SysOperation framework just to mention just a few examples.

Code example

In a SysOperation contract class, the SysOperation framework allows a Query object to be used to query data for the operation. Because the related operation can also be run in "batch", in other words in "unattended" mode without an user interface, the Query object of the class needs to be saved to the database so that the operation can be executed by the batch framework and use the exact same query object that was specified by the user when adding the operation to batch. In other words, the system saves all query filters, joins and other changes to the query and will then use your exact query when executing in batch.

An instance of an class object cannot be saved to the database. So, how does the system save the object to the database? By using Base64 encoding and decoding. The Query object of the contract class is serialized to a binary object using the "pack" method of the Query class. This method returns a binary representation of the query in a container type. All class variables of the Query object are serialized and converted to a binary object (container). The resulting binary object of the Query object is then Base64 encoded to text. This text can be saved to the database as a regular text field.

When the Query object instance is required by the SysOperation framework, the process is reversed. The text is Base64 decoded to binary, the original Query object with all the specified filters and other changes can then be instantiated by using the binary data to construct a Query object.

1. Class variable for the Base64 encoded text.

2. Code that converts the text back to binary object.

3. Code that converts the binary object to text.

Using Boolean OR to make where clause fields optional

Sometimes you have a custom "while select" statement and you need to have one or more of the where clause fields in the statement to be optional. In this example, I will use a simple method to display the customer account and group for customers.

The method below displays the customer accounts for customers and uses the _custGroup argument to filter customers that match the customer group. The method works as expected but there is one drawback: it can only show customers for the specified customer group.

So, what if we want to show info for all customers with this method? Can't we just call the method and pass an empty string? The answer is yes, that's possible but will not work as one would expect. The problem with this is, that if you call the method and pass an empty string for the customer group, the while select statement will return all customers where the customer group is empty. Since the customer group field is mandatory, and all customers have a customer group, the query will return nothing.

Show customer info method

//show customer info with mandatory argument
public static void showCustomerInfo(CustGroupId _custGroup)
{
    CustTable custTable;

    while select custTable
        where custTable.CustGroup == _custGroup
    {
        info(strFmt("%1 - %2", custTable.AccountNum, custTable.CustGroup));
    }

}

How do we solve this problem? Your first thought might be to simply duplicate the while select statement in the method for each argument used in the where clause. In this example two statements, one statement with, and another statement whithout the customer group where clause. And simply add an "if" statement to check if the customer group argument has a value or not. Yes, this will work but this is definitely not the way to go. The reason is because if you have a large while select statement with many lines and a method with multiple arguments. The code will be hard to read and error prone when modifications are done to it.

Optional argument

So, what is the solution? By using the Boolean OR operator in the while select statement. The where clause for customer group is extended with an Boolean OR to contain two parts. The first part of the Boolean OR evaluates the method customer group argument variable, if this part of the Boolean OR evaluates to TRUE, the second part of the OR is simply ignored, effectively ignoring the where clause part for customer group in the second part. However, if the first part of the Boolean OR statement evaluates to FALSE, the second part of the boolean OR is then evaluated which effectively then applies the where clause of the customer group to the statement. Making the where clause optional and having compact and readible code.

//show customer info with optional argument
public static void showCustomerInfo(CustGroupId _custGroupId = '')
{
	CustTable custTable;

	while select custTable
		where (_custGroupId == '' || custTable.CustGroup == _custGroupId)
	{
		info(strFmt("%1 - %2", custTable.AccountNum, custTable.CustGroup));
	}

}

Complete example

You can run the class using the SysClassRunner class.
e.g. https://[your-environment].operations.eu.dynamics.com/?mi=SysClassRunner&cls=MyTestJob

public final class MyTestJob
{
    public static void main(Args _args)
    {
        CustGroupId custGroup;
        Dialog dlg = new Dialog('Enter customer group');
        DialogField fldCustGroup = dlg.addField(extendedTypeStr(CustGroupId));

        if (dlg.run())
        {
            custGroup = fldCustGroup.value();
            MyTestJob::showCustomerInfo(custGroup);
        }
    }

    //show customer info with optional argument
    public static void showCustomerInfo(CustGroupId _custGroup = '')
    {
        CustTable custTable;

        while select custTable
            where (_custGroup == '' || custTable.CustGroup == _custGroup)
        {
            info(strFmt("%1 - %2", custTable.AccountNum, custTable.CustGroup));
        }
    }

}