Alexa, Field Service and Me (Part 5) – Using Azure Service Bus

In my previous post I walked through swapping out Power Automate with Azure Functions for responding to Alexa in our Field Service scenario.

This post is about finalising the code by using Service Bus function to create the Work Order.

In the Power Automate version I used a child flow call to allow the response to happen quickly, without waiting for the numerous calls to establish data then creating the record. This is standard, good practice to improve the response time.

Create the Queue

Firstly, over to portal.azure.com & create a service bus. The service bus is the messaging underpinning for our queue.

Next create a queue within the service bus. Queues are a list of things that your code needs to process. Functions can then be subscribed to the queue to be triggered when something enters the queue

Adding to the Queue

Back in our first function, you only require a few lines to add an item to the queue. The first line gets an environment variable like previously, from the local json file or the application settings within the Azure function.

The object I create here is to simply pass all the details I need to the queue.

var queueClient = (IQueueClient)new QueueClient(Environment.GetEnvironmentVariable("ServiceBusConString"), Environment.GetEnvironmentVariable("queueName"), ReceiveMode.PeekLock, null);
var fsObject = new
{
  email = emailAddress,
  contactId = (Guid)jContactResponse["value"][0]["contactid"],
  intent,
  date,
  device
};
Task messageReturn = SendMessagesAsync(JsonConvert.SerializeObject(fsObject).ToString());

I then call the function SendMessagesAsync, converting the object to a string as I do it. The SendMessageASync is below

private static async Task SendMessagesAsync(string messageToSend)
{
  try
  {
    Message message = new Message(Encoding.UTF8.GetBytes(messageToSend));
    Console.WriteLine("Sending message: " + messageToSend);
    await queueClient.SendAsync(message);
    message = null;
  }
  catch (Exception ex)
  {
    Console.WriteLine(string.Format("{0} :: Exception: {1}", DateTime.Now, ex.Message));
  }
}

Call the method uses an Asynchronous call which I don’t wait for the return. I just assume it works and get on with responding to the user.

Reading the Queue

To read the object from the queue, you need to register the function as a subscriber to the queue.

To do this, the function trigger needs to have a certain format

[FunctionName("AddToFS")]
public static async void AddToFS([ServiceBusTrigger("ccalexa", Connection = "ServiceBusConString")] string myQueueItem, ILogger log, ExecutionContext context)
{

The parameters to the function connect to a queue called ccalexa & a service bus indicated in the application variable “ServiceBusConString”. This signature shows that Microsoft is thinking about moving between environments from the start.

The next part of the function defines the parameters for the call to D365. This leads to parsing the object that is being found on the queue.

JObject woObject = JObject.Parse(myQueueItem);

Guid contactId = (Guid)woObject["contactId"];
var email = woObject["email"];
var intent = woObject["intent"];
string date = (string)woObject["date"];
string device =  woObject["device"].ToString();

Once we have the detail of the item sent in, we can go to D365 and retrieve some records we need to create the Work Order, firstly to the Contact, to retrieve the account associated with it.

var contactResult = await d365Connect.GetAsync("api/data/v9.1/contacts(" + contactId + ")?$select=_parentcustomerid_value");
if (!contactResult.IsSuccessStatusCode)
{
  return;
}

If the return is not success, something went wrong. Forgive me for not doing some proper error trapping here. Next, we work to get the Default Pricelist, from the account & work order type from the intent passed in

JObject contactObject = JObject.Parse(contactResult.Content.ReadAsStringAsync().Result);
var accountId = contactObject["_parentcustomerid_value"];
HttpResponseMessage accountResponse = await d365Connect.GetAsync("api/data/v9.1/accounts(" + accountId.ToString() + ")?$select=_defaultpricelevelid_value");

JObject jaccountResponse = JObject.Parse(accountResponse.Content.ReadAsStringAsync().Result);

Guid priceListId = (Guid)jaccountResponse["_defaultpricelevelid_value"];

HttpResponseMessage woTypeResponse = await d365Connect.GetAsync("api/data/v9.1/msdyn_workordertypes()?$select=msdyn_workordertypeid&$filter=cc_alexaintent eq '" + intent + "'");

JObject jwotResponse = JObject.Parse(woTypeResponse.Content.ReadAsStringAsync().Result);
Guid woTypeId = (Guid)jwotResponse["value"][0]["msdyn_workordertypeid"];

Next, we build up the object to add as a new work order. Line 2 shows binding to a pricelist record. This method is used for type & account too. I also generate a random number for the name to keep consistent with the Flow version.

JObject workOrder = new JObject();
workOrder.Add("msdyn_pricelist@odata.bind", ("/pricelevels(" + priceListId + ")"));

workOrder.Add("msdyn_name", ("AZ" + new Random().Next(4000, 500000)));
workOrder.Add("msdyn_serviceaccount@odata.bind", ("/accounts(" + accountId + ")"));
workOrder.Add("msdyn_systemstatus", 690970000);

workOrder.Add("msdyn_workordertype@odata.bind", ("/msdyn_workordertypes(" + woTypeId + ")"));
workOrder.Add("msdyn_taxable", false);

if (date != string.Empty) workOrder.Add("msdyn_timefrompromised", date);
if (device != string.Empty) workOrder.Add("msdyn_instructions", device);

log.LogInformation(workOrder.ToString());

HttpRequestMessage createWO = new HttpRequestMessage(HttpMethod.Post, d365Connect.BaseAddress.ToString() + "api/data/v9.1/msdyn_workorders");

createWO.Content = new StringContent(workOrder.ToString(), Encoding.UTF8, "application/json");

HttpResponseMessage createWOResp = d365Connect.SendAsync(createWO, HttpCompletionOption.ResponseContentRead).Result;

Finally a post method to the msdyn_workorders entity pushes this as a new work order into the system.

Connecting Alexa to our Function

This is the simplest bit. In the original set of posts, I talked about endpoints. The endpoint needs swapping to the function call.

The URL is retrieved the same way as I did in my demo, from the Azure Function properties in the Azure portal.

Update the Service Endpoint & you are good to go.

The results

When I started this challenge, I wanted to compare and contrast the response time between Flow and Functions. I would assume Functions would be quicker, but what would the difference be?

Caveats here – Both the flow and the function are my code. I am sure that there are better ways of doing both. Both follow each other functionally so it is a fair comparison.

To test, I swapped the end point and then did 5 runs to “warm up the code”. I found , particularly Azure function, took a while to come up to speed. This can be explained by cold starting of functions which will be the case in our scenario. Even flows run faster the second time through.

I then ran the test cycle 10 times then used the monitoring within Alexa to monitor response time. Both sections I checked I was getting work orders being created correctly.

The first blip in the chart is the Flow configuration. This has a P90 (90 percent of the requests where responded within this time) of over 4 seconds. It drops to a more respectable 1 second as the Flow is warmed up.

The second blip is when the configuration swaps to Azure Function. You can see this has a peek around 2 seconds for the first call. then dropping to 300ms for each subsequent call. This is a vast improvement for the responsiveness of your app.

Don’t get me wrong, I am not telling you to revert everything to a Function, it is using the right tool for the job.

Power Automate got me this far, it is great for scenarios where you don’t need an immediate response, it is great to prove that developing Alexa into your toolset is beneficial, but if you want to get serious about the user experience, in this instance, a Function serves you better.

Alexa, Field Service and Me (Part 3) – Creating Work Orders

This is a continuation of my series on a proof of concept to allow Alexa to interact with D365 Field Service

Objectives

  • The Business scenario (Part 1)
  • Create an Alexa Skill (Part 1)
  • Connect the Skill to D365 (Part 2)
  • Use Field Service to book an appointment (This Part)
  • Return the information to Alexa (Part 2)

In this final (unless I need to expand on the scenarios) part of the story, Flow will be used to take the information garnered from the user and create a work order.

Calling a Sub-flow

The original flow was all about Alexa, but what about other voice assistants? Flow is no different that any other programming languages and as such we can use the same concepts to writing decent flows, one of them being re-usability. If the code to write the Work order and book it is generic, it can be re-used when Google or Siri is connected to the application.

To this end, the Flow starts with another HTTP trigger, which is called from the previous flow.

Just like when connection Flow to Alexa, the URL is required in the child to update the caller. Create a new Flow, using the When a HTTP request is received. As content to the JSON Schema, add just enough to get you going and use the output to send an email so that the schema coming from the parent flow can be seen. This is a repart of the logic used to start our parent flow.

Once saved, the URL will be generated in the trigger step. Use this to call the child Flow from the parent. This is the HTTP action. The Method is a POST, the URI is the trigger URL defined in the child Flow. No headers are required. In the body, add in the things that are already known, just to ensure repeat calls to D365 are made. The Intent is also passed in, which has all the details about what the user wanted.

Now ready for testing, ensure both Flows are in Test mode and trigger Alexa. After a little delay, an email should be sent in the child Flow detailing enough information to create a Work Item.

Use this email to populate the JSON as previously, easily creating the schema that is required.

Creating a Work Order

Next, get the Account the contact is associated with. This is done with a call to the D365 Instance using the Common Data Service Connector.

A Work Order needs some fields to be set before it can be save, Work Order Type is one of them. This could be hard coded, but Alexa has supplied the intent. To match the Work order type with the intention in Alexa, a field on the Work Order Type was added, Alexa Intent, which is searched for in the CDS List Records action.

To make the Flow easier to manage and reduce the dual looping, the Work Order Type is returned to a variable

Once the data for the new Work Order is available, create the Work Order using the CDS Create Record connector.

Most of these are obvious, but the one that is not is the Work Order Number. In Field Service, this is automated, with a prefix and a number range. As this doesn’t work in Flow, a work number is generated using a random number, using an expression.

rand(10000,100000)

A few other parts of the work order are populated, helping the service manager to match the time as appropriate.

Alexa, Field Service and Me (Part 2) – Linking to Flow

This is a continuation of my series on a proof of concept to allow Alexa to interact with D365 Field Service

Objectives

  • The Business scenario (Part 1)
  • Create an Alexa Skill (Part 1)
  • Connect the Skill to D365 (This part)
  • Use Field Service to book an appointment (Part 3)
  • Return the information to Alexa (This part)

In this post I will be linking Alexa to D365 and returning some information back to the end user.

Receiving information from Alexa

Alexa interacts with the outside world with a HTTPS request. Once Alexa has determined that it understands the user and they have asked for something of your skill, it posts to the web service you have configured.

That API Guy had a great post where he links his Alexa to his O365 email account and has Alexa read out his new email. This article steps through linking Alexa and Flow, and it showed me how simple that part of the integration is. Microsoft has done the hard work by certifying it’s connections, you just need to create one.

Flow provides several methods of subscribing to HTTP events, but the one we are interested in is at the bottom

The URL is generated for you, and is the bit you need to post into Alexa Skill configuration once we have created the Flow. In the JSON schema, we just want a stub for now. The schema is defined by Alexa and how you configure the skill. It is best to get the connection up and running and use the initial call to substitute the schema later.

Every Flow needs at least one action to be able to save, so to aid testing and understand the JSON sent by Alexa, the first step is to send an email to myself with the content of the call.

Saving the flow and go back to the trigger

A specific trigger URL for our Flow is now generated.

Back in Alexa

In Alexa, on the left in the Skill is Endpoints. Alexa allows different endpoints to the skill depending on the Alexa region as well as a fall back. As this is a POC, using the default is appropriate.

The important part of this is the drop down below the URL copied from Flow, this needs to be the second option “My development endpoint is a sub-domain of a domain that has a wild card certificate from a certificate authority“. Basically, Microsoft has certified all their Flow endpoints with wild card certificates, allowing Amazon to trust that it is genuine.

One saved, Build your skill again. I found every time I touched any configuration or indeed anything in Alexa, I needed to rebuild.

Testing

Ready to test. Alexa allows you to test your connection via the Test tab within your skill.

You have to select Development from the drop down. You can also use this interface to check Production skills.

A word of warning, everytime you build, to test that new build, I found I had to re-select Development in this window by toggling between “Off” and “Development”.

Back in Flow, test your flow by confirming that you will conduct the action.

In the Test panel in Alexa, enter a phrase with your Invocation and Utterance in Alexa entry and if Alexa understands it is for your invocation, your flow should be triggered! Alexa will complain, as our simple flow hasn’t returned anything to it. We’ll worry about that later.

In our simple Flow, I used email as the action, as can be seen below.

This is the raw JSON output of the Alexa skill and this is used to tell Flow what to expect. This way it will provide the rest of the Flow with properties that are more appropriate.

Back in the Flow trigger, select Use sample payload to generate schema, which presents the text entry, where you paste in the email body.

Flow does the hard work now and presents you with a schema for what Alexa is sending your Flow

Authenticating the user

Taking triggers from Alexa, hence a user is all well and good, but they expect a rapid response. Alexa Skills are public domain once you publish it, anyone can enable your skill, hence a little work needs to be done to understand who is calling our skill and whether Big Energy Co supports them.

Which User is it?

The request from Alexa does include a unique reference of the user, but normally businesses like Big Energy Co work on email addresses, which you can get from Alexa, but the user has to give you permission. This can be pre-approved when the end user installs the skill or you can ask for approval.

Alexa needs to be told that your skill would like this permission. This allows Alexa to request this permission for you when your skill is enabled by the user

On the left hand menu, there is a Permissions link, where our skill asks for the email address

To ask Alexa for an users email address is a seperate web call with properties from original message sent from Alexa in the trigger.

Alexa provides each session an access token so that Flow can ask for more information from Alexa in the context of the users session, location, names, email etc. As this is a globally distributed system, the api End point can vary depending on where the user started their session. The URI at the end asks for the email address of the user of the session.

Alexa may refuse to return the email address, because the end user has not given permission for our skill to share this information. If the call to get the email was a success, it means that Alexa has returned the value and the Flow can continue

Without the permission, the call fails and so the Flow captures this. Remember to set a run after for this step to ensure it still runs on failure. Otherwise the flow will fail not so gracefully.

Alexa provides standard permission request functionality, the Flow responds to the original call with this detail in a JSON formatted string, with the permissions the Skill wants listed. Finally, for this failure, let the Flow terminate successfully. The Flow can’t continue without the information.

Does Big Energy Co support the user?

It is not enough that the user has asked us for help, they need to be known to Big Energy, as a contact. Querying D365 for this data is the next step

Using the D365 connector, query the contact entity for the email address that Alexa has given us.

A conditional flow checks to see if the return from the Retrieve Contact step has a single record using an expression below

length(body('Retrieve_Contact')?['value'])

Responding to the user, quickly

Like any Alexa Skill, the user expects an immediate response. Flow in itself is quick, but when it comes to updating or creating records, it can take seconds. The timeout for your response to Alexa is 10 seconds, which isn’t a lot when you want to look up several things to create the appropriate work order.

To get around this, respond to the user once you know you have all the information you need to create the appropriate records in D365. Here, the Flow responds if the contact is in our database. In production, you could do some more checks or just assume that if the contact is known, the logic could create an opportunity to sell them a contract if nothing else. Equally, terminate gracefully after responding that Big Energy Co doesnt know the customer, prompting them to ring the service desk.

In the switch statement, a response to the user is built up. Specific, personalised responses using the data you have retrieved is essential to give the customer an understanding that you have received the request and will respond.

The responses are short and to the point but personalised to the customer and what they asked for with some expressions to add more information if they told Alexa.

This snippet adds after “request for a Service” what the user asked for a service on, relying on the JSON formatted values.

if(equals(triggerBody()?['request']?['intent']?['slots']?['device']?['value'], ''), '', concat(' for your ',triggerBody()?['request']?['intent']?['slots']?['device']?['value']))

This snippet adds to the end a sentence including the date that the user asked for.

if(equals(triggerBody()?['request']?['intent']?['slots']?['date']?['value'],''),'',concat( 'We will endeavour to send an engineer on ',triggerBody()?['request']?['intent']?['slots']?['date']?['value']))

Finally, for the Alexa response part, the Flow returns a response to the user. This combines the body with a little standard text. This is what Alexa will say. You can also add a response to the screen for those devices able to do that.

The final part of this flow goes on and creates the work order. I seperated out the flows using a sub flow, which is discussed in the next part of the blog.

Alexa, Field Service and Me (Part 1)

As we all now have smart devices in our home, linking them to business applications could be a key differentiator between winners and the also rans. This series of posts will demonstrate how I connected Alexa to D365 Field service.

Objectives

  • The Business scenario (this part)
  • Create an Alexa Skill (this part)
  • Connect the Skill to D365 (Part 2)
  • Use Field Service to book an appointment (Part 3)
  • Return the information to Alexa (Part 2)

Our Scenario – Big Energy Co

Our generic big energy company is diversifying into support and maintenance of home energy products, boilers, central heating, plumbing, electricals and numerous other aspects of a consumer’s home life. A client will ring up, tell the support desk that they have an issue with the appliance and the employee will book a suitable engineer in to come out to their home. This is all done via Field Service in D365. Big Energy also have scheduled servicing of Boilers on an annual basis.

What the CTO wants to do is embrace the in home virtual assistant to promote Big Energy as a forward thinking organisation which is at the forefront of technology to allow Big Energy’s customers to book an engineers visit via their smart device. Her expectation is that Alexa, Google Home or Siri will allow a service call to be booked without talking to the support desk. This will allow meaningful interactions with their customers 24/7.

The proof of concept will start with the front runner in the war of virtual assistance, mostly because I have one.

Alexa

If you have read my introduction to LUIS in Alexa Skill concepts are a mirror of the concepts introduced in LUIS. Alexa has Utterances & Intents. Entities are called Intent Slots for Alexa. Alexa has also got another concept, that being the Invocation.

First off, get yourself an Amazon account & sign up for the developer program at developer.amazon.com. This is all free. People can buy skills, but not sure Big Energy’s customers would think this is appropriate.

Use the Create Skill button, enter a name & select a language. I also choose Custom & Provision your own to allow us to send data to Flow which is hosting our integration.

Then I select Start from scratch, none of the others seem to match our scenario

Invocation

An Invocation is the starting point and is the differentiator between you and every other Alexa skill out there. It is effectively a name for your skill, what the user would say to call your skill. I have slightly altered my invocation to put some spaces in there so it is more natural to the end user.

Intents

Like in LUIS, intents are a category or what the user is looking for or asking. You can have many intents per skill. In our scenario around home appliances the user can ask for a service, a repair or an emergency as a starting point. Let’s start with a request for service.

Utterances

Utterances are samples to train Alex to understand the intent in natural language. Add as many utterances as you like as samples of how a person would ask for a service or repair, but ensure they are different.

Slots

In my sample utterances above you can see I have added Slots. Slots are the same as entities in LUIS, data that the user gives us when they are saying their utterance in addition to the type of thing they want.

Each Slot has a type, and I have used the default Amazon types except for device, which is a custom one.

Slot Types

Amazon has 43 of it’s own list types or 6 built in types for numbers & dates, but devices are not in the list. I want to know what type of thing the engineer is going out to fix, not sure I need to be that specific, but would be good to know if I need to send an electrician, gas engineer or plumber. I have added my own Slot Type, called it device and I now list the values I expect.

You also should also enter synonyms, not everyone calls it a telly, just us northeners.

Once you have entered enough intents & utterances, time to build and test. A nice message lets you know when it is done building. Ready for the utterance profiler or testing

Utterance Profiler

In the top right, is a pop out to test your utterances. Enter an utterance you haven’t used before to check you are getting the expected results.

The text I wrote is in blue, it has decided that this is a service intent, with a device of boiler & the day of wednesday.

You can carry on fine tuning your utterances, intents and slots to get the most accurate model. Like any language understanding, this will be an ongoing model.

This is Alexa done, we have configured everything we need to in Alexa, apart from our link to Flow. This needs a bit of pre-work in Flow to activate, in the next post.