Chapter 6 EFM Configuration

Prerequisites

Ensure you have completed all the steps in the following documents before continuing:


Allow the IR829 Roadway Broker to Connect Upstream

  1. Open the regional broker EFM System Administrator web interface here and log in as the efmAdmin user with the password that was set during installation

The “radiation”-style warning in the upper right-hand of the page indicates that there are downstream brokers in quarantine.

  1. Navigate to Permissions > Quarantine > Authorize
  2. Select the IR829 roadway broker from the list and click the Authorize button at the bottom

You should now see the IR829 roadway broker efm-roadway1 listed underneath the Root Broker in the Brokers section on the left-hand of the page.

  1. Select efm-roadway1 from the Brokers section


Add DSLinks to the Roadway Broker

MQTT DSLink

  1. Navigate to Management > links > Install Link
  2. Select MQTT from the link dropdown
  3. Enter mqtt as the name and click Invoke

It should say success True if the operation was successful.

  1. Navigate to Management > links > mqtt > Start Link
  2. Click Invoke


C-Based Serial DSLink

  1. Navigate to Management > links > Install Link from Zip
  2. Click Choose File and select the dslink-c-serial_0.8.0_x86_64.zip DSLink file

  1. Enter serial as the name and click Invoke

It should say success True if the operation was successful.

  1. Navigate to Management > links > serial > Start Link
  2. Click Invoke


Configure Roadway Broker DSLinks

MQTT

  1. Open the regional broker EFM Dataflow Editor web interface here and log in again as the efmAdmin user, if prompted
  2. Navigate to Data > downstream > efm-roadway1 > downstream > mqtt
  3. Under Metrics right-click on Broker Enabled: false, set the value to true, and click Invoke

  1. Right-click on mqtt and select Add Server
  2. Use the following options:

  1. Expand ... > mqtt > local-broker > Subscriptions

This ... > convention means to stay where you were in the Data section, but to dive deeper into the hierarchy. It saves rewriting the entire path when you are not changing your broker focus. In this case, the ... means Data > downstream > efm-roadway1 > downstream

  1. Right-click on Subscriptions and select Subscribe
  2. Use the following options:

  1. Subscribe to another MQTT channel with the following options:

The active MQTT channel subscriptions will show up under the Metrics section.

  1. Select ... > efm-roadway1 > downstream > dataflow
  2. Right-click on dataflow and select Create Dataflow
  3. Name the dataflow clear-local-mqtt-logs and click Invoke

  1. Expand ... > dataflow > clear-local-mqtt-logs
  2. Open the clear-local-mqtt-logs dataflow in the right-hand editor workspace by clicking on its dataflow icon

  1. Expand ... > efm-roadway1 > sys > links > mqtt
  2. Right-click on mqtt and drag Log > Clear to the editor workspace Dataflow section

This is a process you will become very familiar with: dragging various functional “blocks” to the editor workspace to build a dataflow, editing their properties, and connecting their inputs and outputs. This is essentially graphical programming and as you will see can perform fairly complex logic and data operations.

  1. Double-click on the new block and rename clear_log to clear-mqtt-log
  2. Edit the following values in the right-hand Properties section:

As you likely have already figured out, this clear-local-mqtt-logs dataflow will clear the MQTT log every 10 minutes. Without this in place the default MQTT DSLink log level would eventually fill up the small amount of storage of the IR829 EFM instance.


You can see this log in the EFM System Administrator under the Life cycle section for the efm-roadway1 broker’s mqtt link.

  1. Close the clear-local-mqtt-logs dataflow by clicking the x icon next to its name above the editor workspace


Serial

  1. Navigate to ... > efm-roadway1 > downstream > c-serial
  2. Right-click on c-serial and select add connection
  3. Use the following options:

  1. Select ... > efm-roadway1 > downstream > dataflow
  2. Right-click on dataflow and select Create Dataflow
  3. Name the dataflow clear-local-serial-logs and click Invoke

This is another dataflow that will clear a DSLink log. This time it’s for the Serial DSLink, but the reason and process is the same.

  1. Expand ... > dataflow > clear-local-serial-logs
  2. Open the clear-local-serial-logs dataflow in the right-hand editor workspace by clicking on its dataflow icon
  3. Expand ... > efm-roadway1 > sys > links > serial
  4. Right-click on serial and drag Log > Clear to the editor workspace Dataflow section
  5. Double-click on the new block and rename clear_log to clear-serial-log
  6. Edit the following values in the right-hand Properties section:

Note that this time you are setting the log to clear every hour instead of every 10 minutes like with the MQTT log. This is to aid in potential serial connectivity troubleshooting.

  1. Close the clear-local-serial-logs dataflow by clicking the x icon next to its name above the editor workspace

Create DOT Demo Dataflows on the Roadway Broker

Make sure the RWIS/Car Simulator and DMS are connected and running before starting this section.


Publish Local RWIS Data

  1. Select ... > efm-roadway1 > downstream > dataflow
  2. Right-click on dataflow and select Create Dataflow
  3. Name the dataflow publish-local-rwis and click Invoke
  4. Expand ... > dataflow > publish-local-rwis
  5. Open the publish-local-rwis dataflow in the right-hand editor workspace by clicking on its dataflow icon
  6. Expand ... > efm-roadway1 > downstream > mqtt > local-broker > Data > weather

  1. Under the Metrics section, drag roadway-rwis1 to the editor workspace Dataflow section

If you don’t see any data listed under Metrics from roadway1-rwis1 go back to the Raspberry Pi RWIS and Car Simulator document and review the troubleshooting section. Make sure you have correctly set up the MQTT DSLink and MQTT configuration as detailed above as well.


If you do see roadway-rwis1 data listed, you probably noticed that it is changing every 3 seconds. The EFM Dataflow Editor shows data updates in realtime.

  1. Double-click on the new block and rename roadway1_rwis1 to roadway1-rwis1-mqtt
  2. In the right-hand Properties section, mouse over the lastUpdate field until a blue dot appears

  1. Click on the blue dot and check the Pinned box

Pinning a field adds it to that block’s listed fields in the Dataflow section so it is easy to link it to an input field of other blocks.

  1. Navigate to ... > efm-roadway1 > data
  2. Right-click on data and drag publish to the editor workspace Dataflow section

  1. Select the new block and edit the following values in the right-hand Properties section:

Make sure you are updating the Path property toward the bottom and not the path property near the middle of the Properties section. The Path is how other EFM brokers are able to subscribe to the roadway broker’s published data.

  1. Link the roadway1-rwis1-mqtt value field to the publish Value field by dragging the red arrowhead from one block to the open spot on the other

If you ever accidentally link the wrong fields, you can simply double-click the arrowhead and the link will be removed.

  1. Link the roadway1-rwis1-mqtt lastUpdate field to the publish Timestamp field

You can confirm that the data is successfully published by navigating to ... > efm-roadway1 > data and expanding the weather > roadway1-rwis1 entry under the Metrics section. The data values should change every 3 seconds, just like the source data. This is an example of how EFM can publish data upstream as it is received locally without modification. The next dataflow will add some additional filtering logic to its publish activity.

  1. Close the publish-local-rwis dataflow by clicking the x icon next to its name above the editor workspace

Publish Vehicle Safety Reports

  1. Select ... > efm-roadway1 > downstream > dataflow
  2. Right-click on dataflow and select Create Dataflow
  3. Name the dataflow publish-crash-reports and click Invoke
  4. Expand ... > dataflow > publish-crash-reports
  5. Open the publish-crash-reports dataflow in the right-hand editor workspace by clicking on its dataflow icon
  6. Expand ... > efm-roadway1 > downstream > mqtt > local-broker > Data > vehicles

  1. Under the Metrics section, drag roadway-rsu1 to the editor workspace Dataflow section

Once again, if you don’t see any data listed under Metrics from roadway1-rsu1 go back to the Raspberry Pi RWIS and Car Simulator document and review the troubleshooting section as well as the MQTT setup sections above.

  1. Double-click on the new block and rename roadway1_rsu1 to roadway1-rsu1-mqtt
  2. In the right-hand Properties section, mouse over the lastUpdate field until a blue dot appears
  3. Click on the blue dot and check the Pinned box
  4. Under the Blocks section, expand Table Operations
  5. Drag JSON Parser to the editor workspace Dataflow section

This block will parse the JSON data that is received over MQTT and convert it into table format. Take a minute to look at all the different blocks that are available for dataflow logic and processing. As you create progressively more complex dataflows below you will become familiar with many of these blocks.

  1. Double-click on the new block and rename jsonParser to vehicle-table
  2. Link the roadway1-rsu1-mqtt value field to the vehicle-table input field
  3. Navigate to ... > efm-roadway1 > data
  4. Right-click on data and drag publish to the editor workspace Dataflow section
  5. Select the new block and edit the following values in the right-hand Properties section:
  1. Mouse over the enabled field in the Properties section until a blue dot appears
  2. Click on the blue dot and check the Pinned box
  3. Select vehicle-table and click on the Table button for the output field in the Properties section to see the underlying table data

  1. Drag the value in row 0 for the crashed column to the publish enabled field

The resulting field input indicator for the publish block will be gray with a circle indicating it’s from an external resource that can’t be easily represented graphically within the dataflow editor.

This is also how the logic of this dataflow works. The vehicle data is only published when the crashed field is true.

  1. Close the table dialog
  2. Link the roadway1-rsu1-mqtt value field to the publish Value field
  3. Link the roadway1-rsu1-mqtt lastUpdate field to the publish Timestamp field

  1. Place the Car Simulator on its side to simulate a vehicle safety event

You can confirm that the data is successfully published by navigating to ... > efm-roadway1 > data and expanding the vehicles > crash-reports entry under the Metrics section. The data values should change every second, but only when the Car Simulator is not sitting flat on its rubber feet. You can confirm this within the publish-crash-reports dataflow as well, watching the enabled field in the publish block, which should change to true during simulated vehicle safety events and stay false otherwise.


This is an example of how EFM can intelligently filter large input data streams locally before they are published upstream. The next dataflow will use even more logic to tie local RWIS visibility data to local DMS output completely at the roadway.

  1. Close the publish-crash-reports dataflow by clicking the x icon next to its name above the editor workspace

Local RWIS-DMS Integration

  1. Select ... > efm-roadway1 > downstream > dataflow
  2. Right-click on dataflow and select Create Dataflow
  3. Name the dataflow local-rwis-dms and click Invoke
  4. Expand ... > dataflow > local-rwis-dms
  5. Open the local-rwis-dms dataflow in the right-hand editor workspace by clicking on its dataflow icon
  6. Expand ... > efm-roadway1 > downstream > mqtt > local-broker > Data > weather
  7. Under the Metrics section, drag roadway-rwis1 to the editor workspace Dataflow section
  8. Double-click on the new block and rename roadway1_rwis1 to roadway1-rwis1-mqtt
  9. Under the Blocks section, drag Table Operations > JSON Parser to the editor workspace Dataflow section
  10. Double-click on the new block and rename jsonParser to rwis-table
  11. Link the roadway1-rwis1-mqtt value field to the rwis-table input field
  12. Under the Blocks section, drag Logic > If to the editor workspace Dataflow section
  13. Double-click on the new block and rename If to light-level-trigger
  14. Select rwis-table and click on the Table button for the output field in the Properties section to see the underlying table data
  15. Drag the value in row 0 for the light column to the light-level-trigger input 0 field
  16. Close the table dialog
  17. Select light-level-trigger and edit the following values in the right-hand Properties section:

This one if block is providing the main logic for this dataflow. It is evaluating the RWIS light level and setting the message to be displayed on the DMS for either low or regular visibility. The rest of the dataflow takes this output, sends it to the DMS over serial, and publishes the DMS state upstream.


Most dataflows can be broken down logically to an “input-logic-output” cycle. It can be helpful to keep this in mind when trying to understand more complex dataflows. You may have also noticed that data-oriented dataflow blocks (usually used for input and output) are red while logic and operations blocks are orange.

  1. Under the Blocks section, drag String Operations > Split to the editor workspace Dataflow section
  2. Select the split block and edit the following values in the right-hand Properties section:
  1. Link the light-level-trigger output field to the split input field

  1. Under the Blocks section, drag Table Operations > Page to the editor workspace Dataflow section
  2. Double-click on the new block and rename tablePage to table-page
  3. Select table-page and edit the following values in the right-hand Properties section:
  1. Mouse over the currentPage field, click the blue dot, and check the Pinned box
  2. Mouse over the totalPage field, click the blue dot, and check the Pinned box
  3. Link the split output field to the table-page input field
  4. Under the Blocks section, drag Logic > Stop Watch to the editor workspace Dataflow section
  5. Double-click on the new block and rename stopwatch to table-row-loop
  6. Select table-row-loop and edit the following values in the right-hand Properties section:
  1. Mouse over the reset field, click the blue dot, and check the Pinned box
  2. Link the table-row-loop output field to the table-page currentPage field
  3. Under the Blocks section, drag Logic > If to the editor workspace Dataflow section
  4. Double-click on the new block and rename if1 to row-loop-reset
  5. Select row-loop-reset and edit the following values in the right-hand Properties section:
  1. Link the table-row-loop output field to the row-loop-reset input 0 field
  2. Link the table-page totalPage field to the row-loop-reset input 1 field
  3. Link the table-row-loop output field to the row-loop-reset then field
  4. Link the row-loop-reset output field to the table-row-loop reset field
  5. Under the Blocks section, drag Table Operations > Table Row Cells to the editor workspace Dataflow section
  6. Double-click on the new block and rename tableRowCells to select-value
  7. Select select-value and edit the following values in the right-hand Properties section:
  1. Link the table-page output field to the select-value table field

  1. Navigate to ... > efm-roadway1 > downstream > c-serial
  2. Under the Metrics section, right-click on dms-serial and drag send message to the editor workspace Dataflow section

  1. Double-click on the new block and rename SendMessage to send-to-dms
  2. Select send-to-dms and edit the following values in the right-hand Properties section:
  1. Link the select-value value 0 field to the send-to-dms Message field

The DMS should have just changed its display from the 3 green lights to DRIVE SAFE in orange.

  1. Navigate to ... > efm-roadway1 > data
  2. Right-click on data and drag publish to the editor workspace Dataflow section
  3. Select the new block and edit the following values in the right-hand Properties section:
  1. Link the send-to-dms Message field to the publish Value field

  1. Cover up the RWIS to simulate a visibility event and check that the DMS display cycles through FOG AHEAD and USE CAUTION
  2. Remove the RWIS cover and the display should reset to DRIVE SAFE

If the DMS isn’t updating, review the troubleshooting steps outlined in Raspberry Pi Serial DMS , the Serial DSLink setup on the roadway broker, and the IR829 IOS and IOx serial port configuration.

  1. Close the local-rwis-dms dataflow by clicking the x icon next to its name above the editor workspace

Create DOT Demo Dataflows on the Regional Broker

Connect to the ParStream Server

  1. Navigate to Data > downstream > ParStream-Import
  2. Right-click on ParStream-Import and select Add Database
  3. Use the following options:


Save All Crash Data to ParStream

  1. Select Data > downstream > dataflow

We are creating these next dataflows on the regional broker, which is where the ParStream database resides and where the ParStream connection is configured above. This makes sense given the main purpose of the regional broker is to aggregate the data from roadway brokers.

  1. Right-click on dataflow and select Create Dataflow
  2. Name the dataflow save-all-crash-data and click Invoke
  3. Expand ... > dataflow > save-all-crash-data
  4. Open the save-all-crash-data dataflow in the right-hand editor workspace by clicking on its dataflow icon
  5. Under the Blocks section, drag Table Operations > JSON Parser to the editor workspace Dataflow section
  6. Double-click on the new block and rename jsonParser to parsed-crash-data
  7. Navigate to ... > downstream > DQL
  8. Right-click on DQL and drag Query to the editor workspace Dataflow section

  1. Double-click on the new block and rename query to get-all-crash-data
  2. Select get-all-crash-data and edit the following values in the right-hand Properties section:

This is Distributed Query Language (DQL), which provides a powerful, flexible way to extract published data throughout all the brokers within an EFM deployment. This particular DQL query gets any crash-reports data published by any broker the regional broker knows about. In this demo environment, the only other broker is the roadside broker you configured above.

  1. Click the Invoke button associated with the invoke field
  2. Click the Table button associated with the output field
  3. Drag the value in row 0 for the value column to the parsed-crash-data input field

If you don’t see any data in the table, turn the Car Simulator on its side to have it send crash data.

  1. Close the table
  2. Navigate to ... > downstream > ParStream-Import > historian > Tables > crash_data

If you don’t see the crash_data table listed, right-click on historian and select Introspect and then check again.

  1. Right-click on crash_data and drag Insert Row to the editor workspace Dataflow section

  1. Double-click on the new block and rename insertRow to save-crash-data
  2. Select save-crash-data and edit the following values in the right-hand Properties section:
  1. Select get-all-crash-data and click the Table button associated with the output field in the Properties section
  2. Drag the value in row 0 for the timestamp column to the save-crash-data ts field

  1. Close the table
  2. Select parsed-crash-data and click the Table button associated with the output field in the Properties section
  3. Link the table values in row 0 to the associated column field inputs for save-crash-data

  1. Close the table

You will fully confirm whether this dataflow is working correctly when you configure the DOT Demo DGLux dashboard below. For now you should see the save-crash-data fields update when you place the Car Simulator on its side and see a value of INSERTED for the Status field in its Properties section.

  1. Close the save-all-crash-data dataflow by clicking the x icon next to its name above the editor workspace

Save Sampled RWIS Data to ParStream

  1. Select Data > downstream > dataflow
  2. Right-click on dataflow and select Create Dataflow
  3. Name the dataflow save-sampled-rwis-data and click Invoke
  4. Expand ... > dataflow > save-sampled-rwis-data
  5. Open the save-sampled-rwis-data dataflow in the right-hand editor workspace by clicking on its dataflow icon
  6. Under the Blocks section, drag Table Operations > JSON Parser to the editor workspace Dataflow section
  7. Double-click on the new block and rename jsonParser to parsed-rwis-data
  8. Navigate to ... > downstream > DQL
  9. Right-click on DQL and drag Query to the editor workspace Dataflow section
  10. Double-click on the new block and rename query to get-all-rwis-data
  11. Select get-all-rwis-data and edit the following values in the right-hand Properties section:

This DQL query gets any weather data published by any broker the regional broker knows about.

  1. Click the Invoke button associated with the invoke field
  2. Click the Table button associated with the output field
  3. Drag the value in row 0 for the value column to the parsed-rwis-data input field

You should see this table data update every 3 seconds.

  1. Close the table
  2. Under the Blocks section, drag Table Operations > Realtime Recorder to the editor workspace Dataflow section
  3. Double-click on the new block and rename realtimeRecorder to buffered-rwis-data
  4. Click on the small plus icon in the upper-right of the buffered-rwis-data block (3) times to add 3 more name and value fields
  5. Mouse over the blue dot associated with the name 0 field in the Properties section
  6. Click it and check the Pinned box
  7. Mouse over the blue dot associated with the value 0 field in the Properties section
  8. Click it and check the Pinned box
  9. Edit the following values in the right-hand Properties section:
  1. Select parsed-rwis-data and click the Table button associated with the output field in the Properties section
  2. Link the table values in row 0 to the associated column field inputs for buffered-rwis-data

  1. Close the table

If you select the buffered-rwis-data block and click the Table button associated with the output field in its Properties section you will see that it is generating a table with 60 seconds of rolling RWIS data. This table is the data source for the 60-second moving average that will ultimately end up saved into ParStream. This is an example of how EFM can intelligently sample data to drive insight and efficiency or otherwise enforce a DOT’s desired operational policies.

  1. Under the Blocks section, drag Tables Operations > Aggregation to the editor workspace Dataflow section
  2. Double-click the new block and rename tableAggregation to last-ts
  3. Select last-ts and edit the following values in the right-hand Properties section:
  1. Link the buffered-rwis-data output field to the last-ts input field
  2. Under the Blocks, drag Date Time Operations > Date Math to the editor workspace Dataflow section
  3. Double-click the new block and rename dateMath to ts-to-utc
  4. Select ts-to-utc and unpin the following fields by right-clicking on the blue dot and unchecking the Pinned box:

This will make the block much smaller and easier to position in the editor workspace by only exposing the needed input and output fields.

  1. Link the last-ts output field to the ts-to-utc input field

  1. Under the Blocks section, drag Table Operations > Aggregation to the editor workspace Dataflow section
  2. Double-click the new block and rename tableAggregation1 to avg-temp_c
  3. Select avg-temp_c and edit the following values in the right-hand Properties section:
  1. Link the buffered-rwis-data output field to the avg-temp_c input field
  2. Under the Blocks section, drag Table Operations > Aggregation to the editor workspace Dataflow section
  3. Double-click the new block and rename tableAggregation2 to avg-press_hpa
  4. Select avg-press_hpa and edit the following values in the right-hand Properties section:
  1. Link the buffered-rwis-data output field to the avg-press_hpa input field
  2. Under the Blocks section, drag Table Operations > Aggregation to the editor workspace Dataflow section
  3. Double-click the new block and rename tableAggregation3 to avg-light
  4. Select avg-light and edit the following values in the right-hand Properties section:
  1. Link the buffered-rwis-data output field to the avg-light input field

  1. Navigate to ... > downstream > ParStream-Import > historian > Tables > rwis_data
  2. Right-click on rwis_data and drag Insert Row to the editor workspace Dataflow section
  3. Double-click on the new block and rename insertRow to save-rwis-data
  4. Select save-rwis-data and edit the following values in the right-hand Properties section:
  1. Link the ts-to-utc output field to the save-rwis-data ts field
  2. Link the buffered-rwis-data value 0 field to the save-rwis-data station field
  3. Link the avg-temp_c output field to the save-rwis-data temp_c field
  4. Link the avg-press_hpa output field the save-rwis-data press_hpa field
  5. Link the avg-light output field to the save-rwis-data light field

As before, you will fully confirm whether this dataflow is working correctly when you configure the DOT Demo DGLux dashboard below. For now you should see the averaged save-rwis-data fields update every 3 seconds (the ParStream INSERT itself only happens every 60 seconds per the interval field) and see a value of INSERTED for the Status field in its Properties section.

  1. Close the save-sampled-rwis-data dataflow by clicking the x icon next to its name above the editor workspace
  2. Close the Dataflow Editor

Create a DOT Demo Dashboard in DGLux

  1. Browse to the built-in DGLux5 dashboard tool installation that comes with EFM here

The first time you connect to the DGLux dashboard tool it will prompt you to install a license. DGLux is separately licensed from EFM. It is used here to show the power of EFM and give an example of how to extract operational data from the system. Most DOTs will use their own Traffic Management System (TMS) rather than something like DGLux.

  1. Click Request License
  2. Enter your contact information and specify the 180-day Trial License

  1. Click Submit Request

  1. Check your email for the automated license delivery with a subject of Atrius Solution Builder License Generated
  2. Copy the license per the email instructions into the Input license data here: field in the DGLux5 License dialog that should still be open
  3. Click Submit License
  4. The page will reload and you will be prompted to accept the DGLux5 End User License Agreement
  5. Click I Agree
  6. Select CREATE NEW > Default > Blank Project
  7. Click Choose

  1. Enter DOT Dashboard for the project name and click OK

DGLux is a powerful, complicated tool. Rather than send time trying to learn it, you are going to import an existing dashboard to get started as quickly as possible. Feel free to make a copy of the dashboard or create a new DGLux project if you are interested in becoming more familiar with the tool.

  1. Download the following files for the dashboard:

You may have to manually save this to your downloads folder as your web browser will want to display it inline.

  1. In the left-hand Project section, select the DOT Dashboard > assets folder and click the Upload button

  1. Browse to the cisco_logo.png file you downloaded and click Open

  1. The logo file should now appear under the assets folder

  1. In the left-hand Project section, select the DOT Dashboard folder and click the Upload button again
  2. Browse to the index.dg5 file you downloaded and click Open
  3. Click Ok when prompted to overwrite the existing index.dg5 file

  1. Browse to the DOT Dashboard which should now be available here

You should see the DMS display update in real time based on your setup. Cover your RWIS sensor and the display should change. The RWIS graphs should update every 60 seconds. Placing the Car Simulator on its side should trigger new entries in the Vehicle Safety Events table. There are also examples of live roadway video data (public feeds from the Internet) and an example Google Maps traffic view. If any of these aren’t updating or don’t look correct, don’t be afraid to reach out for help.


Also be aware that these are publicly-accessible dashboards, so be careful who you share the URL with and be respectful of others’ dashboards. You wouldn’t want someone clearing your ParStream tables in the middle of a demo.

  1. Close DGLux

On to Chapter 7 Demo Walkthrough

left-chevronright-chevron