The Data Management portal gives data engineers full visibility and full control over the scripts that transform data in the warehouse. From here, scripts can be created, managed, searched, edited, and viewed. All data system connection credentials are managed here as well.
The Data Management portal is divided into the following sections:
- Scripts: All the scripts that transform the data in your warehouse
- Activities: All the activities that are generated from scripts
- Credentials Stored credentials for access to each data system (Redshift, SFTP, etc)
The Scripts section gives a summary of all scripts, their type, and their status (pending or approved). More information on script types (activity steam, enrichment, and customer attribute) can be found here. This is also the place new scripts can be created or existing scripts can be modified.
The Activity Stream and Enrichment sections show all the scripts that have been created and the names of the activities they generate and the activities they enrich, respectively. Keep in mind that a single script can generate or enrich multiple activity types.
Clicking into each script shows the SQL logic that defines it.
All scripts can be created or modified from the scripts section.
- Create a New Script
- Explore the Data Source
- Modify the SQL Logic
- Rename the Stream Table Columns (Recommended)
Select the type of script you wish to create from the left navigation (Activity Stream Script, Enrichment Script, or Customer Attribute Script), then press the blue ** + Create New Script** button to begin defining your script.
The database schemas available use in the script can be explored in the left panel of the SQL editor. Clicking into each schema will show the tables and fields within each. This structure mimics the function any SQL client.
Modify the logic in the SQL editor to define your script logic. The templated format defined in the editor can be used as a starting point for your script. All scripts follow a structured format, which varies based on script type - see the resources below for more information on the required structure.
Resources: SQL Standards for Scripts
Tip: Preview the query output by pressing the Run Query button on the bottom right of the SQL editor. The output will show up to 1,000 rows of your query output, and you can inspect the data from here to ensure it meets your needs.
The Stream Table Columns panel to the left of the Dataset preview can be used to rename the otherwise ambiguous "Feature" columns to something more human-interpretable. The renamed columns will appear in the Narrator UI so users can understand what is in each column.
This step is optional, but highly recommended.
Note: These columns cannot be renamed directly in the SQL query because
link are standard column names for the activity stream. These column names need to remain consistent in order to append (UNION) the new activities to the existing activity stream. For this reason, Narrator provides the option to store meta-data about each activity so that human-interpretable names can still be available.
All scripts must be validated to ensure they meet the standards necessary to add them to the activity stream. Narrator runs a series of validations to help you transform your data and SQL into the right format. Once you feel good about the activity logic you've created, press Validate and Continue to validate it.
Script validation is a two step process:
- Ensure the SQL can run
- Check script requirements and configure activities
Validation Step 1: Validate SQL
This step ensures the SQL can be executed. This step is similar to the validation completed by any SQL client.
Validation Step 2: Check Requirements and Configure Activities
The second step checks that the script meets the requirements to be added to the activity stream.
It checks the following:
Necessary columns are present:
- A timestamp (
ts) is available for every activity
If the validation criteria not met, the script cannot be created and you should go back to modify the logic until this criteria is met. We're here to help if you're still having trouble, feel free to use the Ask Ahmed chat or email email@example.com/docs to get quick support.
Confirm Activities Generated
The last part of the validation confirms the activities that will be generated from the script and provides the option to disable any of them.
Disabling an activity means it will not be added to the activity stream. If you choose to disable an activity, you can always re-enable it from the Ignored Activities section of the Data Management portal.
Now that the script has been validated, the next step is to name and configure it.
It is recommended to give a descriptive name like "Sales Activities" instead of "Script 1" so that others can quickly reference it from list of scripts.
If you would like one script to run after another, then you can select the yes checkbox and use the dropdown to specify the script that should be run before this script.
Configure Processing Method
Select the processing method you would like to use to update your script by selecting one of the four options provided. More information on processing types can be found here.
The last step is to review the details of your script.
If you haven't already added names for the columns headers, you can edit them here. The review step let's you see the validations conducted and the activities that will be generated from your script.
If there's anything you want to change, you can move back to a previous step by using the back link at the upper right.
Finally, if you are happy with your script, then you can Submit for Approval. This will send your script to the Narrator team for review. Usually the review process takes a few hours, but always less than 24 hours.
You can see all scripts awaiting review in the Pending Approval section of the Scripts section. If at any point you wish to edit an script before it has been approved, you can do it from this section.
All scripts can be edited from the Scripts view of the Data Management portal.
Once you locate the script you want to edit, you can use the Edit pencil to the right of the script to begin modifying the script.
Any modifications to scripts go through the same process as newly created script. This includes the two-step validation and review from a member of the Narrator team.
The Activity section gives a summary of all activities that have ever been defined by a script. Each activity is categorized by status.
- Live: Activities currently being updated in the activity stream
- New: Activities added to the activity stream recently
- Ignored: Activities that have been deactivated in a script and therefore not generating any new activities in the activity stream
- Retired: Activities that are no longer being generated by an active script
- Pending: Activities that are associated with scripts awaiting approval
Within each section you can see a list of activities and the scripts that generated them. Finally, you can also edit the script that generates this activity from this page.
The Manage Database Connections section allows you to manage all credentials associated with your data systems in one place. This is where you can store the credentials to access Redshift, SFTP, etc.
All credentials are kept secret and can only be edited by account admins.
Additionally, the Manage Processing section gives you the ability to pause all Narrator processing if needed. This can be helpful if you are conducting a data warehouse migration and need to halt any processing for a short period of time.
Admin users can set permissions for the Data Management portal on the Account Management page. It is recommended to give all users view permissions and limit modify permissions to members of your data engineering team.
Users with view permissions will be able to:
- View all scripts and script definitions
- View all activities
- See pending scripts
Users with modify permissions will be able to:
- Create new scripts
- Modify existing scripts
- Edit connection credentials
- Manage activities
- Pause all data processing