Retrieving and Storing Aggregated Triggered Send Data

Here’s another one of my favorite applications of WSProxy — retrieving and storing aggregated TriggeredSendSummary data in a Data Extension using a Script Activity.

This data is helpful in that it shows aggregated send data for all Triggered Sends — traditional and Journey Builder ones.  A new row is added every time a Triggered Send Definition is published, so it’s also a historical record of publishing events — along with the performance of each “version” of the Triggered Send Definition.  The sum of each of the activity counts represents what you would see in Tracking for a particular Triggered Send Definition.

One nice benefit is that the results also contain the number of emails queued to be sent at a specific point in time.  So if you retrieve this data hourly (with some additional configuration), you could monitor the sending queue over time.

(If you’re looking for something like this for TXM Email sends, you’re stuck with the queue metrics endpoint and aggregating the activity yourself.)

To get started, create a Data Extension with this schema:

Name: TriggeredSendSummary
External Key: TriggeredSendSummary

Next, create a Script Activity in Automation Studio like this.

This script does a Describe of a couple of SOAP objects to get the retrievable properties, sets a filter (it’s like a where-clause in SQL), iterates through the TriggeredSendSummary Object results, does a few more related retrieves for specific attribute, and finally upserts the results into the specified data extension. If you don’t want to retrieve all records every time, you can update the filter criteria on line 81 and get more specific. I’ve generally not restricted it, because it’s not that much data to retrieve.

Finally, schedule this to run on an interval appropriate for your needs.

Once you have data, you can toolbar-download it as needed, query it, export it with an Extract Activity, or lookup and display it in an email or CloudPage with AMPscript.

Salesforce Marketing Cloud Engineer Skills

I’ve spent a fair amount of time thinking about the skill progression/path for someone who’s doing engineering work in the Salesforce Marketing Cloud platform.

This obviously doesn’t apply to everyone working in the platform, but for those of you doing this kind of work in agencies, I think it can be a helpful guide for taking the next step. Certification tests have their place for sure, but these are what I’ve found to be the practical benchmarks.

Recruiters, hiring managers: this is what people like me do in Marketing Cloud.

Download as a CSV

LevelCategorySubcategorySkillDescription
BeginnerEmail StudioAMPscriptBasic Syntaxdemonstrates a working knowledge of AMPscript syntax
BeginnerEmail StudioAMPscriptBasic Functionsdemonstrates a working knowledge of AMPscript functions, what they are and generally what they are used for
IntermediateEmail StudioAMPscriptPersonalization and conditionalsunderstands fault-tolerant methods for personalization and can build conditionals with multiple criteria
IntermediateEmail StudioAMPScriptDebuggingknows how to use the process of elimination and debugging output to troubleshoot AMPscript runtime errors
IntermediateEmail StudioAMPscriptLoopingcan retrieve data from a data extension outside of the send/view context and iterate through and output the results
IntermediateEmail StudioAMPscriptPayload parsingcan demonstrate how to parse a JSON or XML payload and output HTML in a loop
AdvancedEmail StudioAMPscriptAPI functionsis familiar with the SOAP Objects and Methods and how to interact with them using AMPscript
AdvancedEmail StudioAMPscriptSalesforce CRM functionscan demonstrate how to interact with SF CRM using AMPscript API functions
BeginnerEmail StudioSQLBasicsunderstands basic SQL syntax, select statements, datatypes and where clauses
BeginnerEmail StudioSQLJoins/unionsunderstands SQL data relationships and how to produce a single result from multiple data extensions
IntermediateEmail StudioSQLMarketing Cloud data modelis familiar with the system data views and their relationships and can build queries to retrieve and/or summarize subscriber and activity data
IntermediateEmail StudioSQLConversionsknows how the datatypes in SFMC match to T-SQL data types and can convert between them using cast and convert functions.
IntermediateEmail StudioSQLDate mathcan demonstrate how to efficiently select rows between two dates and can calculate past and future dates
IntermediateEmail StudioSQLConditional field valuescan demonstrate the use of case, coalesce, isnull, isnumeric, isdate to conditionally set column values
AdvancedEmail StudioSQLDeduplicating and aggregatingknows how to deduplicate rows in a data extension and the various methods for aggregating data within the constraints of SFMC
AdvancedEmail StudioSQLOptimization & Data Normalizationknows all reasons why queries can fail and can articulate how to address each one, familiar with normalized relational database models
BeginnerEmail StudioHTMLBasicsunderstands the basic anatomy of an HTML document (DOM), also containers, links and other email related elements, can manually re-indent code
IntermediateEmail StudioHTMLContainerscan build a table with multiple columns and rows and understands inline vs block-level elements
IntermediateEmail StudioEmail HTMLPlatform targetingfamiliar with how to target code for specific email platforms
IntermediateEmail StudioCSSBasicsknows how styles cascade and can code external and inline CSS to alter the appearance of HTML elements
IntermediateEmail StudioCSSMedia queriescan translate CSS media query syntax to alter HTML elements for various sized display sizes
IntermediateEmail StudioJavaScriptBasicsunderstands the general JS syntax and data structures and can implement basic HTML5 field validation
IntermediateEmail StudioJavaScriptForms and librariescan craft a basic AJAX form submission and is familiar with common JS libraries
IntermediateEmail StudioServer-side JavaScriptBasicsunderstands when to use Server-side JavaScript and what libraries are available for interacting with SFMC
AdvancedEmail StudioServer-side JavaScriptHelper functionscan retrieve Subscriber and Data Extension data using the Server-side JavaScript helper functions in the context of a CloudPage
AdvancedEmail StudioServer-side JavaScriptContent Syndicationknows how to retrieve content from outside of SFMC and store it for use in SFMC
IntermediateContact BuilderBasicsunderstands how to configure data extensions for use in Journey Builder and MobileConnect
IntermediateJourney BuilderBasicsfamiliar with the methods for starting someone on a journey and how to configure data source for decision splits
IntermediateCloudPagesHTML/CSS Basicsunderstands responsive design and coding
IntermediateCloudPagesOne-Click Unsubscribe pagescan configure a page that accepts the send context and successfully logs an unsub event and displays a confirmation message
IntermediateCloudPagesCode Resource pagescan build a code resource page that returns a JSON object to a referring page
AdvancedCloudPagesData Capture Formsknows how form posts and AJAX work in CloudPages and use the API functions to capture data in SFMC data extensions
AdvancedCloudPagesCustom Preference Centerscan build API calls with AMPscript or Server-side JavaScript to implement a custom preference center
IntermediateMobileConnectBasicsknows how to create dynamic content in MobileConnect messages
AdvancedMobileConnectScriptingcan script MobileConnect messages to read data from a data extension or write data to a data extension
IntermediateSocial StudioBasicsunderstands the basics of Social Studio and knows how to create audiences in Email Studios
IntermediateIntegrationsSales/Service CloudMarketing Cloud Connectfamiliar with how the data models are different between SFMC and SF CRM, what is created when they are integrated
IntermediateIntegrationsAPIREST/SOAP APIcan make a test API call using SOAP or REST from outside of the SFMC platform
AdvancedIntegrationsAPIConsultationhas a level of familiarity with the SFMC web services to consult with a client who is designing an API integration with SFMC
AdvancedIntegrationsAPITransactional Messaging APIcan create a transactional messaging send definition for Email and SMS and can demonstrate how to register webhooks for reporting
AdvancedIntegrationsAPIContent Block SDKknows how to configure a custom content block in an external host and integrate it into Content Builder
AdvancedIntegrationsAPICustom Journey Builder Activitycan build a custom Journey Builder activity that integrates with a third-party system
AdvancedIntegrationsGeneralEinstein Recommendationsunderstands how to configure foundational Einstein recommendations data in SFMC and can advise client teams with tracking integrations

I’d love to hear from you about this. What would you add or remove? Leave a comment below.

Data Extension Tips & Tricks

Here are a few Data Extension (DE) tricks to save you some time:

Faster Times

Avoid adding fields to DEs at the top or bottom of the hour.   Your data center’s server load adversely affects the required back-end update tasks and increase the likelihood of your DE changes timing out.

Fewer Drill-downs

Duplicate DEs to the root folder and then move them in multiples to the final folder structure.  It saves time if you don’t have to drill-down in the folder selector for every DE.

Flexible Copy

If you need to copy a DE, go through the Create process and choose Create from Existing from the Creation Method selector.  This gives you an opportunity to add/update the fields — their names, data types, primary key options and default values — in the copy.

Query Studio Create

You can create generic data extensions using Query Studio.  Sometimes it’s easier to create a DE with a query, copy and update it to suit.  Be sure you reset or remove the 1 day retention in your copy.  This works well with the Create from Existing tip.

A hearty tip-of-the-hat to my DEG team for their contributions.

Direct Links for SFMC pages

Thought I would share these direct links to SFMC pages (in order of my usage).  Can’t remember who showed me these originally, but I use them constantly when moving around the Marketing Cloud interface.  These’ll work regardless of your data center/stack.

Here’s a bookmark file you can download and import into your browser of choice.

Determining Business Unit Subscriber Status

Business unit subscriber status (or business unit membership) has been one of those frustrating and elusive things in SFMC for a long time. Since the platform only stores a business unit-level status for a subscriber if they’ve been sent an email from the child business unit, the various sources of this status in the platform are not accurate.

It wasn’t until Salesforce introduced the _BusinessUnitUnsubscribes System Data View in 2019 that there was, in my opinion, an accurate source of truth for this data.

Here are my go-to queries to arrive at this data set. I’d suggest scheduling a daily Automation with these queries in the parent business unit.

Queries

All Subscribers in all business units

This query creates a row for every subscriber in every business unit. Since subscribers exist at the enterprise level, they could potentially have a send (or unsubscribe) in any child business unit.

You’ll need to update this query to add your own MIDs and business unit names.

Unsubscribes per business unit

This one sets the status for the subscriber if an explicit unsubscribe in a business unit has been recorded.

You’ll need to update the CASE statement for your own set of MIDs and business unit names.
Unsubscribes are the only deliverability status recorded at the business unit level.

Caveats

Due to the dependencies on the System Data Views, if you have more than a few million subscribers, you’ll be hard-pressed to keep these queries running under the 30-minute timeout window. In other words, this solution doesn’t scale very well if you have a large subscriber base.

All of this assumes that your business units are configured to limit unsubscribes to the business unit level (i.e. The “Subscribers will be unsubscribed from this BU only” option is selected in Setup > Business units > Unsubscribes settings).

Uses

The business unit subscriber status data is helpful in a few areas in SFMC.

Data Integrations

If Marketing Cloud is not your database of record for deliverability and status by business unit is a key data point, this can be a good source to extract and load into an external data warehouse.

Segmentation

You can use this data as a source for suppressing opt-outs so your pre-send counts more accurately reflect actual send counts.

Reporting

If you have any reporting needs relating to business unit membership and cross-over you can utilize this data source. I don’t recommend treating SFMC like a reporting platform, but the flexibility of Query Activities and Data Extensions can’t be denied.

Further reading