Guide: Producing a Form Envelope for Integrations

Owned by Clayton Sims
Last updated: Apr 03, 2024 by Krinda Wernicke
5 min read

This document is intended as a helper to technical teams who are seeking to create Form Submission packages for submitting data to CommCareHQ, generally to update one or more cases in a project space.

Creating these documents is the biggest stumbling block for implementing partners. This guide won't go into detail about individual fields or specifications since those are covered elsewhere, but will provide advice about structuring transactions for submission, and what data is expected. 

The best way to get started with a template for integration is to create a CommCare Form in a sample app which completes the actions you want your integration to take, and to review the raw XML in CommCare HQ using the “Raw XML” view / download in the Submit History Report. We strongly, strongly recommend you do that in addition to using these templates.

Note: Forms submitted through the API may only be easy to find with the Raw Forms, Errors & Duplicates report, rather than the Submit History Report.

Base Example Template

The following is a sample template for a Form XML Submission illustrating what data needs to be provided by an integration.

Fields which will need to be provided are written as {{field_name}} for further explanation below, everything else should be submitted as is. 

<?xml version="1.0" ?> <data name="{{form_name}}" version="{{form_version}}" xmlns="{{form_schema_id}}" xmlns:jrm="http://dev.commcarehq.org/jr/xforms"> <n0:meta xmlns:n0="http://openrosa.org/jr/xforms"> <n0:deviceID>{{integration_source_id}}</n0:deviceID> <n0:timeStart>{{time_start}}</n0:timeStart> <n0:timeEnd>{{time_end}}</n0:timeEnd> <n0:username>{{commcare_username}}</n0:username> <n0:userID>{{commcare_user_id}}</n0:userID> <n0:instanceID>{{unique_form_guid}}</n0:instanceID> </n0:meta> {{form_contents}} </data>

Field Explanation Guide

  • form_name - A human readable description of what this form does. Will appear in reports

    • Specific to: Your Integration

    • Suggested Use - A terse describe the function of your submission

    • Example: Automatic Patient Updater

  • form_version - A numeric version for the form schema

    • Specific to: Your Integration

    • Suggested Use - Once your integration is in production, increment this number in your code any time you change the code that generates these blocks

    • Example: 1

  • form_schema_id - A unique ID for the schema of this XML document.  Will be used to group form submissions. XMLNS Schemas look like urls by tradition, but do not need to actually be valid urls.

    • Specific to: Your Integration

    • Suggested Use - Use a different schema ID for each type of integration you build, each document with the same schema ID should have the same document structure

    • Example: http://mycompany.com/commcare/patient_updater

  • integration_source_id - A plaintext description of what engine is generating this form envelope

    • Specific to: Your Integration

    • Suggested Use: Terse description of the integration environment that generated the envelope. If a library was used to generate the envelope, this should name the library.

    • Example: mycompany custom commcare integration

  • time_start - The time the integration run began as an ISO8601 compliant timestamp

    • Specific to: One run of the integration 

    • Suggested Use: Initially just set one timestamp for all time events, or set timestamps to generate as they are requested. Further considerations described at the bottom of this page

    • Example: 2019-09-24T13:55:14.281-04

  • time_end - The time this envelope was finished as an ISO8601 compliant timestamp

    • Specific to: One run of the integration 

    • Suggested Use: Initially just set one timestamp for all time events, or set timestamps to generate as they are requested. Further considerations described at the bottom of this page

    • Example: 2019-09-24T13:55:14.281-04

  • commcare_username - The login username submitting this data.

    • Specific to: The project space where the integration is run

    • Suggested Use: Should be populated with the login username of the user whose credentials will be used to submit this envelope

    • Example: jdoe@mycompany.com

  • commcare_user_id - The machine GUID of the Web User submitting this data 

    • Specific to: The project space where the integration is run

    • Suggested Use: This ID can be identified in a few ways. 

      • Navigating to the CommCareHQ Web Users page, selecting your user, then viewing your GUID in the url bar

      • From the List Web Users API

      • Viewing a form submission submitted by your user in the UI

    • Example: 1e670b9f-7259-41b0-bb2f-b1e2cf824828

  • unique_form_guid - A unique UUIDv4 machine id which represents this form envelope. This ID is used to ensure idempotency about the transaction while allowing resubmission. If you submit an envelope twice with the same id, HQ will treat them as the same submission.

    • Specific to: One run of the integration 

    • Suggested Use: Generate a random UUIDv4 id from a library in your platform for each transaction.

    • Example: 9b107fb4-c993-411b-87de-e2037377264a

  • form_contents - The remainder of the form is an XML document containing arbitrary elements, including structured CaseXML Transactions as explained below

A resulting envelope:

<?xml version="1.0" ?> <data name="Automatic Patient Updater" version="1" xmlns="http://mycompany.com/commcare/patient_updater" xmlns:jrm="http://dev.commcarehq.org/jr/xforms"> <n0:meta xmlns:n0="http://openrosa.org/jr/xforms"> <n0:deviceID>mycompany commcare integration tool</n0:deviceID> <n0:timeStart>2019-09-24T13:55:14.281-04</n0:timeStart> <n0:timeEnd>2019-09-24T13:55:14.281-04</n0:timeEnd> <n0:username>jdoe@mycompany.com</n0:username> <n0:userID>1e670b9f-7259-41b0-bb2f-b1e2cf824828</n0:userID> <n0:instanceID>9b107fb4-c993-411b-87de-e2037377264a</n0:instanceID> </n0:meta> </data>

Case Data

Inside of your form transaction you are probably ultimately intending to inject one or more CaseXML Transactions to change case data. You can read that spec sheet for additional specifics about what transactions can perform.

Two examples of case changes are provided below to illustrate how to encode those transactions. You can include as many casexml transactions as necessary in your envelopes (in the {{form_contents}} section above, but we recommend batching around 50 case transactions per form submission

Some fields are common with those above, new fields are described below the template.

Create a new case

... <n0:case case_id="{{case_id}}" date_modified="{{time_end}}" user_id="{{commcare_username}}" xmlns:n0="http://commcarehq.org/case/transaction/v2"> <n0:create> <n0:case_name>{{case_name}}</n0:case_name> <n0:owner_id>{{owner_id}}</n0:owner_id> <n0:case_type>{{case_type}}</n0:case_type> </n0:create> <n0:update> <n0:{{case_field}}>{{field_value}}</n0:{{case_field}}> ... </n0:update> </n0:case> ...

Field Explanation Guide

  • case_id - A UUIDv4 machine code that uniquely identifies this case

    • Suggested Use - In this case, since we are creating a case, this field should be set to a newly generated UUIDv4 value

    • Example: a1d6916d-3d3c-4b90-bd86-64de292c6ce7

  • case_name - A human readable description of the entity being kept track of with this case object

    • Specific to: Your project configuration

    • Suggested Use - Terse, human readable description

    • Example: Steve Johnson

  • owner_id - The UUIDv4 Machine ID of the entity which owns this case. This is either a CommCare User, a Location, or a Group

    • Specific to: Your project configuration

    • Suggested Use - Which ID this is set to is highly specific to each project, and may require ingesting data from an appropriate API or reviewing form submissions to find a sample owner_id in use

    • Example: 313d21e6-ae44-4fe5-899e-6926b43b969c

  • case_type - The CommCare Case Type for the model that you are creating

    • Specific to: Your project configuration

    • Suggested Use: This should be clear from sample form submissions, or from reviewing the Case Module in the app builder for the type id.

    • Example: patient

The remaining data to be set against the case is configured as a key/value dictionary in the update block, which contains the remaining fields whether a case is being created or updated.

Update an existing case

If a case should be updated rather than created,  the structure of the transaction is the same, but only containing the update section. Fields from the create section can be placed in the update section to change them once the case is already created.


Advanced Usage

These are considerations which aren't important for initial integration rollout or testing, but may prove helpful in long term production use cases

Time Start / End / Etc

There's no specific semantic importance to timeStart and timeEnd values for integrations, and for initial rollout it's generally fine to just create one timestamp when you generate the envelope and reuse it in all fields.

Once things are working, though, setting them intentionally can still be helpful for debugging production issues.

We recommend setting timeStart to the time at which the integration run began or finished retrieving the data that will be submitted, and timeEnd to the time when the envelope was sealed, spooled, and ready to submit. This allows the identification of process ordering or time sequencing issues.