README.md

REDCap-Twilio SMS Notification System

This system automatically sends SMS notifications to participants in a REDCap project after they complete registration. The system monitors for new records in REDCap, waits for a configurable delay period, and then sends a customized SMS message with a survey link.

System Flow

First Run Process

[Start]
   │
   ▼
[Get All Records from REDCap]
   │
   ▼
[Save All Record IDs to existing_records.json]
   │
   ▼
[Exit Program]

On the first run, the system:

1. Retrieves all current record IDs from REDCap
2. Saves them to existing_records.json
3. Exits without sending any messages

Subsequent Runs Process

   ▼
[Check Each Record]
   │
   ├─── If in existing_records.json ──► [Skip]
   │
   └─── If New Record ────────────────► [Process]
   │
   ▼
[Check Registration Status]
   │
   ├─── If Complete ──► [Add to Processing Queue]
   │
   └─── If Incomplete ──► [Skip]
   │
   ▼
[Check Processing Queue]
   │
   ├─── If Delay Complete ──► [Send SMS]
   │                            │
   │                            ▼
   │                         [Save to processed_records.json for logging]
   │                            │
   │                            ▼
   │                         [Save to existing_records.json]
   │
   └─── If Delay Not Complete ──► [Wait]

On subsequent runs, the system:

1. Gets all records from REDCap
2. Identifies new records (not in existing_records.json)
3. For each new record:
   • Checks if registration is complete
   • Adds complete records to the processing queue with current timestamp
4. For each record in the processing queue:
   • Checks if the configured delay time has passed since registration
   • If delay completed:
     • Gets a survey link for the participant
     • Finds the participant's phone number
     • Sends an SMS with the survey link
     • Logs the message in processed_records.json
     • Adds the record ID to existing_records.json
   • If delay not completed:
     • Waits until next run

Key Features

• First Run Detection: Automatically detects first run and initializes system
• New Record Detection: Only processes records not seen before
• Configurable Delay: Waits a configurable time before sending messages
• REDCap Integration: Directly interfaces with REDCap API
• Twilio SMS Integration: Sends SMS messages via Twilio
• Message Logging: Logs all sent messages in REDCap and locally
• Fallback Mechanisms: Includes robust error handling and fallbacks
• Command Line Interface: Supports various commands for maintenance

Configuration

All configuration is stored in a PostgreSQL database but includes fallbacks if database access fails:

REDCap Configuration

• API URL and Token
• Event Name and Repeat Instance

Twilio Configuration

• Account SID and Auth Token
• Phone Number

Processing Configuration

• Registration delay time (minutes)
• Polling interval (seconds)
• Registration status fields to check
• Phone number fields to check
• Survey instruments to use
• Phone formatting options
• Message template

Usage

python app.py               - Run the monitoring process
python app.py force <id>    - Force send a message to record <id>
python app.py sync          - Synchronize processed records with REDCap
python app.py help          - Show help message

Files

• app.py: Main application file
• redcap.py: REDCap integration functions
• existing_records.json: Stores known record IDs
• processed_records.json: Stores message history

Requirements

• Python 3.6+
• REDCap API access
• Twilio account
• PostgreSQL database (optional, for configuration)

Installation

1. Clone the repository
2. Install required packages: pip install -r requirements.txt
3. Configure database connection or edit fallback values
4. Run the application: python app.py

Troubleshooting

If you encounter issues:

1. Check database connection
2. Verify REDCap API access
3. Confirm Twilio credentials
4. Check logs for detailed error messages
5. Use the force command to test specific records

Features

• Automatically monitors REDCap for new registrations
• Sends personalized survey links via SMS
• Implements a 1-minute delay for new participants
• Prevents duplicate message sending
• Maintains a permanent record of sent messages
• Secure credential management using environment variables

Prerequisites

• Python 3.6 or higher
• REDCap API access
• Twilio account with SMS capabilities
• Required Python packages (listed in requirements.txt)

Configuration

REDCap Setup

1. Ensure your REDCap project has:

   • A registration form with a completion field
   • A phone number field for participants
   • An interest form survey

2. The registration completion field should be named "registration_complete" with value "2" when complete

Phone Number Field

The system will look for the phone number in the following fields (in order):

• phone_number
• phone
• mobile
• mobile_number
• cell
• cell_number
• contact_phone
• participant_phone

If your REDCap project uses a different field name, modify the possible_phone_fields list in the code.

Usage

1. Start the monitoring system:

python app.py

2. The system will:
   • Check for new registrations every 10 seconds
   • Wait 1 minute after registration completion
   • Send the survey link via SMS
   • Keep track of sent messages to prevent duplicates

File Structure

• app.py: Main application code
• .env: Environment variables (create this file)
• processed_records.json: Stores records of sent messages
• requirements.txt: Python package dependencies

Error Handling

The system includes comprehensive error handling for:

• API connection issues
• Missing or invalid credentials
• SMS sending failures
• Record processing errors

All errors are logged with detailed information for debugging.

Logging

The system provides detailed logging of:

• Record processing status
• Registration completion checks
• SMS sending attempts
• Error messages
• Available REDCap fields

Security

• Credentials are stored in environment variables
• No sensitive data is stored in the code
• API tokens and keys are never logged
• Processed records are stored locally only

Maintenance

• The system maintains a permanent record of sent messages
• Duplicate messages are prevented even after system restarts
• Failed message attempts are logged for review

Troubleshooting

1. If messages aren't sending:

   • Check Twilio credentials in .env
   • Verify phone number format
   • Check Twilio account balance

2. If records aren't being processed:

   • Verify REDCap API credentials
   • Check registration_complete field value
   • Review available fields in REDCap

3. If the system isn't starting:

   • Verify all environment variables are set
   • Check Python version
   • Ensure all dependencies are installed

Support

For issues or questions:

1. Check the logs for error messages
2. Verify all configuration settings
3. Contact system administrator