README.md

# REDCap-Twilio SMS Notification System

This system automatically sends SMS notifications to participants in a REDCap project after they complete registration. The system monitors for new records in REDCap, waits for a configurable delay period, and then sends a customized SMS message with a survey link.

## System Flow

### First Run Process
```
[Start]
   │
   ▼
[Get All Records from REDCap]
   │
   ▼
[Save All Record IDs to existing_records.json]
   │
   ▼
[Exit Program]
```

On the first run, the system:
1. Retrieves all current record IDs from REDCap
2. Saves them to `existing_records.json`
3. Exits without sending any messages

### Subsequent Runs Process
```
   ▼
[Check Each Record]
   │
   ├─── If in existing_records.json ──► [Skip]
   │
   └─── If New Record ────────────────► [Process]
   │
   ▼
[Check Registration Status]
   │
   ├─── If Complete ──► [Add to Processing Queue]
   │
   └─── If Incomplete ──► [Skip]
   │
   ▼
[Check Processing Queue]
   │
   ├─── If Delay Complete ──► [Send SMS]
   │                            │
   │                            ▼
   │                         [Save to processed_records.json for logging]
   │                            │
   │                            ▼
   │                         [Save to existing_records.json]
   │
   └─── If Delay Not Complete ──► [Wait]
```

On subsequent runs, the system:
1. Gets all records from REDCap
2. Identifies new records (not in `existing_records.json`)
3. For each new record:
   - Checks if registration is complete
   - Adds complete records to the processing queue with current timestamp
4. For each record in the processing queue:
   - Checks if the configured delay time has passed since registration
   - If delay completed:
     - Gets a survey link for the participant
     - Finds the participant's phone number
     - Sends an SMS with the survey link
     - Logs the message in `processed_records.json`
     - Adds the record ID to `existing_records.json`
   - If delay not completed:
     - Waits until next run

## Key Features

- **First Run Detection**: Automatically detects first run and initializes system
- **New Record Detection**: Only processes records not seen before
- **Configurable Delay**: Waits a configurable time before sending messages
- **REDCap Integration**: Directly interfaces with REDCap API
- **Twilio SMS Integration**: Sends SMS messages via Twilio
- **Message Logging**: Logs all sent messages in REDCap and locally
- **Fallback Mechanisms**: Includes robust error handling and fallbacks
- **Command Line Interface**: Supports various commands for maintenance

## Configuration

All configuration is stored in a PostgreSQL database but includes fallbacks if database access fails:

### REDCap Configuration
- API URL and Token
- Event Name and Repeat Instance

### Twilio Configuration
- Account SID and Auth Token
- Phone Number

### Processing Configuration
- Registration delay time (minutes)
- Polling interval (seconds)
- Registration status fields to check
- Phone number fields to check
- Survey instruments to use
- Phone formatting options
- Message template

## Usage

```
python app.py               - Run the monitoring process
python app.py force <id>    - Force send a message to record <id>
python app.py sync          - Synchronize processed records with REDCap
python app.py help          - Show help message
```

## Files

- `app.py`: Main application file
- `redcap.py`: REDCap integration functions
- `existing_records.json`: Stores known record IDs
- `processed_records.json`: Stores message history

## Requirements

- Python 3.6+
- REDCap API access
- Twilio account
- PostgreSQL database (optional, for configuration)

## Installation

1. Clone the repository
2. Install required packages: `pip install -r requirements.txt`
3. Configure database connection or edit fallback values
4. Run the application: `python app.py`

## Troubleshooting

If you encounter issues:
1. Check database connection
2. Verify REDCap API access
3. Confirm Twilio credentials
4. Check logs for detailed error messages
5. Use the `force` command to test specific records

## Features

- Automatically monitors REDCap for new registrations
- Sends personalized survey links via SMS
- Implements a 1-minute delay for new participants
- Prevents duplicate message sending
- Maintains a permanent record of sent messages
- Secure credential management using environment variables

## Prerequisites

- Python 3.6 or higher
- REDCap API access
- Twilio account with SMS capabilities
- Required Python packages (listed in requirements.txt)

## Configuration

### REDCap Setup
1. Ensure your REDCap project has:
   - A registration form with a completion field
   - A phone number field for participants
   - An interest form survey

2. The registration completion field should be named "registration_complete" with value "2" when complete

### Phone Number Field
The system will look for the phone number in the following fields (in order):
- phone_number
- phone
- mobile
- mobile_number
- cell
- cell_number
- contact_phone
- participant_phone

If your REDCap project uses a different field name, modify the `possible_phone_fields` list in the code.

## Usage

1. Start the monitoring system:
```bash
python app.py
```

2. The system will:
   - Check for new registrations every 10 seconds
   - Wait 1 minute after registration completion
   - Send the survey link via SMS
   - Keep track of sent messages to prevent duplicates

## File Structure

- `app.py`: Main application code
- `.env`: Environment variables (create this file)
- `processed_records.json`: Stores records of sent messages
- `requirements.txt`: Python package dependencies

## Error Handling

The system includes comprehensive error handling for:
- API connection issues
- Missing or invalid credentials
- SMS sending failures
- Record processing errors

All errors are logged with detailed information for debugging.

## Logging

The system provides detailed logging of:
- Record processing status
- Registration completion checks
- SMS sending attempts
- Error messages
- Available REDCap fields

## Security

- Credentials are stored in environment variables
- No sensitive data is stored in the code
- API tokens and keys are never logged
- Processed records are stored locally only

## Maintenance

- The system maintains a permanent record of sent messages
- Duplicate messages are prevented even after system restarts
- Failed message attempts are logged for review

## Troubleshooting

1. If messages aren't sending:
   - Check Twilio credentials in .env
   - Verify phone number format
   - Check Twilio account balance

2. If records aren't being processed:
   - Verify REDCap API credentials
   - Check registration_complete field value
   - Review available fields in REDCap

3. If the system isn't starting:
   - Verify all environment variables are set
   - Check Python version
   - Ensure all dependencies are installed

## Support

For issues or questions:
1. Check the logs for error messages
2. Verify all configuration settings
3. Contact system administrator