Databricks

replacing logic apps

I replaced Azure Logic Apps with a Databricks approach for two critical integrations: outbound email delivery and SharePoint - ADLS file transfer. The result is a single engineering surface, consistent monitoring, lower operational cost, and faster runtime.

Logic Apps were costly and hard to scale.

Context and Challenge

The original plan relied on Logic Apps for both email sending and SharePoint file movement. It was easy to start but expensive and hard to scale: Logic App runs multiplied quickly, troubleshooting was fragmented, and changes required duplicating logic across flows.

The Approach

1. Email delivery pipeline
  • Validate and normalize partner‑provided metadata (Excel source, non‑negotiable).

  • Generate canonical instructions and lifecycle records.

  • Send via SendGrid with retries, batching, and failure capture.

2. SharePoint transfer module
  • Support Download, Upload, and Delete actions via Graph API.

  • Use service principal auth and chunked upload sessions.

  • Apply safe deletion rules and extension filters.

Move to Databricks
 jobs orchestrated
 by ADF.

What I Built

  • Logic Apps replacement: Databricks jobs cover the same functional scope, with stronger reliability and observability.

  • Template‑driven execution: Job parameters control source/target paths, filters, and actions without code changes.

  • Lifecycle tracking: Unified Delta tables for validation, delivery, and failure states.

  • Resilient IO: Retry‑enabled SendGrid session; Graph API pagination and chunked uploads.

  • Operational safety: Extension filters, delete‑contents‑only mode, and explicit rerun options.

Reusable
email + SharePoint transfer modules.

500 EUR/month
to 2 EUR/month.

Results

  • Cost reduction: Email delivery dropped from 500 EUR/month to 2 EUR/month.

  • Similar cost drop for SharePoint transfers after removing Logic App usage.

  • Faster operations with fewer moving parts and consolidated logging.

  • Scalable onboarding: New flows added via configuration rather than new Logic Apps.

let's work together

Send info