Python · Terraform · SFTP · Docker

A file transfer pipeline you can actually fail closed

Files land in an inbox. Some are empty. Some are the same file dropped twice. Some look fine until SFTP is down. This lab is a sanitized stand-in for the managed file transfer work I do — landing zone, validate, transform, deliver, audit.

Inbox, validate, and SFTP delivery with Terraform landing zones and an audit trail

Production MFT is not a happy-path script. Partners drop CSV, JSON, and HL7 into a landing zone. A worker has to decide, every time: is this file real, is it a duplicate, and did delivery actually succeed? If any of those answers is no, the original stays put and the failure is recorded — not swallowed by a retry that nobody can reconstruct later.

file-transfer-pipeline is that shape without client data or vendor lock-in. Terraform stands up a local Docker landing zone (and an AWS S3 sketch). Python does the rest.

The path a file takes

Every inbound file goes through the same five steps:

  1. Land in data/inbox/ (or an S3 prefix in the AWS sketch).
  2. Validate size, encoding, and format. HL7 gets a fast MSH gate here; deeper parsing lives in the companion HL7 toolkit.
  3. Transform a good file into a canonical JSON payload.
  4. Deliver to a local sink or SFTP.
  5. Audit the attempt in SQLite, including SHA-256 so the same content is not delivered twice.

What I wanted to show failing

The interesting cases are the ones that turn into tickets at 2 a.m. if you only test the green path:

Failure What the pipeline does
Empty or huge file Reject with a stable error code
Bad encoding / invalid JSON / header-only CSV Quarantine plus a .error.json sidecar
HL7 without MSH-9 or MSH-10 Reject before delivery
Same content dropped twice Skip via SHA-256 audit key
Downstream SFTP outage Keep the original in failed/ and record the error

Infra as code, not a console screenshot

I used Terraform at work, so the lab does too. infra/local uses the Docker provider to bring up SFTP and Postgres. infra/aws sketches S3 inbox / processed / failed buckets plus IAM — CI runs terraform validate; you should not apply it without a backend and an account. 

cd infra/local
terraform init
terraform plan

Run it

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
cp samples/partners.csv data/inbox/
file-pipeline --root data
pytest -q

After docker compose up -d or terraform apply in infra/local:

file-pipeline --root data --sftp

What this is not

It is not a MOVEit or GoAnywhere clone, and it is not production. It is the interviewable version of the job: a worker that fails closed, leaves an audit trail, and treats landing zones as code. The HL7 companion post covers the parser that explains why a message is bad after this pipeline’s MSH gate.

GitHub repo HL7 toolkit post Back to projects