AWS’s Agentic Data Ingestion is a reference solution for automating parts of ETL—extracting, transforming and loading data. Amazon Bedrock models propose field mappings and transformation code; agents on Amazon Bedrock AgentCore and Step Functions coordinate the workflow, and AWS Glue runs the jobs. AWS describes S3 uploads that trigger individual runs and JDBC database loads that run on demand or on a schedule. AWS’s implementation guide describes a ten-stage process with human approval gates.
Before adapting the workflow, check that its input cadence, review points and test data fit your pipeline. They shape how the sample operates, not just how it is deployed.
How files and database sources enter the workflow
For files, each upload to Amazon S3 starts a separate run. The sample supports CSV, TSV, pipe-delimited text, JSON, Parquet, Avro, ORC and Excel .xlsx files; an Excel run processes one worksheet. For databases, the sample connects through JDBC to PostgreSQL, Amazon Aurora, MySQL, MariaDB, Oracle or SQL Server. A database load can run on demand or on a schedule. Incremental loads use a timestamp or numeric watermark to select rows newer than the previous run.
| Input path | Run cadence | Test data |
| Files in Amazon S3 | One run per uploaded file | A 100-row sample, tested in a Glue interactive session using the job’s Spark version |
| JDBC database | On demand or scheduled; incremental loads use a timestamp or numeric watermark | Rows generated from the schema, because the test session cannot access the database |
That cadence is event-triggered for files and scheduled or on demand for database loads, rather than continuous streaming. For outputs, the sample can write Apache Iceberg tables in S3 Tables, relational databases through JDBC, or partitioned Parquet files to an S3 bucket. S3 Tables supports append, overwrite and merge operations.
The ten stages from input to Glue job
The reference workflow proceeds in this order:
- Validate the file. Check the incoming file before processing.
- Extract its schema. Identify the fields and their types.
- Compare schemas. Assess the incoming structure against the target.
- Map fields. Propose which source fields correspond to target fields.
- Generate transformation code. Create the transformation needed for the mapped data.
- Test the code. Run the transformation through the applicable test path.
- Provision target infrastructure. Prepare the destination resources.
- Create the Glue job. Set up the job that will run the transformation.
- Record quality checks. Capture checks such as record counts, nulls and schema compliance.
- Run the pipeline. Execute the Glue job.
Where human review and code controls apply
Approval gates follow field mapping and code testing. Mapping review can be skipped only when every proposed field pair has high confidence and a matching name supports it; reused mappings default to review. The gates fail closed if the workflow cannot determine whether a review is required, record the reviewer’s identity and time out after 72 hours.
The model does not write the whole job script. Reviewed Jinja2 templates provide target-specific scaffolding; the model typically supplies a 5–15-line transformation block. The workflow checks that block against an allowlist and validates the rendered job’s required parameters before creating the Glue job.
File transformations are tested in an AWS Glue interactive session against 100 sample rows, using the same Spark version as the job. Database tests use rows generated from the schema rather than records from the source database. That distinction matters when assessing what a passing test covers: the JDBC test exercises generated inputs, not a read from the connected database.
A self-healing agent can start one retry for certain schema or type errors and out-of-memory failures. It skips a retry if the failed job may already have written data, or if retrying cannot fix the problem, such as a permissions error.
Deployment requirements and regional setup
The sample requires AWS permissions in the deployment Region and in us-east-1, access to Claude Sonnet 4.6, Node.js 20 or later, Python 3.12 or later, the AWS CLI and Docker. You also need ARM64 image-building capability or QEMU, about 100 GB of free disk space and an authenticator app. Deployment creates eight CloudFormation stacks. The first model call requires one-time account setup with the deploying user’s credentials; without it, the AI steps fail with AccessDeniedException.
The sample uses the us.anthropic.claude-sonnet-4-6 cross-Region inference profile, which AWS describes as available only from US AWS Regions. Deployments outside the US need a different model ID. The sample also uses Cognito sign-in with multifactor authentication enabled by default, but all signed-in users have full dashboard access; identity federation and role-based dashboard access are not included.
For security, the sample tests generated code in isolation under a read-only role. Pipeline jobs and agent runtimes use separate IAM roles, with scoped permissions and a permissions boundary for pipeline jobs. Data is encrypted in transit and at rest, while JDBC credentials are referenced through Secrets Manager ARNs.
What AWS’s cost estimates cover
AWS estimates a sample-scale run at $0.10–$0.25 and an idle deployment at $15–$20 per month. The estimates use public list prices for a test deployment in us-west-2 with seven- or eight-column sample files.
| Cost item | AWS estimate | Scope |
| Sample-scale run | $0.10–$0.25 per run | Sample files in the us-west-2 test deployment |
| Idle deployment | $15–$20 per month | Sample deployment, with no run cadence specified |
| Bedrock model calls | $0.05–$0.14 per run | Sample-scale run |
| Glue interactive-session tests | $0.01–$0.02 per attempt | Two G.1X workers in the test deployment |
| Glue job | About $0.02 per run | About 0.04 DPU-hours |
| AgentCore runtime, Step Functions, DynamoDB, S3 and CloudWatch | Under $0.05 per run combined | Sample-scale run |
AWS attributes about $14 of the idle estimate to two WAF web ACLs and rules, up to $5 to CloudWatch depending on free-tier use, and about $1 to storage and one secret. These are estimates for the stated sample setup, not a workload-independent rate.