Your Terraform Runs as AI Memory, Without Leaking a Single Secret

Building an HCP Terraform connector for cognee, where the hardest part was not the sync but keeping passwords out of memory.

When an infrastructure change fails at 2 a.m., the first question is always “Why did the last networking apply fail, and has this happened before?” The answer is somewhere in HCP Terraform’s run history, spread across plan logs, error messages and commit messages.

For Mergetober, cognee’s integration hackathon, I built a connector for HCP Terraform (formerly Terraform Cloud), issue #4717. It syncs workspaces and runs into cognee so you can ask questions like that one. The pull request is cognee-community #406.

The issue had one clear warning in it: plan output can contain secrets. That warning shaped the whole design.

The setup in brief

cognee is an open-source memory layer for AI agents. cognee.remember(...) builds a knowledge graph from documents, and cognee.recall(...) answers questions from it. dlt is a Python data loading library, and the connector is a dlt resource: a generator of documents and "tombstones" that tell cognee what to forget.

import os
import cognee
from cognee_community_connector_terraform_cloud import terraform_cloud_source
await cognee.remember(
    terraform_cloud_source(os.environ["TFE_ORGANIZATION"], token=os.environ["TFE_TOKEN"]),
    dataset_name="terraform",
    primary_key="id",
    write_disposition="merge",  # REQUIRED, the add pipeline defaults to "replace"
    max_rows_per_table=0,
    self_improvement=False,
)
answer = await cognee.recall("Why did the last networking apply fail?", datasets=["terraform"])

The environment variable names TFE_ORGANIZATION, TFE_TOKEN and TFE_HOSTNAME are the same ones the official tfe Terraform provider uses, so they feel familiar to anyone who already works with HCP Terraform. A read-only team token is enough, and HCP Europe and Terraform Enterprise hosts work too.

What gets stored

The connector produces two kinds of documents:

• One per workspace: its name, organization, Terraform version, execution mode, connected repository and branch, working directory, auto-apply setting, tags and description. It deliberately leaves out fields that change on every run, like the resource count and lock state. Otherwise the workspace document would change after every apply, and cognee would re-process it for nothing.
• One per finished run: status, operation, what triggered it, the commit, the plan and apply counts, the planned resource changes, and any error diagnostics.

You can choose which workspaces to sync by name, by wildcard like "*-prod", by tags, or by project. Speculative runs, the plan-only runs created for pull requests, are opt-in.

Why plan output is dangerous

Terraform hides values you mark as sensitive. Everything else is printed in plain text in the plan. A database password passed as a normal variable, a connection string, an access key in a resource argument: all of it can end up in a plan log.

If the connector stored plan logs as they are, it would copy those secrets into a knowledge graph that anyone with access to the dataset can query. So there are two separate layers of protection.

Layer 1: keep a summary, not the plan. The connector reads the plan and apply logs, both plain text and Terraform’s structured JSON output, and keeps only three things: the resource addresses with their action (aws_db_instance.main must be replaced), the Plan: summary line, and the summaries of any errors. Attribute values are never read at all. Most secrets never get near a document.

Layer 2: scrub everything anyway. Every text field the connector emits, including run messages, commit messages, workspace descriptions and the plan summary, goes through a function called redact.scrub first. It strips terminal color codes (so they cannot split a secret into pieces that slip past a pattern), then replaces anything that looks like:

• private keys
• AWS access keys
• GitHub, GitLab, Slack and HCP Terraform tokens
• JWTs and Azure account keys
• credentials inside URLs, like postgres://admin:hunter22@db/app
• secret-looking assignments such as password = ... or client_secret: ...

with a marker like [REDACTED:aws_access_key]. The number of redactions is reported in the sync stats, so you can see it working. The plan log download links are signed URLs; they are used once, never stored or logged, and the API token is never sent to them.

The README is honest about the limit: pattern matching lowers the risk, it cannot promise to catch every format. That is exactly why layer 1 exists.

The bug: a password with spaces

An independent review of my PRs found a hole in layer 2, and it was a good one.

The pattern for secret assignments matched a value up to the first space. So this line:

password = "correct horse battery staple"

matched nothing at all. The value has spaces, so the pattern never found its end, and the whole password went through untouched. The same was true for export DB_PASSWORD="my secret pass". Single-word passwords were redacted fine, which is why my tests passed.

The fix parses quoted and unquoted values separately. A quoted value now runs to its closing quote, spaces and escaped quotes included, and is redacted as a whole. A test checks it both on the scrubber directly and through a full sync, with a multi-word password planted in a workspace description and a run message.

While testing the fix, I found something else. I timed the scrubber on adversarial input, and a long line of dotted text like a.a.a.a...password took 2.9 seconds at 8,000 characters, and grew quadratically from there. A large log line could stall a sync for minutes. The cause was an unbounded part of the pattern that tried every possible starting point. Limiting how long a key name can be brought the same input down to 0.05 seconds, and a regression test now keeps it there.

Syncing runs efficiently

The run listing endpoints are rate limited to 30 requests per minute, and they have no “created after” filter. So the connector lists runs newest first, 100 per page, and stops paging as soon as it reaches runs older than the last sync. The API documentation does not actually state the sort order, so the connector only stops early while the answers really are newest first. If they ever are not, it keeps paging instead of silently missing runs.

A run is only ingested once it finishes. Runs that are still planning or applying are remembered and checked again later with GET /runs/{id}. Otherwise a single run would be processed once for every status it passed through.

What the live test found

I ran the connector against a real, free HCP Terraform organization with two tagged workspaces and real runs. The very first live sync failed.

The organization-wide run listing answered HTTP 400 “Invalid include parameter” for plan and apply. Only the single-run endpoint accepts those includes. Every sync would have failed, while every test passed, because my test fake happily accepted the includes.

The fix lists runs without includes and reads each new finished run once through /runs/{id}, which does accept them. More importantly, the test fake now rejects those includes exactly like the real API, so the mistake cannot come back unnoticed.
After the fix, the live run looked like this:

Forgetting deleted workspaces safely

Runs cannot be deleted on their own in HCP Terraform. So the connector forgets at the workspace level: a workspace that is deleted, no longer visible, or deselected is removed along with all of its runs.

A workspace listing that suddenly comes back empty forgets nothing until the next run confirms it, which is the last row in the table above. A page missing its data list raises an error instead of being read as "nothing exists".

Lessons

1. When data can contain secrets, use two layers. Avoid reading the dangerous data at all, then scrub what is left anyway.
2. Test your patterns with realistic secrets. “hunter2” is not how real passwords look. Real ones have spaces, quotes and symbols.
3. Time your regular expressions on hostile input. A pattern that works can still be slow enough to stall everything.
4. Your fake API should reject what the real API rejects. My tests passed while every real sync failed.

The connector has 77 tests that run on Python 3.10, 3.11 and 3.13 without an HCP Terraform account.

Links

• Pull request: topoteretes/cognee-community #406
• Issue: topoteretes/cognee #4717
• cognee: github.com/topoteretes/cognee

Built for Mergetober 2026, cognee’s open-source integration hackathon.