ffiec-data-connect (Python)
Output formats
list vs pandas vs polars; date format options; null-handling with pd.NA and np.nan; per-column dtypes.
Every collect_* function accepts three shape-and-format parameters:
output_type— shape of the returned object (list of dicts, DataFrame, raw bytes, etc.).date_output_format— type of the date column (string as-returned,"YYYYMMDD", or adatetimeobject).force_null_types— how null values are represented in numeric columns (pd.NAvsnp.nan).
All three are accepted on every collect_* method for API symmetry, so you
can switch methods without hitting TypeError on an unrecognized kwarg. That
said, not all combinations are meaningful — raw-bytes outputs bypass the
parsing layer entirely, and list-of-strings methods have no typed null columns
to apply force_null_types to. The library emits a UserWarning at runtime
whenever a parameter is passed but can’t take effect, so you know to drop it.
See Ignored-parameter warnings below for the
full matrix.
How types are detected
Types come from the XBRL unit reference on each fact in the FFIEC response, not from Python introspection. The mapping is fixed:
| XBRL unit | Example value | Python type | Notes |
|---|---|---|---|
USD | "1500000" | np.int64 | Monetary. Divided by 1,000 (see below). |
PURE | "1.25" | np.float64 | Ratios, percentages. |
NON-MONETARY | "0.85" | np.float64 | Non-monetary numerics. |
| Boolean | "true" / "false" | np.bool_ | Boolean indicators. |
| Other | "text" | str | Free text. |
The detected type is preserved in the data_type column of the output, so
you always know which of int_data / float_data / bool_data / str_data
to read.
USD scaling
USD values are scaled down by 1,000 via integer division (//). A Call
Report line item reported as 1,500,000 in the raw XBRL ends up as 1500 in
int_data. This matches how the FFIEC itself presents Call Report amounts in
their UI — figures are in thousands of dollars unless otherwise labeled.
Multiply by 1,000 if you need raw dollars.
output_type
| Value | Returns | Available on | When to use |
|---|---|---|---|
"list" | list[dict] (or list[str] for dates-only endpoints) | all 7 methods | Piping into JSON, custom processing. |
"pandas" | pandas.DataFrame | all 7 methods | Default for analysis. |
"polars" | polars.DataFrame | all 7 methods (requires the optional [polars] extra) | Large result sets, better memory profile. |
"xbrl" | bytes — raw UTF-8 XBRL XML (starts with <?xml) | collect_data, collect_ubpr_facsimile_data | Downstream XBRL processing, archival. |
"pdf" | bytes — raw PDF file (starts with %PDF) | collect_data only (UBPR endpoint is XBRL-only per FFIEC) | Auditor-readable archive per quarter. |
"bytes"is a deprecated alias for"xbrl". Oncollect_ubpr_facsimile_datait’s translated transparently and emits aDeprecationWarning. On every other method it raisesValidationErrorafter the warning (was silently misbehaving before 3.0.0rc4). Migrate to"xbrl".
Polars requires the extra
output_type="polars" without the polars dependency installed raises
ValidationError with an install hint. Previously the enhanced-layer
methods silently fell back to a list on missing polars — fixed in 3.0.0
for consistency with collect_data.
pip install 'ffiec-data-connect[polars]'
List output
Each row is a plain dict:
{
"mdrm": "RCONK280",
"rssd": "37",
"quarter": "6/30/2022",
"data_type": "int",
"int_data": 0,
"float_data": None,
"bool_data": None,
"str_data": None,
}
One — and only one — of int_data / float_data / bool_data / str_data is
populated per row, indicated by data_type.
Pandas output
DataFrame with nullable dtypes:
| Column | dtype | Nullable? |
|---|---|---|
mdrm | object | No |
rssd | object | No |
quarter | object or datetime64[ns] | No |
data_type | object | No |
int_data | Int64 | Yes (pd.NA) |
float_data | float64 | Yes (NaN) |
bool_data | boolean | Yes (pd.NA) |
str_data | string | Yes (pd.NA) |
Int64 (capital-I) is the nullable integer dtype. Regular int64 can’t hold NA,
which is why the library defaults to Int64 — it preserves integer display even when
other rows in the column are null.
Polars output
Polars types map directly: Int64, Float64, Boolean, Utf8, Date. All
support nulls natively.
Raw-bytes outputs: "xbrl" and "pdf"
Both return the FFIEC server’s response verbatim — no parsing, no normalization. Suitable for archival, downstream XBRL tooling, or auditor handoff.
# Raw UTF-8 XBRL (available on collect_data and collect_ubpr_facsimile_data)
xbrl_bytes = collect_data(
creds, reporting_period="12/31/2024", rssd_id="480228",
series="call", output_type="xbrl",
)
assert xbrl_bytes.startswith(b"<?xml")
Path("480228_2024Q4.xbrl").write_bytes(xbrl_bytes)
# Raw PDF (collect_data, Call Report series only)
pdf_bytes = collect_data(
creds, reporting_period="12/31/2024", rssd_id="480228",
series="call", output_type="pdf",
)
assert pdf_bytes.startswith(b"%PDF")
Path("480228_2024Q4.pdf").write_bytes(pdf_bytes)
output_type="pdf" with series="ubpr" raises ValidationError locally
before any network call — the UBPR endpoint is XBRL-only per the FFIEC spec.
"xbrl" normalizes the UTF-8 BOM that the UBPR endpoint emits, so
responses from both facsimile endpoints arrive with a clean <?xml prolog.
If you were relying on the BOM (unlikely), decode with encoding="utf-8-sig"
would still work — the stripped bytes round-trip through UTF-8 cleanly.
date_output_format
Controls the type of the quarter / datetime column (wherever the method
returns date data).
| Value | Example | Python/pandas type |
|---|---|---|
"string_original" (default) | "12/31/2023" | str / object |
"string_yyyymmdd" | "20231231" | str / object |
"python_format" | datetime(2023,12,31, tzinfo=America/New_York) | tz-aware datetime |
"python_format" is what you want for any time-series analysis — it lets
pandas handle resampling, joins on date, and rolling windows without
string-parsing.
Timezone. FFIEC emits all of its date/time values in Washington, DC
local time but sends no tz marker on the wire. In "python_format" mode,
the library labels the result as America/New_York (via zoneinfo), so
DST is handled correctly — a July submission carries EDT (UTC-4) and a
January one carries EST (UTC-5). This means you can compare timestamps
across quarters, mix them with UTC data (.astimezone(timezone.utc)), or
convert to any other zone without reattaching a tz yourself. If you pass
an already-tz-aware datetime into the library, your tz is preserved —
we only label naive values.
Availability note. Prior to 3.0.0, date_output_format was a stub
on three methods — collect_reporting_periods, collect_ubpr_reporting_periods,
and collect_filers_submission_date_time. Passing a non-default value
had no effect (the library silently returned the "string_original"
shape). 3.0.0 implements the conversion and attaches the
America/New_York timezone label; all three now honor the parameter.
Error handling for bad dates
In "python_format" mode, a date string the library can’t parse raises
ValidationError. The alternative — silently returning a string — would
violate the documented return type and break downstream .year / .month
access. In the string modes, unparseable input passes through unchanged
with a logger.debug trail.
force_null_types — integer display
The most common gripe: integers printed as 100.0 instead of 100. Root
cause: when a pandas column mixes integers and np.nan, pandas upcasts
everything to float.
The library defaults to pd.NA (the pandas-native null), which preserves
Int64 dtype. You can override on methods that actually produce typed
columns:
| Value | Null marker | Integer column dtype |
|---|---|---|
None (default) | pd.NA | Int64 (displays 100) |
"pandas" | pd.NA | Int64 (same as default) |
"numpy" | np.nan | float64 (displays 100.0) |
Set force_null_types="numpy" only when you’re feeding data into legacy
code that expects np.nan. For new analysis, leave it at the default.
Where force_null_types has an effect
force_null_types is accepted on all 7 collect_* methods (so you
can pass it without a TypeError when switching methods), but it only
does real work on the two that produce typed numeric DataFrames:
| Method | force_null_types effect |
|---|---|
collect_data | Real — switches int_data column null marker |
collect_ubpr_facsimile_data | Real — same |
collect_reporting_periods | No-op (list of date strings, no typed columns) — emits UserWarning |
collect_ubpr_reporting_periods | No-op — emits UserWarning |
collect_filers_since_date | No-op (list of RSSD IDs) — emits UserWarning |
collect_filers_submission_date_time | No-op (string/datetime columns, no numerics) — emits UserWarning |
collect_filers_on_reporting_period | No-op (panel metadata, no numerics) — emits UserWarning |
The warnings (new in 3.0.0) surface the no-op at runtime rather than leaving it buried in the docstring.
data = collect_data(
creds,
rssd_id="480228",
reporting_period="12/31/2025",
series="call",
output_type="pandas",
force_null_types="pandas", # explicit — or omit for same behavior
)
Ignored-parameter warnings
The library surfaces three categories of silently-ignored parameter
combinations at runtime as UserWarnings. If you see one, drop the
argument or change output_type — the library is telling you your value
has no effect.
| Combination | Why it’s a no-op | Emitted by (new in 3.0.0) |
|---|---|---|
output_type="xbrl" or "pdf" + force_null_types=<non-None> | Raw-bytes outputs bypass parsing — null-type selection can’t apply | collect_data, collect_ubpr_facsimile_data |
output_type="xbrl" or "pdf" + date_output_format=<non-default> | Same reason — date formatting can’t apply to bytes | collect_data |
force_null_types=<non-None> on a method with no typed null columns | Listed above — 5 methods accept the parameter for API symmetry only | collect_reporting_periods, collect_ubpr_reporting_periods, collect_filers_since_date, collect_filers_submission_date_time, collect_filers_on_reporting_period |
To silence:
import warnings
warnings.filterwarnings("ignore", category=UserWarning, module="ffiec_data_connect")
But the cleaner fix is almost always to drop the argument — if the warning fires, your code has a value that’s going nowhere.
Pattern: pivot typed columns into a single value column
For analysis it’s often more useful to have one value column than four typed ones:
import pandas as pd
def collapse_value(df: pd.DataFrame) -> pd.DataFrame:
"""Collapse int_data/float_data/bool_data/str_data into one 'value' column."""
out = df.copy()
out["value"] = out["int_data"].astype("object")
out["value"] = out["value"].fillna(out["float_data"].astype("object"))
out["value"] = out["value"].fillna(out["bool_data"].astype("object"))
out["value"] = out["value"].fillna(out["str_data"].astype("object"))
return out[["mdrm", "rssd", "quarter", "data_type", "value"]]
This is not built in because it loses dtype — one of the library’s correctness guarantees. Do it at the edge of your pipeline, not in the middle.
See also
- Bulk download — combining many collect_data calls.
- Troubleshooting § Integer Display — if you’re seeing
.0.