New Features in cendat ver 0.4.1

This post covers some new cendat features.
Python
cendat
Census
API
Author

Lance Couzens

Published

August 22, 2025

New Version, New Features

Since the introductory post, I’ve added some new features to make cendat more robust, user-friendly, and capable:

  1. The biggest update is the addition of the tabulate() method for the CenDatResponse objects returned by get_data().

  2. The CenDatResponse methods to_polars() and to_pandas() got new functionality

    • Automatic concatenation via the new concat parameter, which defaults to False for backward compatibility with prior versions
    • Destringing–the Census API returns all data in strings, which prevents polars and pandas from inferring the schema. You can now set destring=True to force type conversion on the raw response data and allow polars and pandas to infer the schema automatically. This parameter defaults to False, and the schema may still be specified by the user via schema_overrides with or without destringing.

  3. The user can now specify the API timeout count in seconds via timeout in the get_data() method to accommodate longer requests–the default is 30 seconds.

  4. The get_data() method gained the preview_only parameter (boolean, defaults to False). If True, the number of API queries required to satisfy the request will be determined and messaged, but the queries will not be executed. This is a handy precursor step when dealing with very granular or unfamiliar geographic summary levels which could yield tens of thousands of queries.

  5. Pattern matching in the list_variables() method has been extended to the “concept” field.

Please consult the GitHub or PyPi repository landing pages for a full library reference–here is the syntax for tabulate():

tabulate(self, *variables, strat_by=None, weight_var=None, weight_div=None, where=None, logic=all, digits=1)

Generates and prints a frequency table.

  • *variables (str): One or more column names to include in the tabulation.
  • strat_by (str, optional): A column name to stratify the results by. Percentages and cumulative stats will be calculated within each stratum. Defaults to None.
  • weight_var (str, optional): The name of the column to use for weighting. If None, each row has a weight of 1. Defaults to None.
  • weight_div (int, optional): A positive integer to divide the weight by, useful for pooled tabulations across multiple product vintages. weight_var must be provided if this is used. Defaults to None.
  • where (str | list[str], optional): A string or list of strings representing conditions to filter the data before tabulation. Each condition should be in a format like "variable operator value" (e.g., "AGE > 30"). Defaults to None.
  • logic (callable): The function to apply when multiple where conditions are provided. Use all for AND logic (default) or any for OR logic.
  • digits (int): The number of decimal places to display for floating-point numbers in the output table. Defaults to 1.

Examples

These primarilly illustrate usage of the new tabulate() method.

Quick ACS Aggregate Analysis

Here we want to see how many counties in the U.S., as of 2023, had a million or more residents, and how many people lived in those counties.

%%time

from cendat import CenDatHelper
from dotenv import load_dotenv
import os

load_dotenv()

cdh = CenDatHelper(key=os.getenv('CENSUS_API_KEY'))

cdh.list_products(years=[2023], patterns=r"/acs/acs1\)")
cdh.set_products()
cdh.set_variables("B01001_001E") # total population
cdh.set_geos("050") # counties
response = cdh.get_data()

# how many counties
response.tabulate("state", where="B01001_001E > 1_000_000")

# how many people in those counties
response.tabulate("state", weight_var="B01001_001E", where="B01001_001E > 1_000_000")
✅ API key loaded successfully.
✅ Product set: 'ACS 1-Year Detailed Tables (2023/acs/acs1)' (Vintage: [2023])
✅ Variables set:
  - Product: ACS 1-Year Detailed Tables (2023/acs/acs1) (Vintage: [2023])
    Variables: B01001_001E
✅ Geographies set: 'county' (requires `within` for: state)
✅ Parameters created for 1 geo-variable combinations.
ℹ️ Fetching parent geographies for 'county'...
✅ Found 52 combinations for 'county' within the specified scope.
ℹ️ Making 52 API call(s)...
shape: (18, 5)
┌───────┬────┬──────┬──────┬────────┐
│ state ┆  n ┆  pct ┆ cumn ┆ cumpct │
╞═══════╪════╪══════╪══════╪════════╡
│    04 ┆  2 ┆  4.2 ┆    2 ┆    4.2 │
│    06 ┆ 10 ┆ 20.8 ┆   12 ┆   25.0 │
│    12 ┆  6 ┆ 12.5 ┆   18 ┆   37.5 │
│    13 ┆  1 ┆  2.1 ┆   19 ┆   39.6 │
│    17 ┆  1 ┆  2.1 ┆   20 ┆   41.7 │
│    24 ┆  1 ┆  2.1 ┆   21 ┆   43.8 │
│    25 ┆  1 ┆  2.1 ┆   22 ┆   45.8 │
│    26 ┆  2 ┆  4.2 ┆   24 ┆   50.0 │
│    27 ┆  1 ┆  2.1 ┆   25 ┆   52.1 │
│    32 ┆  1 ┆  2.1 ┆   26 ┆   54.2 │
│    36 ┆  6 ┆ 12.5 ┆   32 ┆   66.7 │
│    37 ┆  2 ┆  4.2 ┆   34 ┆   70.8 │
│    39 ┆  2 ┆  4.2 ┆   36 ┆   75.0 │
│    42 ┆  2 ┆  4.2 ┆   38 ┆   79.2 │
│    48 ┆  7 ┆ 14.6 ┆   45 ┆   93.8 │
│    49 ┆  1 ┆  2.1 ┆   46 ┆   95.8 │
│    51 ┆  1 ┆  2.1 ┆   47 ┆   97.9 │
│    53 ┆  1 ┆  2.1 ┆   48 ┆  100.0 │
└───────┴────┴──────┴──────┴────────┘
shape: (18, 5)
┌───────┬────────────┬──────┬────────────┬────────┐
│ state ┆          n ┆  pct ┆       cumn ┆ cumpct │
╞═══════╪════════════╪══════╪════════════╪════════╡
│    04 ┆  5,649,033 ┆  5.8 ┆  5,649,033 ┆    5.8 │
│    06 ┆ 28,013,381 ┆ 28.7 ┆ 33,662,414 ┆   34.5 │
│    12 ┆ 10,221,001 ┆ 10.5 ┆ 43,883,415 ┆   45.0 │
│    13 ┆  1,079,105 ┆  1.1 ┆ 44,962,520 ┆   46.1 │
│    17 ┆  5,087,072 ┆  5.2 ┆ 50,049,592 ┆   51.3 │
│    24 ┆  1,058,474 ┆  1.1 ┆ 51,108,066 ┆   52.4 │
│    25 ┆  1,623,952 ┆  1.7 ┆ 52,732,018 ┆   54.1 │
│    26 ┆  3,021,595 ┆  3.1 ┆ 55,753,613 ┆   57.1 │
│    27 ┆  1,258,713 ┆  1.3 ┆ 57,012,326 ┆   58.4 │
│    32 ┆  2,336,573 ┆  2.4 ┆ 59,348,899 ┆   60.8 │
│    36 ┆ 10,672,233 ┆ 10.9 ┆ 70,021,132 ┆   71.8 │
│    37 ┆  2,353,976 ┆  2.4 ┆ 72,375,108 ┆   74.2 │
│    39 ┆  2,559,151 ┆  2.6 ┆ 74,934,259 ┆   76.8 │
│    42 ┆  2,775,367 ┆  2.8 ┆ 77,709,626 ┆   79.7 │
│    48 ┆ 15,250,132 ┆ 15.6 ┆ 92,959,758 ┆   95.3 │
│    49 ┆  1,185,813 ┆  1.2 ┆ 94,145,571 ┆   96.5 │
│    51 ┆  1,141,878 ┆  1.2 ┆ 95,287,449 ┆   97.7 │
│    53 ┆  2,271,380 ┆  2.3 ┆ 97,558,829 ┆  100.0 │
└───────┴────────────┴──────┴────────────┴────────┘
CPU times: user 685 ms, sys: 177 ms, total: 863 ms
Wall time: 8.78 s

In 2023 there were 48 counties with populations over a million (mostly in California, Florida, New York, and Texas), and the total population across those counties was nearly 98 million.

CPS Microdata Tabulation

In this example, we’ll pull microdata from the Current Population Survey’s tobacco supplements for 2022 and 2023, and compare the proportions of established daily smokers across a selection of states. We’ll start with just 2022 to limit the output in our variable search. We’re looking for variables that indicate the respondent has smooked at least 100 cigarettes and is a current every day smoker.

cdh.list_products(years=[2022], patterns="/cps/tobacco")
cdh.set_products()
for var in cdh.list_variables(patterns=[r"smoke.*every day", "100"], logic=any):
    print(var['name'], var['label'])
✅ Product set: 'Current Population Survey: Tobacco Use Supplement (2022/cps/tobacco/sep)' (Vintage: [2022])
PEB7C How long have you smoked every day
PEC7D How long did you smoke every day
PEC7A Current some day smokers ever smoke every day for 6 months
PEA1 Smoked at least 100 cigarettes
PEA3 Categorize now smoke: every day  some days  not at all.
PEC8 Current some day smoker was every day  some days  or not at all 12 months ago
PTC7E Howm many cigs smoked each day when smoking every day
PEH6 Smoked every day  some days  or not at all
PEH5 For how long smoked EVERY DAY

It looks like we’re interested in PEA1 and PEA3–we’ll inspect the full variable dictionaries to make sure.

cdh.list_variables(patterns=["PEA1", "PEA3"], logic=any, match_in="name")
[{'name': 'PEA1',
  'label': 'Smoked at least 100 cigarettes',
  'concept': 'N/A',
  'group': 'N/A',
  'values': {'item': {'-2': "Don't Know",
    '2': 'No',
    '-1': 'Not in universe',
    '-3': 'Refused',
    '1': 'Yes',
    '-9': 'No Answer'}},
  'type': 'int',
  'sugg_wgt': 'PWNRWGT',
  'product': 'Current Population Survey: Tobacco Use Supplement (2022/cps/tobacco/sep)',
  'vintage': [2022],
  'url': 'http://api.census.gov/data/2022/cps/tobacco/sep'},
 {'name': 'PEA3',
  'label': 'Categorize now smoke: every day  some days  not at all.',
  'concept': 'N/A',
  'group': 'N/A',
  'values': {'item': {'2': 'Some_days',
    '1': 'Every_day',
    '-2': "Don't Know",
    '3': 'Not_at_all',
    '-3': 'Refused',
    '-9': 'No Answer',
    '-1': 'Not in universe'}},
  'type': 'int',
  'sugg_wgt': 'PWNRWGT',
  'product': 'Current Population Survey: Tobacco Use Supplement (2022/cps/tobacco/sep)',
  'vintage': [2022],
  'url': 'http://api.census.gov/data/2022/cps/tobacco/sep'}]

These variables get at what we want, and since we’re interested in the proportion among all adults, we can ignore the out of universe responsdents (likely never smokers that didn’t make it past a gate question… this is just an example, so we don’t need to dig any deeper).

Note that we can also see the suggested weights to use for analysis of these variables, so we’ll include that in our set variables list. First, though, we’ll expand our products to cover 2023 as well.

cdh.list_products(years=[2022, 2023], patterns="/cps/tobacco")
cdh.set_products()
cdh.set_variables(["PEA1", "PEA3", "PWNRWGT"])
✅ Product set: 'Current Population Survey: Tobacco Use Supplement (2022/cps/tobacco/sep)' (Vintage: [2022])
✅ Product set: 'Current Population Survey: Tobacco Use Supplement (2023/cps/tobacco/jan)' (Vintage: [2023])
✅ Product set: 'Current Population Survey: Tobacco Use Supplement (2023/cps/tobacco/may)' (Vintage: [2023])
✅ Variables set:
  - Product: Current Population Survey: Tobacco Use Supplement (2022/cps/tobacco/sep) (Vintage: [2022])
    Variables: PEA1, PEA3, PWNRWGT
  - Product: Current Population Survey: Tobacco Use Supplement (2023/cps/tobacco/jan) (Vintage: [2023])
    Variables: PEA1, PEA3, PWNRWGT
  - Product: Current Population Survey: Tobacco Use Supplement (2023/cps/tobacco/may) (Vintage: [2023])
    Variables: PEA1, PEA3, PWNRWGT

Next, we’ll set our geography of interest–states, specified by description (vs sumlev)–and get our data. Since this is a microdata request, we have to explicitly specify the geographies we want, corresponding to the summary level we set in set_geos(). We’ll specify our states in the within argument of get_data(). Let’s assume we’re interested in comparing the proportions between California (06) and Texas (48).

cdh.set_geos("state", "desc")
response = cdh.get_data(within={'state': ['06', '48']})
response.tabulate(
    "PEA1", "PEA3",
    strat_by="state",
    weight_var="PWNRWGT",
    weight_div=3,
)
✅ Geographies set: 'state'
✅ Parameters created for 3 geo-variable combinations.
ℹ️ Making 6 API call(s)...
shape: (43, 7)
┌───────┬──────┬──────┬──────────────┬──────┬──────────────┬────────┐
│ state ┆ PEA1 ┆ PEA3 ┆            n ┆  pct ┆         cumn ┆ cumpct │
╞═══════╪══════╪══════╪══════════════╪══════╪══════════════╪════════╡
│    48 ┆   -3 ┆   -3 ┆     12,151.4 ┆  0.1 ┆     12,151.4 ┆    0.1 │
│    48 ┆   -3 ┆   -2 ┆      7,862.0 ┆  0.0 ┆     20,013.4 ┆    0.1 │
│    48 ┆   -3 ┆   -1 ┆     31,283.4 ┆  0.1 ┆     51,296.9 ┆    0.2 │
│    48 ┆   -3 ┆    3 ┆      6,011.1 ┆  0.0 ┆     57,308.0 ┆    0.3 │
│    48 ┆   -2 ┆   -9 ┆      7,299.8 ┆  0.0 ┆     64,607.8 ┆    0.3 │
│    48 ┆   -2 ┆   -2 ┆      1,871.9 ┆  0.0 ┆     66,479.6 ┆    0.3 │
│    48 ┆   -2 ┆   -1 ┆     42,634.2 ┆  0.2 ┆    109,113.9 ┆    0.5 │
│    48 ┆   -2 ┆    2 ┆      2,698.4 ┆  0.0 ┆    111,812.3 ┆    0.5 │
│    48 ┆   -2 ┆    3 ┆     11,474.7 ┆  0.1 ┆    123,287.0 ┆    0.6 │
│    48 ┆   -1 ┆   -1 ┆          0.0 ┆  0.0 ┆    123,287.0 ┆    0.6 │
│    48 ┆    1 ┆   -9 ┆     12,524.6 ┆  0.1 ┆    135,811.6 ┆    0.6 │
│    48 ┆    1 ┆   -3 ┆      6,074.9 ┆  0.0 ┆    141,886.5 ┆    0.6 │
│    48 ┆    1 ┆   -2 ┆      2,451.6 ┆  0.0 ┆    144,338.1 ┆    0.6 │
│    48 ┆    1 ┆    1 ┆  1,230,269.3 ┆  5.5 ┆  1,374,607.4 ┆    6.2 │
│    48 ┆    1 ┆    2 ┆    521,851.7 ┆  2.3 ┆  1,896,459.0 ┆    8.5 │
│    48 ┆    1 ┆    3 ┆  2,993,540.7 ┆ 13.4 ┆  4,889,999.7 ┆   21.9 │
│    48 ┆    2 ┆   -9 ┆      9,233.3 ┆  0.0 ┆  4,899,233.0 ┆   21.9 │
│    48 ┆    2 ┆   -1 ┆ 11,019,066.7 ┆ 49.3 ┆ 15,918,299.7 ┆   71.2 │
│    48 ┆    2 ┆    2 ┆     63,562.7 ┆  0.3 ┆ 15,981,862.4 ┆   71.5 │
│    48 ┆    2 ┆    3 ┆  6,364,974.8 ┆ 28.5 ┆ 22,346,837.2 ┆  100.0 │
│     6 ┆   -9 ┆   -1 ┆      2,588.4 ┆  0.0 ┆      2,588.4 ┆    0.0 │
│     6 ┆   -3 ┆   -3 ┆      4,518.2 ┆  0.0 ┆      7,106.6 ┆    0.0 │
│     6 ┆   -3 ┆   -1 ┆     38,037.9 ┆  0.1 ┆     45,144.6 ┆    0.2 │
│     6 ┆   -3 ┆    3 ┆      3,679.8 ┆  0.0 ┆     48,824.4 ┆    0.2 │
│     6 ┆   -2 ┆   -9 ┆      2,126.6 ┆  0.0 ┆     50,951.0 ┆    0.2 │
│     6 ┆   -2 ┆   -3 ┆      1,726.2 ┆  0.0 ┆     52,677.2 ┆    0.2 │
│     6 ┆   -2 ┆   -2 ┆     13,550.3 ┆  0.0 ┆     66,227.5 ┆    0.2 │
│     6 ┆   -2 ┆   -1 ┆    100,758.4 ┆  0.3 ┆    166,985.9 ┆    0.6 │
│     6 ┆   -2 ┆    3 ┆     16,578.8 ┆  0.1 ┆    183,564.7 ┆    0.6 │
│     6 ┆   -1 ┆   -1 ┆          0.0 ┆  0.0 ┆    183,564.7 ┆    0.6 │
│     6 ┆    1 ┆   -9 ┆     11,301.9 ┆  0.0 ┆    194,866.6 ┆    0.6 │
│     6 ┆    1 ┆   -3 ┆     14,985.0 ┆  0.0 ┆    209,851.7 ┆    0.7 │
│     6 ┆    1 ┆   -2 ┆     14,227.3 ┆  0.0 ┆    224,078.9 ┆    0.7 │
│     6 ┆    1 ┆    1 ┆  1,054,186.4 ┆  3.5 ┆  1,278,265.3 ┆    4.3 │
│     6 ┆    1 ┆    2 ┆    576,434.2 ┆  1.9 ┆  1,854,699.5 ┆    6.2 │
│     6 ┆    1 ┆    3 ┆  3,532,156.8 ┆ 11.8 ┆  5,386,856.3 ┆   18.0 │
│     6 ┆    2 ┆   -9 ┆     18,986.8 ┆  0.1 ┆  5,405,843.1 ┆   18.0 │
│     6 ┆    2 ┆   -3 ┆      4,104.9 ┆  0.0 ┆  5,409,947.9 ┆   18.0 │
│     6 ┆    2 ┆   -2 ┆      7,485.7 ┆  0.0 ┆  5,417,433.7 ┆   18.1 │
│     6 ┆    2 ┆   -1 ┆ 16,344,706.7 ┆ 54.5 ┆ 21,762,140.4 ┆   72.5 │
│     6 ┆    2 ┆    1 ┆     13,137.2 ┆  0.0 ┆ 21,775,277.6 ┆   72.6 │
│     6 ┆    2 ┆    2 ┆     74,384.9 ┆  0.2 ┆ 21,849,662.4 ┆   72.8 │
│     6 ┆    2 ┆    3 ┆  8,153,180.9 ┆ 27.2 ┆ 30,002,843.3 ┆  100.0 │
└───────┴──────┴──────┴──────────────┴──────┴──────────────┴────────┘

We’re interested in the rows where PEA1 == 1 (have smoked 100+ cigarettes) and PEA3 == 1 (currently smoke every day). Since we’re pooling three supplement waves, we need to divide the weights by 3 to get totals that make sense. We also stratify by state to get results that are comparable across states.

In 2022/2023, 5.5% of adults in Texas were estimated to be established, daily cigarette smokers. The figure in California was 3.5%.

Citation

BibTeX citation:
@online{couzens2025,
  author = {Couzens, Lance},
  title = {New {Features} in Cendat Ver 0.4.1},
  date = {2025-08-22},
  url = {https://mostlyunoriginal.github.io/posts/2025-08-22-cendat-new-features/},
  langid = {en}
}
For attribution, please cite this work as:
Couzens, Lance. 2025. “New Features in Cendat Ver 0.4.1.” August 22, 2025. https://mostlyunoriginal.github.io/posts/2025-08-22-cendat-new-features/.