cendat ver 0.5.0

A Lifeline for when You’re Drowning in Variables

When you’re not sure exactly what you’re looking for, finding the right variables can be tough, especially for huge products like the ACS. And while list_variables was meant to help with that, it wasn’t enough on its own. That’s why ver 0.5.0 has three new methods: list_groups, set_groups, and describe_groups. The first of these work almost exactly like the corresponding *_variables methods, except we can use them to focus our variable exploration to thematically contiguous sets of variables as an intermediate step. In other words, we can explore and lock in the concepts by which variables are grouped before we lock in our variable selections. The describe_groups method provides hierarchical listings of nested variable relationships within groups and in many cases will obviate the need for list_variables altogether. Let’s look at an example where we want to focus on poverty in the ACS 5 year product for 2023.

import os
from dotenv import load_dotenv
from cendat import CenDatHelper

load_dotenv()

cdh = CenDatHelper(key=os.getenv("CENSUS_API_KEY"))

cdh.list_products(2023, r"acs/acs5\)")
cdh.set_products()
for group in cdh.list_groups(patterns=["sex by age", "poverty"]):
    print(group['name'], group['description'])

✅ API key loaded successfully.
✅ Product set: 'ACS 5-Year Detailed Tables (2023/acs/acs5)' (Vintage: [2023])
B17007 Poverty Status in the Past 12 Months of Unrelated Individuals 15 Years and Over by Sex by Age
B17001 Poverty Status in the Past 12 Months by Sex by Age
B17001G Poverty Status in the Past 12 Months by Sex by Age (Two or More Races)
B17001H Poverty Status in the Past 12 Months by Sex by Age (White Alone, Not Hispanic or Latino)
B17001E Poverty Status in the Past 12 Months by Sex by Age (Native Hawaiian and Other Pacific Islander Alone)
B17001F Poverty Status in the Past 12 Months by Sex by Age (Some Other Race Alone)
B17001C Poverty Status in the Past 12 Months by Sex by Age (American Indian and Alaska Native Alone)
B17001D Poverty Status in the Past 12 Months by Sex by Age (Asian Alone)
B17001A Poverty Status in the Past 12 Months by Sex by Age (White Alone)
B17001B Poverty Status in the Past 12 Months by Sex by Age (Black or African American Alone)
B17001I Poverty Status in the Past 12 Months by Sex by Age (Hispanic or Latino)

We’re interested in poverty overall, so we can lock in group B17001. Then we’ll invoke describe_groups to see what the variable relationships look like.

cdh.set_groups("B17001")
cdh.describe_groups()

✅ Groups set: B17001

--- Group: B17001 (Poverty Status in the Past 12 Months by Sex by Age) ---

  Product: ACS 5-Year Detailed Tables (2023/acs/acs5) (Vintage: 2023)
      B17001_001E: Total:
        B17001_002E: Income in the past 12 months below poverty level:
          B17001_003E: Male:
            B17001_004E: Under 5 years
            B17001_005E: 5 years
            B17001_006E: 6 to 11 years
            B17001_007E: 12 to 14 years
            B17001_008E: 15 years
            B17001_009E: 16 and 17 years
            B17001_010E: 18 to 24 years
            B17001_011E: 25 to 34 years
            B17001_012E: 35 to 44 years
            B17001_013E: 45 to 54 years
            B17001_014E: 55 to 64 years
            B17001_015E: 65 to 74 years
            B17001_016E: 75 years and over
          B17001_017E: Female:
            B17001_018E: Under 5 years
            B17001_019E: 5 years
            B17001_020E: 6 to 11 years
            B17001_021E: 12 to 14 years
            B17001_022E: 15 years
            B17001_023E: 16 and 17 years
            B17001_024E: 18 to 24 years
            B17001_025E: 25 to 34 years
            B17001_026E: 35 to 44 years
            B17001_027E: 45 to 54 years
            B17001_028E: 55 to 64 years
            B17001_029E: 65 to 74 years
            B17001_030E: 75 years and over
        B17001_031E: Income in the past 12 months at or above poverty level:
          B17001_032E: Male:
            B17001_033E: Under 5 years
            B17001_034E: 5 years
            B17001_035E: 6 to 11 years
            B17001_036E: 12 to 14 years
            B17001_037E: 15 years
            B17001_038E: 16 and 17 years
            B17001_039E: 18 to 24 years
            B17001_040E: 25 to 34 years
            B17001_041E: 35 to 44 years
            B17001_042E: 45 to 54 years
            B17001_043E: 55 to 64 years
            B17001_044E: 65 to 74 years
            B17001_045E: 75 years and over
          B17001_046E: Female:
            B17001_047E: Under 5 years
            B17001_048E: 5 years
            B17001_049E: 6 to 11 years
            B17001_050E: 12 to 14 years
            B17001_051E: 15 years
            B17001_052E: 16 and 17 years
            B17001_053E: 18 to 24 years
            B17001_054E: 25 to 34 years
            B17001_055E: 35 to 44 years
            B17001_056E: 45 to 54 years
            B17001_057E: 55 to 64 years
            B17001_058E: 65 to 74 years
            B17001_059E: 75 years and over

Okay, this is cool, right?! The nesting makes it much easier to see how these variables are related to one another. Let’s say we want the total overall and the total below poverty: B17001_001E and B17001_002E. We can set those and proceed like normal. We’ll get data for all U.S. places.

cdh.set_variables(["B17001_001E", "B17001_002E"])
cdh.set_geos("place", "desc")

response = cdh.get_data(
    include_names=True,
    include_attributes=True,
    include_geoids=True
)

✅ Variables set:
  - Product: ACS 5-Year Detailed Tables (2023/acs/acs5) (Vintage: [2023])
    Variables: B17001_001E, B17001_002E
✅ Geographies set: 'place' (requires `within` for: state)
✅ Parameters created for 1 geo-variable combinations.
✅ Found 1 combinations. Building API queries...
ℹ️ Making 1 API call(s)...

Note that we have some new parameters in get_data. Like include_names, which I described in the last post, include_attributes and include_geoids allow us to bring in variables that don’t show up in our variable listings because they’re either standard summary level attributes or attributes of the statistics themselves. In ver 1.0.0 I’m going to change the defaults on include_names and include_geoids to True, but for now they all default to False for compatibility with prior versions. GEO_ID especially will be needed for the upcoming polygon features.

df = response.to_polars(
    concat=True,
    schema_overrides={
        'B17001_001E': int,
        'B17001_001M': int,
        'B17001_002E': int,
        'B17001_002M': int,        
    }
)
df.glimpse()

Rows: 32325
Columns: 16
$ NAME          <str> 'Abanda CDP, Alabama', 'Abbeville city, Alabama', 'Adamsville city, Alabama', 'Addison town, Alabama', 'Akron town, Alabama', 'Alabaster city, Alabama', 'Albertville city, Alabama', 'Alexander City city, Alabama', 'Alexandria CDP, Alabama', 'Aliceville city, Alabama'
$ GEO_ID        <str> '1600000US0100100', '1600000US0100124', '1600000US0100460', '1600000US0100484', '1600000US0100676', '1600000US0100820', '1600000US0100988', '1600000US0101132', '1600000US0101180', '1600000US0101228'
$ B17001_001E   <i64> 48, 2306, 4235, 651, 354, 33224, 22180, 14240, 3420, 2338
$ B17001_002E   <i64> 0, 508, 798, 76, 38, 1768, 3795, 3102, 182, 1037
$ B17001_001EA <null> None, None, None, None, None, None, None, None, None, None
$ B17001_002EA <null> None, None, None, None, None, None, None, None, None, None
$ B17001_001MA <null> None, None, None, None, None, None, None, None, None, None
$ B17001_001M   <i64> 71, 300, 69, 176, 227, 90, 158, 46, 639, 459
$ B17001_002M   <i64> 13, 170, 435, 53, 40, 465, 994, 554, 139, 369
$ B17001_002MA <null> None, None, None, None, None, None, None, None, None, None
$ state         <str> '01', '01', '01', '01', '01', '01', '01', '01', '01', '01'
$ place         <str> '00100', '00124', '00460', '00484', '00676', '00820', '00988', '01132', '01180', '01228'
$ product       <str> 'ACS 5-Year Detailed Tables (2023/acs/acs5)', 'ACS 5-Year Detailed Tables (2023/acs/acs5)', 'ACS 5-Year Detailed Tables (2023/acs/acs5)', 'ACS 5-Year Detailed Tables (2023/acs/acs5)', 'ACS 5-Year Detailed Tables (2023/acs/acs5)', 'ACS 5-Year Detailed Tables (2023/acs/acs5)', 'ACS 5-Year Detailed Tables (2023/acs/acs5)', 'ACS 5-Year Detailed Tables (2023/acs/acs5)', 'ACS 5-Year Detailed Tables (2023/acs/acs5)', 'ACS 5-Year Detailed Tables (2023/acs/acs5)'
$ vintage       <i32> 2023, 2023, 2023, 2023, 2023, 2023, 2023, 2023, 2023, 2023
$ sumlev        <str> '160', '160', '160', '160', '160', '160', '160', '160', '160', '160'
$ desc          <str> 'place', 'place', 'place', 'place', 'place', 'place', 'place', 'place', 'place', 'place'