Inside Out Security Blog   /  

Beyond Privacy and DSARs: Public Data Requests (FOI) Are the Law in 50 States

Beyond Privacy and DSARs: Public Data Requests (FOI) Are the Law in 50 States


    Happy 2020!  The New Year brings Californians under the California Consumer Privacy Act (CCPA). CA consumers can ask state-based companies for all relevant data, and to request that companies delete their data. Not in compliance with CCPA? Don’t panic yet. California will not be enforcing the law for another six months. If you’re looking for solutions, we can help out.

    The CCPA also gives us an opportunity to talk about another type of law in effect in all 50 states. They are known collectively as public data access laws and are modeled on the federal government’s Freedom of Information Act (FOIA).  Private companies are excluded from FOI-type laws of course.

    Get the Free Essential Guide to US Data Protection Compliance and Regulations

    State governments, generally the executive and legislative branches but also state agencies, have to make their records available upon request to the public. State court have their own rules for accessing court records. To make matters more complicated, local city governments can have their variant of FOI laws. And there are exemptions built into all these FOI laws to restrict access under certain conditions.

    Freedom of Information for the People

    US states enacted their own FOI legislation back in the 1970s, mirroring the federal law. Generally, a particular state organization (branch of government or agency) would have to initially respond to a data request within a given period of time under a state law. The cost of these requests can vary depending on  how much staff time is involved. If you want to get a feel for these laws, you can check your own state’s public or open record act (see below for a snippet of New York’s own FOI law).

    exemptions in New York’s FOI statute
    Some of  the many exemptions in New York’s FOI statute.

    One interesting point to make is that while private companies are now facing data subject access requests or DSARs for the first time, state- and federal-level entities have long had to deal with these types of requests. While the new privacy laws, such as CCPA, call for free access to data (allowing for “reasonable charges” when the requests are excessive), the state FOI laws have too often been associated with highs administrative fees for the requestor: for example this case in Michigan involves processing fees over of over $40k!

    These fees are typically based on the hourly rate of the lowest-paid government worker involved. Key point: without automation, costs for manually searching through huge file systems and databases add up quickly!

    Along Comes MuckRock for FOI Automation

    Thankfully, there’s a great resource to understand how well states are doing in dealing with public record requests. I refer you to Muckrock, which is an non-profit that tracks FOI requests through their own clever web-based system. It’s worth nothing that even with help from Muckrock state public access requests have their own quirks, and you may still need the services of an attorney to help out with more complicated submissions.

    In any case, Muckrock provides a single portal to access the separate FOI online sites, along with providing useful information on exemptions, required response times, average fees, and some other stats on previous requests. If you want to get a handle on New Jersey’s FOI law, you can click at,  and the same URL pattern repeats for other states.

    average response times
    Cool graphic, courtesy of Muckrock, showing average response times (in days) for FOI requests.

    What if you needed to drill down a little and look at individual agencies with a state? Muckrock can help here as well. They provide RESTful Web APIs and a Github repository with sample Pythons scripts showing how to use the interface to access a ginormous database covering every federal, state, and local government agency in the US!

    Let’s say you wanted to learn the number of and the average delay in processing FOI requests to the NJ State Police. You’d use the agency variant of the API and fill in with the agency id, which happens to be 839: Click on the link to see the results in Muckrock’s visual interface.

    NJ State Police in this example
    Muckrock’s own visual interface displays results of agency lookup. NJ State Police in this example.

    And the answer is that 60 requests have been tracked, only 17 completed, and there’s an average response time of 60 days. Not bad, but not great either.

    How did I know the NJ State Police was assigned a Muckrock agency id of 839? Based on poking around  the APIs, I learned a brute-force approach was required, and so I downloaded the whole shebang using their script. This effectively dumps the  database of agency ids along with stats.  With over 14,000 data rows, it takes forever. However, I’ve done all the dirty work, and you can find the Muckrock  database, as a .csv file, in my Github repository.

    Yeah, it has the stats of the time of this writing. But six months from now you can search for the agency id in your Excel spreadsheet and then pull in the current stats using the API I showed above.

    And the Top Ten Agencies Are …

    I’m convinced this new decade will launch a new awareness of privacy among consumers with the CCPA helping to spur copycat legislation across the US. As I mentioned above, governmental agencies are under their own rules, but I suspect they’ll be more FOI requests, and new pressure to improve performance. If these agencies don’t, at least in some states, there can be significant penalties.

    With the dataset I placed in my Github repository, you can do some of your own interesting  analysis. I’ll close this post with an interesting factoid that answers the question: Which agencies at the state level  — excluding cities and local governments — have the longest responses for  completing FOI requests?

    Keeping in mind that this represents requests submitted through Muckrook, the answer is:

    ID Agency Average response time (days) Success rate (%) Number of requests Number of requests completed
    274 Massachusetts State Police 86 34 255 82
    354 New York State Police 84 21 79 16
    372 Virginia State Police 11 10 79 7
    435 Pennsylvania State Police 59 23 72 16
    423 Illinois State Police 28 39 71 28
    633 Michigan State Police 28 22 66 14
    827 Delaware State Police 109 25 64 15
    839 New Jersey State Police 60 30 60 17
    790 Connecticut State Police 137 25 59 14
    850 Vermont State Police 58 46 58 27



    What you should do now

    Below are three ways we can help you begin your journey to reducing data risk at your company:

    1. Schedule a demo session with us, where we can show you around, answer your questions, and help you see if Varonis is right for you.
    2. Download our free report and learn the risks associated with SaaS data exposure.
    3. Share this blog post with someone you know who'd enjoy reading it. Share it with them via email, LinkedIn, Twitter, Reddit, or Facebook.

    We're Varonis.

    We've been keeping the world's most valuable data out of enemy hands since 2005 with our market-leading data security platform.

    How it works