Is there any frame of reference to compare this to? I.e what is the normal fluctuation/movement of that 'datasets available' number? For instance, it dropped by 18k between 9/13 and 10/13.
That’s interesting, I was looking at the ICE ERO statistics and not only do they show up as not having a published data source, only metadata, the corresponding dashboards at ice.gov/statistics seem to not be working. I don’t know what the state of either of those was before I looked yesterday, so I don’t know if this is a recent change or not.
I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did, especially in 2024 due to rule changes which roughly doubled the number of normal deportations. I was further surprised to find out that there were roughly 3.5 times more deportations under Biden than under the first Trump administration.
I suspect that what’s happening is that leadership under Biden did the sensible thing and very quietly snuck up on priority illegal aliens, whereas with Trump, all the media attention is causing similar people to circle the wagons and proactively take steps to avoid enforcement actions.
A big factor is that Trump actually got rid of the prioritization scheme in deportation courts that processes violent criminals (and similar) more quickly than people caught with a broken taillight.
He had the net effect of increasing the amount of time a dangerous immigrant would be inside the country while the system got clogged with all sorts of cases that the American public broadly agrees hardly matter.
> I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s
There were also more deportations the year before Trump's first term than the year of his first term.
Consider the chart of border encounters. This one shows the past through 2023 [0] and in 2024 we had 2.1 million encounters at the southern border [1]. It appears that deportations are a function of border encounters, and border encounters increased substantially during Biden's administration.
First Trump administration had a very strong pattern of talk and maybe a few actions… and then they seem to forget what they were doing and the result is little to no action.
Trump loves to talk about trade but when it came to China he put up some token tariffs and announced “I didn’t say it would be easy” and then did nothing the rest of his term.
> I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did
Really? It was very widely reported. Just wait until you see how many people Obama deported
"I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did..."
I think this is an essentially meaningless number until you also include the total number of illegals that were let in during the same period for each administration.
The more important number is how many people attempted crossing, which jumped significantly during Biden. This probably has something to do with their phenomenally cratered economies during COVID, multiple destabilized governments in LatAm, and severe climate migration.
That's a big part of why the Biden administration (on day one) proposed a bill that created a large economic development grant for LatAm while also modernizing border security.
Complete nothingburger. The malicious action would be to subtly change key datasets to promote some kind of biased narrative. It wouldn't be the removal of less than 1% of the datasets.
Comical to think the Trump admin would be the ones "scrubbing" data when they passed the OPEN Government Data Act as part of the Foundations for Evidence Based Policymaking Act at the beginning of 2019. The law that requires agencies to publish their data to the site!
Trump must really be playing 4d chess if one of the first actions in office is to "scrub" a few obscure and yet unidentified datasets from data.gov.
You can look at catalog.data.gov it shows totals. I'm comparing January 14th to today.
The biggest loss I see is Organizations - Department of Energy, 5473 to 3647.
I also see under Bureaus - Energy Programs, 4347 to 2521.
These are overlapping categories (-1826 on both).
There are others, but they seem smaller, a few hundred at most.
Looking closer, there was a major increase quite recently before this decrease. On January 8th there were 3617 DoE datasets. On the 14th it was up to 5473. Now we're back down to just 30 more than on the 8th.
Could it have been a publishing mistake, or some order to undo recent publications for review by the new admin?
I can't tell from the Internet Archive. The datasets that went away seem to have been added quite recently, between January 8th and 14th. There isn't a suitable capture between January 8th and 25th to see which tags or categories changed within the DoE datasets.
Perhaps the marker indicating where you are is supposed to work it's way up the list? I don't think what you've said here - in this thread - has been unreasonable.
I'm not sure if it's nefarious or not, but I would have to lean on the side of "If the goverment is quietly removing information then probably it's not for the benefit of the public.
Also from their own site at data.gov
> The United States Government’s open data site is designed to unleash the power of government open data to inform decisions by the public and policymakers, drive innovation and economic activity, achieve agency missions, and strengthen the foundation of an open and transparent government.
The part " to inform decisions by the public and policymakers" in particular stands out.
Anyway data sets is not exactly expensive to keep hosted for a government body so there's not really any excuse to scrub them unless they're polluted in some way ?
Even if it's polluted, you can mark it as is, with reasoning and keep it up, so you can learn what dirty/bad data is, and build quality detectors with the information contained in/with the "dirty" or "low quality" data set.
Open Data initiatives generally build upon the premise of immutability.
Data Sets can be alive, i.e. new data can be streamed into them, but they won't be purged, shortened or slimmed down.
You can expect more historical data to be opened as the digitization and organization continues, and don't expect this data to be randomly being removed from archives.
Even going one step further, if this data sets are refined and formatted, they are versioned, so you can always see the previous versions (e.g.: https://www.zenodo.org).
If you're starting to remove any data set from a data pool w/o any explanation and announcement, this means there's a new "Department of Truth" in the works, 99% of the time.
Is there any frame of reference to compare this to? I.e what is the normal fluctuation/movement of that 'datasets available' number? For instance, it dropped by 18k between 9/13 and 10/13.
I found https://uc3.cdlib.org/data-mirror/ which was a mirror a few years ago, but it's been taken down due to a lack of ongoing funding.
It would be a great start to have a mirror of just the metadata, to at least see what is being removed.
That’s interesting, I was looking at the ICE ERO statistics and not only do they show up as not having a published data source, only metadata, the corresponding dashboards at ice.gov/statistics seem to not be working. I don’t know what the state of either of those was before I looked yesterday, so I don’t know if this is a recent change or not.
I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did, especially in 2024 due to rule changes which roughly doubled the number of normal deportations. I was further surprised to find out that there were roughly 3.5 times more deportations under Biden than under the first Trump administration.
I suspect that what’s happening is that leadership under Biden did the sensible thing and very quietly snuck up on priority illegal aliens, whereas with Trump, all the media attention is causing similar people to circle the wagons and proactively take steps to avoid enforcement actions.
A big factor is that Trump actually got rid of the prioritization scheme in deportation courts that processes violent criminals (and similar) more quickly than people caught with a broken taillight.
He had the net effect of increasing the amount of time a dangerous immigrant would be inside the country while the system got clogged with all sorts of cases that the American public broadly agrees hardly matter.
It's like a cartoon version of a shitty manager.
https://www.themarshallproject.org/2019/07/16/trump-tried-to...
> I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s
There were also more deportations the year before Trump's first term than the year of his first term.
Consider the chart of border encounters. This one shows the past through 2023 [0] and in 2024 we had 2.1 million encounters at the southern border [1]. It appears that deportations are a function of border encounters, and border encounters increased substantially during Biden's administration.
0. https://www.nytimes.com/interactive/2023/10/29/us/illegal-bo...
1. https://www.migrationpolicy.org/news/fy2024-us-border-encoun....
Seems to be working for me: https://www.ice.gov/spotlight/statistics
> Biden’s administration deported far more people than Trump’s did
It seems like Trump’s biggest problem is, like himself, the people he put in charge are C-players, so getting anything complex done is hard.
Yes and: Infighting.
I once reported to a guy that set all us minions (product managers) against each other. It was awful.
First Trump administration had a very strong pattern of talk and maybe a few actions… and then they seem to forget what they were doing and the result is little to no action.
Trump loves to talk about trade but when it came to China he put up some token tariffs and announced “I didn’t say it would be easy” and then did nothing the rest of his term.
and they weren't token tariffs.
they were mostly ineffective tariffs, in the "tried so hard and got so far but in the end it doesn't even matter" sense, but he wasn't all talk
Trump killed the Trans Pacific Partnership.
(This was a surprise to me. I'm grateful that I'm not at risk of being extradited to rot in a chinese prison for reverse engineering some IoT POS.)
How would TPP have put you at risk for Chinese prison?
> I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did
Really? It was very widely reported. Just wait until you see how many people Obama deported
"I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did..."
I think this is an essentially meaningless number until you also include the total number of illegals that were let in during the same period for each administration.
The more important number is how many people attempted crossing, which jumped significantly during Biden. This probably has something to do with their phenomenally cratered economies during COVID, multiple destabilized governments in LatAm, and severe climate migration.
That's a big part of why the Biden administration (on day one) proposed a bill that created a large economic development grant for LatAm while also modernizing border security.
Complete nothingburger. The malicious action would be to subtly change key datasets to promote some kind of biased narrative. It wouldn't be the removal of less than 1% of the datasets.
Comical to think the Trump admin would be the ones "scrubbing" data when they passed the OPEN Government Data Act as part of the Foundations for Evidence Based Policymaking Act at the beginning of 2019. The law that requires agencies to publish their data to the site!
Trump must really be playing 4d chess if one of the first actions in office is to "scrub" a few obscure and yet unidentified datasets from data.gov.
It seems like some groups already expected this from efforts during the previous Trump admin. This article mentions a few, and more recent preparations: https://www.scientificamerican.com/article/scientists-scramb...
Do we have any idea what datasets have gone missing? More details please from anyone who has them.
You can look at catalog.data.gov it shows totals. I'm comparing January 14th to today.
The biggest loss I see is Organizations - Department of Energy, 5473 to 3647. I also see under Bureaus - Energy Programs, 4347 to 2521. These are overlapping categories (-1826 on both).
There are others, but they seem smaller, a few hundred at most.
Looking closer, there was a major increase quite recently before this decrease. On January 8th there were 3617 DoE datasets. On the 14th it was up to 5473. Now we're back down to just 30 more than on the 8th.
Could it have been a publishing mistake, or some order to undo recent publications for review by the new admin?
This seems like a super important observation.
If the number of dataset saw a massive jump (50%), then back down a week later, that seems more like the correction of an error.
Maybe its related to climate change and advantages of renewables?
I know DoE does "other things", but I don't expect these to be public anyway.
I can't tell from the Internet Archive. The datasets that went away seem to have been added quite recently, between January 8th and 14th. There isn't a suitable capture between January 8th and 25th to see which tags or categories changed within the DoE datasets.
Common Crawl's latest crawl was Jan 12th-25th, and the index is available.
[flagged]
This seems like some extreme jumping to conclusions, especially if no politically-notable or popular data sets appear to be touched.
For example, suppose the list of New York schools during 2020 [0] is removed. Is that -1 to the total indicative of something nefarious?
[0] https://catalog.data.gov/dataset/2020-doe-high-school-direct...
"it's not a nazi salute, don't be ridiculous"
"they're not building a camp in Guantanamo, don't be ridiculous"
"they're not getting rid of gov workers en mass, don't be ridiculous"
"they're not going to blanket pardon January 6th rioters, don't be ridiculous"
"they're not going to deport international students who protested against Israel, don't be ridiculous"
"they're not going to deregulate industries and set us back decades in term of ecological progress, don't be ridiculous"
"they're not scrubbing open gov data, don't be ridiculous" < you are here
Indeed, it would be a shame to jump to conclusions, we have 0 hints of what's happening
> < you are here
Your guesses about me are hilariously wrong.
Ex: https://news.ycombinator.com/item?id=42850668 , https://news.ycombinator.com/item?id=42850761, https://news.ycombinator.com/item?id=42816654
So trim those excessive confidence levels, and consider what you might change to improve your process.
Perhaps the marker indicating where you are is supposed to work it's way up the list? I don't think what you've said here - in this thread - has been unreasonable.
I'm not sure if it's nefarious or not, but I would have to lean on the side of "If the goverment is quietly removing information then probably it's not for the benefit of the public.
Also from their own site at data.gov
> The United States Government’s open data site is designed to unleash the power of government open data to inform decisions by the public and policymakers, drive innovation and economic activity, achieve agency missions, and strengthen the foundation of an open and transparent government.
The part " to inform decisions by the public and policymakers" in particular stands out.
Anyway data sets is not exactly expensive to keep hosted for a government body so there's not really any excuse to scrub them unless they're polluted in some way ?
Even if it's polluted, you can mark it as is, with reasoning and keep it up, so you can learn what dirty/bad data is, and build quality detectors with the information contained in/with the "dirty" or "low quality" data set.
Open Data initiatives generally build upon the premise of immutability.
Data Sets can be alive, i.e. new data can be streamed into them, but they won't be purged, shortened or slimmed down.
You can expect more historical data to be opened as the digitization and organization continues, and don't expect this data to be randomly being removed from archives.
Even going one step further, if this data sets are refined and formatted, they are versioned, so you can always see the previous versions (e.g.: https://www.zenodo.org).
If you're starting to remove any data set from a data pool w/o any explanation and announcement, this means there's a new "Department of Truth" in the works, 99% of the time.
I once read that occasionally a new Pharoah would make a point of smashing all prior evidence of a prior ruler(s). Obelisks, murals, etc.
agreed, this is very likely just incompetence or bureaucracy at play, i doubt anyone in tr*mp's administration cares about the data being removed