Is there any frame of reference to compare this to? I.e what is the normal fluctuation/movement of that 'datasets available' number? For instance, it dropped by 18k between 9/13 and 10/13.
I’m on mobile at the moment so I haven’t been able to dig too deep, but here’s [0] a query that returns the metadata for everything modified since Jan 1st (add more rows at the end if it’s cut off). I don’t think it shows deletes directly, but it can be compared to the current catalog. API documentation here [1]
That’s interesting, I was looking at the ICE ERO statistics and not only do they show up as not having a published data source, only metadata, the corresponding dashboards at ice.gov/statistics seem to not be working. I don’t know what the state of either of those was before I looked yesterday, so I don’t know if this is a recent change or not.
I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did, especially in 2024 due to rule changes which roughly doubled the number of normal deportations. I was further surprised to find out that there were roughly 3.5 times more deportations under Biden than under the first Trump administration.
I suspect that what’s happening is that leadership under Biden did the sensible thing and very quietly snuck up on priority illegal aliens, whereas with Trump, all the media attention is causing similar people to circle the wagons and proactively take steps to avoid enforcement actions.
A big factor is that Trump actually got rid of the prioritization scheme in deportation courts that processes violent criminals (and similar) more quickly than people caught with a broken taillight.
He had the net effect of increasing the amount of time a dangerous immigrant would be inside the country while the system got clogged with all sorts of cases that the American public broadly agrees hardly matter.
The Tories in the UK messed up processes around immigration in a similar way.
Break the process and it looks like you have a bigger problem, so it's a political win - if you don't care about actually doing the thing you say you care about.
> guy has no capacity nor will to think in second order effects
Maybe we're looking for the wrong effects. Most crime in America is not committed by illegal immigants. Getting rid of violent criminals through immigration courts faster isn't going to result in any crime drop a voter would notice.
The part people will notice is wage inflation. And that's, simply, a numbers game. Furthermore, it's not violent criminals who are competing with Americans for jobs, it's the law-abiding hard-working ones. The ones whose worst offence since crossing the border was having a taillight out.
If you're being Machiavellian, the target is those honest migrants. The ones on dole or committing crimes aren't creating enough problems. Their role is rhetorical. (And honestly, if they get worse, it isn't like people will vote for Democrats. Perverse incentive.)
I tend to default to Hanlon's razor [1]. But it's difficult to call a guy who has won the Presidency twice, in remarkably different situations, an idiot.
Eh, “blame the minorities to get elected, fail to solve actual problems” is the easiest political play in the book.
It doesn’t take a genius to come up with that one.
Says more about voters and our information environment than it does about Trump.
Actual solutions to complex problems tend to be complex. Fake solutions can get away with being 3 word slogans. The latter has a huge structural advantage in politics, especially in our current information environment where people can’t focus on a single problem for more than 5 seconds at a time.
> I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s
There were also more deportations the year before Trump's first term than the year of his first term.
Consider the chart of border encounters. This one shows the past through 2023 [0] and in 2024 we had 2.1 million encounters at the southern border [1]. It appears that deportations are a function of border encounters, and border encounters increased substantially during Biden's administration.
From what I could gather here, the largest "deportment" are the rejected ones at southern border entry points. That number is increasing year over year, regardless of administration: same foreigner trying over and over again.
Ive constructed a Sankey diagram of all the flow of visa-less immigration.
First Trump administration had a very strong pattern of talk and maybe a few actions… and then they seem to forget what they were doing and the result is little to no action.
Trump loves to talk about trade but when it came to China he put up some token tariffs and announced “I didn’t say it would be easy” and then did nothing the rest of his term.
> Because of the copyright enforcement section of the TPP
The point of TPP was containing China. China wasn't invited. Trump blew it up resulting in a trade bloc designed to exclude China turning into one that excludes us [1].
I agree it's good we have archives of all the sensible takes on deporting criminals with due process and not mass rounding up people based on skin color or last name at a very high expense to the taxpayers with poor results.
I think this is one of those times where two people agree on the evidence but not the resulting position.
It's very easy to fight straw men like all progressives are against all deportation and border enforcement.
> I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did
Really? It was very widely reported. Just wait until you see how many people Obama deported
"I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did..."
I think this is an essentially meaningless number until you also include the total number of illegals that were let in during the same period for each administration.
The more important number is how many people attempted crossing, which jumped significantly during Biden. This probably has something to do with their phenomenally cratered economies during COVID, multiple destabilized governments in LatAm, and severe climate migration.
That's a big part of why the Biden administration (on day one) proposed a bill that created a large economic development grant for LatAm while also modernizing border security.
No, 85% of apprehensions don't result in release into the country. That is a number that you'll have a hard time substantiating. Allegedly Mayorkas said it to the CBP union, according to the CBP union.
Below is an actual analysis of actual data from the right-leaning Cato Institute. The real number is somewhere in the 40-50% range and has been stable through most administrations (and, shockingly, Trump released more illegal immigrants into the country than Biden).
In addition to letting immigrants into the country at a higher rate, Trump also specifically slowed down deportation processing for violent immigrants, keeping them in the country, free (as is required by the Constitution's Equal Protection and Due Process clauses) for longer.
When trying to understand non-localized, large scale phenomena, yeah, you should defer to data and statistics. That’s literally why we invented them. We discovered over and over that people’s intuitions about what they were seeing were completely unreliable.
Why do you think statistics exists and all this machinery for capturing and processing data?
Statistics often exist to bamboozle and provide false and misleading context to otherwise extremely unpopular policy.
In this case, for example, to justify actions (or lack thereof) that will sway voter demographics in your favor over the medium term.
You can look at catalog.data.gov it shows totals. I'm comparing January 14th to today.
The biggest loss I see is Organizations - Department of Energy, 5473 to 3647.
I also see under Bureaus - Energy Programs, 4347 to 2521.
These are overlapping categories (-1826 on both).
There are others, but they seem smaller, a few hundred at most.
Looking closer, there was a major increase quite recently before this decrease. On January 8th there were 3617 DoE datasets. On the 14th it was up to 5473. Now we're back down to just 30 more than on the 8th.
Could it have been a publishing mistake, or some order to undo recent publications for review by the new admin?
I can't tell from the Internet Archive. The datasets that went away seem to have been added quite recently, between January 8th and 14th. There isn't a suitable capture between January 8th and 25th to see which tags or categories changed within the DoE datasets.
Perhaps the marker indicating where you are is supposed to work it's way up the list? I don't think what you've said here - in this thread - has been unreasonable.
That comment also seems very reasonable to me and probably doesn't deserve the response it got.
The other one does make sense to me though (but would be better in a different context) and echoes my own personal frustration with the "don't be ridiculous" dismissal of concern for things going on in the face of a blatant pattern (which perhaps you don't agree is a pattern with your implication that the higher things in that list haven't been dismissed as ridiculous then immediately happened).
I'm not sure if it's nefarious or not, but I would have to lean on the side of "If the goverment is quietly removing information then probably it's not for the benefit of the public.
Also from their own site at data.gov
> The United States Government’s open data site is designed to unleash the power of government open data to inform decisions by the public and policymakers, drive innovation and economic activity, achieve agency missions, and strengthen the foundation of an open and transparent government.
The part " to inform decisions by the public and policymakers" in particular stands out.
Anyway data sets is not exactly expensive to keep hosted for a government body so there's not really any excuse to scrub them unless they're polluted in some way ?
Even if it's polluted, you can mark it as is, with reasoning and keep it up, so you can learn what dirty/bad data is, and build quality detectors with the information contained in/with the "dirty" or "low quality" data set.
Open Data initiatives generally build upon the premise of immutability.
Data Sets can be alive, i.e. new data can be streamed into them, but they won't be purged, shortened or slimmed down.
You can expect more historical data to be opened as the digitization and organization continues, and don't expect this data to be randomly being removed from archives.
Even going one step further, if this data sets are refined and formatted, they are versioned, so you can always see the previous versions (e.g.: https://www.zenodo.org).
If you're starting to remove any data set from a data pool w/o any explanation and announcement, this means there's a new "Department of Truth" in the works, 99% of the time.
Complete nothingburger. The malicious action would be to subtly change key datasets to promote some kind of biased narrative. It wouldn't be the removal of less than 1% of the datasets.
Comical to think the Trump admin would be the ones "scrubbing" data when they passed the OPEN Government Data Act as part of the Foundations for Evidence Based Policymaking Act at the beginning of 2019. The law that requires agencies to publish their data to the site!
Trump must really be playing 4d chess if one of the first actions in office is to "scrub" a few obscure and yet unidentified datasets from data.gov.
I found https://uc3.cdlib.org/data-mirror/ which was a mirror a few years ago, but it's been taken down due to a lack of ongoing funding.
It would be a great start to have a mirror of just the metadata, to at least see what is being removed.
Is there any frame of reference to compare this to? I.e what is the normal fluctuation/movement of that 'datasets available' number? For instance, it dropped by 18k between 9/13 and 10/13.
The API might be a good place to start.
I’m on mobile at the moment so I haven’t been able to dig too deep, but here’s [0] a query that returns the metadata for everything modified since Jan 1st (add more rows at the end if it’s cut off). I don’t think it shows deletes directly, but it can be compared to the current catalog. API documentation here [1]
[0] https://catalog.data.gov/api/3/action/package_search?q=metad...
[1] https://data.gov/developers/apis/
*fixed first url
That’s interesting, I was looking at the ICE ERO statistics and not only do they show up as not having a published data source, only metadata, the corresponding dashboards at ice.gov/statistics seem to not be working. I don’t know what the state of either of those was before I looked yesterday, so I don’t know if this is a recent change or not.
I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did, especially in 2024 due to rule changes which roughly doubled the number of normal deportations. I was further surprised to find out that there were roughly 3.5 times more deportations under Biden than under the first Trump administration.
I suspect that what’s happening is that leadership under Biden did the sensible thing and very quietly snuck up on priority illegal aliens, whereas with Trump, all the media attention is causing similar people to circle the wagons and proactively take steps to avoid enforcement actions.
A big factor is that Trump actually got rid of the prioritization scheme in deportation courts that processes violent criminals (and similar) more quickly than people caught with a broken taillight.
He had the net effect of increasing the amount of time a dangerous immigrant would be inside the country while the system got clogged with all sorts of cases that the American public broadly agrees hardly matter.
It's like a cartoon version of a shitty manager.
https://www.themarshallproject.org/2019/07/16/trump-tried-to...
The Tories in the UK messed up processes around immigration in a similar way.
Break the process and it looks like you have a bigger problem, so it's a political win - if you don't care about actually doing the thing you say you care about.
The guy has no capacity nor will to think in second order effects.
> guy has no capacity nor will to think in second order effects
Maybe we're looking for the wrong effects. Most crime in America is not committed by illegal immigants. Getting rid of violent criminals through immigration courts faster isn't going to result in any crime drop a voter would notice.
The part people will notice is wage inflation. And that's, simply, a numbers game. Furthermore, it's not violent criminals who are competing with Americans for jobs, it's the law-abiding hard-working ones. The ones whose worst offence since crossing the border was having a taillight out.
If you're being Machiavellian, the target is those honest migrants. The ones on dole or committing crimes aren't creating enough problems. Their role is rhetorical. (And honestly, if they get worse, it isn't like people will vote for Democrats. Perverse incentive.)
I tend to default to Hanlon's razor [1]. But it's difficult to call a guy who has won the Presidency twice, in remarkably different situations, an idiot.
[1]https://en.wikipedia.org/wiki/Hanlon%27s_razor
Eh, “blame the minorities to get elected, fail to solve actual problems” is the easiest political play in the book.
It doesn’t take a genius to come up with that one.
Says more about voters and our information environment than it does about Trump.
Actual solutions to complex problems tend to be complex. Fake solutions can get away with being 3 word slogans. The latter has a huge structural advantage in politics, especially in our current information environment where people can’t focus on a single problem for more than 5 seconds at a time.
Yep. MAGA is the party of ineffective virtue signalers.
He does not care and does not have to care. It is about looking good to his supporters, getting admiration and being able to hurt who he dislikes.
It is not about actually achieving something.
Or a blatant saboteur.
Are you an ex-wife trans immigrant f us all over?
> I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s
There were also more deportations the year before Trump's first term than the year of his first term.
Consider the chart of border encounters. This one shows the past through 2023 [0] and in 2024 we had 2.1 million encounters at the southern border [1]. It appears that deportations are a function of border encounters, and border encounters increased substantially during Biden's administration.
0. https://www.nytimes.com/interactive/2023/10/29/us/illegal-bo...
1. https://www.migrationpolicy.org/news/fy2024-us-border-encoun....
Seems to be working for me: https://www.ice.gov/spotlight/statistics
From what I could gather here, the largest "deportment" are the rejected ones at southern border entry points. That number is increasing year over year, regardless of administration: same foreigner trying over and over again.
Ive constructed a Sankey diagram of all the flow of visa-less immigration.
Direct citations included.
https://github.com/egberts/immigration
> Biden’s administration deported far more people than Trump’s did
It seems like Trump’s biggest problem is, like himself, the people he put in charge are C-players, so getting anything complex done is hard.
Yes and: Infighting.
I once reported to a guy that set all us minions (product managers) against each other. It was awful.
First Trump administration had a very strong pattern of talk and maybe a few actions… and then they seem to forget what they were doing and the result is little to no action.
Trump loves to talk about trade but when it came to China he put up some token tariffs and announced “I didn’t say it would be easy” and then did nothing the rest of his term.
and they weren't token tariffs.
they were mostly ineffective tariffs, in the "tried so hard and got so far but in the end it doesn't even matter" sense, but he wasn't all talk
If you do a thing and it doesn’t work and don’t even try to fix it… still all talk.
Trump killed the Trans Pacific Partnership.
(This was a surprise to me. I'm grateful that I'm not at risk of being extradited to rot in a chinese prison for reverse engineering some IoT POS.)
How would TPP have put you at risk for Chinese prison?
Because of the copyright enforcement section of the TPP?
> Because of the copyright enforcement section of the TPP
The point of TPP was containing China. China wasn't invited. Trump blew it up resulting in a trade bloc designed to exclude China turning into one that excludes us [1].
Trump's voters don't like trade.
[1] https://en.wikipedia.org/wiki/Comprehensive_and_Progressive_...
You are aware that the TPP excluded China BY DESIGN, right?
The TPP was meant to economically counterbalance China by working closely with their neighbors.
Without it their incentive was to cut deals to work closer economically with China…
Trump’s isolationist inclinations empowers America’s enemies.
It's upsetting to the progressives but Biden and Obama both deported far more than Trump did.
Thankfully we have archives and youtube, entirely searchable, where from 2008-2012 the democrats were strongly for sending people back home.
I agree it's good we have archives of all the sensible takes on deporting criminals with due process and not mass rounding up people based on skin color or last name at a very high expense to the taxpayers with poor results.
I think this is one of those times where two people agree on the evidence but not the resulting position.
It's very easy to fight straw men like all progressives are against all deportation and border enforcement.
> It's upsetting to the progressives but Biden and Obama both deported far more than Trump did.
Is it ? Outside of the extreme progressive ideologies , what about the progressive platform would be against deportations?
Well, if you categorize everything you disagree with as extreme, none.
> I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did
Really? It was very widely reported. Just wait until you see how many people Obama deported
"I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did..."
I think this is an essentially meaningless number until you also include the total number of illegals that were let in during the same period for each administration.
The more important number is how many people attempted crossing, which jumped significantly during Biden. This probably has something to do with their phenomenally cratered economies during COVID, multiple destabilized governments in LatAm, and severe climate migration.
That's a big part of why the Biden administration (on day one) proposed a bill that created a large economic development grant for LatAm while also modernizing border security.
[flagged]
Oof, rumor mill strikes again.
No, 85% of apprehensions don't result in release into the country. That is a number that you'll have a hard time substantiating. Allegedly Mayorkas said it to the CBP union, according to the CBP union.
Below is an actual analysis of actual data from the right-leaning Cato Institute. The real number is somewhere in the 40-50% range and has been stable through most administrations (and, shockingly, Trump released more illegal immigrants into the country than Biden).
In addition to letting immigrants into the country at a higher rate, Trump also specifically slowed down deportation processing for violent immigrants, keeping them in the country, free (as is required by the Constitution's Equal Protection and Due Process clauses) for longer.
https://www.cato.org/blog/new-data-show-migrants-were-more-l...
https://www.themarshallproject.org/2019/07/16/trump-tried-to...
I guess people need to ignore their lying eyes.....
When trying to understand non-localized, large scale phenomena, yeah, you should defer to data and statistics. That’s literally why we invented them. We discovered over and over that people’s intuitions about what they were seeing were completely unreliable.
Why do you think statistics exists and all this machinery for capturing and processing data?
Statistics often exist to bamboozle and provide false and misleading context to otherwise extremely unpopular policy. In this case, for example, to justify actions (or lack thereof) that will sway voter demographics in your favor over the medium term.
Do we have any idea what datasets have gone missing? More details please from anyone who has them.
You can look at catalog.data.gov it shows totals. I'm comparing January 14th to today.
The biggest loss I see is Organizations - Department of Energy, 5473 to 3647. I also see under Bureaus - Energy Programs, 4347 to 2521. These are overlapping categories (-1826 on both).
There are others, but they seem smaller, a few hundred at most.
Looking closer, there was a major increase quite recently before this decrease. On January 8th there were 3617 DoE datasets. On the 14th it was up to 5473. Now we're back down to just 30 more than on the 8th.
Could it have been a publishing mistake, or some order to undo recent publications for review by the new admin?
This seems like a super important observation.
If the number of dataset saw a massive jump (50%), then back down a week later, that seems more like the correction of an error.
Maybe its related to climate change and advantages of renewables?
I know DoE does "other things", but I don't expect these to be public anyway.
I can't tell from the Internet Archive. The datasets that went away seem to have been added quite recently, between January 8th and 14th. There isn't a suitable capture between January 8th and 25th to see which tags or categories changed within the DoE datasets.
Common Crawl's latest crawl was Jan 12th-25th, and the index is available.
[flagged]
It seems like some groups already expected this from efforts during the previous Trump admin. This article mentions a few, and more recent preparations: https://www.scientificamerican.com/article/scientists-scramb...
TDS going hard here
[flagged]
"it's not a nazi salute, don't be ridiculous"
"they're not building a camp in Guantanamo, don't be ridiculous"
"they're not getting rid of gov workers en mass, don't be ridiculous"
"they're not going to blanket pardon January 6th rioters, don't be ridiculous"
"they're not going to deport international students who protested against Israel, don't be ridiculous"
"they're not going to deregulate industries and set us back decades in term of ecological progress, don't be ridiculous"
"they're not scrubbing open gov data, don't be ridiculous" < you are here
Indeed, it would be a shame to jump to conclusions, we have 0 hints of what's happening
> < you are here
Your guesses about me are hilariously wrong.
Ex: https://news.ycombinator.com/item?id=42850668 , https://news.ycombinator.com/item?id=42850761, https://news.ycombinator.com/item?id=42816654
So trim those excessive confidence levels, and consider what you might change to improve your process.
Perhaps the marker indicating where you are is supposed to work it's way up the list? I don't think what you've said here - in this thread - has been unreasonable.
That comment also seems very reasonable to me and probably doesn't deserve the response it got.
The other one does make sense to me though (but would be better in a different context) and echoes my own personal frustration with the "don't be ridiculous" dismissal of concern for things going on in the face of a blatant pattern (which perhaps you don't agree is a pattern with your implication that the higher things in that list haven't been dismissed as ridiculous then immediately happened).
[flagged]
"they're not rounding up Jews and gassing them en masse, don't be ridiculous" < a lot of people during Adolf Hitler's regime
I'm not sure if it's nefarious or not, but I would have to lean on the side of "If the goverment is quietly removing information then probably it's not for the benefit of the public.
Also from their own site at data.gov
> The United States Government’s open data site is designed to unleash the power of government open data to inform decisions by the public and policymakers, drive innovation and economic activity, achieve agency missions, and strengthen the foundation of an open and transparent government.
The part " to inform decisions by the public and policymakers" in particular stands out.
Anyway data sets is not exactly expensive to keep hosted for a government body so there's not really any excuse to scrub them unless they're polluted in some way ?
Even if it's polluted, you can mark it as is, with reasoning and keep it up, so you can learn what dirty/bad data is, and build quality detectors with the information contained in/with the "dirty" or "low quality" data set.
Open Data initiatives generally build upon the premise of immutability.
Data Sets can be alive, i.e. new data can be streamed into them, but they won't be purged, shortened or slimmed down.
You can expect more historical data to be opened as the digitization and organization continues, and don't expect this data to be randomly being removed from archives.
Even going one step further, if this data sets are refined and formatted, they are versioned, so you can always see the previous versions (e.g.: https://www.zenodo.org).
If you're starting to remove any data set from a data pool w/o any explanation and announcement, this means there's a new "Department of Truth" in the works, 99% of the time.
I once read that occasionally a new Pharoah would make a point of smashing all prior evidence of a prior ruler(s). Obelisks, murals, etc.
agreed, this is very likely just incompetence or bureaucracy at play, i doubt anyone in tr*mp's administration cares about the data being removed
Complete nothingburger. The malicious action would be to subtly change key datasets to promote some kind of biased narrative. It wouldn't be the removal of less than 1% of the datasets.
Comical to think the Trump admin would be the ones "scrubbing" data when they passed the OPEN Government Data Act as part of the Foundations for Evidence Based Policymaking Act at the beginning of 2019. The law that requires agencies to publish their data to the site!
Trump must really be playing 4d chess if one of the first actions in office is to "scrub" a few obscure and yet unidentified datasets from data.gov.
Why would they remove any datasets?
Also calling SVI obscure is quite hilarious.