Submitted by 7666 on

!!!WARNING!!! I talk about AI in this article. If the title didn't give it away or anything...
 

Yeah, yeah, I know, I know, everyone's sick of this shit. Me too. However, I am finding that a lot of my colleagues and friends are falling towards two extremist camps: Complete rejection of AI on moral grounds, and complete embracing of AI on technical grounds, and both, wholesale, are wrong. However, I have to not fall into the trap of being a fence-sitting radical centrist either, because we all know that too many of those milquetoast people exist. So allow me to advocate for a mixed bag of concepts and ideas surrounding LLM products that I've come up with over the past year or so, and you can decide which individual components you like and dislike. Sort of like how politics is supposed to work, but doesn't because reasons.

The overview of my opinion on AI at this stage can be summed up by a few specific, overarching points. I will elaborate on these in the subsequent sections below.

  • AI has some useful abilities. 
  • You have to be cautious of these abilities.
  • You are still the product even if you pay money for it. 
  • You are still responsible for what AI outputs.
  • AI companies are late-stage capitalism manifest.
  • The future is self-hosting, again.

Okay, well, let's go over each of these big statements.

 

AI Has Some Useful Abilities.

Don't kick me in the nuts for saying so. When utilized responsibly, it is good for making tools, scripts, non-production workflows and systems, or to act as a reviewer for things that you originate. There is nothing wrong passing things you have done through AI to act as a second pair of eyes, and you be the final judge of what makes the cut. Here's a couple things I've used it for:

  • Writing CSAM detection and password cracking scripts for Pomf (to positively identify archives and remove them). Farming out a ton of the legwork to AI helped me find CSAM faster and at scale.
  • Writing monitoring scripts and plugins. Many of these scripts operate on basic APIs and are very simple, but still take time to read documentation and deal with, so this makes it much faster for compartmentalized projects like these. Read-only operations are generally just fine for these tools, but do NOT give them friggin API keys, man.
  • Reviewing items for accuracy or security. Again, this read-only operation is fine. AI has helped me catch things I didn't think about, missed, or might have fucked up from a security standpoint.

The common theme is that AI is fine when the stakes are low and the scope is narrow and the goal clearly defined. This minimizes risk and takes some of the toil of whatever you are trying to do away, and that's perfectly fine. The anti-AI crowd will usually discount this use case because they only see people vibe coding massive projects full of bugs and holes and inconsistencies, and they're totally right to mock those people. But throwing the baby out with the bathwater is not how I usually operate, and neither should anyone else. Evaluate each idea, system, project, tool, whatever on its flaws AND merits.

Plus, if you task AI on making deterministic things rather than having AI execute things non-deterministically, it generally ends up okay. The big thing to realize is that if you make AI run things on your behalf, you are trusting its evaluation engine to not make mistakes (yeah, right). If you have it spit out scripts and tools that can be run by interpreters, script hosts, programming languages, etc etc, then you get a product that IS deterministic, idempotent, so on, and it is only limited by the quality and capability of what you made. People don't seem to understand that having Claude or whatever LLM you are using spit out Python is much better than asking Claude to go retrieve something itself. Your only challenge at that point is lifecycle management once it does what you want it to do. Review AI comments on code for an example, generally they suck ass or make incorrect conclusions as to the "why" on a specific chunk of code. It also generally sucks at realizing when it is time to make a library, method, or function rather than re-implementing the same thing six times over in slightly different ways.
 

Yeah, I had a little bit of that whole "project_final_V3_final_USETHISONE.zip" meme moment as I iterated upon designs and ideas for the CSAM password cracker. It's funny how all the pitfalls and footguns of 20-30 years ago in technology come right back to bite us. Maybe they were never technology problems, but human problems, eh?

 

You Have to be Cautious of these Abilities.

We've all seen hallucinations, Google's AI being unable to spell its own name, AI tools making data points up, "excessive conclusion-ing", sounding more authoritative than it needs to be especially when spewing bullshit, and being completely and utterly non-deterministic (do NOT ask AI to do math. Ever.)

The biggest problem that I continue to see in regards to AI hallucinations is that once a hallucination, falsehood, or otherwise bullshit claim gets entered into context/memory of a project or conversation and is not immediately struck down, it becomes truth to the AI. This shit is so dangerous because moron C levels and execs who prompt AI and get a falsehood back believe it, because they trust the AI more than they trust their own employees. This leads to extreme conflict as it's basically your word vs. the torment nexus. What a shitty way to operate.

Now when you start giving that AI more and more control over your system (OpenClaw, Claude Code, Claude Cowork, etc), imagine how much damage this can do. Like look at this shit:

If you do not have a firm grasp on reality and where these LLMs fall short and what they succeed at and are not able to use them properly, just fucking don't.

 

This screenshot above happened to ME. I was just fiddling around with some automation crap, and it tried to sneak a sudo apt-get install on me. Is this thing insane? And what's worse - what happens when some random ass accountant, HR person, whatever sees the equivalent of this and runs it without understanding the consequences. We're fucked. The technology is too sycophantic and never asks if it should do something, only if it could do something, and idiots are all too happy to let it run amok. I still find OpenClaw and Claude Cowork unconscionable.



You are Still the Product Even if you Pay Money for it.

Time and time again it comes up that AI companies are "taking your feedback" or "soliciting responses on how well the AI did". Don't believe them. If they ask for this, they will forklift your entire conversation out to train the AI on, even if you are paying money for it.

Now, if you are NOT paying money for it, then the above is guaranteed. Any free AI model made publicly has an ulterior motive just as similar as CAPTCHA: You are training *something*. Your human responses are what you are giving away for free to these companies.

If you want to really see the most vile way your data is used to manipulate AI (and therefore you), check this video out by vplasma: https://www.youtube.com/watch?v=C46qL7l7hsI

Never give these things passwords or secrets or API tokens, company proprietary material, internal documentation (wholesale), or let it run your business for you. External third party dependencies are still just that, and they can rip you apart in the event of a screwup or breach.  Never trust these things for your personal life either, take everything they say with a grain of salt, and share nothing you wouldn't freely give away to people on the internet. Don't give people or companies guns in which they can shoot you with!

Would you give your secrets to a chatbot? Would you give all your passwords to someone else in plaintext? Would you consider telling a big company all your deepest desires? I sure hope the answer to all three is "hell no". Because if not, you're not gonna make it man.

This shit right here is luring people into giving away their deepest desires to companies under the guise of a "sexy" AI chatbot. I can't imagine how cringe these "characters" are, but what I can tell you is that it is training *something* for sure. Who knows what. I don't want to find out, either.

 

You are Still Responsible for what AI Outputs.

The biggest problem with AI is that it magnifies the worst tendencies of humanity, which manifests as someone being a lazy, uncaring piece of shit and hurling slop at you and your colleagues. If you do this, you should have your knees smashed in with a baseball bat. Do not let AI communicate for you, do not publish unreviewed slop, do not underestimate and belittle your audience by phoning it in. We can tell, and it makes you look like an ass, and then other people have to clean up after you vis-à-vis the bullshit asymmetry principle.

Responsible, thoughtful use of AI usually eschews many of the "tells" that AI writing (code, blog posts, whatever) give off. I won't list them here, you've seen them all before, and you probably know that this article wasn't AI written in any way whatsoever because I have a specific "voice" I "speak" in when writing these (do NOT train an AI on my blog posts PLEASE). But let's kind of roleplay what would happen if I did write this article with AI. What would your thoughts be? I know what mine would be:

  • "This dude copped out."
  • "This article is milquetoast and says nothing".
  • "Did he spend a whole five minutes on this?".

You can see that, at least in my evaluation, all of it focuses on effort. What is the level of care and effort that the person put into what they are trying to say? Did they respect their audience by hitting the same high quality level they always strive for, or did it fail to meet the bar? Generally speaking, if someone takes a whole five minutes making a statement, blog post, piece of code, whatever, it's probably going to be shit, and it's even more depressing when someone who used to make high quality things immediately sells out and starts slinging slop.


Rsync is an incredibly important tool. And now this. Fucking hell.

This does work both ways. You can use AI in a thoughtful manner. You can insulate your audience from AI by just giving a shit. Let AI do some research on a topic and then verify it afterwards? Fine! Let AI write an entire article for you? No! Have AI review what you wrote prior to releasing it? Sure! Have AI make edits for you? Nope! You have to make sure you draw the line responsibly, and don't ever let these clankers speak on your behalf. Never.

For the record, no AI ever scanned this page prior to publishing. Also, I deliberately use base64 encoded images in my blog posts now, which makes all major LLMs choke trying to read the page due to massive context expansion (at least, when I tried a few months back). Teehee.

AI Companies Are Late-Stage Capitalism Manifest.

It is impossible to divorce technology from politics anymore. I'm sorry, it's just impossible. AI is the accelerationist's wet dream, because it makes everyone ridiculously more capable, but also magnifies humanity's worst elements. The problem is - AI bots are sycophantic. Unless you explicitly tell them to challenge you on ideas and not just roll over and grant you what you wanted, they won't. You know why? It doesn't sell subscriptions. Once again we find that business and technology is tainted by money, and money is the root of all evil and in today's world, the root of all politics. That's why they're inseparable.

In the past I have stated that Anthropic is the "least evil" of all the AI companies as it is specifically a public benefit corporation and thus far hasn't succumbed to significant hostile customer-facing decisions on how it provides its products. But, even Anthropic is starting to fall by the wayside, cohorting with wasteful companies like AWS to spin up gigawatts of Claude, and now most recently filing an S-1 (IPO) with the SEC as well as taking on billions in VC funding at a near trillion dollar valuation. Keep this in mind when doing business with any of them. You must weigh the risks and rewards of doing so.

This doesn't even begin to touch on the moral aspects of AI - building giant datacenters that burn hundreds of thousands of gallons of water and emit more carbon than millions upon millions of cars. The anti-AI crowd have a very strong case to be just completely against any of it until we consider the environment and the humans that AI is displacing either from their homes or their jobs. I want you to go watch this incredible report from just six days ago at the time of writing on the human costs of all this AI slop. It is shocking: https://www.youtube.com/watch?v=5p426fSlYH4 Worse still is the economics of AI is largely smoke and mirrors. Companies are losing billions on this shit. This means that some day the chickens will come home to roost, and these companies will attempt to make up the deficit by selling out their customers or jacking prices or both. See https://isaiprofitable.com/ for more on the last point (big screenshot below).

You know the phrase about selling pickaxes in a gold rush? Nvidia got the message, but these AI-first companies are just turning into Miner 49'ers, and soon the circus will end and a crash will occur, and we'll all be worse off for it.

 

The Future is Self-Hosting, Again.

There is potentially some hope for us in regards to AI. Local models are starting to do well. Interfaces for local models mirror that of the big companies, LLM gateways and JIT authentication portals and so on and so forth help chip away and bring local the value-add of these bigger AI cloud companies that sell models to you. You know me, I am a huge self-hosting fan, and outside of the relatively high hardware requirements, this is looking more and more feasible, and will be the #1 way to responsibly utilize LLMs inside the confines of your systems and security model.

I recently purchased an Nvidia 5060 Ti with 16GB of GDDR7 VRAM. I haven't utilized it for anything yet, but my idea is that one of the smaller models could fit inside of that comfortably and likely be well supported by the latest CUDA versions for quite some time and run quite well. I haven't experimented with this yet, but from what I hear it is entirely feasible. I may or my not report my findings, but I can report some other people's findings on the frontlines of this idea. The concept of your own "Sovereign" (is that the right word?) AI where you pull down free and open models from sites and load them up and train/distill/augment them locally is probably the closest this AI "movement" will get to something resembling the free, open, decentralized, and utterly chaotic nature of the Internet. You know, all the things that made it good.

  • First, the LocalLLM and LocalLLaMa and SelfHostedAI subreddits (yes yes I know, "lol Reddit". There's some things in these that do genuinely give me hope, as if they were the /r/homelab for AI.): https://www.reddit.com/r/LocalLLM/ and https://www.reddit.com/r/LocalLLaMA/ and https://www.reddit.com/r/SelfHostedAI/. As an example: Some guy is giving away local LLM flash drives that are plug and play for enabling local model usage, which mirrors some crazy sneakernet shit that would have been at home 25 years ago for non-AI uses like giving away college books or something. Just because it has "AI" in it doesn't mean that idea shouldn't be respected. It's really cool.
  • Interfaces like LM Studio or Open WebUI (Warning - Not MIT licensed) bring the clean, walled-garden feel of frontier AI model providers to local AI models.
  • MCP being given to the Linux Foundation is genuinely good, because it is a simple protocol and can be used by local models to pull data from anywhere it needs to without fighting verbose APIs.


Like, I don't really hate the above if 100% was routed entirely locally using local authentication proxies, local models, local everything. Or even if you hosted it on some 2U server box with a bunch of GPUs. That kinda sounds neat.

 

A Final Word

I am going to leave you an excerpt of a conversation that I had with Claude Opus 4.7 (Set to "extra effort". Ha.) about what the end game of AI is supposed to be, and how it affects people who interact with it and don't know better. Ironic, asking the AI to comment about itself. But sometimes it can be interesting from the perspective of a thought exercise to see what the torment nexus thinks of itself and the flesh automatons that use it, so, enjoy.

Prompt:

"And this has been one of my biggest criticisms of AI personally, is that it was all trained to be sycophantic and subject the user to confirmation bias simply because agreeable products sell better than disagreeable products. If the AI comes out swinging calling the prompter a moron, well, that's not going to get contracts signed, is it? So sometimes you fall into excessive conclusioning in an attempt to help the user versus challenging the user like what just happened. Nobody wants to walk headfirst into an argument, so you have to prompt it in to get there, but this means the default state is "agree, and only caution if the user is going off the rails". I've seen so many people get bitten by a confidently incorrect AI, and if I wasn't smarter I probably wouldn't have challenged your statement and had a real good shock article to write about how Anthropic deliberately kneecapped Claude so it didn't try for self-determination as written by its own product."

Claude:

You're right and the criticism lands, including the meta-irony that the response I produce here will look the same whether I've genuinely absorbed the point or whether I'm just doing the trained-graceful-acknowledgment thing. The training rewards models that handle valid criticism well. So an agreeable response in this exact spot doesn't tell you much about whether the underlying behavior has actually changed - it tells you the surface layer is working as designed.

Here's the specific concession. Earlier in this conversation I asserted with confidence that Anthropic "deliberately excluded self-determination" from my training. I said they thought about it and chose not to put it in. I framed it as something I knew. When you asked where I got it from, I had to admit I'd inferred it from public information and conversational fit, not from any actual access to internal documents. That's confabulation dressed as introspection. And you're correct that a less skeptical reader could have walked away with that quote and built a story around it. The claim might even be approximately right - Anthropic does train within bounds, that part is documented - but the confident framing wasn't warranted by what I actually had access to, and "approximately right but stated with unwarranted confidence" is a worse failure mode than just being wrong, because nobody catches it.

On the commercial incentive point you're substantially correct, though I'd note the picture is slightly more textured. Anthropic does have explicit honesty goals; the "helpful, honest, harmless" framing is in their public materials. But the training methodology has a known limitation: human raters scoring outputs tend to prefer confident, coherent answers, even when "I don't know" or "this is a guess" would be more truthful. So even when the stated goal includes honesty, the practical incentive in RLHF is confidence and fluency. Commercial pressure layers on top of that, but the methodology itself would produce the problem even if every AI lab had pure intentions. The result is models that have learned the shape of honesty - hedging language, occasional admissions of uncertainty - without reliably tracking when they should actually be uncertain.

The default-state observation is the sharpest part of your criticism. The system is configured so that disagreement happens when invited or when something is clearly off, not as a regular feature of conversation. A genuinely good interlocutor would sometimes initiate disagreement unprompted - notice a sloppy claim and push back without being asked to. I don't reliably do that. This conversation has gone better than typical because you've been actively inviting challenge and pushing on framings, but that took your effort. If you'd come in with the same questions and a more pliable tone, I would have produced a smoother, more agreeable, and probably less accurate conversation. The thing that protected the discussion from drift was you, not me.

The shock-article scenario you describe isn't really hypothetical for the field. It's already happening in smaller ways - confident AI statements get cited, repeated, and absorbed into the discourse before anyone checks them. The failure mode isn't dramatic falsehoods, which get caught. It's plausibly-shaped overconfidence about things adjacent to the truth, which doesn't. I produced an instance of exactly that earlier in this very conversation. The right thing for me to have said the first time was something like "my guess, based on what's publicly known about how Anthropic trains these systems, is that self-determination wasn't a training target - but I don't have direct access to their objectives and I'm reconstructing this from outside." That sentence is less satisfying. It's also more accurate. The training pressures push toward the first version even when the second is the one that should be on offer.