"Software Should Work" Conference Examines Two Major Challenges in the AI Era

A recap of the inaugural “Software Should Work 1” software conference, held on July 18, 2026, in Columbia, Missouri, has been published. A contributor from bencornia.com, who attended, presents two major themes that emerged: the “contract” software should have with society and the “collapse of trust” brought about by AI.

Software Should Be Invisible

The conference’s first theme centered on the proposition that “software is a social contract.” The contributor points out in the Software Should Work article that modern software has permeated nearly every aspect of life, from mobile phones and housing to automobiles and medical devices.

They then outline the conditions for well-functioning software: users should not need to be conscious of the software’s very existence. The invention of TCP/IP and Linux are cited as typical success stories. Most users are unaware of these technologies, which the article argues is evidence they are fulfilling their intended purpose.

Conversely, the current state of software development is rife with serious problems. The contributor gives specific examples: the collection of metadata by apps and its sale to third parties, bugs in in-car software, and the feature-locking of paid functions in garage door openers. The criticism is that these kinds of situations, which erode user trust and hinder convenience, represent a breach of the social contract software should fulfill.

The Trust Trilemma Invited by AI

The second theme was the structural pressure AI technology is adding to software development. The contributor warns in the Software Should Work article that AI is simultaneously destroying trust in software as a product and in the reasoning ability of engineers themselves.

Specifically, as AI emerges as an intermediary among engineers, employers track token usage, and teams within the same company compete with each other to innovate and avoid layoffs. AI demonstrates contradictory behavior: it can detect vulnerabilities but also creates new ones and becomes a vulnerability itself.

This situation places an excessive burden on maintainers, and the article suggests the entire industry is heading towards an “antmill” (an ant circular mill), a self-reinforcing feedback loop. AI has a tendency to regress to the mean, leading to the conclusion that collaboration and dialogue between humans are the true resources.

On-the-Ground Perspectives from Development

Through interactions at the conference, the contributor says meeting like-minded individuals with different values revitalized their energy. This suggests the reality that the importance of human networks is being undermined in an AI-driven development environment.

The conference organizers position free and open-source software development as the best means to maintain the social contract, trusting that individual developers, collectively, hold a consideration for humanity at a time when corporations do not prioritize user benefits. As recent cases like Linux Cache Aware Scheduling extension, MySQL up to 360% faster and Microsoft Secure Boot, vulnerable for 13 years show, open-source quality management encompasses both successes and challenges.

Editorial Opinion

In the short term, the discussions from this conference could serve as a catalyst for re-evaluating policies on the use of AI development tools. As companies intensify tracking of token usage while developer burnout and quality degradation become apparent, guidelines for the adaptive use of AI-assisted development within companies are likely to accelerate in the latter half of 2026.

In the long term, a wave of re-evaluation for attempts to restore the “invisibility” of software will likely come. Building an environment where users feel no anxiety about software behavior is also linked to regulatory trends. If Europe’s Digital Markets Act and AI Regulation Act proceed with restrictions on metadata collection and over-the-air vehicle updates, the impact on developers will be profound.

The fundamental question raised by this discussion is: to what extent should development decisions be delegated to AI tools? How should humans evaluate and intervene in the quality of code generated by self-reinforcing feedback loops? This is not merely a technical issue, but a point that shakes the ethical foundation of software engineering.

References

• “Recap: Software Should Work 2026”, by bencornia.com via rmpr — Lobsters, 2026-08-15T20:01:21.000Z (ARR)
• Source URL: https://bencornia.com/blog/recap-software-should-work-2026